Build your test library

This page helps you fill a project’s test case library without typing it in. At the end the library holds a case for every automated test in your repos, a suite for each CI job that runs them, and the manual cases your repos and tickets already describe, and it stays in step each time code merges.

By Dawn builds it as a proposal. One of your machines reads each repo’s tests, CI config and QA files, and By Dawn turns what it found into cases. Nothing lands in the library until a project admin has reviewed every item and committed it. As everywhere in By Dawn, your tests run on your CI, your own machines or by a person, never on By Dawn’s servers, and building the library runs none of them.

It’s in Tests›, then Cases, under Build my test library.

Before you start

• Test management has to be on for your organization. Without it the Cases tab says Test management is not on your plan. Contact support to turn it on.
• You need to be a project admin, or an org admin or owner. Everyone else sees “A project admin builds the library.”
• A machine has to be online. Each repo is read on one of your organization’s shared machines, which clones it from its remote. See Local agent for adding one.

Start it

1. Open the project’s Tests page and choose the Cases tab. In an empty library it shows Start from what your repos already have; otherwise the button is at the top of the case list.

2. Choose Build my test library. The Build library tab opens, listing what the proposal will hold:

   • every automated test, as a case, filed by repo, file and describe block;
   • a suite for each CI job that runs tests, with where it launches, and the one line to add where a job sends no report;
   • the manual cases in your Gherkin features, QA checklists and test sheets, and in this project’s QA scripts.

3. Start it. The tab shows Scanning and lists your repos, each with its own state: Waiting for a machine, Scanning on a machine, Scanned or Failed. The page fills in as each scan finishes, so you can leave and come back.

A project has one proposal at a time. While one is scanning or waiting for review, the button reads Review your test library proposal and opens it. A project with no repos is ready at once, and holds only its QA scripts.

What it reads

Each repo is read at its default branch.

• Automated tests, without running them. The machine asks each runner for its list of tests, such as vitest list, playwright test --list, pytest --collect-only, go test -list or dotnet test --list-tests, and reads the titles straight from the test files where a runner can’t list them (Jest and bun test). Listing may install the repo’s dependencies and build it first, as the runner needs.
• CI jobs, from the CI files alone: .github/workflows, .gitlab-ci.yml, azure-pipelines.yml, .circleci/config.yml and Jenkinsfile. No job runs.
• Manual test files: Gherkin .feature files, TESTING.md, QA*.md, anything under docs/qa/, and CSV test sheets. Up to 200 files and 10 MB a repo are read, and up to 5,000 cases from them.
• QA scripts that agents left on the project’s tickets, from tickets whose QA step passed. They land in a QA scripts folder, each linked to its ticket. If a ticket has several documents, label the script “QA script” so By Dawn knows which one to read.

A repo that couldn’t be read says why, such as “The repo has no remote a machine can clone.” The rest of the proposal is still there, but that repo’s automated cases and suites are missing from it. If no machine takes a scan, By Dawn keeps trying for about half an hour, and gives up on a scan no machine finishes within 45 minutes.

Review the proposal

When every repo has been read, the proposal is Ready for review. Its first line counts what it found, such as “1,284 cases · 6 suites · 41 manual”, and a second line counts your decisions: “412 accepted · 3 rejected · 869 to decide”. Every item has to be accepted or rejected before the proposal can be committed.

It has four sections, each a tab with its count: Automated, Suites, Manual and Gaps. Each shows a tree on the left, such as repo, then file, then describe block for automated tests, and the items on the right, with a search box and a filter for To decide, Accepted or Rejected.

Decide as broadly or as narrowly as you like:

• Everything: Accept all, Reject all, or Accept the rest once you’ve rejected what you don’t want.
• A section: Accept all or Reject all on its tab, such as every suite.
• A folder: the accept and reject buttons on a folder in the tree, such as one test file.
• Items: Accept or Reject on each, or tick several and decide them together. Choose the same button again to put an item back to undecided.

What each section holds:

• Automated: one item per test, marked New, Updated, Renamed, Gone or Back against what the library already has. Rejecting a new test keeps it out of the library for good, so later syncs won’t add it back (see Let an excluded test back in). Rejecting any other change only skips it this time, because the code owns those and the next merge applies them.
• Suites: one per CI job that runs tests, with its kind (unit, integration, end-to-end or smoke), how it fans out (such as “3 shards · node: 20, 22 · 6 legs”) and whether By Dawn can launch it. A job that Writes no test report shows the one line to add, or the command to change, so its results reach By Dawn, with a copy button.
• Manual: one case per scenario, checklist or row, with its first steps. One that matches a case you already imported says “Updates an existing case imported from the same source.”
• Gaps: the parts of your code that no test touches. They aren’t worked out yet, so this section says Gaps aren’t computed yet.

Commit it

When nothing is left to decide, choose Commit to the library. By Dawn checks each accepted change against the library as it is now, so a case someone edited while you were reviewing is never overwritten; a change that no longer applies waits for the next scan. Then it lands everything in one go:

• the accepted automated tests become cases;
• the accepted suites are created, with where they launch if your plan includes launching tests;
• the accepted manual cases are added, in their folders;
• every repo you accepted tests from starts staying in step on each merge (below).

The tab then says “The library has landed”, with what it added, and Open the library opens the Cases tab.

If you’d rather not land it, choose Discard. Nothing in it lands and every decision on it is lost, and you can build the library again whenever you like.

Stay in step after a merge

Once a repo’s tests have landed, each merge to its default branch has one of your machines read the repo’s tests again, and By Dawn applies what changed by itself: new tests become cases, renamed ones follow, and removed ones are marked gone. The code owns where a test is and what it’s called. What people wrote stays theirs: a sync never changes a case’s title, description, priority, tags, links or folder.

A test you rejected as new stays out. Running Build my test library again later proposes only what the merges haven’t already brought in, such as a repo added to the project since, and new QA files.

Let an excluded test back in

Open the last proposal from the Build library tab (See the last proposal), find the test in Automated and choose Allow again. The repo’s next scan adds it as a case.

Improve the cases with an agent

Cases built from your code have the test’s own name as their title. To make them readable, the Build my test library button in the Tests page’s header starts an agent on your machine that reads your code and proposes plain titles, step-by-step cases, feature tags, links to the tickets behind each test, and draft cases for what no test covers. It never changes your code. Its proposals arrive as one set to review and apply: see Review changes an agent proposes.

FAQ

Does building the library run my tests?

No. The machine lists the tests and reads files, and runs nothing. Listing may install the repo’s dependencies or build it, because some runners need that to list their tests, but no test runs.

A repo says Failed.

Its message says why, such as no machine taking the scan or no remote to clone. Its tests and suites are missing from this proposal, but the rest is fine. Discard the proposal and build again once the machine or the remote is sorted, or commit what’s there and build again later for the missing repo.

It says a newer scan has landed.

A merge synced one of the repos while you were reviewing, so the proposal is out of date. Discard it and build the library again.

Can I bring in cases from Testmo or TestRail too?

Yes, with an import: see Move from Testmo or TestRail.

Next

• Test cases: work with the library you’ve built.
• Run tests by hand: run its manual cases.
• Send test results from CI: the line a suite with no report needs.