Test results
This page explains how By Dawn uses your test results, and how to get them into By Dawn. By the end you’ll know which way to send them suits your repos, and what each part of By Dawn needs from them.
By Dawn reads the report files your test runners already write, such as JUnit XML from Vitest or pytest, TRX from
dotnet test, and lcov or Cobertura coverage. It shows them on the ticket and the pull request, gives a worker the
failing tests when CI is red, and spots tests that fail one time and pass the next. One rule holds throughout: your
tests run on your CI, your own machines or by a person, never on By Dawn’s servers. By Dawn only reads what they wrote.
This page is about automated results. To write test cases, run them by hand and explore without a script, see Test cases, Run tests by hand and Exploratory sessions, which need test management as well.
Turn on test results
Section titled “Turn on test results”Test results depend on your organization’s plan. Without them, a ticket has no Tests section and a pull request has no Tests tab, and anything your CI sends is refused. Your plan isn’t something you can change in Settings, then Organization, then Billing yet, so contact support to turn test results on. Plans and billing has the other features that depend on the plan.
Choose how results get in
Section titled “Choose how results get in”There are three ways in, and they work side by side. Results from all three land on the same ticket and pull request, so you can start with one and add another later.
| The worker’s own runs | saw-agent tests submit |
GitHub artifact pickup | |
|---|---|---|---|
| What it is | By Dawn records the tests a ticket’s worker runs at its Tests and build pass step | One step at the end of a CI job sends its reports | By Dawn reads the reports your workflows already upload as artifacts |
| Works with | Go, pytest, dotnet test, Vitest, bun test and Jest |
Any CI: GitHub Actions, GitLab CI, Azure Pipelines, Jenkins, CircleCI, Buildkite | GitHub Actions |
| What you set up | Nothing, for most repos | A Tests — submit results API key and a step in each test job | An owner or admin approves GitHub’s Actions: read for By Dawn’s app |
| History | From now on | From now on | The last 30 days too |
| Coverage | No | Yes | Yes |
| Where it’s explained | Results from the worker’s runs | Send test results from CI | Pick up results from GitHub Actions |
Each part of By Dawn reads a different mix of them:
| What you get | Worker’s runs | CI (submit or pickup) |
|---|---|---|
| The Tests panel on a ticket and a pull request | Yes, as Worker run | Yes, as CI |
| Failing tests for the worker when CI is red | No | Needed |
| Flaky tests and their Flaky tickets | Yes | Yes |
| A quiet re-run when only known-flaky tests failed | No | Needed, from every red job |
| Fails on main next to a failure | No | Yes, from runs on the default branch |
| Coverage | No | Yes |
So start with the worker’s own runs, which need nothing, and add a CI path when you want the worker to get CI’s
failing tests. If your repos are on GitHub and their workflows already upload JUnit, TRX or coverage files as
artifacts, artifact pickup needs no CI change. GitLab CI, Azure Pipelines and CircleCI have pickup of their own,
which Set up your CI explains. Otherwise, add saw-agent tests submit to your test jobs. If your
runner doesn’t write a report yet, Make your test runner write reports has its flag.
Send results from runs on your default branch too, not only from pull requests. That’s how By Dawn knows a test
already fails on main, so it doesn’t blame the change.
Read the results on a ticket or pull request
Section titled “Read the results on a ticket or pull request”On a ticket, the Tests section sits under Pull requests. Its line says how the ticket’s tests stand, such as 2 failing, 412 passing or No results yet, and adds the number of repos when there’s more than one. On the Pull requests page, a pull request has a Tests tab next to Overview and Review, showing only that pull request’s runs. Test results on the Checks card, and next to the checks in the code view, opens the same list.
Each repo gets a card for the newest commit that was tested:
- A verdict: Failing, Passing, Reading reports while By Dawn reads them, Couldn’t read when no report could be read, or No tests when the reports held none.
- The commit and branch, the counts (such as “2 failed · 1 flaky · 409 passed · 1 skipped”), and where the run came from: Worker run, CI, Upload or Manual run, with the job, the number of shards and the attempt. Open CI run opens the run in your CI.
- The failed tests, under Failed. Each shows its suite and name, file and line, how long it took and any retries. Open one to read its full message and trace. Only the first 20 are listed until you choose Show N more failures.
- Flaky tests, under Flaky — passed on a retry.
- Badges on a test: Flaky for a test By Dawn knows is flaky, Quarantined for one an admin has quarantined, and Fails on main when it also failed in the newest run on the default branch, “so likely not this change”.
- History: the commits tested before, newest first, with how each went. Six show, and Show N older shows the rest.
A commit tested more than once, such as by the worker and then by CI, or by a re-run, shows a chip for each run. Choose one to see it. Shards of one CI run read as one run. Load older runs at the bottom reads further back.
Before anything has arrived, the section says No test results on this ticket yet and lists the ways in. While a run is still being read, the panel checks again every few seconds.
What the worker gets when CI is red
Section titled “What the worker gets when CI is red”When CI on a ticket’s pull request fails, By Dawn sends the worker back to fix it (see Watching CI). If your CI sent its results, the worker gets the failing tests rather than the end of a log: up to 10 of them, each with its file and line, its message, the end of its trace, whether it also fails on the default branch and whether it’s known to be flaky. A test that passed in another run of the same commit is marked too, so the worker doesn’t chase a flake.
This comes only from CI’s results, sent by saw-agent tests submit or artifact pickup, never from the worker’s own
runs, because the failure the worker has to fix is CI’s. If CI’s results are still on their way, By Dawn waits up to five
minutes for them. If the repo’s CI sends no results, the worker reads the failing checks’ logs, as before. The
worker can read the whole list again at any time (saw tests failures), up to 50 failing tests a pull request.
Flaky tests
Section titled “Flaky tests”A test is flaky when it passed and failed on the same commit, whichever ways the results came in, or failed and then passed on a retry within one run. By Dawn keeps watching for 30 days: a test with no new sign of flakiness in that time stops being counted as flaky.
When a test first turns flaky, By Dawn files one ticket for it, Flaky: and the test’s name, in the test’s project. It’s related to the ticket whose run caught it, if there was one, and holds what happened, the test’s file and its recent results. If the test turns flaky again later, or breaks, By Dawn comments on that ticket rather than filing another, so it collects the test’s history. Fix the test from it like any other ticket.
Flaky tickets are part of Tests, so only an organization with test results on its plan gets them. A project admin can switch them off in Project settings › Tests: File Flaky tickets automatically. With it off, By Dawn still marks flaky tests and proposes quarantine on the Tests page’s Health tab, but files no ticket and comments on none.
One run that fails five or more tests which passed on the same commit in another run is noisy: that’s the machine it ran on, not the tests. A noisy run’s failures don’t make any test flaky, propose quarantine or file a ticket. A test that fails on its own in another run is still caught.
When CI on a pull request is red only because of tests By Dawn already knows are flaky, By Dawn re-runs the failed jobs instead of waking the worker, and the card says Re-running: known-flaky tests. It does that at most twice for a commit. It needs your CI’s results from every red job, so By Dawn can see that every failure is a known flake. A red job that sent nothing, such as lint or a build that broke, goes to the worker as usual.
A test that failed on the default branch, outside any pull request, after passing in the run before, is only suspected of being flaky. That’s a lead, not a verdict, so By Dawn never re-runs CI for it.
Quarantine
Section titled “Quarantine”The Flaky ticket proposes quarantining the test. Quarantine is a project admin’s (or an org admin’s) decision, and it changes only By Dawn’s own Get CI green step: a quarantined test’s failures no longer hold a card in Building or wake a worker. GitHub’s required checks and merging are untouched, and still need CI that is actually green.
A quarantined test that fails at three commits in a row, with no pass, isn’t flaky any more but broken, so it holds By Dawn’s gate again until it passes or the quarantine is lifted. Quarantining and lifting are noted on the test’s Flaky ticket. The app shows a Quarantined badge on the test, but doesn’t have a control for quarantining one yet.
Coverage
Section titled “Coverage”Send a coverage file with your results, and By Dawn records the covered lines, branches and functions with the run, for the run and for each file. Coverage comes from your CI: the worker’s own runs don’t record it. The Tests panel doesn’t show coverage yet, and nothing in By Dawn gates on it, so your own coverage checks in CI keep working as they do.
How long By Dawn keeps results
Section titled “How long By Dawn keeps results”By Dawn keeps each test’s individual results for 90 days. Runs, their totals and their coverage are kept after that, and so is each raw report file you sent, so you can always see what a run sent.
Does By Dawn run my tests?
No. Your tests run on your CI, your own machines or by a person. The worker runs them on the machine it’s working on, which is one of yours. By Dawn only reads the reports. It can start a suite on your CI for you, from Run tests, but the run is still your CI’s (Launch test runs).
Can By Dawn's results turn a red check green on GitHub?
No. Test results and quarantine affect only By Dawn’s own steps, such as Get CI green. GitHub’s required checks and your branch rules still decide whether a pull request can merge.
I don't see a Tests section on my tickets.
Test results aren’t on for your organization yet, so the section and the pull request’s Tests tab are hidden. See Turn on test results.
The empty Tests section says saw tests submit. Is that a different command?
No, it’s the same one. In CI it’s run as saw-agent tests submit, from the saw-agent download. See
Send test results from CI.
A run says Couldn’t read. What now?
None of the reports that run sent could be read, and the run says why, such as a file that isn’t a format By Dawn reads. A run with several reports still counts the ones that could be read. The formats By Dawn reads lists what works.
Why are there two runs on one commit?
Each run that tested the commit is kept: the worker’s, CI’s, and each re-run of CI as a new attempt. Choose a chip to see one. Shards of one CI run are one run, not several.
- Results from the worker’s runs: what the worker records on its own, and how to make sure it can.
- Send test results from CI: one step for any CI.
- Pick up results from GitHub Actions: no CI change, for repos on GitHub.
- Set up your CI: pickup, results and launching on GitLab CI, Azure Pipelines, Jenkins, CircleCI and Buildkite.
- Make your test runner write reports: the flag for each runner.
- Build your test library: turn the tests in your repos into a case library.
- Plan testing for a release or cycle: what has to pass before a release or cycle is done.