# Test results

> How By Dawn reads the test results your own CI and machines produce, the three ways to send them, and what the Tests panel, the worker and flaky-test handling do with them.

This page explains how By Dawn uses your test results, and how to get them into By Dawn. By the end you'll know which way to
send them suits your repos, and what each part of By Dawn needs from them.

By Dawn reads the report files your test runners already write, such as JUnit XML from Vitest or pytest, TRX from
`dotnet test`, and lcov or Cobertura coverage. It shows them on the ticket and the pull request, gives a worker the
failing tests when CI is red, and spots tests that fail one time and pass the next. One rule holds throughout: **your
tests run on your CI, your own machines or by a person, never on By Dawn's servers.** By Dawn only reads what they wrote.

This page is about automated results. To write test cases, run them by hand and explore without a script, see
[Test cases](https://docs.bydawn.com/test-cases/), [Run tests by hand](https://docs.bydawn.com/test-runs/) and [Exploratory sessions](https://docs.bydawn.com/test-sessions/), which need
test management as well.

## Turn on test results

Test results depend on your organization's plan. Without them, a ticket has no **Tests** section and a pull request
has no **Tests** tab, and anything your CI sends is refused. Your plan isn't something you can change in
**Settings › Organization › Billing** yet, so [contact support](https://app.dev.sawrun.com/support) to
turn test results on. [Plans and billing](https://docs.bydawn.com/billing/) has the other features that depend on the plan.

## Choose how results get in

There are three ways in, and they work side by side. Results from all three land on the same ticket and pull
request, so you can start with one and add another later.

| | The worker's own runs | `saw-agent tests submit` | GitHub artifact pickup |
|---|---|---|---|
| **What it is** | By Dawn records the tests a ticket's worker runs at its **Tests and build pass** step | One step at the end of a CI job sends its reports | By Dawn reads the reports your workflows already upload as artifacts |
| **Works with** | Go, pytest, `dotnet test`, Vitest, `bun test` and Jest | Any CI: GitHub Actions, GitLab CI, Azure Pipelines, Jenkins, CircleCI, Buildkite | GitHub Actions |
| **What you set up** | Nothing, for most repos | A **Tests — submit results** API key and a step in each test job | An owner or admin approves GitHub's **Actions: read** for By Dawn's app |
| **History** | From now on | From now on | The last 30 days too |
| **Coverage** | No | Yes | Yes |
| **Where it's explained** | [Results from the worker's runs](https://docs.bydawn.com/tests-worker-runs/) | [Send test results from CI](https://docs.bydawn.com/developers/tests-submit/) | [Pick up results from GitHub Actions](https://docs.bydawn.com/tests-github-artifacts/) |

Each part of By Dawn reads a different mix of them:

| What you get | Worker's runs | CI (submit or pickup) |
|---|---|---|
| [The Tests panel](#read-the-results-on-a-ticket-or-pull-request) on a ticket and a pull request | Yes, as **Worker run** | Yes, as **CI** |
| [Failing tests for the worker](#what-the-worker-gets-when-ci-is-red) when CI is red | No | **Needed** |
| [Flaky tests](#flaky-tests) and their Flaky tickets | Yes | Yes |
| [A quiet re-run](#flaky-tests) when only known-flaky tests failed | No | **Needed**, from every red job |
| **Fails on main** next to a failure | No | Yes, from runs on the default branch |
| [Coverage](#coverage) | No | Yes |

So start with the worker's own runs, which need nothing, and add a CI path when you want the worker to get CI's
failing tests. If your repos are on GitHub and their workflows already upload JUnit, TRX or coverage files as
artifacts, artifact pickup needs no CI change. GitLab CI, Azure Pipelines and CircleCI have pickup of their own,
which [Set up your CI](https://docs.bydawn.com/tests-ci/) explains. Otherwise, add `saw-agent tests submit` to your test jobs. If your
runner doesn't write a report yet, [Make your test runner write reports](https://docs.bydawn.com/developers/test-reports/) has its flag.

Send results from runs on your default branch too, not only from pull requests. That's how By Dawn knows a test
already fails on `main`, so it doesn't blame the change.

## Read the results on a ticket or pull request

On a ticket, the **Tests** section sits under **Pull requests**. Its line says how the ticket's tests stand, such as
**2 failing**, **412 passing** or **No results yet**, and adds the number of repos when there's more than one. On
the **Pull requests** page, a pull request has a **Tests** tab next to **Overview** and **Review**, showing only
that pull request's runs. **Test results** on the **Checks** card, and next to the checks in the code view, opens
the same list.

Each repo gets a card for the newest commit that was tested:

- **A verdict**: **Failing**, **Passing**, **Reading reports** while By Dawn reads them, **Couldn’t read** when no report
  could be read, or **No tests** when the reports held none.
- **The commit and branch**, the counts (such as "2 failed · 1 flaky · 409 passed · 1 skipped"), and where the run
  came from: **Worker run**, **CI**, **Upload** or **Manual run**, with the job, the number of shards and the attempt.
  **Open CI run** opens the run in your CI.
- **The failed tests**, under **Failed**. Each shows its suite and name, file and line, how long it took and any
  retries. Open one to read its full message and trace. Only the first 20 are listed until you choose **Show N more
  failures**.
- **Flaky tests**, under **Flaky — passed on a retry**.
- **Badges** on a test: **Flaky** for a test By Dawn knows is flaky, **Quarantined** for one an admin has quarantined, and
  **Fails on main** when it also failed in the newest run on the default branch, "so likely not this change".
- **History**: the commits tested before, newest first, with how each went. Six show, and **Show N older** shows the
  rest.

A commit tested more than once, such as by the worker and then by CI, or by a re-run, shows a chip for each run.
Choose one to see it. Shards of one CI run read as one run. **Load older runs** at the bottom reads further back.

Before anything has arrived, the section says **No test results on this ticket yet** and lists the ways in. While a
run is still being read, the panel checks again every few seconds.

## What the worker gets when CI is red

When CI on a ticket's pull request fails, By Dawn sends the worker back to fix it (see
[Watching CI](https://docs.bydawn.com/pull-requests/#watching-ci)). If your CI sent its results, the worker gets the failing tests rather
than the end of a log: up to 10 of them, each with its file and line, its message, the end of its trace, whether it
also fails on the default branch and whether it's known to be flaky. A test that passed in another run of the same
commit is marked too, so the worker doesn't chase a flake.

This comes only from CI's results, sent by `saw-agent tests submit` or artifact pickup, never from the worker's own
runs, because the failure the worker has to fix is CI's. If CI's results are still on their way, By Dawn waits up to five
minutes for them. If the repo's CI sends no results, the worker reads the failing checks' logs, as before. The
worker can read the whole list again at any time (`saw tests failures`), up to 50 failing tests a pull request.

## Flaky tests

A test is **flaky** when it passed and failed on the same commit, whichever ways the results came in, or failed and
then passed on a retry within one run. By Dawn keeps watching for 30 days: a test with no new sign of flakiness in that
time stops being counted as flaky.

When a test first turns flaky, By Dawn files one ticket for it, **Flaky:** and the test's name, in the test's project.
It's related to the ticket whose run caught it, if there was one, and holds what happened, the test's file and its
recent results. If the test turns flaky again later, or breaks, By Dawn comments on that ticket rather than filing
another, so it collects the test's history. Fix the test from it like any other ticket.

Flaky tickets are part of Tests, so only an organization with test results on its plan gets them. A project admin
can switch them off in **Project settings › Tests**: **File Flaky tickets automatically**. With it off, By Dawn still
marks flaky tests and proposes quarantine on the Tests page's **Health** tab, but files no ticket and comments on
none.

One run that fails five or more tests which passed on the same commit in another run is **noisy**: that's the
machine it ran on, not the tests. A noisy run's failures don't make any test flaky, propose quarantine or file a
ticket. A test that fails on its own in another run is still caught.

When CI on a pull request is red only because of tests By Dawn already knows are flaky, By Dawn re-runs the failed jobs
instead of waking the worker, and the card says **Re-running: known-flaky tests**. It does that at most twice for a
commit. It needs your CI's results from every red job, so By Dawn can see that every failure is a known flake. A red job
that sent nothing, such as lint or a build that broke, goes to the worker as usual.

A test that failed on the default branch, outside any pull request, after passing in the run before, is only
**suspected** of being flaky. That's a lead, not a verdict, so By Dawn never re-runs CI for it.

### Quarantine

The Flaky ticket proposes quarantining the test. Quarantine is a project admin's (or an org admin's) decision, and
it changes only By Dawn's own **Get CI green** step: a quarantined test's failures no longer hold a card in
**Building** or wake a worker.
GitHub's required checks and merging are untouched, and still need CI that is actually green.

A quarantined test that fails at three commits in a row, with no pass, isn't flaky any more but **broken**, so it
holds By Dawn's gate again until it passes or the quarantine is lifted. Quarantining and lifting are noted on the test's
Flaky ticket. The app shows a **Quarantined** badge on the test, but doesn't have a control for quarantining one
yet.

## Coverage

Send a coverage file with your results, and By Dawn records the covered lines, branches and functions with the run,
for the run and for each file. Coverage comes from your CI: the worker's own runs don't record it. The Tests panel
doesn't show coverage yet, and nothing in By Dawn gates on it, so your own coverage checks in CI keep working as they do.

## How long By Dawn keeps results

By Dawn keeps each test's individual results for 90 days. Runs, their totals and their coverage are kept after that,
and so is each raw report file you sent, so you can always see what a run sent.

## FAQ

**Does By Dawn run my tests?**

No. Your tests run on your CI, your own machines or by a person. The worker runs them on the machine it's working
on, which is one of yours. By Dawn only reads the reports. It can start a suite on your CI for you, from **Run tests**,
but the run is still your CI's ([Launch test runs](https://docs.bydawn.com/tests-launch/)).

**Can By Dawn's results turn a red check green on GitHub?**

No. Test results and quarantine affect only By Dawn's own steps, such as **Get CI green**. GitHub's required checks and
your branch rules still decide whether a pull request can merge.

**I don't see a Tests section on my tickets.**

Test results aren't on for your organization yet, so the section and the pull request's **Tests** tab are hidden.
See [Turn on test results](#turn-on-test-results).

**The empty Tests section says saw tests submit. Is that a different command?**

No, it's the same one. In CI it's run as `saw-agent tests submit`, from the `saw-agent` download. See
[Send test results from CI](https://docs.bydawn.com/developers/tests-submit/).

**A run says Couldn’t read. What now?**

None of the reports that run sent could be read, and the run says why, such as a file that isn't a format By Dawn reads.
A run with several reports still counts the ones that could be read. [The formats By Dawn
reads](https://docs.bydawn.com/developers/test-reports/#the-formats-by-dawn-reads) lists what works.

**Why are there two runs on one commit?**

Each run that tested the commit is kept: the worker's, CI's, and each re-run of CI as a new attempt. Choose a chip to
see one. Shards of one CI run are one run, not several.

## Next

- [Results from the worker's runs](https://docs.bydawn.com/tests-worker-runs/): what the worker records on its own, and how to make sure it
  can.
- [Send test results from CI](https://docs.bydawn.com/developers/tests-submit/): one step for any CI.
- [Pick up results from GitHub Actions](https://docs.bydawn.com/tests-github-artifacts/): no CI change, for repos on GitHub.
- [Set up your CI](https://docs.bydawn.com/tests-ci/): pickup, results and launching on GitLab CI, Azure Pipelines, Jenkins, CircleCI and
  Buildkite.
- [Make your test runner write reports](https://docs.bydawn.com/developers/test-reports/): the flag for each runner.
- [Build your test library](https://docs.bydawn.com/test-library/): turn the tests in your repos into a case library.
- [Plan testing for a release or cycle](https://docs.bydawn.com/test-plans/): what has to pass before a release or cycle is done.

Source: https://docs.bydawn.com/tests/