# Make your test runner write reports

> The flags that make Vitest, Jest, Bun, Playwright, pytest, Go, .NET, Java and Ruby write the test results and coverage files By Dawn reads.

This page has the flag or setting that makes your test runner write a report By Dawn can read, for results and for
coverage. By the end, your test job leaves a results file (and a coverage file, if you want coverage) that
[`saw-agent tests submit`](https://docs.bydawn.com/developers/tests-submit/) or [GitHub artifact pickup](https://docs.bydawn.com/tests-github-artifacts/) can
send to By Dawn.

Most runners print to the terminal by default and write a machine-readable report only when you ask. By Dawn reads the
common report formats rather than one of its own, so you add a flag, not a plugin for By Dawn. Nothing here runs your
tests anywhere new: they still run on your CI or your own machines.

## The formats By Dawn reads

| Format | Holds | What it looks like | Written by |
|---|---|---|---|
| JUnit XML (`junit`) | results | an XML file whose root is `<testsuites>` or `<testsuite>` | almost every runner |
| TRX (`trx`) | results | an XML file whose root is `<TestRun>` | `dotnet test` |
| CTRF (`ctrf`) | results | a JSON file with a `results.tests` array | CTRF reporters for most runners |
| `go test -json` (`go-test-json`) | results | one JSON event per line | `go test -json` |
| lcov (`lcov`) | coverage | `SF:` … `end_of_record` records, usually `lcov.info` | Istanbul (Vitest, Jest), Bun, pytest-cov, coverlet |
| Cobertura XML (`cobertura`) | coverage | an XML file whose root is `<coverage>` | coverage.py, coverlet, Istanbul, ReportGenerator |
| Go coverprofile (`go-coverprofile`) | coverage | a text file whose first line is `mode: set`, `count` or `atomic` | `go test -coverprofile` |
| coverlet JSON (`coverlet-json`) | coverage | coverlet's own `coverage.json` | coverlet |

By Dawn doesn't read JaCoCo, Clover or OpenCover XML for coverage, or NUnit's or xUnit's own XML for results. Use the
alternatives each section below gives.

From a results file By Dawn reads each test's name and the suites or classes around it, its file and line, its
duration, whether it passed, failed or was skipped, how many times it was retried, and a failing test's message and
trace. A test that failed and then passed on a retry is recorded as passed and flaky. From a coverage file By Dawn reads
covered and total lines, branches and functions for each file.

## Vitest

Add the JUnit reporter next to the default one, and point it at a file:

```sh
npx vitest run --reporter=default --reporter=junit --outputFile.junit=reports/junit.xml
```

Or set it once in `vitest.config.ts`:

```ts
export default defineConfig({
  test: {
    reporters: ['default', 'junit'],
    outputFile: { junit: 'reports/junit.xml' },
    coverage: { reporter: ['text', 'lcov'] },
  },
});
```

For coverage, run with `--coverage`. The `lcov` reporter writes `coverage/lcov.info`. A Vitest JUnit file doesn't
say which Vitest project a test belongs to, so if you run several projects, send each one's report with
`--variant project=<name>`.

## Jest

Jest writes JUnit through the `jest-junit` package:

```sh
npm install --save-dev jest-junit
JEST_JUNIT_OUTPUT_DIR=reports JEST_JUNIT_ADD_FILE_ATTRIBUTE=true npx jest --reporters=default --reporters=jest-junit
```

That writes `reports/junit.xml`. `JEST_JUNIT_ADD_FILE_ATTRIBUTE=true` adds each test's file, which By Dawn shows next to
a failure. For coverage, add `--coverage --coverageReporters=lcov`, which writes `coverage/lcov.info`.

## Bun

```sh
bun test --reporter=junit --reporter-outfile=reports/junit.xml
```

For coverage, add `--coverage --coverage-reporter=lcov`, which writes `coverage/lcov.info`.

## Playwright

Use the built-in JUnit reporter, in `playwright.config.ts`:

```ts
export default defineConfig({
  reporter: [['list'], ['junit', { outputFile: 'reports/playwright-junit.xml' }]],
});
```

Or on the command line: `PLAYWRIGHT_JUNIT_OUTPUT_FILE=reports/playwright-junit.xml npx playwright test --reporter=junit`.

By Dawn keeps each Playwright project (such as `chromium` and `webkit`) apart, so the same test in two browsers is two
results. Playwright writes file paths relative to its `testDir`, so if that isn't the repo root, send the report
with `--root <testDir>`, such as `--root e2e`.

## pytest

```sh
pytest --junitxml=reports/junit-pytest.xml
```

For coverage, use `pytest-cov`. `--cov --cov-report=xml:coverage.xml` writes Cobertura, and
`--cov-report=lcov:coverage/lcov.info` writes lcov. Either works.

## Go

`go test -json` is a results format By Dawn reads, so redirect it to a file:

```sh
go test -json -coverprofile=coverage.out ./... > reports/go-test.json
```

`-coverprofile` writes the coverage. Build failures come through too: a package that doesn't compile is reported as
a build failure, with the compiler's error, rather than as a run with no tests.

If you'd rather have JUnit, or you want [artifact pickup](https://docs.bydawn.com/tests-github-artifacts/) to find your results, use
[gotestsum](https://github.com/gotestyourself/gotestsum), which runs the same tests:

```sh
gotestsum --junitfile reports/junit-go.xml -- -coverprofile=coverage.out ./...
```

Benchmarks aren't tests, so By Dawn leaves them out.

## .NET

`dotnet test` writes TRX with the built-in logger:

```sh
dotnet test --logger trx --results-directory reports
```

By Dawn reads the method's class and name as the test, and keeps each row of a theory or data-driven test apart, by its
arguments.

For coverage, use coverlet. With the `coverlet.collector` package, the collector writes Cobertura:

```sh
dotnet test --logger trx --collect:"XPlat Code Coverage" --results-directory reports
```

It writes `reports/<guid>/coverage.cobertura.xml`, so send it with `--coverage 'reports/**/coverage.cobertura.xml'`.
With `coverlet.msbuild` instead, `/p:CollectCoverage=true` writes coverlet's `coverage.json`, which By Dawn reads too.
If you prefer JUnit, the `JunitXml.TestLogger` package adds `--logger "junit;LogFilePath=reports/junit.xml"`.

## Java

Maven Surefire and Gradle both write JUnit XML on every run, with no flag:

- **Maven:** `target/surefire-reports/TEST-*.xml`. In a multi-module build, send `'**/target/surefire-reports/TEST-*.xml'`.
- **Gradle:** `build/test-results/test/TEST-*.xml`.

Surefire's reruns (`-Dsurefire.rerunFailingTestsCount=2`) are read as retries, so a test that passed on a rerun is
recorded as flaky.

By Dawn doesn't read JaCoCo XML. To send coverage, convert it to Cobertura with
[ReportGenerator](https://github.com/danielpalme/ReportGenerator)
(`-reports:target/site/jacoco/jacoco.xml -reporttypes:Cobertura`), or send results only.

## Ruby

For RSpec, add the `rspec_junit_formatter` gem:

```sh
bundle exec rspec --format progress --format RspecJunitFormatter --out reports/rspec-junit.xml
```

For coverage, use SimpleCov with the `simplecov-cobertura` gem, which writes `coverage/coverage.xml`:

```ruby
require 'simplecov'
require 'simplecov-cobertura'
SimpleCov.formatter = SimpleCov::Formatter::CoberturaFormatter
SimpleCov.start
```

## Any other runner

Look for a JUnit XML reporter: nearly every test runner has one, built in or as a package. A CTRF reporter works
too. Leave `--runner` out for a runner that isn't in its list, and By Dawn names the tests from the JUnit file as it is.

## Name files so every path finds them

`saw-agent tests submit` works out each file's format from its content, so it doesn't mind what a file is called.
[Artifact pickup](https://docs.bydawn.com/tests-github-artifacts/#which-files-it-picks-up) goes by names first. If you might use both, pick
names that suit pickup:

- put `junit` in a JUnit file's name, such as `junit-pytest.xml`
- start a CTRF file's name with `ctrf`, such as `ctrf-report.json`
- keep lcov's usual `lcov.info`
- start a Cobertura file's name with `coverage` or `cobertura`

`go test -json` output and coverlet's `coverage.json` aren't picked up from artifacts, so send them with
`saw-agent tests submit`.

## Limits

| Limit | Value |
|---|---|
| One report file | 20 MB |
| Reports in one run | 32 |
| One run's results formats | one, plus any number of coverage files |
| Tests in one report | 100,000 (retries count) |
| Results in one run | 200,000 |
| A failure's message | its first 4,000 characters |
| A failure's trace | its last 16 KB |
| Files in one coverage report | 50,000. By Dawn keeps the 5,000 with the most uncovered lines. |

By Dawn doesn't process DTDs or XML entities: a report that declares a DTD with an internal subset isn't read.

> **When a report can't be read:** A report By Dawn can't read is set aside on its own, with the reason, and the run's other reports still count. The raw
> file is kept on the run either way. A run fails only when none of its reports could be read, and then the run says
> why.

## Next

- [Send test results from CI](https://docs.bydawn.com/developers/tests-submit/): send these files to By Dawn from any CI.
- [Pick up results from GitHub Actions](https://docs.bydawn.com/tests-github-artifacts/): let By Dawn find them in your workflow artifacts.
- [Test results](https://docs.bydawn.com/tests/): what By Dawn does with them.

Source: https://docs.bydawn.com/developers/test-reports/