Make your test runner write reports

This page has the flag or setting that makes your test runner write a report By Dawn can read, for results and for coverage. By the end, your test job leaves a results file (and a coverage file, if you want coverage) that saw-agent tests submit or GitHub artifact pickup can send to By Dawn.

Most runners print to the terminal by default and write a machine-readable report only when you ask. By Dawn reads the common report formats rather than one of its own, so you add a flag, not a plugin for By Dawn. Nothing here runs your tests anywhere new: they still run on your CI or your own machines.

The formats By Dawn reads

Format	Holds	What it looks like	Written by
JUnit XML (junit)	results	an XML file whose root is <testsuites> or <testsuite>	almost every runner
TRX (trx)	results	an XML file whose root is <TestRun>	dotnet test
CTRF (ctrf)	results	a JSON file with a results.tests array	CTRF reporters for most runners
go test -json (go-test-json)	results	one JSON event per line	go test -json
lcov (lcov)	coverage	SF: … end_of_record records, usually lcov.info	Istanbul (Vitest, Jest), Bun, pytest-cov, coverlet
Cobertura XML (cobertura)	coverage	an XML file whose root is <coverage>	coverage.py, coverlet, Istanbul, ReportGenerator
Go coverprofile (go-coverprofile)	coverage	a text file whose first line is mode: set, count or atomic	go test -coverprofile
coverlet JSON (coverlet-json)	coverage	coverlet’s own coverage.json	coverlet

By Dawn doesn’t read JaCoCo, Clover or OpenCover XML for coverage, or NUnit’s or xUnit’s own XML for results. Use the alternatives each section below gives.

From a results file By Dawn reads each test’s name and the suites or classes around it, its file and line, its duration, whether it passed, failed or was skipped, how many times it was retried, and a failing test’s message and trace. A test that failed and then passed on a retry is recorded as passed and flaky. From a coverage file By Dawn reads covered and total lines, branches and functions for each file.

Vitest

Add the JUnit reporter next to the default one, and point it at a file:

npx vitest run --reporter=default --reporter=junit --outputFile.junit=reports/junit.xml

Or set it once in vitest.config.ts:

export default defineConfig({
  test: {
    reporters: ['default', 'junit'],
    outputFile: { junit: 'reports/junit.xml' },
    coverage: { reporter: ['text', 'lcov'] },
  },
});

For coverage, run with --coverage. The lcov reporter writes coverage/lcov.info. A Vitest JUnit file doesn’t say which Vitest project a test belongs to, so if you run several projects, send each one’s report with --variant project=<name>.

Jest

Jest writes JUnit through the jest-junit package:

npm install --save-dev jest-junit
JEST_JUNIT_OUTPUT_DIR=reports JEST_JUNIT_ADD_FILE_ATTRIBUTE=true npx jest --reporters=default --reporters=jest-junit

That writes reports/junit.xml. JEST_JUNIT_ADD_FILE_ATTRIBUTE=true adds each test’s file, which By Dawn shows next to a failure. For coverage, add --coverage --coverageReporters=lcov, which writes coverage/lcov.info.

Bun

bun test --reporter=junit --reporter-outfile=reports/junit.xml

For coverage, add --coverage --coverage-reporter=lcov, which writes coverage/lcov.info.

Playwright

Use the built-in JUnit reporter, in playwright.config.ts:

export default defineConfig({
  reporter: [['list'], ['junit', { outputFile: 'reports/playwright-junit.xml' }]],
});

Or on the command line: PLAYWRIGHT_JUNIT_OUTPUT_FILE=reports/playwright-junit.xml npx playwright test --reporter=junit.

By Dawn keeps each Playwright project (such as chromium and webkit) apart, so the same test in two browsers is two results. Playwright writes file paths relative to its testDir, so if that isn’t the repo root, send the report with --root <testDir>, such as --root e2e.

pytest

pytest --junitxml=reports/junit-pytest.xml

For coverage, use pytest-cov. --cov --cov-report=xml:coverage.xml writes Cobertura, and --cov-report=lcov:coverage/lcov.info writes lcov. Either works.

Go

go test -json is a results format By Dawn reads, so redirect it to a file:

go test -json -coverprofile=coverage.out ./... > reports/go-test.json

-coverprofile writes the coverage. Build failures come through too: a package that doesn’t compile is reported as a build failure, with the compiler’s error, rather than as a run with no tests.

If you’d rather have JUnit, or you want artifact pickup to find your results, use gotestsum, which runs the same tests:

gotestsum --junitfile reports/junit-go.xml -- -coverprofile=coverage.out ./...

Benchmarks aren’t tests, so By Dawn leaves them out.

.NET

dotnet test writes TRX with the built-in logger:

dotnet test --logger trx --results-directory reports

By Dawn reads the method’s class and name as the test, and keeps each row of a theory or data-driven test apart, by its arguments.

For coverage, use coverlet. With the coverlet.collector package, the collector writes Cobertura:

dotnet test --logger trx --collect:"XPlat Code Coverage" --results-directory reports

It writes reports/<guid>/coverage.cobertura.xml, so send it with --coverage 'reports/**/coverage.cobertura.xml'. With coverlet.msbuild instead, /p:CollectCoverage=true writes coverlet’s coverage.json, which By Dawn reads too. If you prefer JUnit, the JunitXml.TestLogger package adds --logger "junit;LogFilePath=reports/junit.xml".

Java

Maven Surefire and Gradle both write JUnit XML on every run, with no flag:

• Maven: target/surefire-reports/TEST-*.xml. In a multi-module build, send '**/target/surefire-reports/TEST-*.xml'.
• Gradle: build/test-results/test/TEST-*.xml.

Surefire’s reruns (-Dsurefire.rerunFailingTestsCount=2) are read as retries, so a test that passed on a rerun is recorded as flaky.

By Dawn doesn’t read JaCoCo XML. To send coverage, convert it to Cobertura with ReportGenerator (-reports:target/site/jacoco/jacoco.xml -reporttypes:Cobertura), or send results only.

Ruby

For RSpec, add the rspec_junit_formatter gem:

bundle exec rspec --format progress --format RspecJunitFormatter --out reports/rspec-junit.xml

For coverage, use SimpleCov with the simplecov-cobertura gem, which writes coverage/coverage.xml:

require 'simplecov'
require 'simplecov-cobertura'
SimpleCov.formatter = SimpleCov::Formatter::CoberturaFormatter
SimpleCov.start

Any other runner

Look for a JUnit XML reporter: nearly every test runner has one, built in or as a package. A CTRF reporter works too. Leave --runner out for a runner that isn’t in its list, and By Dawn names the tests from the JUnit file as it is.

Name files so every path finds them

saw-agent tests submit works out each file’s format from its content, so it doesn’t mind what a file is called. Artifact pickup goes by names first. If you might use both, pick names that suit pickup:

• put junit in a JUnit file’s name, such as junit-pytest.xml
• start a CTRF file’s name with ctrf, such as ctrf-report.json
• keep lcov’s usual lcov.info
• start a Cobertura file’s name with coverage or cobertura

go test -json output and coverlet’s coverage.json aren’t picked up from artifacts, so send them with saw-agent tests submit.

Limits

Limit	Value
One report file	20 MB
Reports in one run	32
One run’s results formats	one, plus any number of coverage files
Tests in one report	100,000 (retries count)
Results in one run	200,000
A failure’s message	its first 4,000 characters
A failure’s trace	its last 16 KB
Files in one coverage report	50,000. By Dawn keeps the 5,000 with the most uncovered lines.

By Dawn doesn’t process DTDs or XML entities: a report that declares a DTD with an internal subset isn’t read.

When a report can't be read

A report By Dawn can’t read is set aside on its own, with the reason, and the run’s other reports still count. The raw file is kept on the run either way. A run fails only when none of its reports could be read, and then the run says why.

Next

• Send test results from CI: send these files to By Dawn from any CI.
• Pick up results from GitHub Actions: let By Dawn find them in your workflow artifacts.
• Test results: what By Dawn does with them.