Skip to content
Docs
About By Dawn Contact supportOpen By Dawn
  1. Send test results
  2. Send test results from CI

Send test results from CI

This page shows you how to send a CI job’s test results to By Dawn. By the end, each pipeline run shows up as one test run on its ticket and pull request, however many jobs and shards it took, and By Dawn’s worker gets the failing tests when CI is red.

The command is saw-agent tests submit. It runs after your tests, in the same job, and sends the report files your test runner already wrote: JUnit XML, TRX, CTRF or go test -json results, and lcov, Cobertura, Go coverprofile or coverlet coverage. By Dawn reads them itself, so you don’t convert anything. Your tests still run on your CI, never on By Dawn’s servers. The command only carries what the runner wrote. (You may see it written saw tests submit. It’s the same command, in the saw-agent binary.)

If your runner doesn’t write one of those files yet, Make your test runner write reports has the flag for each runner. If your repo is on GitHub and already uploads its reports as workflow artifacts, By Dawn can pick them up with no CI change at all. Test results compares the ways in.

Before you start

Section titled “Before you start”

You need three things:

  • An API key with the Tests access. An owner or admin creates it in Settings › Organization › Security, on the API keys card, with Access set to Tests — submit results (“Sends test results from CI (saw-agent tests submit) — no board access.”). That key’s only scope is tests:write: it can send test runs and can’t read or change anything else, so it’s safe to hand to CI. Store it as a CI secret named SAW_API_KEY. API keys walks through creating one.
  • Your project’s ID, such as project-3a86…. Store it as a CI variable named SAW_PROJECT. The app doesn’t show project IDs yet. List your projects with a board key in Try it, or with GET /orgs/{orgId}/projects as in the Quickstart, and copy the project’s id.
  • The repo connected to that project, under the same name. By Dawn finds the repo by name, and by default the command uses your CI’s repository name (saw-ui for LenderCom/saw-ui). If the repo has a different name in By Dawn, pass --repo.

Test results also have to be on for your organization. If they aren’t, the upload is refused with entitlement_required (Turn on test results).

Install it

Section titled “Install it”

The CI install is one line. It downloads the newest stable release for Linux x64 into the working directory:

Terminal window
curl -fsSL https://github.com/LenderCom/saw-agent-releases/releases/latest/download/saw-agent_linux_amd64.tar.gz | tar -xz

The archive holds a single saw-agent binary. Swap linux_amd64 for linux_arm64, darwin_amd64 or darwin_arm64 to match your runner. There’s no Windows build of this archive, so run the step on a Linux or macOS runner. Pairing the agent with your organization, as a remote agent needs, isn’t required: this command needs only the key.

Add it to your CI

Section titled “Add it to your CI”

Add the step after your tests. Make it run when the tests fail, too, because a red job’s failures are the results worth having. Each tab sets the key and project from your CI’s secret store, and sets SAW_API_URL: the command sends to https://workspace.sawrun.com unless you set it, and these docs use the dev workspace, https://workspace.dev.sawrun.com. If By Dawn already picks up your CI’s results by itself (GitHub Actions, or the others in Set up your CI), you don’t need the step as well.

Add a step to the job that runs your tests. !cancelled() runs it whether the tests passed or failed. In a sharded matrix, give each shard its place with --shard. (On GitHub Actions you can skip the key and the project ID altogether: sign in with GitHub’s OIDC token instead.)

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        shard: [1, 2, 3]
    steps:
      - uses: actions/checkout@v4
      - run: npx vitest run --shard=${{ matrix.shard }}/3 --reporter=default --reporter=junit --outputFile.junit=reports/junit.xml
      - name: Send test results to SAW
        if: ${{ !cancelled() }}
        env:
          SAW_API_KEY: ${{ secrets.SAW_API_KEY }}
          SAW_PROJECT: ${{ vars.SAW_PROJECT }}
          SAW_API_URL: https://workspace.dev.sawrun.com
        run: |
          curl -fsSL https://github.com/LenderCom/saw-agent-releases/releases/latest/download/saw-agent_linux_amd64.tar.gz | tar -xz
          ./saw-agent tests submit --results 'reports/*.xml' --shard ${{ matrix.shard }}/3

When it works, the step prints one line for each run it made:

saw-agent tests submit: run trun-7c1e… group tgrp-41d0… state=ready tests=412 passed=409 failed=2 skipped=1 flaky=1 reports=r0=parsed

state is ready once By Dawn has read the reports, ingesting while it still is, or failed if no report could be read. flaky counts tests that failed and then passed on a retry, and they’re counted in passed too. Each part after reports= is one file you sent: r0, r1 and so on for results, c0 and on for coverage. A part By Dawn couldn’t read says rejected, and the run’s other files still count.

Sign in with GitHub’s OIDC token

Section titled “Sign in with GitHub’s OIDC token”

On GitHub Actions the command can send results with no By Dawn key stored anywhere. GitHub gives every job that asks for one a signed OIDC token: a short-lived JSON Web Token naming the job’s repository, branch and run. When SAW_API_KEY isn’t set and the job has permissions: id-token: write, the command asks GitHub for a token for the sawrun audience and exchanges it at POST /auth/token/github-oidc. By Dawn checks GitHub’s signature, then looks for a GitHub Actions trust your organization added for that repository. If one matches, By Dawn issues the same 15-minute access token a Tests API key would get.

An owner or admin adds the trust in API keys. It names the repository, the By Dawn project its results go to, and optionally a branch or environment. The trust names the project, so --project is optional. Pass it anyway if the repository is trusted for more than one project. By Dawn shows the step to paste when you add the trust.

jobs:
  test:
    runs-on: ubuntu-latest
    permissions:
      id-token: write   # lets the job ask GitHub for its OIDC token
      contents: read
    steps:
      - uses: actions/checkout@v4
      - run: npx vitest run --reporter=default --reporter=junit --outputFile.junit=reports/junit.xml
      - name: Send test results to SAW
        if: ${{ !cancelled() }}
        env:
          SAW_API_URL: https://workspace.dev.sawrun.com
        run: |
          curl -fsSL https://github.com/LenderCom/saw-agent-releases/releases/latest/download/saw-agent_linux_amd64.tar.gz | tar -xz
          ./saw-agent tests submit --results 'reports/*.xml'
  • permissions: id-token: write is what lets GitHub hand the job a token. Setting permissions replaces the defaults, so list contents: read too, or actions/checkout can’t read the repository. Give it to the job that sends results, not the whole workflow.
  • A trust with an environment only admits a job that declares that environment:. GitHub puts the environment in the token only for a job that declares one.
  • --auth github-oidc insists on the token, and --auth api-key on the key. By default the key wins when SAW_API_KEY is set.

What a trust admits

Section titled “What a trust admits”

A trust with no branch or environment admits any workflow in the repository, the same reach a repository secret holding a key would have. A pull_request workflow started from a fork gets no token, so it can’t send results. A pull_request_target or workflow_run workflow is different. It runs in your repository, on your default branch, with the permissions you give it, so it gets a token that matches a main trust. If such a job checks out and runs the pull request’s code, that code can send results as you. So:

  • Don’t give id-token: write to a job that runs a fork’s code.
  • Do use a protected environment, with required reviewers, to narrow a trust. A branch filter narrows a trust to the runs on that branch, but a pull_request_target run is on your default branch too.
  • Add a new trust after renaming or re-creating the repository on GitHub. A trust matches the repository by its name, and from its first use by its GitHub ID too. So a renamed repository no longer matches, and one deleted and made again under the same name isn’t trusted.

When the token is refused

Section titled “When the token is refused”

A refusal exits 4 and names the fix:

The command says Means
“this job has no GitHub OIDC token … give the job permissions: id-token: write” The job has no id-token: write, or it’s a pull_request run from a fork. With --project or SAW_PROJECT set, it says “SAW_API_KEY is not set” instead, and names the same permission.
“SAW refused the GitHub OIDC token … github_oidc_no_trust” No trust admits this repository on this branch or environment. Check the trust’s repository, branch and environment, that the job declares the environment, and that the trust isn’t revoked or expired.
“… github_oidc_ambiguous_trust” The repository is trusted for more than one project. Pass --project.
“… github_oidc_invalid_token” The token isn’t GitHub’s, has expired, or was requested for another audience. The message says which check failed.

How the uploads become one run

Section titled “How the uploads become one run”

Every job and every shard sends its own upload. By Dawn puts together the uploads that tested the same commit in the same pipeline run, so a run split across three shards and a coverage job reads as one test run on the ticket.

  • The run ID (--run) ties them together. It defaults to your CI’s ID for the whole pipeline run, such as GitHub’s run ID or GitLab’s pipeline ID, so jobs of one pipeline match without you doing anything.
  • The attempt (--attempt) keeps a re-run apart. Re-running a pipeline on the same commit is a new attempt, and By Dawn shows it as its own run rather than mixing old and new results.
  • The job and shard (--job, --shard) tell uploads within a run apart. Sending the same repo, commit, run ID, attempt, job and shard again gives you back the run By Dawn already made instead of a second one. That’s what makes a retry safe, and why each matrix leg needs its own job or shard (see Do and don’t).

When By Dawn launched the run, it handed the workflow the launch’s ID as the saw_launch_id input. Put it in the step’s env, such as SAW_LAUNCH_ID: ${{ inputs.saw_launch_id }}, and pass it as --run "$SAW_LAUNCH_ID": every shard then shares it, and the run fills that launch. Don’t write ${{ inputs.saw_launch_id }} into the run: script itself, since GitHub pastes it in before the shell reads it and a crafted input could run commands next to your key. On a run nobody launched the input is empty, and --run falls back to your CI’s ID.

By Dawn ties the run to its pull request by the pull request’s number, or else by its branch, and from there to the pull request’s ticket. Pass --ticket only if you need to name the ticket yourself.

One upload holds one results format. If your globs find JUnit and TRX files, the command sends one upload per format and adds the format to the job’s name, such as unit (junit) and unit (trx). Coverage files go with the first.

What it reads from your CI

Section titled “What it reads from your CI”

The command recognises six CI systems from their own environment variables and fills in everything else from them. A flag you pass always wins over what it detects. These are the values that decide how uploads fit together:

CI Run ID Attempt Shards
GitHub Actions GITHUB_RUN_ID GITHUB_RUN_ATTEMPT — (pass --shard)
GitLab CI CI_PIPELINE_ID — CI_NODE_INDEX of CI_NODE_TOTAL
Azure Pipelines BUILD_BUILDID SYSTEM_JOBATTEMPT SYSTEM_JOBPOSITIONINPHASE of SYSTEM_TOTALJOBSINPHASE
Jenkins BUILD_TAG — —
CircleCI CIRCLE_WORKFLOW_ID — CIRCLE_NODE_INDEX of CIRCLE_NODE_TOTAL
Buildkite BUILDKITE_BUILD_ID BUILDKITE_RETRY_COUNT + 1 BUILDKITE_PARALLEL_JOB of BUILDKITE_PARALLEL_JOB_COUNT

The job’s name comes from GITHUB_JOB, CI_JOB_NAME, SYSTEM_JOBDISPLAYNAME, STAGE_NAME (else JOB_BASE_NAME), CIRCLE_JOB or BUILDKITE_LABEL. The commit, branch and pull request number come from each CI’s usual variables, such as GITHUB_SHA, CI_COMMIT_BRANCH or CHANGE_ID. On a pull request or merge request build (GitHub, GitLab and Azure Pipelines), the command sends the pull request’s own head commit and branch, not the temporary merge commit the CI checked out, so the run lands on the commit By Dawn knows.

It also takes the repository’s name, the checkout’s directory and a link back to the CI run from the same place, and the link shows as Open CI run on the run in By Dawn. Where a CI has no attempt, it’s 1. Outside a CI it recognises, the commit comes from git rev-parse HEAD and the run is recorded as an upload.

Flags

Section titled “Flags”

Only --project (or SAW_PROJECT) and at least one --results or --coverage are required. With GitHub OIDC, --project is optional too, because the trust names the project.

Flag Environment What it does
--project <id> SAW_PROJECT The By Dawn project the run belongs to.
--results <glob> Results files. Repeat it, or separate globs with commas. ** matches any depth.
--coverage <glob> Coverage files, the same way.
--repo <name> SAW_REPO The repo’s name in By Dawn. Defaults to your CI’s repository name, then the origin remote’s.
--sha <sha> The commit tested, all 40 (or 64) characters. Defaults to your CI’s, then git rev-parse HEAD.
--branch <name> The branch. Defaults to your CI’s.
--pr <number> The pull request’s number. Defaults to your CI’s.
--job <name> The job’s name. Defaults to your CI’s. Give each matrix leg its own.
--run <id> The ID that ties uploads into one run. Defaults to your CI’s pipeline run ID. In a run By Dawn launched, pass the saw_launch_id input.
--launch <id> The By Dawn launch these results fill. Defaults to --run when it’s a launch’s ID, such as tlaunch-6d0f….
--attempt <n> The run’s attempt. Defaults to your CI’s, else 1.
--shard <i>/<n> This upload’s shard, counting from 1, such as 2/3. At most 256 shards.
--format <f> Skip format detection for the results files: junit, trx, ctrf or go-test-json.
--coverage-format <f> The same for coverage: lcov, cobertura, go-coverprofile or coverlet-json.
--runner <r> The runner that wrote the results: vitest, jest, bun, playwright, pytest, go, dotnet or junit. By Dawn names tests the way that runner does.
--root <dir> The repo-relative directory the runner’s paths start from, such as a monorepo package or Playwright’s testDir.
--checkout-dir <dir> Where the repo is checked out, so By Dawn can turn absolute paths into repo paths. Defaults to your CI’s workspace, then git’s top level.
--variant <v> A variant for every result, such as os=windows, so a matrix leg’s tests stay apart from the others’.
--ticket <id> The ticket the run belongs to. By default By Dawn ties it by branch or pull request.
--api <url> SAW_API_URL By Dawn’s workspace API. Defaults to https://workspace.sawrun.com.
--accounts-url <url> SAW_ACCOUNTS_URL By Dawn’s accounts API, where the key is exchanged. Defaults to the workspace host with accounts. in place of workspace..
--auth <mode> api-key, github-oidc or auto (the default): the key when SAW_API_KEY is set, else the GitHub Actions job’s OIDC token, as Sign in with GitHub’s OIDC token describes.
--allow-missing A glob that matches nothing warns instead of failing.
--dry-run Print what would be sent, and send nothing. Needs no key.
--json Print each run By Dawn made as JSON.

The key always comes from SAW_API_KEY. The command exchanges it for a 15-minute access token, as Authentication describes, and prints neither. On GitHub Actions with no key, the command can sign in with the job’s OIDC token instead, and then --project is optional (Sign in with GitHub’s OIDC token).

The command works out each file’s format from its first few kilobytes, so --format is only for a file it can’t place. A file over 20 MB, or more than 32 files in one upload, is refused before anything is sent.

Try it before CI does

Section titled “Try it before CI does”

--dry-run shows what the command would send, from your own machine, without a key:

Terminal window
./saw-agent tests submit --project project-3a86… --results 'reports/**/*.xml' --coverage coverage/lcov.info --dry-run
upload 1 of 1 (dry run — nothing sent)
meta: {
  "repoName": "web",
  "sha": "4f9c0d1e…",
  "source": "upload",
  "attempt": 1,
  "reports": [ … ]
}
  r0  junit           reports/unit/junit.xml
  c0  lcov            coverage/lcov.info

Outside a CI it recognises, the command names the repo after your origin remote and records the run as an upload rather than a CI run.

Exit codes and errors

Section titled “Exit codes and errors”

Every failure prints one line that starts saw-agent tests submit: and says what went wrong.

Exit code Means For example
0 Sent. Or, with --allow-missing, nothing was found to send.
2 A flag is wrong, or the project, repo or commit couldn’t be found. no project — give --project <id> or set SAW_PROJECT (the SAW project's id)
3 A glob matched no file. "coverage/lcov.info" matched no file (--allow-missing to send what was found)
4 No key, or By Dawn refused it (or refused the GitHub OIDC token). SAW_API_KEY is not set — an org API key with the tests:write scope, or on GitHub Actions …
5 By Dawn refused the upload. Its error slug and a hint follow. SAW refused the upload: no repo named nope in the project (HTTP 422 validation_error) — check --repo is the repo's name in SAW
6 By Dawn couldn’t be reached, or stayed busy. cannot send the reports after 5 tries: HTTP 503
7 A file is over 20 MB or isn’t a format By Dawn reads, or an upload would carry over 32 files. reports/out.xml is not a report format SAW reads — give --format if it is one

The command retries a dropped connection, a 429 and any 5xx itself, up to five tries, waiting 1, 2, 4 and then 8 seconds, or as long as Retry-After asks. Retrying is safe, because By Dawn answers a repeated upload with the run it already made. A refusal (exit 5) isn’t retried, because sending it again would get the same answer:

  • entitlement_required: test results aren’t on for your organization.
  • forbidden: the key doesn’t have the Tests access, or the project is restricted. A key never reaches a restricted project.
  • 404 not_found: --project isn’t a project ID in your organization.
  • 422 validation_error: the message says which field. Most often the repo’s name doesn’t match By Dawn’s.

Do and don’t

Section titled “Do and don’t”
  • Do quote your globs (--results 'reports/*.xml'). Unquoted, the shell expands them first, and the command stops with “quote a glob so the shell does not expand it”.
  • Do run the step when tests fail (!cancelled(), after_script, succeededOrFailed(), post { always }, when: always). A failing run is the one By Dawn’s worker needs.
  • Decide whether the step may fail the job. It exits non-zero when it can’t send, which you’ll want to see while you set it up. Once it works, add || true (or GitHub’s continue-on-error: true) if a By Dawn outage should never turn a green job red.
  • Do use --allow-missing where the tests may not run at all, such as a job that fails during install. Without it, a glob that matches nothing exits 3, and the job’s log shows that rather than the real failure.
  • Do give each matrix leg its own job, shard or variant. On GitHub Actions, every leg of a matrix shares one GITHUB_JOB. Two legs that send the same job and shard read as a retry of one upload, and By Dawn keeps only the first. Pass --shard for shards of one suite, or --job "test (${{ matrix.os }})" and --variant os=${{ matrix.os }} for legs that each run the whole suite.
  • Do name the job as your CI does if you can. When CI is red because of tests By Dawn already knows are flaky, By Dawn re-runs the job quietly instead of waking the worker, but only if every red job sent results. By Dawn matches an upload to a red job by its name, or by the name’s start for a matrix (unit matches unit (ubuntu-latest, 20)).
  • Don’t send reports from a cancelled run. They hold the tests that ran before the cancel, and nothing else.

Next

Section titled “Next”