Set up your CI

This page shows you how to set up GitLab CI, Azure Pipelines, Jenkins, CircleCI and Buildkite for By Dawn’s Tests. By the end, each finished pipeline’s test results reach the ticket and pull request they belong to, and your pipeline accepts the inputs By Dawn sends when it launches a suite.

There are three things to set up for each system, and the sections below take them in turn:

• How By Dawn reaches it. GitLab CI and Azure Pipelines use connections you may already have in By Dawn. Jenkins, CircleCI and Buildkite each need a CI connection: an account and an API token an org admin adds once.
• How results come back. By Dawn picks up results from GitLab CI, Azure Pipelines and CircleCI by itself, from the reports your jobs already keep. Any of the five can also send them with saw-agent tests submit, and Jenkins and Buildkite do for now: Buildkite’s pickup starts with launching on it.
• How By Dawn launches on it. A launch starts your pipeline with four inputs, the launch’s ID among them. Launching on these five isn’t available yet, but each section says what a target will need and how a pipeline accepts the inputs, so you can get it ready.

Your tests run on your CI, never on By Dawn’s servers. By Dawn only reads what your jobs wrote, and starts a pipeline you’ve told it to start. GitHub Actions has its own pages: Pick up results from GitHub Actions and Launch test runs.

	How By Dawn reaches it	How results come back	Launching
GitLab CI	The project’s GitLab connection	By Dawn reads finished pipelines every 5 minutes	Not available yet
Azure Pipelines	The organization’s Azure DevOps connection	By Dawn reads finished builds every 5 minutes	Not available yet
Jenkins	A CI connection	saw-agent tests submit only	Not available yet
CircleCI	A CI connection	CircleCI’s webhook, as each job finishes	Not available yet
Buildkite	A CI connection	saw-agent tests submit, until launching starts its pickup	Not available yet

Before you start

• Test results have to be on for your organization (Turn on test results). Without them By Dawn picks up nothing, and refuses what your CI sends.
• CI connections need launching too. Settings›, then Organization›, then CI connections is part of launching test runs, so without it the page shows Launching test runs is not on your plan. Contact support to turn it on. That includes CircleCI’s webhook, so on a plan without launching, CircleCI sends its results with saw-agent tests submit.
• The repo is connected to a By Dawn project on the Repos page. By Dawn files each result under the repo whose remote matches the pipeline’s, so a pipeline of a repo By Dawn doesn’t have is skipped.

Add a CI connection

Jenkins, CircleCI and Buildkite don’t have an app By Dawn can install, so By Dawn signs in to them with an API token. A CI connection holds that token for your whole organization. Every member can see the list, but only an org admin or owner can add, change, test or delete a connection, so a member who needs one asks an admin.

1. Open Settings›, then Organization›, then CI connections and choose Add connection.

2. Pick the CI system: Jenkins, CircleCI or Buildkite. It can’t be changed later.

3. Give it a Name, such as Jenkins build farm. Two connections in an organization can’t share a name.

4. Fill in the fields for that system:

   	Base URL	Username	API token
   Jenkins	Your Jenkins address, with any folder path, such as https://jenkins.example.com/	The Jenkins user the token belongs to	An API token of that user, from their Security settings in Jenkins
   CircleCI	Empty for circleci.com, or your CircleCI server’s address	—	A personal API token, from User settings › Personal API tokens in CircleCI
   Buildkite	— (always api.buildkite.com)	—	An API access token with read and write builds access, from Personal settings › API access tokens

5. Choose Add connection. By Dawn stores the token sealed, so nobody sees it again, not even an admin, and tests the connection straight away.

The token acts as its user, so By Dawn can do whatever that user can in your CI. Use a dedicated user with access to only the jobs and projects you want By Dawn to see, rather than your own account.

By Dawn only calls public https addresses. A Jenkins or CircleCI server behind your firewall needs a public https endpoint, because By Dawn never calls a private, internal or local address, so that a connection can’t be pointed at By Dawn’s own network.

Test a connection

Each connection says Token saved or No token yet, and when it was last verified. Choose Test to check it again: By Dawn makes one read as the token’s user and says Connected as that user, or what went wrong:

The test says	What to do
Jenkins didn’t accept the token.	Check the username and token, then replace the token.
… accepted the token, but it doesn’t have the access SAW needs.	Give the token’s user more access in your CI, or use another user’s token.
Nothing answered as … at that address.	Check the base URL.
SAW couldn’t reach … at that address.	Check the base URL, and that your CI is up and reachable from the internet.
SAW only calls public https addresses	The address is private or internal. Give By Dawn a public https endpoint.
The saved token was revoked.	Replace the token.

For a Jenkins test to pass, Jenkins has to know the user: a Jenkins that answers anonymous callers fails the test rather than passing as nobody.

Replace the token or delete a connection

Replace token puts a new token in at once, and By Dawn stops using the old one. Revoke the old one in your CI yourself. Edit changes the name, username or address. A new address needs the token again, because By Dawn only ever sends a token to the address it was given for, so the saved one is removed and the connection is tested with the new one.

Delete revokes the saved token and stops By Dawn starting runs through the connection. Launch targets that use it fail until you pick another connection for them.

GitLab CI

Connect GitLab

By Dawn reads your pipelines through the project’s GitLab connection, the same one a GitLab import uses. If the project doesn’t have one, a project admin adds it in Settings›, then Import & sync›, then New import, as Import from GitLab explains, for gitlab.com or your own GitLab. The connection asks GitLab for the api scope and acts as the GitLab user who approved it, so By Dawn reads the pipelines that user can see.

A connection that’s still a draft, or has ended, isn’t used. If your import has ended, connect GitLab again for Tests.

What By Dawn picks up

Every 5 minutes By Dawn reads the finished pipelines of each repo in the project whose remote is on that GitLab: on any branch, passed, failed or cancelled. The first read goes back 30 days. GitLab doesn’t tell By Dawn when a pipeline ends, which is why By Dawn looks on a cadence.

In each pipeline, By Dawn looks at the latest attempt of each job that passed or failed, if the job kept an artifact and either declared a test or coverage report (artifacts:reports:junit, cobertura or coverage_report) or has a name that says it holds test output. It reads the reports in two places:

• The job’s artifacts (artifacts:paths), matched by file name the same way as GitHub’s. A job whose name starts with saw-test-results has every file read by its content, whatever it’s called.
• GitLab’s own test report, when the artifacts hold no results but the job declared artifacts:reports:junit. By Dawn reads the report GitLab built from your JUnit files instead.

Coverage only comes from the artifacts, so keep your coverage file under artifacts:paths as well as artifacts:reports. Add when: always, so a job whose tests failed still keeps its reports:

test:
  script:
    - npx vitest run --coverage --reporter=default --reporter=junit --outputFile.junit=reports/junit.xml
  artifacts:
    when: always
    paths:
      - reports/
      - coverage/cobertura-coverage.xml
    reports:
      junit: reports/junit.xml

Each job becomes one run in By Dawn, on the pipeline’s commit and branch, and the jobs of one pipeline read as one stitched run. A job you retry is recorded as a new attempt. The GitHub limits hold here too, such as 100 MB for one job’s artifacts.

To send results from a job yourself instead, add saw-agent tests submit to it. Pick one way per job, because a job that does both records its results twice.

Launch on GitLab CI

Launching on GitLab CI isn’t available yet: the app lists it under Runs on as not available yet. When it is, a project admin adds the target, because it runs pipelines as the GitLab connection’s user. The target names:

• the GitLab connection it runs through
• the Pipeline: the GitLab project, by its full path (group/sub/project) or its numeric ID
• a Branch or tag. GitLab starts a pipeline at a branch or tag, not at a commit, so a launch pinned to a commit needs that commit to be the branch’s head, as on GitHub.

By Dawn creates the pipeline with GitLab’s API and hands it saw_launch_id, saw_suite, saw_environment and saw_configuration as pipeline variables, with each input of the target’s own. Inputs become GitLab variables, so their names take letters, digits and _ only. You don’t declare the variables, but GitLab has to let the connection’s user set them: under Settings › CI/CD › Variables in GitLab, allow pipeline variables for at least that user’s role. The user also needs the Developer role in the project, and leave to push to the branch if it’s protected.

Jobs read the variables like any other. A job with rules has to run for a pipeline started through the API:

e2e:
  rules:
    - if: $CI_PIPELINE_SOURCE == "api"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - npx playwright test --reporter=junit
  variables:
    PLAYWRIGHT_JUNIT_OUTPUT_NAME: reports/junit.xml
    TARGET_ENV: $saw_environment
  artifacts:
    when: always
    reports:
      junit: reports/junit.xml

By Dawn picks up a pipeline it launched as soon as it ends and files the results under the launch, so the job needs no submit step.

Azure Pipelines

Connect Azure DevOps

By Dawn reads your builds through your organization’s Azure DevOps connection, the one it uses for repos and pull requests. An org owner or admin makes it once in Settings›, then Organization›, then Azure DevOps (Repos has more). It covers repos on dev.azure.com in the connected Azure DevOps organization. A repo on *.visualstudio.com, Azure DevOps Server or another Azure DevOps organization isn’t read.

What By Dawn picks up

Every 5 minutes By Dawn reads the builds that finished in the last two hours, for every Azure DevOps repo connected to By Dawn: on any branch, whatever their result. Azure DevOps doesn’t tell By Dawn when a build ends, which is why By Dawn looks on a cadence. When you connect a repo, By Dawn also reads its last 30 days of builds, up to 200.

By Dawn reads two things from a build:

• Its published test runs. Each test run your pipeline publishes, with the PublishTestResults task or VSTest, becomes one run in By Dawn, named after the test run. By Dawn reads the run’s attached report file if it can, and otherwise the results Azure DevOps recorded.
• Its artifacts, matched by artifact name and file name the same way as GitHub’s. When the build published test runs, By Dawn takes only coverage from its artifacts, apart from an artifact whose name starts with saw-test-results.

So a pipeline that already publishes its test results needs no change:

- script: npx vitest run --reporter=default --reporter=junit --outputFile.junit=reports/junit.xml
  displayName: Test
- task: PublishTestResults@2
  condition: succeededOrFailed()
  inputs:
    testResultsFormat: JUnit
    testResultsFiles: reports/junit.xml

Results land on the build’s commit and branch. For a pull request build, that’s the pull request’s own head commit, not the merge commit Azure DevOps builds.

To send results from a job yourself instead, add saw-agent tests submit. Pick one way per pipeline, because a build that does both records its results twice.

Launch on Azure Pipelines

Launching on Azure Pipelines isn’t available yet: the app lists it under Runs on as not available yet. When it is, a project admin adds the target, which names:

• the Pipeline: its ID (the definitionId in its URL) or its exact name. It has to build the target’s repo from Azure Repos, so a target can’t start another repo’s pipeline.
• a Branch or tag. Azure Pipelines runs at an exact commit, so unlike GitHub, a launch can pin any commit on it.

By Dawn starts a run of the pipeline with your organization’s Azure DevOps connection, which needs permission to queue builds of that pipeline. It hands the pipeline saw_launch_id, saw_suite, saw_environment and saw_configuration, with each input of the target’s own, as runtime parameters. Azure DevOps refuses a run that sends a parameter the YAML doesn’t declare, so declare all four, even the ones your tests don’t use. Give each a default, so a run By Dawn didn’t start, such as a push, still queues. A single space will do, because saw-agent treats a blank --run as no launch:

parameters:
  - name: saw_launch_id
    type: string
    default: ' '
  - name: saw_suite
    type: string
    default: ' '
  - name: saw_environment
    type: string
    default: ' '
  - name: saw_configuration
    type: string
    default: ' '

steps:
  - script: npx playwright test --reporter=junit
    displayName: Test
    env:
      PLAYWRIGHT_JUNIT_OUTPUT_NAME: reports/junit.xml
      TARGET_ENV: ${{ parameters.saw_environment }}
  - task: PublishTestResults@2
    condition: succeededOrFailed()
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: reports/junit.xml

By Dawn picks up a build it launched as soon as it ends and files its test runs under the launch, so the pipeline needs no submit step. If you send results with saw-agent tests submit instead, pass --run the launch ID through env, as SAW_LAUNCH_ID: ${{ parameters.saw_launch_id }}, and use "$SAW_LAUNCH_ID" in the script. Azure Pipelines pastes a ${{ }} expression into the script before the shell reads it, so env keeps a value from ever being read as a command.

Jenkins

Connect Jenkins

Add a CI connection for Jenkins, with its address, a user and that user’s API token. Your Jenkins has to be reachable from the internet over https.

Send results

By Dawn doesn’t pick up results from Jenkins, so every Jenkins job sends them with saw-agent tests submit, from a post { always { … } } block. Send test results from CI has the pipeline, under Jenkins.

Launch on Jenkins

Launching on Jenkins isn’t available yet. The app offers Jenkins under Runs on, but By Dawn refuses to save a Jenkins target until launching on Jenkins is turned on. When it is, By Dawn builds a job with Jenkins’ buildWithParameters through a Jenkins connection, as the connection’s user. That user can build whatever it can see, a deploy job too, so By Dawn builds only the jobs on the project’s list, with only the parameters the list names.

A project admin or org admin keeps that list in Settings›, then This project›, then Test launches, under Jenkins jobs SAW may build. Until the project has one, By Dawn builds no Jenkins job in it.

1. Choose Add job and give the Job path as Jenkins’ folders show it, such as team/portal/e2e, not its URL. For a multibranch job, the path ends with the branch.

2. Under Parameters a launch may send, keep By Dawn’s six: saw_launch_id, saw_suite, saw_environment, saw_configuration, saw_ref and saw_sha. A new row starts with them, and without them no launch target can use the job. Add any parameter of your own a target should set, up to twelve in all.

3. Add What it does, such as “Playwright end-to-end tests”, if it helps, and choose Save. A project lists at most 20 jobs.

A Jenkins target names the Jenkins connection and the Job. By Dawn checks the list again at every launch, so taking a job or a parameter off it stops its targets launching.

The job declares saw_launch_id, saw_suite, saw_environment and saw_configuration as string parameters, with each parameter of the target’s own, and saw_ref and saw_sha if it wants the branch and commit the launch asked for. Jenkins drops a parameter the job doesn’t declare. Then it passes saw_launch_id to saw-agent tests submit as --run, which files the results under the launch:

pipeline {
  agent any
  parameters {
    string(name: 'saw_launch_id', defaultValue: '')
    string(name: 'saw_suite', defaultValue: '')
    string(name: 'saw_environment', defaultValue: '')
    string(name: 'saw_configuration', defaultValue: '')
  }
  environment {
    SAW_PROJECT = 'project-3a86…'
    SAW_API_URL = 'https://workspace.dev.sawrun.com'
  }
  stages {
    stage('Test') {
      steps {
        sh 'PLAYWRIGHT_JUNIT_OUTPUT_NAME=reports/junit.xml TARGET_ENV="$saw_environment" npx playwright test --reporter=junit'
      }
    }
  }
  post {
    always {
      withCredentials([string(credentialsId: 'saw-api-key', variable: 'SAW_API_KEY')]) {
        sh '''
          curl -fsSL https://github.com/LenderCom/saw-agent-releases/releases/latest/download/saw-agent_linux_amd64.tar.gz | tar -xz
          ./saw-agent tests submit --results 'reports/*.xml' --job e2e --run "$saw_launch_id"
        '''
      }
    }
  }
}

The ''' keeps Groovy from filling in $saw_launch_id itself, so the shell reads it from the build’s environment. On a build By Dawn didn’t start the parameter is empty, and --run falls back to the build’s own tag.

Jenkins answers a build request with a queue item, not a build, so By Dawn finds the build by its saw_launch_id. A job that doesn’t declare it can’t be found, and the launch fails after 10 minutes. The connection’s user needs the Job › Build permission to start the job, and Overall › Read and Job › Read to follow it.

CircleCI

Connect CircleCI

Add a CI connection for CircleCI, with a personal API token. Use a token whose user is an organization admin in CircleCI if you can: that’s what lets By Dawn add its webhook to your projects itself.

Add By Dawn’s webhook

CircleCI tells By Dawn each time a job finishes, through an outbound webhook on each CircleCI project. When you test the connection, By Dawn adds a webhook named SawRun test results to the CircleCI project of each of your organization’s repos on GitHub or Bitbucket, and the connection lists them: Webhook added or Not hooked. It does the same for a repo you connect later. If this deployment of By Dawn has no public webhook address, the connection says so, and CircleCI’s results reach By Dawn only through saw-agent tests submit.

By Dawn can’t add the webhook to a project in two cases, and the connection says which:

• The token’s user isn’t a CircleCI organization admin. Only an admin may add webhooks. Use an admin’s token, or add the webhook by hand.
• The project is named by ID, such as circleci/…/…, as projects set up through CircleCI’s GitHub App or its GitLab integration are. By Dawn can’t work out the project from the repo, so add the webhook by hand.

To add it by hand, choose Signing secret on the connection, which shows the Webhook URL and the Signing secret. Then, in CircleCI:

1. Open the project’s Project Settings › Webhooks and choose Add Webhook.

2. Paste the URL and the secret. Tick Workflow Completed and Job Completed, and keep certificate verification on.

3. Save the webhook.

By Dawn checks every delivery against the secret, so a webhook that isn’t signed with it is refused. Rotate secret makes a new one and updates the webhooks By Dawn added itself. Webhooks you added by hand stop working until you paste the new secret into each, so rotate only when you need to, such as when the secret has leaked.

What By Dawn picks up

When a job passes or fails, By Dawn reads its reports:

• Its stored artifacts (store_artifacts), matched by file name the same way as GitHub’s. A job whose name starts with saw-test-results has every file read by its content.
• Its test metadata (store_test_results), when its artifacts hold no results, up to 20,000 tests a job. Coverage still comes from its artifacts.

So a job that stores its results already needs no change:

jobs:
  test:
    docker:
      - image: cimg/node:22.11
    steps:
      - checkout
      - run: npx vitest run --reporter=default --reporter=junit --outputFile.junit=reports/junit.xml
      - store_test_results:
          path: reports

Each job becomes one run in By Dawn, on the pipeline’s commit and branch, and the jobs of one pipeline read as one stitched run. Re-running a workflow records a new attempt. The GitHub limits hold here too.

If you can’t add the webhook, send the results with saw-agent tests submit instead. Pick one way per job, because a job that does both records its results twice.

Launch on CircleCI

Launching on CircleCI isn’t available yet: the app lists it under Runs on as not available yet. When it is, a project admin adds the target, which names:

• the CircleCI connection it runs through. The token’s user needs write access to the project.
• the Pipeline: the project slug from CircleCI’s Project Settings, such as gh/acme/portal or circleci/<org-id>/<project-id>
• a Branch or tag. CircleCI starts a pipeline at a branch or tag, not at a commit, so By Dawn checks the commit the pipeline checked out. If the launch was pinned to another one, By Dawn cancels the pipeline and the launch fails, saying which commit the branch is at.

By Dawn triggers the pipeline with saw_launch_id, saw_suite, saw_environment and saw_configuration as pipeline parameters, with each input of the target’s own. CircleCI refuses a parameter the config doesn’t declare, so declare each one as a string with a default of "":

version: 2.1

parameters:
  saw_launch_id: { type: string, default: "" }
  saw_suite: { type: string, default: "" }
  saw_environment: { type: string, default: "" }
  saw_configuration: { type: string, default: "" }

jobs:
  e2e:
    docker:
      - image: mcr.microsoft.com/playwright:v1.49.0-noble
    environment:
      PLAYWRIGHT_JUNIT_OUTPUT_NAME: reports/junit.xml
      TARGET_ENV: << pipeline.parameters.saw_environment >>
    steps:
      - checkout
      - run: npx playwright test --reporter=junit
      - store_test_results:
          path: reports

A workflow with filters or a when condition has to run for the branch the launch starts, or CircleCI starts no workflow and the launch fails. With the webhook in place, By Dawn files the jobs of a pipeline it launched under the launch, so the job needs no submit step.

Buildkite

Connect Buildkite

Add a CI connection for Buildkite, with an API access token that has read and write builds access to your Buildkite organization. By Dawn always calls api.buildkite.com.

Send results

Buildkite doesn’t tell By Dawn when a build ends, so By Dawn reads builds of the pipelines your Buildkite launch targets name, every 5 minutes. Until launching on Buildkite is available, no target names a pipeline yet, so send results with saw-agent tests submit from the step that ran the tests. Send test results from CI has the step, under Buildkite.

Launch on Buildkite

Launching on Buildkite isn’t available yet: the app lists it under Runs on as not available yet. When it is, a project admin adds the target, which names:

• the Buildkite connection it runs through
• the Pipeline: organization-slug/pipeline-slug, as the pipeline’s page shows it. buildkite.com/acme/portal-e2e is acme/portal-e2e.
• a Branch or tag. Buildkite builds an exact commit, so a launch can pin any commit on it.

By Dawn creates a build with saw_launch_id, saw_suite, saw_environment and saw_configuration as environment variables and build meta-data, with saw_ref and saw_sha when the launch names a branch or commit, and each input of the target’s own. You don’t declare them. Inputs become environment variables, so their names take letters, digits and _ only, and can’t be one of Buildkite’s own BUILDKITE_ variables or one that changes how a program starts, such as PATH, LD_PRELOAD or NODE_OPTIONS.

From then on, By Dawn picks up the target’s pipeline the way it does CircleCI’s jobs: when a build it launched ends, and every 5 minutes for the pipeline’s other builds, 30 days back the first time. It reads each passed or failed job’s uploaded artifacts (buildkite-agent artifact upload), matched by path the same way as GitHub’s, and a path under saw-test-results/ has every file read by its content. Each job is one run in By Dawn, the jobs of one build read as one stitched run, and a step with parallelism becomes the shards of one job:

steps:
  - label: "e2e"
    command: |
      TARGET_ENV="$$saw_environment" npx playwright test --reporter=junit
    env:
      PLAYWRIGHT_JUNIT_OUTPUT_NAME: reports/junit.xml
    artifact_paths: "reports/junit.xml"

The $$ keeps Buildkite from filling in the variable when it uploads the pipeline, so the shell reads it when the step runs. A build By Dawn launched files its results under the launch, so the step needs no submit step. If you send results with saw-agent tests submit instead, pass --run "$$saw_launch_id".

FAQ

Why isn't launching available on my CI yet?

By Dawn only starts a pipeline you’ve fenced: a target a project admin saved for GitLab CI, Azure Pipelines, CircleCI and Buildkite, and the project’s job list for Jenkins, as the workflow allowlist fences GitHub Actions. The app offers each system under Runs on once By Dawn launches on it, and until then a launch target runs on GitHub Actions. Results don’t wait: they come back from all five today.

My pipeline finished, but no results appear.

Check that test results are on for your organization, that the repo is connected to a By Dawn project on the Repos page under the same remote, and that the job kept its reports with names By Dawn looks for. GitLab and Azure Pipelines are read every 5 minutes, so give a pipeline a few minutes. For CircleCI, check that the connection says Webhook added for the project.

Can I use pickup and saw-agent tests submit together?

Yes, but not in the same job. Both are CI results, so a job that does both records each report twice, as two runs on the same commit. Pick one way per job.

Does By Dawn need a token for GitLab CI or Azure Pipelines?

No. It uses the project’s GitLab connection and your organization’s Azure DevOps connection, so there’s no token to paste. CI connections are only for Jenkins, CircleCI and Buildkite.

Our Jenkins is only on our internal network. Can By Dawn reach it?

Not for launching, because By Dawn’s servers only call public https addresses. Your jobs can still send results with saw-agent tests submit, because that call goes from your Jenkins to By Dawn.

Does a launch use my CI minutes?

Yes. The pipeline runs on your CI like any other run, on your runners or agents and your minutes.

Next

• Send test results from CI: the key, the project ID and the flags for saw-agent tests submit in each CI.
• Launch test runs: suites, targets, Run tests and following a launch.
• Test results: what By Dawn does with the results once they’re in.
• Make your test runner write reports: get your runner to write a file By Dawn reads.