Launch test runs

This page shows you how to start a test suite on your own CI from By Dawn, by a button rather than a push. By the end you’ll have a suite with a workflow it launches, and each launch on the suite’s page with its state and the results the workflow sent back.

A launch is By Dawn asking your CI to run one suite once. By Dawn starts the workflow and passes it a few inputs, the workflow runs your tests and sends its reports back, and By Dawn files them under the launch. Your tests run on your CI, never on By Dawn’s servers. By Dawn only starts the run and reads what it wrote. Suites and their launches are in Tests›, then Suites.

Launching works with GitHub Actions today. Set up your CI gets GitLab CI, Azure Pipelines, Jenkins, CircleCI and Buildkite pipelines ready for it. An agent can launch a suite too, and so can a test plan.

Before you start

• Launching tests has to be on for your organization, and test management for the Suites tab itself. Without launching, a suite shows Launching test runs is not on your plan where its targets and launches would be. Contact support to turn it on.
• Members of the project make suites, add targets, launch and cancel. Someone with view-only access can follow the launches.
• A project admin or an org admin allows the workflow, because By Dawn only starts workflows on the project’s allowlist (below).
• The repo is on GitHub, connected to the project, with By Dawn’s GitHub App installed on its account.
• The workflow sends its results to By Dawn, with saw-agent tests submit or as artifacts By Dawn picks up. The steps below add the one line a launch needs.

Make a suite

A suite is a saved set of cases you run together, such as Smoke or Full regression. On the Suites tab, choose New suite:

• Name, such as Smoke. Two suites in a project can’t share a name.
• Description: what it checks and when to run it.
• Regression suite: By Dawn drafts every regression suite into each release’s and cycle’s test plan.
• Cases it takes: folders (From folders, with Include their sub-folders), single cases to add or Leave out, and Only cases that are… a kind, priority, status or tag. With no folder and no case picked, it takes the whole library. A preview counts the cases as you go, such as “48 cases · 40 automated, 8 manual”.

Choose Create suite. The suite opens beside the list, which shows each suite’s cases and where it runs, such as “48 cases · nowhere to run yet”. The pencil edits it, and Archive suite retires one you no longer run. An archived suite says Archived — it can’t launch until you choose Restore suite.

Allow the workflow

By Dawn only starts a workflow that’s on the project’s allowlist, with only the inputs the list allows. That’s what stops anyone, person or agent, from starting an arbitrary workflow with arbitrary inputs from By Dawn.

1. Open Settings›, then This project›, then GitHub and find Workflows agents may start. Only a project admin or an org admin can change it.

2. Choose Add workflow and give the Workflow file, such as e2e.yml. It’s the file’s name under .github/workflows/, without the folder.

3. Under Inputs agents may set, list By Dawn’s four inputs: saw_launch_id, saw_suite, saw_environment, saw_configuration. Add any input of your own that a launch should set, such as browser, up to ten in all.

4. Add What it does, such as “Playwright end-to-end tests”, if it helps. The launch target’s workflow picker shows it.

5. Choose Save.

Until you change it, the list is SAW’s default, which holds only update-baselines.yml, so a new project can’t launch any suite. Saving your own list replaces the default, so add update-baselines.yml back if your agents use it. The same list lets a ticket’s agent start the workflow on its own branch (Start CI workflows).

Accept By Dawn’s inputs in the workflow

By Dawn starts the workflow through GitHub’s workflow_dispatch trigger, with four inputs. GitHub refuses a dispatch that sends an input the workflow doesn’t declare, so declare all four, even the ones your tests don’t use:

Input	What By Dawn sends
saw_launch_id	The launch’s ID, such as tlaunch-6d0f…. Send it back with the results.
saw_suite	The suite’s name.
saw_environment	The test environment’s name, such as Staging, or nothing if the launch names none.
saw_configuration	The configuration’s name, such as Chrome, or nothing.

Then pass the launch ID to saw-agent tests submit as --run. That ties the results to the launch, and keeps every shard of the run as one test run:

on:
  push:
    branches: [main]
  workflow_dispatch:
    inputs:
      saw_launch_id: { type: string, required: false }
      saw_suite: { type: string, required: false }
      saw_environment: { type: string, required: false }
      saw_configuration: { type: string, required: false }

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx playwright test --reporter=junit
        env:
          PLAYWRIGHT_JUNIT_OUTPUT_NAME: reports/junit.xml
          TARGET_ENV: ${{ inputs.saw_environment }}
      - name: Send test results to SAW
        if: ${{ !cancelled() }}
        env:
          SAW_API_KEY: ${{ secrets.SAW_API_KEY }}
          SAW_PROJECT: ${{ vars.SAW_PROJECT }}
          SAW_API_URL: https://workspace.dev.sawrun.com
          SAW_LAUNCH_ID: ${{ inputs.saw_launch_id }}
        run: |
          curl -fsSL https://github.com/LenderCom/saw-agent-releases/releases/latest/download/saw-agent_linux_amd64.tar.gz | tar -xz
          ./saw-agent tests submit --results 'reports/*.xml' --run "$SAW_LAUNCH_ID"

The launch ID reaches the script through env, not as ${{ inputs.saw_launch_id }} written into run:: GitHub pastes a ${{ }} expression into the script before the shell reads it, so anyone who can start the workflow could end the quotes and run their own commands next to SAW_API_KEY.

On a push the input is empty, so --run falls back to the workflow run’s own ID and the step works as before. Send test results from CI explains the key, the project ID and the other flags.

If By Dawn already picks up this workflow’s artifacts, you don’t need the submit step: GitHub tells By Dawn which run it started, so the artifacts land on the launch by themselves.

Add a launch target

A launch target says where a suite runs: which workflow, in which repo, at which branch. A suite can have several, such as one on staging and one on production, and one of them is its default.

Open the suite on the Suites tab. Under Where it runs, choose Add target:

• Name, such as CI · staging.
• Runs on: GitHub Actions. The other CI systems and Your machines say not available yet.
• Repo: the project’s repo that holds the workflow.
• Workflow: picked from the allowlist, each with what it does. Under it, the target lists the inputs of your own the workflow’s entry Allows. If the entry is missing one of By Dawn’s four inputs, it says “Its allowlist entry must allow” and names them, and Edit the allowlist in Settings → GitHub takes you there. If the allowlist is empty, it says The project’s workflow allowlist is empty, so SAW can’t start any workflow yet.
• Branch or tag: where the workflow runs. Blank is the repo’s default branch.
• Timeout (minutes): how long a launch may take, counted from when the launch is queued, before By Dawn gives up on it and cancels the run. From 1 to 1,440, and 60 unless you change it.
• Environment and Configuration: the project’s test environments and configurations, the same ones a manual run uses, or None. By Dawn sends their names as saw_environment and saw_configuration.
• Inputs: inputs of your own, sent on every launch, such as browser = chromium. Up to ten, each allowed by the workflow’s entry. Names starting saw_ are By Dawn’s own, and values are plain text: no quotes, $, backticks, backslashes or line breaks.
• Default target: the one Run tests picks first. A suite’s first target is its default.

Choose Add target. The target is listed under Where it runs, with what it runs and where, such as “e2e.yml · saw-ui @ main · Staging · 60 min timeout”. The pencil edits it, except Runs on. The bin removes it, and the launches it made keep their record.

By Dawn checks the allowlist again at every launch, so taking a workflow or an input off the list stops its targets launching until it’s back.

Run tests

Choose Run tests on the suite, or in the Tests page’s header to pick the suite there. The dialog asks:

• Suite, and Where: its targets, with its default picked.
• Environment and Configuration, the target’s to start with.
• Branch, tag or commit: blank runs at the target’s branch. A full commit ID pins the commit, but GitHub only starts a workflow at a branch or tag, so the commit has to be the head of the target’s branch.
• Extra inputs: inputs of your own for this launch only, over the target’s. The workflow’s allowlist entry must allow each one.

Choose Run tests. The dialog closes and the new launch opens at the top of the suite’s Launches, as Queued. Within about 15 seconds By Dawn starts the workflow, and the launch moves to Running once GitHub has the run.

If By Dawn can’t launch, the dialog says why, and nothing is started:

The dialog says	Why, and what to do
This suite has nowhere to run yet	It has no launch target. Add a target opens the form.
The workflow isn’t allowed to start	The workflow, or an input the launch sends, is off the allowlist. A project admin adds it in Settings›, then This project›, then GitHub.
SAW can’t launch on this provider yet	The target runs on a system By Dawn doesn’t launch on yet. Pick a GitHub Actions target.
This suite is archived	Restore it to run it again.
Couldn’t start the run	Anything else, with By Dawn’s reason.

Follow a launch

The suite’s Launches list its launches, newest first, and Show older launches pages back. Each says what it ran and where, such as “e2e.yml on GitHub Actions”, the commit or branch, when it started and for how long, and its state. Open on GitHub opens the workflow run. A live launch updates every few seconds.

State	Means
Queued	By Dawn is about to start it.
Starting	By Dawn is asking GitHub to start the run.
Running	The run has started.
Completed	The run ended and sent its results.
Failed	The run couldn’t start, or ended without sending results. The launch says why.
Cancelled	Someone stopped it.
Timed out	It didn’t end within the target’s timeout. By Dawn asked GitHub to cancel the run.

The state is about the run, not the tests: a launch whose tests failed is still Completed. Its results say how the tests went.

Open a launch to see more: the repo, environment and configuration; what started it and who, such as “Run tests · Dana Kim” or “Agent · A SAW worker”; what GitHub said of the run, such as CI said "failure"; its ticket, if an agent launched it; and its results, the test runs it filled, read the way a ticket’s Tests panel reads them. Until the workflow sends them, it says No results yet: they show here once the run sends them.

By Dawn gives a run that has ended ten minutes for its results to arrive. If none do, the launch fails with “The workflow run ended without sending any results to this launch.” Results that arrive later still complete it.

To stop a launch that’s still going, choose Cancel on it. A launch By Dawn hasn’t started yet is cancelled at once. One that’s running asks GitHub to cancel the workflow run.

Launch from a test plan

An automated item in a test plan launches its suite on one of its targets, and the launch counts toward the plan. Plan testing for a release or cycle explains it.

Launch from an agent

A ticket’s agent can launch a suite itself, such as an end-to-end suite before it opens a pull request:

saw tests run checkout-e2e --wait

It names the suite by its name or ID, and runs on the suite’s default target unless it names another. With no commit or branch, it launches at the head of the ticket’s own branch. --wait follows the launch until it ends. The launch shows on the suite as started by Agent, with a link to the ticket. An agent can only read and cancel the launches its own ticket started, and every rule above applies to it too: the target, the allowlist and your plan.

FAQ

The launch says Completed, but tests failed. Is that right?

Yes. Completed means the workflow ran to the end and sent its results, whatever they say. Open the launch to see which tests failed.

Why did my launch fail with "GitHub refused to start"?

The workflow has no workflow_dispatch trigger, or doesn’t declare an input the launch sends. Declare By Dawn’s four inputs and each input of your own, as in Accept By Dawn’s inputs, and merge it to the branch the target runs on.

Why does it say GitHub has no workflow?

The workflow file isn’t on the branch the launch ran at. Check the target’s Branch or tag, and that the file’s name on the allowlist matches the file in .github/workflows/.

Why can't I pin an older commit?

GitHub starts a workflow at a branch or tag, not at a commit. By Dawn refuses a commit that isn’t the head of the branch the launch runs at, rather than test something else. Launch at a branch or tag whose head is that commit.

Why are GitLab CI and Your machines greyed out?

The app launches on GitHub Actions today, and the other choices say not available yet until they can. Jenkins is listed too, but By Dawn refuses a Jenkins target until launching on Jenkins is turned on. Set up your CI says what each of them will need, so you can get your pipelines ready.

Can a schedule, a pull request or a deploy start a suite?

Not from the app yet. A launch starts from Run tests, a test plan or an agent.

Does a launch use my CI minutes?

Yes. The workflow runs on your GitHub Actions like any other run, on your runners and your minutes.

Next

• Send test results from CI: the key, the project ID and the flags for the results step.
• Test results: how By Dawn reads the results a launch brings back.
• Plan testing for a release or cycle: launch your regression suites as part of a test plan.
• Set up your CI: results and launching on GitLab CI, Azure Pipelines, Jenkins, CircleCI and Buildkite.
• Agent access: what else the workflow allowlist lets agents start.