# Launch test runs

> Run a test suite on your own GitHub Actions from By Dawn — save the suite, allow its workflow, accept the inputs By Dawn sends, press Run tests, and follow the launch until its results are in.

This page shows you how to start a test suite on your own CI from By Dawn, by a button rather than a push. By the end
you'll have a suite with a workflow it launches, and each launch on the suite's page with its state and the results
the workflow sent back.

A **launch** is By Dawn asking your CI to run one suite once. By Dawn starts the workflow and passes it a few inputs, the
workflow runs your tests and sends its reports back, and By Dawn files them under the launch. **Your tests run on your
CI, never on By Dawn's servers.** By Dawn only starts the run and reads what it wrote. Suites and their launches are in
**Tests › Suites**.

Launching works with GitHub Actions today. [Set up your CI](https://docs.bydawn.com/tests-ci/) gets GitLab CI, Azure Pipelines, Jenkins,
CircleCI and Buildkite pipelines ready for it. An agent can launch a suite too, and so can a
[test plan](https://docs.bydawn.com/test-plans/#run-the-plan).

## Before you start

- **Launching tests has to be on** for your organization, and test management for the **Suites** tab itself.
  Without launching, a suite shows **Launching test runs is not on your plan** where its targets and launches would
  be. [Contact support](https://app.dev.sawrun.com/support) to turn it on.
- **Members of the project make suites, add targets, launch and cancel.** Someone with view-only access can follow
  the launches.
- **A project admin or an org admin allows the workflow**, because By Dawn only starts workflows on the project's
  allowlist (below).
- **The repo is on GitHub**, connected to the project, with By Dawn's GitHub App installed on its account.
- **The workflow sends its results to By Dawn**, with [`saw-agent tests submit`](https://docs.bydawn.com/developers/tests-submit/) or as
  artifacts By Dawn [picks up](https://docs.bydawn.com/tests-github-artifacts/). The steps below add the one line a launch needs.

## Make a suite

A suite is a saved set of cases you run together, such as Smoke or Full regression. On the **Suites** tab, choose
**New suite**:

- **Name**, such as `Smoke`. Two suites in a project can't share a name.
- **Description**: what it checks and when to run it.
- **Regression suite**: By Dawn drafts every regression suite into each release's and cycle's
  [test plan](https://docs.bydawn.com/test-plans/#mark-your-regression-suites).
- **Cases it takes**: folders (**From folders**, with **Include their sub-folders**), single cases to add or
  **Leave out**, and **Only cases that are…** a kind, priority, status or tag. With no folder and no case picked, it
  takes the whole library. A preview counts the cases as you go, such as "48 cases · 40 automated, 8 manual".

Choose **Create suite**. The suite opens beside the list, which shows each suite's cases and where it runs, such as
"48 cases · nowhere to run yet". The pencil edits it, and **Archive suite** retires one you no longer run. An
archived suite says **Archived — it can't launch** until you choose **Restore suite**.

## Allow the workflow

By Dawn only starts a workflow that's on the project's allowlist, with only the inputs the list allows. That's what
stops anyone, person or agent, from starting an arbitrary workflow with arbitrary inputs from By Dawn.

1. Open **Settings › This project › GitHub** and find **Workflows agents may start**. Only a project
   admin or an org admin can change it.

2. Choose **Add workflow** and give the **Workflow file**, such as `e2e.yml`. It's the file's name under
   `.github/workflows/`, without the folder.

3. Under **Inputs agents may set**, list By Dawn's four inputs: `saw_launch_id, saw_suite, saw_environment,
   saw_configuration`. Add any input of your own that a launch should set, such as `browser`, up to ten in all.

4. Add **What it does**, such as "Playwright end-to-end tests", if it helps. The launch target's workflow picker
   shows it.

5. Choose **Save**.

Until you change it, the list is **SAW's default**, which holds only `update-baselines.yml`, so a new project can't
launch any suite. Saving your own list replaces the default, so add `update-baselines.yml` back if your agents use
it. The same list lets a ticket's agent start the workflow on its own branch
([Start CI workflows](https://docs.bydawn.com/agent-access/#start-ci-workflows)).

## Accept By Dawn's inputs in the workflow

By Dawn starts the workflow through GitHub's `workflow_dispatch` trigger, with four inputs. GitHub refuses a dispatch
that sends an input the workflow doesn't declare, so declare all four, even the ones your tests don't use:

| Input | What By Dawn sends |
|---|---|
| `saw_launch_id` | The launch's ID, such as `tlaunch-6d0f…`. Send it back with the results. |
| `saw_suite` | The suite's name. |
| `saw_environment` | The test environment's name, such as `Staging`, or nothing if the launch names none. |
| `saw_configuration` | The configuration's name, such as `Chrome`, or nothing. |

Then pass the launch ID to `saw-agent tests submit` as `--run`. That ties the results to the launch, and keeps every
shard of the run as one test run:

```yaml
on:
  push:
    branches: [main]
  workflow_dispatch:
    inputs:
      saw_launch_id: { type: string, required: false }
      saw_suite: { type: string, required: false }
      saw_environment: { type: string, required: false }
      saw_configuration: { type: string, required: false }

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx playwright test --reporter=junit
        env:
          PLAYWRIGHT_JUNIT_OUTPUT_NAME: reports/junit.xml
          TARGET_ENV: ${{ inputs.saw_environment }}
      - name: Send test results to SAW
        if: ${{ !cancelled() }}
        env:
          SAW_API_KEY: ${{ secrets.SAW_API_KEY }}
          SAW_PROJECT: ${{ vars.SAW_PROJECT }}
          SAW_API_URL: https://workspace.dev.sawrun.com
          SAW_LAUNCH_ID: ${{ inputs.saw_launch_id }}
        run: |
          curl -fsSL https://github.com/LenderCom/saw-agent-releases/releases/latest/download/saw-agent_linux_amd64.tar.gz | tar -xz
          ./saw-agent tests submit --results 'reports/*.xml' --run "$SAW_LAUNCH_ID"
```

The launch ID reaches the script through `env`, not as `${{ inputs.saw_launch_id }}` written into `run:`: GitHub
pastes a `${{ }}` expression into the script before the shell reads it, so anyone who can start the workflow could
end the quotes and run their own commands next to `SAW_API_KEY`.

On a push the input is empty, so `--run` falls back to the workflow run's own ID and the step works as before.
[Send test results from CI](https://docs.bydawn.com/developers/tests-submit/) explains the key, the project ID and the other flags.

If By Dawn already [picks up this workflow's artifacts](https://docs.bydawn.com/tests-github-artifacts/), you don't need the submit step:
GitHub tells By Dawn which run it started, so the artifacts land on the launch by themselves.

## Add a launch target

A launch target says where a suite runs: which workflow, in which repo, at which branch. A suite can have several,
such as one on staging and one on production, and one of them is its default.

Open the suite on the **Suites** tab. Under **Where it runs**, choose **Add target**:

- **Name**, such as `CI · staging`.
- **Runs on**: **GitHub Actions**. The other CI systems and **Your machines** say **not available yet**.
- **Repo**: the project's repo that holds the workflow.
- **Workflow**: picked from the allowlist, each with what it does. Under it, the target lists the inputs of your own
  the workflow's entry **Allows**. If the entry is missing one of By Dawn's four inputs, it says "Its allowlist entry must
  allow" and names them, and **Edit the allowlist in Settings → GitHub** takes you there. If the allowlist is empty,
  it says **The project's workflow allowlist is empty, so SAW can't start any workflow yet.**
- **Branch or tag**: where the workflow runs. Blank is the repo's default branch.
- **Timeout (minutes)**: how long a launch may take, counted from when the launch is queued, before By Dawn gives up
  on it and cancels the run. From 1 to 1,440, and 60 unless you change it.
- **Environment** and **Configuration**: the project's test environments and configurations, the same ones a
  [manual run](https://docs.bydawn.com/test-runs/#start-a-run) uses, or **None**. By Dawn sends their names as `saw_environment` and
  `saw_configuration`.
- **Inputs**: inputs of your own, sent on every launch, such as `browser` = `chromium`. Up to ten, each allowed by the
  workflow's entry. Names starting `saw_` are By Dawn's own, and values are plain text: no quotes, `$`, backticks,
  backslashes or line breaks.
- **Default target**: the one **Run tests** picks first. A suite's first target is its default.

Choose **Add target**. The target is listed under **Where it runs**, with what it runs and where, such as
"e2e.yml · saw-ui @ main · Staging · 60 min timeout". The pencil edits it, except **Runs on**. The bin removes it, and
the launches it made keep their record.

By Dawn checks the allowlist again at every launch, so taking a workflow or an input off the list stops its targets
launching until it's back.

## Run tests

Choose **Run tests** on the suite, or in the **Tests** page's header to pick the suite there. The dialog asks:

- **Suite**, and **Where**: its targets, with its default picked.
- **Environment** and **Configuration**, the target's to start with.
- **Branch, tag or commit**: blank runs at the target's branch. A full commit ID pins the commit, but GitHub only
  starts a workflow at a branch or tag, so the commit has to be the head of the target's branch.
- **Extra inputs**: inputs of your own for this launch only, over the target's. The workflow's allowlist entry must
  allow each one.

Choose **Run tests**. The dialog closes and the new launch opens at the top of the suite's **Launches**, as
**Queued**. Within about 15 seconds By Dawn starts the workflow, and the launch moves to **Running** once GitHub has
the run.

If By Dawn can't launch, the dialog says why, and nothing is started:

| The dialog says | Why, and what to do |
|---|---|
| **This suite has nowhere to run yet** | It has no launch target. **Add a target** opens the form. |
| **The workflow isn't allowed to start** | The workflow, or an input the launch sends, is off the allowlist. A project admin adds it in **Settings › This project › GitHub**. |
| **SAW can't launch on this provider yet** | The target runs on a system By Dawn doesn't launch on yet. Pick a GitHub Actions target. |
| **This suite is archived** | Restore it to run it again. |
| **Couldn't start the run** | Anything else, with By Dawn's reason. |

## Follow a launch

The suite's **Launches** list its launches, newest first, and **Show older launches** pages back. Each says what it
ran and where, such as "e2e.yml on GitHub Actions", the commit or branch, when it started and for how long, and its
state. **Open on GitHub** opens the workflow run. A live launch updates every few seconds.

| State | Means |
|---|---|
| **Queued** | By Dawn is about to start it. |
| **Starting** | By Dawn is asking GitHub to start the run. |
| **Running** | The run has started. |
| **Completed** | The run ended and sent its results. |
| **Failed** | The run couldn't start, or ended without sending results. The launch says why. |
| **Cancelled** | Someone stopped it. |
| **Timed out** | It didn't end within the target's timeout. By Dawn asked GitHub to cancel the run. |

The state is about the run, not the tests: a launch whose tests failed is still **Completed**. Its results say how
the tests went.

Open a launch to see more: the repo, environment and configuration; what started it and who, such as "Run tests ·
Dana Kim" or "Agent · A SAW worker"; what GitHub said of the run, such as `CI said "failure"`; its ticket, if an agent
launched it; and its **results**, the test runs it filled, read the way a ticket's Tests panel reads them. Until the
workflow sends them, it says **No results yet: they show here once the run sends them.**

By Dawn gives a run that has ended ten minutes for its results to arrive. If none do, the launch fails with "The workflow
run ended without sending any results to this launch." Results that arrive later still complete it.

To stop a launch that's still going, choose **Cancel** on it. A launch By Dawn hasn't started yet is cancelled at once.
One that's running asks GitHub to cancel the workflow run.

## Launch from a test plan

An automated item in a test plan launches its suite on one of its targets, and the launch counts toward the plan.
[Plan testing for a release or cycle](https://docs.bydawn.com/test-plans/#run-the-plan) explains it.

## Launch from an agent

A ticket's agent can launch a suite itself, such as an end-to-end suite before it opens a pull request:

```sh
saw tests run checkout-e2e --wait
```

It names the suite by its name or ID, and runs on the suite's default target unless it names another. With no commit
or branch, it launches at the head of the ticket's own branch. `--wait` follows the launch until it ends. The launch
shows on the suite as started by **Agent**, with a link to the ticket. An agent can only read and cancel the launches
its own ticket started, and every rule above applies to it too: the target, the allowlist and your plan.

## FAQ

**The launch says Completed, but tests failed. Is that right?**

Yes. **Completed** means the workflow ran to the end and sent its results, whatever they say. Open the launch to see
which tests failed.

**Why did my launch fail with "GitHub refused to start"?**

The workflow has no `workflow_dispatch` trigger, or doesn't declare an input the launch sends. Declare By Dawn's four
inputs and each input of your own, as in [Accept By Dawn's inputs](#accept-by-dawns-inputs-in-the-workflow), and merge it to
the branch the target runs on.

**Why does it say GitHub has no workflow?**

The workflow file isn't on the branch the launch ran at. Check the target's **Branch or tag**, and that the file's
name on the allowlist matches the file in `.github/workflows/`.

**Why can't I pin an older commit?**

GitHub starts a workflow at a branch or tag, not at a commit. By Dawn refuses a commit that isn't the head of the branch
the launch runs at, rather than test something else. Launch at a branch or tag whose head is that commit.

**Why are GitLab CI and Your machines greyed out?**

The app launches on GitHub Actions today, and the other choices say **not available yet** until they can. **Jenkins**
is listed too, but By Dawn refuses a Jenkins target until launching on Jenkins is turned on. [Set up your CI](https://docs.bydawn.com/tests-ci/)
says what each of them will need, so you can get your pipelines ready.

**Can a schedule, a pull request or a deploy start a suite?**

Not from the app yet. A launch starts from **Run tests**, a test plan or an agent.

**Does a launch use my CI minutes?**

Yes. The workflow runs on your GitHub Actions like any other run, on your runners and your minutes.

## Next

- [Send test results from CI](https://docs.bydawn.com/developers/tests-submit/): the key, the project ID and the flags for the results
  step.
- [Test results](https://docs.bydawn.com/tests/): how By Dawn reads the results a launch brings back.
- [Plan testing for a release or cycle](https://docs.bydawn.com/test-plans/): launch your regression suites as part of a test plan.
- [Set up your CI](https://docs.bydawn.com/tests-ci/): results and launching on GitLab CI, Azure Pipelines, Jenkins, CircleCI and Buildkite.
- [Agent access](https://docs.bydawn.com/agent-access/#start-ci-workflows): what else the workflow allowlist lets agents start.

Source: https://docs.bydawn.com/tests-launch/