Plan testing for a release or cycle
This page helps you plan the testing a release or a cycle needs before you call it done: the smoke and regression suites to launch, and the cases people run by hand. At the end you’ll have a test plan that By Dawn drafted and you edited, with its runs started and its progress and pass rate on the release’s or cycle’s page.
A test plan groups the runs that sign off one release or one cycle. Each line of it is an item: a suite launched on your own CI or machines, or a set of cases people run by hand in a manual run. By Dawn never runs your tests itself, so the plan only tracks what your CI, your machines and your team record. Plans are in Tests, then Plans.
Before you start
Section titled “Before you start”- Test management has to be on for your organization. Without it, the Plans tab and the release’s and cycle’s Tests tab say Test management is not on your plan. Working through the manual runs a plan starts needs test results too, as the Runs tab does. Contact support to turn them on.
- Launching a suite from a plan needs launching tests on your plan as well, and a suite with somewhere to launch (Launch test runs). Without launching you can still attach a launch your CI started.
- Members of the project start and edit plans. Someone with view-only access can follow them.
- The plan drafts from your library, so it helps to have cases linked to the tickets you’re shipping, and your regression suites marked (below).
Start a plan
Section titled “Start a plan”A release or a cycle has one plan. The quickest way to start it is from its own page:
-
Open the release or the cycle, and its Tests tab.
-
Choose Draft a test plan.
By Dawn drafts the plan straight away and the tab shows it. You can also start one on the Plans tab: choose New plan, pick A release or A cycle under For, then the release or cycle itself. The list only offers the ones without a plan, and no completed cycles. Leave Have SAW draft it ticked and choose Draft the plan, or clear it and choose Start an empty plan to add every item yourself.
A new plan is named after what it signs off, such as “v2.4 test plan”, and starts as a Draft. Edit on the plan changes its name and adds a description, such as what signing it off means.
What By Dawn drafts
Section titled “What By Dawn drafts”By Dawn drafts from three places, and each item says which with its badge and a line on why it’s there:
- Linked cases: the manual cases linked to the release’s or cycle’s tickets, or to their epics, plus the cases linked to the release itself. They become one manual item, Cases linked to this release (or cycle).
- Regression suite: each of the project’s regression suites. A suite that has automated cases and somewhere to launch becomes an automated item. A suite with manual cases becomes a manual item too, “suite by hand”.
- Last failures: the cases that failed or were blocked in the plan before this one, which is the previous release by release date or the previous cycle. They become one manual item, Failed last time. An automated item that failed in that plan comes back as well. When there’s no plan before this one, By Dawn takes the cases whose latest manual result anywhere was failed or blocked.
Linked cases and last failures only take draft and active cases, never archived ones. A plan holds up to 200 items.
Mark your regression suites
Section titled “Mark your regression suites”By Dawn only drafts the suites you’ve marked as regression suites. On the Suites tab, open a suite and tick Regression suite. By Dawn drafts every regression suite into each release’s and cycle’s plan from then on, and the suite list shows a Regression badge on it. A project can mark more than one.
Change the items
Section titled “Change the items”The draft is yours to edit. The plan lists its items in order, each with its status, its kind, its badge and the counts of its checks.
To add one, choose Add item:
- Kind: Manual for cases a person runs, or Automated for a suite launched on your own CI or machines. Session is shown but can’t be picked yet. You can’t change an item’s kind later, so add a new item instead.
- Name, and a Description if it helps.
- Cases from, for a manual item: Folders and tags, the same picker a manual run uses, or A saved suite. An automated item always takes a Suite, and Runs on picks one of its launch targets or leaves The suite’s default target.
- For a manual item: the Environment, the Configurations (starting the item starts one run per configuration ticked), Assign its cases to and Due.
- Required to sign off: on by default. An optional item is run and reported, but doesn’t hold up the release’s or cycle’s readiness. The plan marks it Optional.
The pencil on an item edits it the same way. A drafted item that someone edits gets an Edited badge, and By Dawn leaves it alone from then on. Its picked cases show as “The cases SAW drafted”, and Pick by folders and tags instead swaps them for a selection of your own.
The arrows move an item up or down. The bin removes it, after you confirm. A drafted item you remove stays removed: By Dawn remembers its cases and its suite, so a re-draft doesn’t bring them back.
Re-draft
Section titled “Re-draft”Tickets join a release and cases get linked after you draft, so choose Re-draft to have By Dawn add what’s new since. It only adds: new linked cases and failures go into the drafted item that holds them, unless someone has edited it, and new regression suites become items. It never removes an item or changes an edited one. A message says what it added or extended, and those items are highlighted. “Nothing new to add” means the plan has everything already.
Run the plan
Section titled “Run the plan”Each item has Start. What happens depends on its kind:
- A manual item starts a manual run of its cases, one per configuration it names, assigned and due as the item says. Under Build, a release plan’s runs test The release, a cycle plan’s No build, or pick A commit with its repo and full commit id. People then work through the runs on the Runs tab.
- An automated item launches its suite on your own CI or machine. Branch or tag is optional and the target’s default otherwise. Results land in the plan as your CI reports them.
Starting an item moves a draft plan to Active. The button then reads Start again, which starts another round, such as a regression pass after the fixes land.
Open an item’s runs line, such as “2 runs”, to see the runs and launches it groups. Open takes you to one, and the unlink icon detaches it. If a run or launch already exists, such as a launch your CI started on its own, choose Attach a run or Attach a launch and pick it, so it counts toward the plan.
Read the progress
Section titled “Read the progress”The top of the plan sums it up:
- Pass rate: checks passed out of those passed, failed or blocked. Skipped and untested checks don’t count, so it reads — until something has a result.
- Items passed, Checks failed, Blocked and Untested, with a bar of every check by result.
A check is one case on one configuration of a manual run, or one automated test in a launch’s results. Only its latest result counts: a newer run of the same configuration, or a retest, replaces an older result. An item with nothing started yet counts every case it would run as untested.
Each item’s status follows from its checks:
| Status | When |
|---|---|
| Not started | Nothing has run for it yet. |
| In progress | A check is still untested, or a launch is still running. |
| Failed | A check failed or was blocked, or a launch failed without reporting. |
| Passed | Every check passed or was skipped. |
Is the release ready?
Section titled “Is the release ready?”A release’s plan passes when every required item has passed: each of its checks passed or was skipped, with nothing failed, blocked or untested, and nothing still running. Optional items are shown but never hold it up.
The release’s readiness has a Testing check for this. The Migrations & deploy tab shows it as passing, a warning or failing, and the Ship dialog says which:
| The Ship dialog says | When |
|---|---|
| Test plan passes | The plan passes, or the release has no plan and the project has no test gate. |
| Test plan has not passed | The plan hasn’t passed yet. It’s a warning, so it doesn’t stop a ship. |
| Test gate closed: the plan has not passed | The project’s test gate is on and the plan hasn’t passed. It fails, and nothing gets the release to Released until the gate opens. |
The test gate
Section titled “The test gate”A project can turn on a test gate, so a release can’t be shipped until its plan passes. It’s off by default. While it’s closed, the Ship dialog offers no Ship anyway: it says The test gate is closed and holds the button. Completing a cycle can’t ship the release either, so untick it to complete the cycle without shipping. Changing the release’s status to a released one is refused too. With the gate on, a release with no plan counts as not passed.
The gate opens once the plan passes, or when a person signs off what’s keeping it from passing. A sign-off stops covering as soon as something new keeps the plan from passing, such as a new failure. The gate can also name folders whose manual cases need a result recorded by a person, so a result from a worker or the AI tester doesn’t count there. Only a person can sign off, turn the gate off or drop one of its folders. An agent can’t.
There’s no switch in the app for the gate, its folders or a sign-off yet, and none for the ✅ Tested section a project can add to its release notes. Until there is, the plans themselves work the same with the gate off: the Testing line warns rather than blocks.
Complete the plan
Section titled “Complete the plan”The buttons at the top move the plan along: Activate a draft, Complete an active plan when testing is done, and Archive one you no longer need. A completed plan can be reopened with Reopen, and an archived one with Restore, but a plan never goes back to Draft.
A completed or archived plan is read-only: its items and results stay, but you reopen it to change an item or start more runs. Delete removes the plan and its items. The runs and launches it grouped stay, with their results, and the release or cycle can start a new plan.
See it from the release or cycle
Section titled “See it from the release or cycle”The release’s and the cycle’s Tests tab show their plan at a glance: its state, pass rate, failed, blocked and untested checks, and each item with its badge and status. Open in Tests opens the plan to edit it.
The Plans tab lists every plan in the project, newest first, with what it signs off, its items passed and its pass rate. It shows Draft or active plans to start with. The filters narrow it to Releases or Cycles or to one state, and the search finds a plan by its name or by its release’s or cycle’s.
Why didn't my regression suite get an automated item?
By Dawn drafts an automated item only for a suite with automated cases and at least one launch target. Add a target on the Suites tab, then choose Re-draft. A suite with manual cases still gets its “by hand” item.
Can a release have two plans?
No. A release or a cycle has one plan. Add items to it, or use Start again for another round. To start over, delete the plan and draft a new one.
Does removing an item delete its runs?
No. The runs and launches stay, with their results. They just stop counting toward the plan. The same goes for detaching one.
Why can't I ship, even with Ship anyway?
The project’s test gate is closed: the release’s plan hasn’t passed. Finish its required items, or have a person sign off the exceptions. Marking an item Optional in the plan also stops it holding up the release, but an agent can’t do that while the gate is on.
Why can't I pick Session?
Exploratory sessions can’t be planned yet. Run one from the Sessions tab in the meantime.
- Run tests by hand: work through the manual runs a plan starts.
- Test cases: link cases to tickets so the next plan drafts them.
- Releases and the project changelog: ship the release your plan signs off.
- Cycles: plan and follow the cycle.