Run tests by hand

This page helps you run test cases by hand with your team: a regression pass before a release, say, or checking a fix on staging. At the end you’ll have a manual run with a result for every case and a bug ticket for every failure, each with the steps that led to it.

A manual run is a set of cases from the library that people work through against one build, step by step, recording what happened. It records what a person did: By Dawn runs nothing itself. Runs are in Tests›, then Runs.

Before you start

• Test management and test results both have to be on for your organization. Without one, the Runs tab says which isn’t on your plan. Contact support to turn them on.
• Members of the project start and work in runs. Someone with view-only access can follow them.
• The cases you’ll run need to be in the library, as Active cases, or as drafts if you include them.

Start a run

1. On the Runs tab, choose New run.

2. Give it a Name, such as Release 4.2 regression, and a Description if it helps.

3. Pick the cases. Tick the folders to run (their sub-folders come too unless you clear Include sub-folders), and type tags under Only cases tagged… to narrow them. With nothing ticked, the run takes every active manual case. Include drafts adds draft cases, and Include automated tests adds the automated tests your CI reports. The dialog counts the cases as you go; a run holds at most 5,000. If the project has suites, Start from A saved suite takes a suite’s cases instead.

4. Say what’s being tested:

   • Build: A commit (the repo and its full commit ID), A release, or No build.
   • Environment: where the testers run it, such as Staging. New environment… adds one, with its URL.
   • Configurations, if the project has any, such as browsers. Tick several to start one run on each, grouped together, so each browser gets its own results.

5. Choose who does it under Assign every case to, or leave it at Nobody yet and assign later. Add a Due date if there’s a deadline.

6. Choose Start run. The run opens.

A run holds up to 5,000 cases. Each case is pinned to the revision it had when the run started, so a tester reads the steps everyone agreed, even if someone edits the case meanwhile.

Follow a run

The Runs tab lists runs that aren’t closed, newest first, with a bar for each one’s progress, such as “12 of 40 done · 2 failed · 1 blocked · 3 to retest”. A run is Open until its first result, then In progress, then Closed. Filter the list by status, by who it’s assigned to (Assigned to me) and by environment. A run past its due date says Overdue.

Open a run to see its cases, grouped by folder, each with its result and who it’s assigned to. Results from other testers show up as they record them.

Assign the cases

Assign… hands out the cases still to do: pick a person, which results to include (Untested and Retest to start with) and, if you like, one folder. To assign particular cases, tick them and choose Assign to…. Assigning to Nobody (unassign) takes them back.

Cases can also go to The AI tester, an agent that works through them in a browser on one of your machines. Its results are marked AI-run, because they’re weaker evidence than a person’s.

Add or remove cases

Add cases adds more with the same picker, skipping any already in the run. Remove from run, at the bottom of a case, removes one that has no result yet.

Record results

Open a case to run it. The view shows its preconditions and its steps, each with what to do and what to expect.

• Each step: mark it Passed, Failed, Blocked or Skipped, and Add a note of what happened. Choose the same result again to clear it.
• The whole case is set from its steps once every step has a result: failed if any step failed, then blocked if any was blocked, passed otherwise. Skipped steps don’t fail it. Set Result for the whole case yourself to override it, or to record a case with no steps.
• Actual result: what actually happened, which a bug ticket quotes. Note is for anything else.
• Screenshots: paste one with ⌘V (CtrlV on Windows and Linux) to put it on the step you’re on, or Attach or drag files: images, logs, .har and .json files, up to 20 a step and 50 a case.

Everything saves as you go, and the timer at the top adds the time you spent on the case. Then Next case, or Next untested to skip to the next case still to do.

Keyboard shortcuts

The run is built to be worked from the keyboard. ? shows these at any time:

To	Press
Move to the next or previous step	j k
Mark the step passed, failed, blocked or skipped	p f b s
Mark the whole case	⇧p ⇧f ⇧b ⇧s
Next case	n or ]
Previous case	⇧n or [
Write the actual result	a
Write a note	e
File a bug	x
Paste a screenshot onto the step	⌘V

They don’t fire while you’re typing in a field, so a note can’t mark a step by accident.

File a bug from a failure

On a failed or blocked case, File bug (x) files a ticket in the project with the details already written. By Dawn fills in the case’s steps with each one’s result, marks the step where it failed here, and adds the actual result, your notes, and the build, environment and configuration. The title starts as “Failed:” and the case’s title, so change it if you can say more, and add anything above what By Dawn fills in. Screenshots are copied onto the ticket unless you turn that off.

If a ticket already tracks the problem, choose Link a ticket that already tracks it instead. Either way, the case shows the ticket, so the next person sees it’s known. Write the actual result before you file, because the ticket quotes it.

Retest

When the fixes are in, choose Retest failures to send every failed and blocked case round again, or tick some cases and choose Retest. Each goes back to the same person as Retest, with its attempt number up by one. Its step results, actual result and time start again, and the earlier attempt stays under Earlier attempts. The case’s note, files and bugs stay.

Tick Use each case’s latest revision to retest with the steps as they read now, if the case was corrected since the run started.

Close the run

When every case has a result, choose Close run. Its results become read-only, though you can still file or link a bug from a case. Reopen opens it again to record more.

Each case’s Runs tab in the library lists the runs it was in, and a ticket the case is linked to shows its latest manual result under Cases covering this.

FAQ

Can I change the build or the cases of a run?

Edit changes its name, description, environment and due date. The build is fixed, because the results are about that build. Add cases with Add cases, and start a new run to test another build.

Why don't I see File bug?

It’s on failed and blocked cases. Mark the case first.

Do manual runs show in a ticket's Tests section?

No. A ticket’s Tests section is for automated results. A manual result shows against the case, in the ticket’s Cases covering this section.

Where do configurations come from?

They’re the project’s list of things a run can vary on, such as browsers or devices. If the project has none, the Configurations choice doesn’t appear.

Next

• Exploratory sessions: test without a script, in a timebox.
• Plan testing for a release or cycle: group the runs that sign off a release or cycle.
• Test cases: fix a case’s steps after a run shows they’re wrong.