Running Tests in Batches
A batch run runs many saved tests at once, from the Tests grid, and shows how each one did. Use it before a version goes live, after a source has changed, or whenever you want to know that a set of rules still behaves as expected. Each test runs exactly as it was saved, and its outcome is recorded on the test. The batch also tells you which tests changed since their previous run, so a rule that suddenly behaves differently stands out.
A batch run uses these parts of the Rule Repository:
- The checkboxes on the Tests grid: one on every row, and one in the grid's heading (Select All Tests) that selects every test the filters match, on every page.
- Run Selected Tests..., at the bottom left of the grid, and Total selected, which counts the tests you have selected.
- The Pay Attention confirmation, which lists the tests that would call your application's real methods.
- The batch panel, which takes the place of the filters while the batch runs. It holds Back to Tests, Stop, a progress bar with one segment per test, and counters: Total, Ran, and one for each outcome.
- The batch grid, which takes the place of the Tests grid. It has one row per test with the columns ID, Name, Outcome, Change, Reason and Duration.
Selecting Tests
- Click the Tests tab.
- Use the filters to narrow the list to the tests you want, for example All rules... for one rule's tests.
- Check the box on each test you want to run, or check the box in the grid's heading to select every test the filters match. Total selected shows how many you have.
Your selection stays as you page through the grid or change the filters, so you can build it up from several searches. You can run at most 350 tests at a time; if you select more, a warning asks you to select fewer.
Running the Batch
- Click Run Selected Tests....
- If any selected test would call one of your application's real methods, the Pay Attention confirmation lists those tests and their calls. Choose:
- Run All to run every test anyway,
- Skip These Tests to run the others and skip these, or
- Cancel to run nothing.
Without permission to run tests that call real methods, you can only skip them.
- The filters give way to the batch panel, and the tests run one after another. Each row appears when its test starts and fills in when it finishes.
- To end the batch early, click Stop. The test already running finishes; every test after it is marked Not Run.
While a batch runs, the tabs and + New are locked.
Reading the Results
Each row shows:
- Outcome:
- Passed or Failed for an Expectation test;
- True or False, the rule's result, for an Exploration test;
- Error for a run that did not finish;
- Skipped for a test you chose to skip;
- Not Run for a test that a Stop cut off.
- Change, compared with the test's previous run:
- First run when the test never ran before;
- No Change when the outcome is the same;
- "was" and the previous outcome (for example, "was Passed") when it differs, in red when it got worse and green when it got better;
- "Changed" and the previous result when an Exploration test's result moved.
- Reason: for an error, the error's first line; for a failed test, what was missed (for example, "Expected True, got False"); for a skipped test, the calls it would have made.
- Duration: how long the run took.
When the batch ends, every outcome counter above zero becomes a button: click Failed, for example, to see only the failed tests, and click Total to see them all again. Click a finished row to open its run in the Test Result dialog; Back to Test there opens the test itself.
Going Back to the Tests Grid
Click Back to Tests. The filters and the Tests grid come back, showing each test's new outcome, and your selection is kept.
What a Batch Records
- Each test's outcome is recorded on the test as its last run, with your name and the time. The test itself is not changed.
- One entry is added to User Activity for the whole batch, with its counts, for example "A batch of 6 tests was run: 1 passed, 1 failed, 1 True, 2 with errors, 1 skipped."
Batch runs are not available on phones.