Retrying Failed Scenarios

Re-running failed scenarios is not generally recommended as this can hide performance issues in the site or application under test. A failing test should result in either remedial work to improve the test or a bug ticket.

However, there are some circumstances where it may become necessary:

• Slow environment issues that are not a priority to be resolved.
• That might be it! We always like to consider other users' perspectives though, so if you have other valid use cases, please share with us!

Retry Failed Scenario Immediately

The cucumber flag --retry can be used in the CLI to immediately rerun a failing scenario after it has finished running.

The number of desired attempts can be passed through as part of the command line:

Ruby

$ cucumber features --retry 2

Parallel:

$ parallel_cucumber features -o '--retry 2'

JavaScript

$ ./node_modules/.bin/cucumber-js features --retry 2

Further documentation: https://github.com/cucumber/cucumber-js/blob/main/docs/retry.md

TypeScript

$ cucumber-js features --retry 2

Further documentation: https://github.com/cucumber/cucumber-js/blob/main/docs/retry.md

Scenarios that have had multiple attempts will be flagged in the Halo Test Run Report with a warning icon and the number of attempts completed.

Rerun All Failed Scenarios

In order to rerun all failed scenarios after a completed first test run, set the rerun flag in the reporting.yml configuration file to true.

This will generate a list of failing scenarios in the rerun subfolder of the remote_test_results folder.

To rerun the failing scenarios listed in the file, simply feed the file into the following CI job:

Ruby

$ cucumber ./remote_test_results/rerun/@rerun.log

JavaScript

$ cucumber-js ./remote_test_results/rerun/@rerun.log

Further documentation: https://github.com/cucumber/cucumber-js/blob/main/docs/rerun.md

TypeScript

$ cucumber-js ./remote_test_results/rerun/@rerun.log

Further documentation: https://github.com/cucumber/cucumber-js/blob/main/docs/rerun.md

We recommend using a separate Halo label for your rerun job so that the history of failed scenarios on the main test run is preserved.

The reporting in Halo will not merge the two test runs.