Test Rail Integration

Test Evolve integrates with Test Rail to link Scenarios in Spark to Test Cases in Test Rail. It can be used to update existing test runs in a project, or create a new test run.

Instructions

1. Enable testrail results upload in the reporting.yml configuration file.

default_config/reporting.yml

testrail:
  enabled: true
  domain: example-domain
  project_id: 1
  test_run_id: 0

Configure domain to be the domain for your Test Rail account. For example if your Test Rail url is example-domain.testrail.io then the domain configuration should be example-domain.

Configure project_id to be the id of the Test Rail project that the data should be pushed to. This can be found in Test Rail as the P number as shown below:

Configure test_run_id to be the id of the Test Rail test run that the data should be pushed to. This can be found in Test Rail as the R number as shown below:

note

If you want to create a new test run in Test Rail with each test run in Spark then set test_run_id to 0.

2. Add your Test Rail credentials to the accounts.yml configuration file.

The credentials can also be added as environment variables.

3. Configure Test Rail to allow API connections. This can be completed in the API tab of Site Settings in your Test Rail account. Check the Enable API checkbox as shown below:

4. Add Cucumber tags to your Scenarios in Spark in order to link them to the Test Rail test cases.

Each Scenario should be tagged with the prefix @testrail- followed by the id of the test case in Test Rail. The test case id can be found in Test Rail as the C number as shown below:

For example, if you have a test case with id of 42 in Test Rail that should map to a Scenario in Spark called Example Checkout Flow then the Scenario should be tagged in the feature file as follows:

@testrail-42
Scenario: Example Checkout Flow
  Given ...
  When ...
  Then ...

5. Execute your test scenarios in Spark.

6. View the test run in Test Rail.

• If updating an existing test run, the test case will be updated with the latest status.
• If creating a new test run, all linked test cases will appear under a new test run in Test Rail, with the Scenario status.