Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

What is the best way to integrate TestRail with Auto test execution?


#1

Hi.
We have addd code to our auto tests that can push test execution status into TestRail if auto test has TestRail’s test case ID.
To keep results I created TestPlan with two test runs for different platforms (configurations).
The problem is how to define test set?
I filtered test cases using our Automation field (value Automated).
But when we automate next test case it will not be included into Test Run! We are running auto tests every night and can’t update Test Run manually every day.

Is there a way to force Test Run update it’s content by the filter?

What are expected use cases for integration of auto tests into TestRail?


#2

Hello Konstantin,

Thanks for your posting. One option would be to automate the test case selection as part of your code. For example, you can get all automated test cases with get_cases and a type_id filter:

http://docs.gurock.com/testrail-api2/reference-cases#get_cases

(assuming your Automated test type uses type ID 1)

You can then simply specify the selection when creating new test runs or plans by passing the case_ids option and is_include = false:

http://docs.gurock.com/testrail-api2/reference-runs#add_run

(just see Request Example)

I hope this helps!

Regards,
Tobias


#3

OK. I got it - so idea is to create TestRail’s test run for every automated test run. I was just thinking it would pollute our Testrails account. But it sounds as a solution. We’ll try to implement it - thanks!


#4

You can alternatively also use a single test run and then simply change/adjust the test case selection (using the same approach: get_cases and then update_run instead of add_run). This might even be preferred as you mentioned (depending on the amount of test runs you create). You could look into periodically creating a new test run (e.g. every week or so) and reuse the same run over multiple automation runs.

We are happy to help in case you have any further questions about this.

Regards,
Tobias


#5

Hello.
We started creating new Test Run for every auto tests - but now we have another problem - all failed tests has no information about known defects :frowning:
I was expecting that Defects All will show me all known defects for test but this field is empty.

Is there a way to see all known defects for tests? If no - please consider it as a feature request. Having auto test report with N% failed w/o details if there is any new defect is really useless.


#6

Hello Konstantin,

The Defects All field would only show defects of the same test run but you can use the defect related reports on the Reports tab to target multiple test runs. This can look as (also taking into account the references in this example):

If this works better for you, you can also keep using a single test run as mentioned in my previous reply.

Regards,
Tobias