Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Using discinct test plan for each build receive to complete a test certification


#1

Hi,

I am currently testing the possibility to use several test plan, one for each “build” of the product we would receive, to complete the testing effort described by the first Test Plan created.

What I am considering to do is to create initially a test plan that include everything that needs to be tested and name it with the version/build info we have:

“Product Version 1.0 Build 001” --> 333 test added.

As we test we would mark the test as passed. After a week, we would receive a second build and I would like to create a new Plan containing the remaining tests (untested). This is possible using the RERUN option on the current test plan. Let’s assume we have completed 133 tests with build 001, the new test plan would look like

“Product Version 1.0 Build 002” —> 200 test added

The first test plan would be closed.

After a second week, lets assume that we receive the third and final build and that we still have 100 test to execute. Using the Rerun option it will be easy to have our new test plan created and it will look like:

“Product Vesion 1.0 Build 003” --> 100 test added

The second test plan would be closed.

All tests will be executed and at the end the 3rd test plan will be closed.

Everything works well to archive such scenario: testrail provide me an easy way to do this with the rerun option, and even the test assignation could be made automatically if done using a proper way.

What is not working well is the report at the Milestone level. Although I know I have tested everything I wanted to with a PASSED status, the report show me that only 53% of the things has passed and 47% not tested:

I was hoping that closing the test plan would not take into consideration the number of untested tests.

Would you have some recommendation of the best way I could use TestRail to accomplish such objective? I could, in theory, remove the “untested” entries from the Test Plan after I have created the new ones to test the new build, but that appear to be a long manual task…

Let me know if there is a better way to do this in TestRail. Or a different approach to identify distinct builds of a certification effort.

Thanks

Pierre


#2

Hello Pierre,

Thanks for your posting. If this is critical, you could really look into removing those untested tests from the case selection (before closing them and after starting a rerun) and this would result in a milestone with 100%. Otherwise, you would also see from the stats that all tests have passed (no failed, blocked, etc.).

You can also use the Results > Comparison for Cases report to get a combined coverage across all runs and this looks as follows (this would show 100% if all tests were passed in one of the runs and this should be what you are looking for):

(see the Latest column on the right)

I hope this helps!

Cheers,
Tobias


#3

Hi Tobias,

Workflow discussed in this thread is something that I have been putting together with automation in mind.
I have checked API documentation for did not find anything that I can use to initiate “Rerun” of the Test Plan… In my case I would like actually to do “clone” of the Template Test Plan (basically “Rerun” with all test cases selected) with different name.
Do you have any other ideas/suggestions that can help me with this?

Please let me know.

Thank you,

Rinat


#4

Hi Rinat,

There’s no direct Rerun API method but you can emulate the same with calls to get_plan and add_plan. If you use a custom selection for your runs, you might also need to call get_tests per test run to find out the case selection
but otherwise get_plan returns all information you need to start a new test plan.

I hope this helps!

Cheers,
Tobias