I am currently testing the possibility to use several test plan, one for each “build” of the product we would receive, to complete the testing effort described by the first Test Plan created.
What I am considering to do is to create initially a test plan that include everything that needs to be tested and name it with the version/build info we have:
“Product Version 1.0 Build 001” --> 333 test added.
As we test we would mark the test as passed. After a week, we would receive a second build and I would like to create a new Plan containing the remaining tests (untested). This is possible using the RERUN option on the current test plan. Let’s assume we have completed 133 tests with build 001, the new test plan would look like
“Product Version 1.0 Build 002” —> 200 test added
The first test plan would be closed.
After a second week, lets assume that we receive the third and final build and that we still have 100 test to execute. Using the Rerun option it will be easy to have our new test plan created and it will look like:
“Product Vesion 1.0 Build 003” --> 100 test added
The second test plan would be closed.
All tests will be executed and at the end the 3rd test plan will be closed.
Everything works well to archive such scenario: testrail provide me an easy way to do this with the rerun option, and even the test assignation could be made automatically if done using a proper way.
What is not working well is the report at the Milestone level. Although I know I have tested everything I wanted to with a PASSED status, the report show me that only 53% of the things has passed and 47% not tested:
I was hoping that closing the test plan would not take into consideration the number of untested tests.
Would you have some recommendation of the best way I could use TestRail to accomplish such objective? I could, in theory, remove the “untested” entries from the Test Plan after I have created the new ones to test the new build, but that appear to be a long manual task…
Let me know if there is a better way to do this in TestRail. Or a different approach to identify distinct builds of a certification effort.