Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Test Run Suite continues to add new test cases


#1

o I have developed a test run from a Test Suite using ->Run Test
Problem: New tests added to this test suite since the Run was created, continue to be added to the Test Run.

Question: Is it possible to have an entire suite for a run “Frozen” for that run? I find this to be an obvious need without having to mark each test case of the suite for that run so that new test cases added don’t sneak in to the run.

Request: For any Test Run, “Freeze” any and all test cases/suites in place for that run.


#2

Hello,

Thanks for your posting. If you choose a custom test case selection for the test run (“select cases” link on the run form), new test cases won’t automatically added to this test run. Please note that changes to the selected test cases (e.g. when you change the description or steps) will still propagate to the test runs, as long as they are still active (i.e. not closed via the lock icon when viewing a test run/plan).

I hope this helps and please let me know in case you have any further questions.

Regards,
Tobias


#3

Much, much thanks for replying Tobias. This is along the lines of what I figured. There is one slight additional caveat I think to your statement, however.
One must INDIVIDUALLY select each and every test case for no further test cases to be added to the run. Because if you select a Section within a suite (ie. a grouping of testcases within a suite) by checking the section header box to select all those test cases in that section, any test cases that are added to that section – even after creating a run – will be added after you create that run, will be added to that run. Whether this is by design or not, that is apparently what is happening.

If modifications or additions are allowed in a test run, then I would like to put in a feature request.
Please have a checkbox that the user checks on or off for “Prevent further modifications or addition of test cases to this run”, or “Make this run a snapshot of selected test cases/sections/suites”. My QA environment is of the type where, “What you get now is what is to be tested” and we are surprised to see test cases change in a run if modified, or test cases added if a section of a suite is selected.

Let me know if my request is too confusing. But regardless, your responsiveness is impressive and very much appreciated.

Tom Costales (Nook Digital Education)


#4

Hello Tom,

Thanks for the update. Test cases are only added to active test runs with the “include-all” option set and this is automatically disabled when you use a custom test case selection. It doesn’t matter if you select an entire section, the whole test suite or the test cases one by one (I’ve just tested this again). Once you use a custom test case selection via the “select cases” link, new test cases are not automatically added to those test runs. It would be great if you could retry this.

Regarding your other point: yes, this is something that we actually already have on our list. Our currently preferred approach would also be a simple freeze option/checkbox when creating a test run/plan and we would like to look into this for a future version.

Regards,
Tobias


#5

Much thank Tobias. Good info and I will re-verify and get back to this post regardless. Thanks again.
–Tom
PS If this is an error on my part (sounds like it is), then the freeze request for test case modifications can wait, but glad you have it on the todo list.


#6

You are welcome, Tom. Looking forward to your feedback.

Regards,
Tobias