(sorry for a long topic)
I’m looking at a scenario where I use lots of configurations.
Imagine the Test Runs below and please ignore any unsupported combinations.
The goal is to get a coverage of all tests used in Plan (included in Runs “Feature A” and “Feature B”)
with respect to the configurations.
Feature A contains cases in one section, Feature B cases belong to another section.
There might be overlaps and skipped tests, but in theory let’s say they represent at least different sections.
Normally there would be many more Test Runs covering different Features/Sections.
The “Results (Comparison for Cases)” - report suggests to use Max 10 test runs unless I use the “Similar option”, which always will be a too low number due to the number of permutations and Runs, and in this case will not show any Feature A - results.
Also, the report includes all cases that exists by default, not just the ones used in the Plan.
If I use the report feature standing in a Plan, I’d expect to see the executions only for included tests. As a report viewer I do not know what tests were selected (unless I examine every Run), so I can not do a valid filtering in the Report Options area to optimize the output.
The report also becomes very wide when the number of runs increase. I’d like a more vertical approach to fit all configurations, i.e.
Feature A (Run)
Config1 Config2 Config3 Section A Case 1 P F P Case 2 P P P Case 3 P P P Case 4 P P F
Feature B (Run)
Config1 Config2 Config3 Section A Section B Case 5 U P P Case 6 P P P Case 7 P P P
Is my approach wrong when I let different Runs represent tests for different Sections, and I should
always include all tests in each Run?
Perhaps I can build a custom report for this vertical approach.
Note that even if I include all tests in a Run the same would apply if I need to have runs for each build.
I’ve not seen any filtering options for Version field either.
So in general I’d like something like this (perhaps possible to do as a custom thing here)
Feature A (Run, builds B1 and B2)
Config1 Config2 Config3 B1 B2 B1 B2 B1 B2 Section A Case 1 P F F P F P Case 2 P P P P F P Case 3 P P P P P P Case 4 P P P P P F