Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.

Reporting with multiple configurations and Test Runs



(sorry for a long topic)

I’m looking at a scenario where I use lots of configurations.
Imagine the Test Runs below and please ignore any unsupported combinations.

The goal is to get a coverage of all tests used in Plan (included in Runs “Feature A” and “Feature B”)
with respect to the configurations.
Feature A contains cases in one section, Feature B cases belong to another section.
There might be overlaps and skipped tests, but in theory let’s say they represent at least different sections.

Normally there would be many more Test Runs covering different Features/Sections.

The “Results (Comparison for Cases)” - report suggests to use Max 10 test runs unless I use the “Similar option”, which always will be a too low number due to the number of permutations and Runs, and in this case will not show any Feature A - results.

Also, the report includes all cases that exists by default, not just the ones used in the Plan.
If I use the report feature standing in a Plan, I’d expect to see the executions only for included tests. As a report viewer I do not know what tests were selected (unless I examine every Run), so I can not do a valid filtering in the Report Options area to optimize the output.

The report also becomes very wide when the number of runs increase. I’d like a more vertical approach to fit all configurations, i.e.

Feature A (Run)

		Config1    Config2    Config3

Section A
  Case 1              P          F          P
  Case 2              P          P          P
  Case 3              P          P          P
  Case 4              P          P          F

Feature B (Run)

	             Config1    Config2    Config3

Section A
    Section B
	   Case 5         U          P          P
	   Case 6         P          P          P
	   Case 7         P          P          P

Is my approach wrong when I let different Runs represent tests for different Sections, and I should
always include all tests in each Run?
Perhaps I can build a custom report for this vertical approach.

Note that even if I include all tests in a Run the same would apply if I need to have runs for each build.

I’ve not seen any filtering options for Version field either.

So in general I’d like something like this (perhaps possible to do as a custom thing here)

Feature A (Run, builds B1 and B2)

		    Config1     Config2      Config3
                     B1 B2       B1 B2       B1 B2      
Section A
  Case 1              P F         F P         F P
  Case 2              P P         P P         F P
  Case 3              P P         P P         P P
  Case 4              P P         P P         P F


Hello Magnus,

Thanks for your posting! The Comparison for Cases report would be the correct report for this and you can certainly run this with more than 10 test runs. The generated report can be quite wide because it uses a dedicated column for each test run and this adds about 100 pixel per test run. But this view also makes it very easy to compare results for test cases over all runs/configurations and also gives you a good overview of problematic areas (e.g. groups of test cases with many failed results).

The report currently includes all cases of the case repository or test suite, but you can select the sections you want to include (if your project uses the single-suite mode) or use the filters on the Test Cases tab. I can see how limiting the view to just the cases that are included in the selected runs make the report more compact and I’m happy to add this to our todo list. You can already see the test cases that are not included via the status/latest column (the cell is empty for those cases).



Following up on this thread. My issue is when the test cases included in a Test Plan are associated with a Jira ticket. When I am in Jira and I see the list of test results, they all have the same the name, is there any way to include there which configuration has been used?
For example:
My Test Plan
TestCaseTitle - Configuration 1
TestCaseTitle - Configuration 2
TestCaseTitle - Configuration 3

After I run this test plan, in Jira I see:
TestCaseTitle - Passed
TestCaseTitle - Failed
TestCaseTitle - Failed

but no information regarding which configuration was used. Is there any way to accomplish this?
Thank you!


Hi Tamara,

Thanks for your posting. Yes, you can expand the test result and this would show all run & result details including the configuration: