Hypothetically, let’s say I have a bunch of test cases that can be run on both Windows & Linux. So I create a test plan, and for my run I add a configuration for both platforms, and execute my tests.
At the end of the release I want to ask this question: ‘What is my pass rate on Windows compared to Linux’
The only thing I can do today is look at the comparison for cases report and create an algorithm to try and calculate the data displayed, but because its in an html format, this is very difficult to do.
Is there a way, using a custom report/api that I can actually query the configuration ID in a run and report on it?
This will help other questions like ‘What configurations do we run against often’ ‘Did we test on xxx configuration during xxx milestone’, etc…
I am trying to push my company to use configurations… and they are great to organize the test runs, but if I cannot report on them then it makes it difficult.