Join 34,000+ subscribers and receive articles from our
blog about software quality, testing, QA and security.

Organizing Test Results


#1

I am having some issues in organizing my test plan with test runs. We use the api to create a plan entry and update the results on a nightly basis. The main issue is that test plans contains aggregate test failures. For example i have a run in which 3 test fails then the next day they all pass but the overall pie chart still shows three failures with a percentage. Is there a different way to organize my test plan such that runs of test cases that fail one day but pass the next are shown as passed. Attached is a screen shot to demonstrate how my pass rate is only 55% but my last run shows 95%.

In this example we had several tests fail for like 3 days in a row but then the run on 01-13 they all pass. Its the same set of tests that were run in previous days. The problem is the pie chart still only shows 55% passed which i would like to show 95% passed. This could be as designed for test plans but maybe i should use test suites?

I guess one way to do this is to update the previous runs with the passed result for the tests that are now passing but it seems like too much manual work. Plus we would looks some history in the results report.

Another workaround I see is to always update the same test run and add results to the same run but then i think i would loose the data in the test results report.

If i only do one run and update results every time then i would only see i bar chart and not the daily runs for the results across days etc…

Maybe I am not using the concept of a test plan correctly maybe i need a suite or something else. Any ideas let me know.


#2

Hi Joe,

Thanks for your posting. Yes, you can look into reusing the same test runs inside the plan (instead of creating new runs via add_plan_entry) and this would override previous results. The previously failed tests would still be in the test result history but the pie charts statistics would only the current status (most recent test runs).

Cheers,
Tobias