Q: Is there a way to change the reporting of the built in reports, as well as the displayed graphs (pie and bar charts) in the Project / Milestone / Test Plans - Tests, to be displayed in “Estimate” time instead of the current number of test cases?
In our QA organization, we look at the estimated time, not the number of test cases to determine if we are on schedule (or behind / ahead). Currently, all the Built in Reports and the graphs displayed in the Milestones, Test Plans and Test areas focus on # of Test Cases in a particular state, which in our case is useless as it has no real relevance to how much testing is left in any given test phase / milestone.
An Example of how our QA teams load TestRail for any given milestone / project (schedules run between 2 weeks and up to 6-9 month test schedules):
Test Cell A: Loads 5000-10000 TC’s with estimated times of 15sec - 2 minutes, with a small number of test cases being 4 hours to over the weekend runs.
Test Cell B: Loads 500-1000 TC’s and each test cases averages 2-3 hours.
Test Cell C: Loads 100-500 TC’s with estimated times of minutes to days
Test Cell D: Loads 100-500 TC’s with estimated times of 4hrs to 1-2 weeks.
Note: All test cells have a subsection of performance / load testing that tend to be between 4hrs - 3 to 5 days in duration. While this might only be 2-3 percent of the test cases under each milestone, it makes up 25-50% of the actual test hours.
In our case, seeing that 83% of the test cases have been completed in some form (Passed / Failed / etc) due to the execution rate of Test Cell A, really does not represent how far along we are in the test phase (Milestone). Since Test Cell A, can execute test cases at a rate of 720 test case per 24 hours, in the first 2-3 weeks of a 3 month test cycle, we already showing a completion of ~40% of execution, while in reality, the overall project is only 10-15% complete after 3 weeks.
Currently we are creating custom reports (by way of warning from Gurock Software), to get the information we need. As noted in one of the previous TestRail updates, Gurock allowed the nesting of Functions in a Test Suite, which broke all of our custom reports for the users that utilized this new function (their data was missed). In addition, we now have the ability to document steps as individual tasks with a Pass / Fail criteria. We have not tested this function as it relates to our custom reports, but it might break our existing custom reports again.
To Net it out, it would be nice if we could customize the built-in reports and graphs from TestRail in some form of a configuration portlet, rather than finding out in the middle of a project that our custom reports are broken due to new function being added (which is great), but not knowing the side effects until multiple weeks into a project, or after having to perform a more in depth smoke test on the pre-production server to validate functionality (which means going and creating fake projects and using all the new function to determine what may or may not break our custom reports. Also, it would be nice to actually use the reporting features of TestRail, rather than use custom reports and export the data to third party tools to massage the data into something our organization can use.
Any help or ideas would be greatly appreciated.