Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Forecasting Discrepancy Between Milestone and Test Run


#1

I’m currently running TestRail version 5.3.0.3603:

Background:

  • Milestone Start Date is 9/19/2016 and shows a forecasted completion date of 10/20/2016.
  • There is only 1 test run in this milestone, and it shows a forecasted completion date of 11/6/2016.
  • With only a single test run in a Milestone, I expected the forecasted end dates to be the same.
  • I’m surmising that the discrepancy in forecasted completion dates is due to the Milestone having a start date of 9/19/2016 (which is correct) vs the Test Run having a Start Date of 9/16/2016 (as displayed in the Progress Burn Down Chart for that individual Test Run) which is not correct.

My problem:

  • The test run started on 9/19/2016, the same as the Milestone, and not 9/16/2016.
  • There is no test case activity logged prior to 9/19/2016…everthing started on 9/19.

Questions:

  • Why would the Test Run Progress Burn Down Chart indicate a start date prior to any activity actually occurring?
  • Can I change the start date of the Test Run to match the test case activity and Milestone Start Date (9/19/2016) so that both views agree on the forecast?

Thanx,
Kris


#2

Hi Kris,

Thanks for your posting. The Progress page of the run/plan pages use the creation date of the run/plan to calculate the forecast/projected completion date and the algorithm on the milestone page is a bit more advanced. The milestone page uses the start date of the milestone (a run/plan doesn’t have a start date in the same sense) and also looks at the actual results and not only creation date of the runs/plans. This is often a bit more accurate than the run/plan progress page in the beginning but evens out if the run/plan is active for more than a few days. It’s already planned to update the progress page for runs/plans as well and use better start dates if available.

Cheers,
Tobias


#3

So I just tested this. I had a sub-milestone I created yesterday (11/18), and within it, one test run. I set the milestone when I created it yesterday to start today (11/19).

This morning, I started the sub-milestone. However, looking at the Progress of the sub-milestone, it shows that I started it yesterday (11/18). This is indicating the same behavior as the test plans/run, aka, nothing seems to honor the start date in regards to the Progress See here:

This is critical for us to get accurate duration. I realize the burndown will converge once progress has been going on for a bit, but the milestones and test plans/run should honor this start date.

Is this going to be fixed in the next release?

-Brian


#4

Hi Brian,

Thanks for your posting. If there are no test results yet, TestRail will use the creation date of the first run or the start date of the milestone, whichever is later. If the milestone and runs do have test results, TestRail uses the creation date of the first result across all runs instead. Can you check if it makes a difference if you add a test result?

Cheers,
Tobias


#5

Ok, adding one test result did trigger the milestone to begin today, and not last week. As you said above, the test plan though still shows as start on 11/18, while the sub-milestone now correctly shows a start date of today, 11/21.

-Brian


#6

Hi Brian,

Test plans and runs always use the creation date currently and the advanced logic is only available with milestones at this point. This evens out for longer running plans/runs usually but it’s planned to extend this to plans & runs as well going forward. The milestone or sub-milestone would already have the more accurate numbers in the meantime.

Cheers,
Tobias