Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

New successful execution of test not showing up next to a JIRA bug


#1

As we are getting deeper in our utilization of TestRail we are seeing a scenario where new test execution are not reported inside of a given JIRA task created previous from a past execution of a test.

Basically we have the Test Case 001 not linked/reference to any JIRA used to test a particular functionality of the software.

We have created a test run T001 to execute this test against build 001. We saw a bug and reported it as JIRA-001. The T001 failed status receive the Defect value of JIRA-001.

At this point, when we go to JIRA-001 we see the TestRail:Results of the failed execution of T001. The integration with JIRA is as expected.

Later we receive Build 002, and created a new test run containing our case 001, registered as T002. We execute the test and reported is as PASSED. Everyone is happy the bug is resolved.

Meanwhile, refreshing JIRA-001, we don’t see the Passed execution of T002, only the failed one remains reported in JIRA.

As a test, I have updated the Case 001 and put JIRA-001 as a reference. Doing this would provide me both the failed and success results after refreshing JIRA-001.

But I was not expecting to have to update the reference field to get the new execution status. Was my assumption improper or did we forgot something in our utilization of TestRail with JIRA?

PS: From a TestRail perspective things are properly reported, i.e at T002 we see it as succesful (obviously) but clicking on the Defect tab will show us that C001 failed with T001, has been reported as JIRA-001 and we can see the up to date status of JIRA-001 (just great). But from a JIRA perspective we don’t see the update status.


#2

Hello Pierre,

Thanks for your reply! Happy to explain the behavior:

TestRail would show all results in the Results panel that are linked by the Defects or References field, and also related results of tests that have linked results. For example, if you add a result to T001 and link it via the Defects field, TestRail would also automatically include newer results for the same test (T001). This only applies to the same test and would not apply to separate tests (e.g. T002).

So, you can either link the common case via the References field (like you mentioned), or reference the original defect as part of the passed test result in T002 to see this result in JIRA as well (via the Defects field). You can quickly look up related tests and defects as part of the History & Context/Defects tab to find previous defects.

I hope this helps!

Cheers,
Tobias


#3

Hello Tobias,

So, if I understand it properly, when we are explicitly retesting the bug JIRA-001 (from my example), and want JIRA to show a record about it, we should register our Success status and include the “Defect” value of JIRA-001.

I think this explain the current observations and the proper way to use TestRail. I wish it would be possible to update the passed tests we just recorded to include the Defect value.

What it not clear from the way we currently use TestRail is what scenario would lead to register a new status for the T001 test, other then making an error while recording previous status. When we receive a new build of our application I would create a new Test Run to test it, causing a new series of test to be generated (i.e. T002 instead of T001 to refer to Case C001). Is there a different way to proceed to retest C001/T001 without generating a new Test ID? If not, we will follow the approach of assigning the defect while retesting a fix.

Thanks,
Pierre


#4

Hello Pierre,

Thanks for your reply!

So, if I understand it properly, when we are explicitly retesting the
bug JIRA-001 (from my example), and want JIRA to show a record about it,
we should register our Success status and include the “Defect” value of
JIRA-001.

Yes, that’s the easiest way to link JIRA-001 to the new test result and the recommended approach in this case. If you add the test result as part of the same test (T001 vs T002), TestRail would automatically show the result as part of the result list. This is not the case if you start another test/run (T002) and linking the defects via the Defects field works best in this case. You can simply view related/previous tests/results & defects on the History & Context/Defects tabs as mentioned.

Regarding the new build and T001 vs T002: if you go through the original test run again instead of starting a new test run, you would still test against T001 and this approach is also used often. Both approaches have pros and cons and we recommend using the approach your team likes best. For example, reusing the same test run would make the defect linking simpler as mentioned (more automatic) and it’s easier to retest tests and see previous defects and results. On the other hand, starting a separate test run gives you a better picture of how a specific build performed and also makes it easier to compare different builds over time (e.g. with the Results > Comparison for Cases report from the Reports tab).

I hope this helps!

Cheers,
Tobias