As we are getting deeper in our utilization of TestRail we are seeing a scenario where new test execution are not reported inside of a given JIRA task created previous from a past execution of a test.
Basically we have the Test Case 001 not linked/reference to any JIRA used to test a particular functionality of the software.
We have created a test run T001 to execute this test against build 001. We saw a bug and reported it as JIRA-001. The T001 failed status receive the Defect value of JIRA-001.
At this point, when we go to JIRA-001 we see the TestRail:Results of the failed execution of T001. The integration with JIRA is as expected.
Later we receive Build 002, and created a new test run containing our case 001, registered as T002. We execute the test and reported is as PASSED. Everyone is happy the bug is resolved.
Meanwhile, refreshing JIRA-001, we don’t see the Passed execution of T002, only the failed one remains reported in JIRA.
As a test, I have updated the Case 001 and put JIRA-001 as a reference. Doing this would provide me both the failed and success results after refreshing JIRA-001.
But I was not expecting to have to update the reference field to get the new execution status. Was my assumption improper or did we forgot something in our utilization of TestRail with JIRA?
PS: From a TestRail perspective things are properly reported, i.e at T002 we see it as succesful (obviously) but clicking on the Defect tab will show us that C001 failed with T001, has been reported as JIRA-001 and we can see the up to date status of JIRA-001 (just great). But from a JIRA perspective we don’t see the update status.