Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Using exploratory session to retest fix give unexpected result when we record multiple test results


#1

Hi,

We are using exploratory session to retest fixes received that don’t have a test case but some test results seems to affect other ones in an unexpected way.

Here is what we did.

In the past, during an exploratory session we report a new defect. A Test result has been created with a Failed status and we set the defect to JIRA-123.

After receiving a fix, during an exploratory session we recorded a PASSED test result and set the defect to JIRA-123.

So far so good, looking at JIRA-123 we see the failed and new passed execution status.

But during the same exploratory session we saw a bug and reported it as a new test results set to FAILED with defect JIRA-345.

Looking at JIRA-345, we can see the test result from TestRail.

Meanwhile, when we look again at JIRA-123, we can see a 3rd result : the failed one coming from the 2nd test result recorded…

While would we see in JIRA-123 a test result set for JIRA-234?

Thanks,

Pierre


#2

Hi Pierre,

Thanks for your posting. Sure, happy to explain this behavior. The TestRail: Results panel in JIRA shows all linked but also related results. For example, you only linked the JIRA-123 result to the Failed status but TestRail automatically shows newer results of the same tests (that’s why you also the Passed result). As another (Failed) result was added to the same test on top of that, TestRail would now show this result as well. JIRA-345 would only show the linked result (because the other results were added before).

You can see the details of each result by expanding it and you can then also see the linked JIRA-345.

Cheers,
Tobias


#3

Hi Tobias,

Thanks for your answer. I personally think this behavior not very suitable in the context of exploratory session as the test result might have no link together. This was the case in our last exploratory session and cause JIRA to report failure for things not related at all. We just closed that JIRA-123 and the most recent status appearing in JIRA is FAILED. I wish we could opt-out from this behavior for Exploratory session as it might jeopardize our utilization of this nice feature.

Thanks,

Pierre


#4

Thanks for your feedback, Pierre. We will make sure to look into this and I agree that it can make sense to handle this a bit smarter in certain situations (e.g. if a result is actually linked to a different issue).

Cheers,
Tobias