Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Testrail to Jira integration - referencing a test case to multiple stories and getting the correct results


#1

Hi,

We’re trying set up Testrail to Jira integration and we’re having some difficulty with determining the workflow of test cases to stories in Jira.

In our Scrum model, we have a PSI (potentially shippable increment) with 4 sprints: PSI2 Sprint1, PSI2 Sprint2, etc.

We have stories in our backlog in Jira that are put into the sprint and we link the test cases to the stories using the reference field in Testrail and we’ve already created our Testrail project and our test run.

Ex:
PSI2 Sprint1
- Jira Story: HIP-101
- Test Cases: HIP-TCS-100.101, HIP-TCS-100.102, HIP-TCS.100.103

Now, with the integration, we execute HIP-TCS-100.101 and 102 in Sprint1.

We create PSI2 Sprint 2 and regression test 100.101 and 100.102 and execute 100.103. We get:
- 100.101 - fail (which passed in Sprint 1)
- 100.102 - pass (which passed in Sprint 1)
- 100.103 - pass

PSI2 finishes and we move to PSI3. Essentially, a brand new project and all the HIP-TCS-100.10x test cases are now regression tests for PSI3.

We create a test run in Testrail, copy in our HIP test cases, create a story for regression testing and reference the Jira story.

The problem: the Jira story now has the results from PSI2. I would have thought we would have had blank results and ready to start over. Additionally, the reference for the original test case to Jira story has changed to the new Jira story so results for the old story could now be wrong.

What are we missing? Do we have to create new test cases each time? The ID of the new test run and the copied test cases is different than the original runs and the test results are ‘Untested’ however, for the linked Jira story, they are showing the original test case ID and the results.

We must be missing something basic here?

Thanks in advance for the help.


#2

Hello Rick,

Thanks for your posting! Could you provide a simpler example? I’m not sure if I fully understood this but JIRA issues can either be linked via the References (test cases) or Defects field (test results). The Results section in JIRA will either include indirect results of linked cases (References) or direct results linked via the Defects field. This also includes results of previous runs/sprints as long as the issue and case are still linked and reference each other.

Cheers,
Tobias


#3

Tobias,

Thanks for the quick response. Let’s see if a few pictures can help. I think it’s just a matter of understanding the right workflow to get what we need here.

Starting with what our Agile development model looks like. It’s pretty standard: Shippable increments on a time basis with a series of sprints:

This shows new test cases that become regression test cases and going from one increment to the next with the test cases becoming regression test cases and running ‘net new’ on the next increment. i.e. we the regression test cases are now ‘untested’. A story lives across the PSI as it’s developed and tested until delivered.

A real time view looks like this:

Here we see that test case is created, has passed and the test case IDs match. All is good. For the remainder of the sprints in the PSI, we’d simply rerun this test case for regression testing and if it fails, we’d know it as it’s tied to the story for the PSI.

Ok, so now we finish PSI3, ship the product and we move on to PSI4, however, the test cases we created in PSI3 are now regression tests. We create a new test run, copy the test cases and Testrail shows them as ‘Untested’. Because we copied them, Testrail is referencing the old story from the previous effort. We change that to the new story. Testrail shows it as untested. Jira is referencing the OLD test case results and test case ID.

In the example above, the Testrail shows the test case ID as T3764. Jira shows it as T3763 which is the ID from the previous run.

So now we have confusion. The story from PSI3 no longer has the Testrail references, the new story from PSI4 has the results from the PSI3 run but my new test run has test cases that are untested.

Does that help?

Thanks,
-Rick


#4

Hi Rick,

Yep, that helps, thanks a lot! Do you copy the actual test cases in TestRail or just create new test runs and use the same cases across PSIs? The test cases use the C# IDs. The actual tests in test runs are instances of those tests and use the T# ID scheme.

If you use the same cases in TestRail and change the original Reference from JIRA-101 to JIRA-102, then the association between JIRA-101 and the previous results is lost and you would no longer see the results in JIRA (since they are no longer linked in TestRail). One option in your case might to include both JIRA-101 and JIRA-102 as part of the References, would this maybe work? You can link multiple issues to a case by separating them by comma (JIRA-101, JIRA-102) and the entered results would then appear under both JIRA-101 and JIRA-102 in JIRA.

Cheers,
Tobias


#5

Hi Tobias,
Sorry to pile on this old question, but Rick has done a terrific job explaining the situation.

We have the same situation. Our testers will edit the Test Case and append a new JIRA ID to the existing list of IDs in the reference field.

Now this list is growing like crazy for some tests that are executed every sprint.
Also, the JIRA will show the latest result, so the test result in the JIRA is not accurate for all the earlier test runs.

If we could somehow add the reference in the T# (Test), rather than the C# (Test Case), I think this will work better. Any thoughts?

Let me know if there’s a better way to handle this.


#6

Hi Tobias,

We are also concerned about the growing list of JIRA ID references for a single test case. Many test cases can be re-executed across various stories in JIRA leading to a very long list and unneeded status being populated across stories.

Based on the following thread:

It is recommended not to copy or duplicate test suites and potentially use the defect field as a reference to stories. We were thinking about using the copy test suite workflow to reset the reference field but this is not advised. Using the defect field is not a viable option as this is populated during execution time and hurts planning and mapping test case execution to stories.

We were looking and wondering if the same functionality that pharkare is suggesting exists or can be implemented. Can we add references on the Test Run level as opposed to the Test Suite identifier Level.

Thank you


#7

Hi Prakash, Dio,

Thanks for your feedback on this! Yes, we understand that there are also use cases where the references list is more dynamic and we even require some sort of versioning. Managing this on the test & run level in addition to the case level could make sense and we will make sure to review this again for a future version. An alternative might be to use the defects field instead in the meantime to link tests & results to issues. Would this be an option?

Cheers,
Tobias


#8

Hi Tobias,

We’ve investigated the alternative of using the defects field and it’s not really solving the problem.

As we’ve developed our workflows and tried multiple options to find a solution and still not having success. Granted we are migrating from one workflow model to another (more scrum and CI based) but it seems as if there’s just not a solution that can easily accommodate our needs here.

The current design of references means that we have pages and pages of test results that are listed in the Jira plug in. For example, consider this use case:

USE CASE 1

I have a suite of 22 test cases for a feature, let’s call it Backup. In our old test case management tool, we assign the 22 test cases, as a suite, to the build we are going to test and track status in the test case management tool. It’s a bit cumbersome without the Jira integration because you then have to go look up the test case results by looking some text reference in the JIra user story. So, what does this look like assuming our Jira project is named BACK.

BACK-100 - As a user I want to backup

  • BACK-TCS-101 - test backup
  • BACK-TCS-102 - test two backups
  • BACK-TCS-103 - restore backups
    .
    .
  • BACK-TCS-122 - delete backups

What’s the status of testing BACK-100? I look at the summary status of BACK-100 which summarizes all the test cases.

Now, with Testrail, I reference BACK-100 in my 22 BACK-TCS-101-122 test cases and when I look at BACK-100, I will see the 22 test cases (spread out over a couple of pages on the plugin section for Testrail).

Not so bad until you do this with multiple test runs. See the screenshot below where I have 6 of 10 pages of results linked in now.

Now, theoretically, the last result is at the top so this isn’t so bad but still, it’s going to get cumbersome quick. Additionally, I have to scroll through all the test cases to see if they all passed.

For another example:

USE CASE 2

I have a test suite that has 100 test cases. I will run that test suite 7 times during a sprint. For our workflow we:

  • Open a story for each execution to track the work for that test suite
  • Over the course of a PSI (which is 4 sprints), we will have 28 executions of the test suite
  • In the current model, this means I will have 28 references to track in the test cases
  • when I move on to the next PSI, I’m testing the same 100 test cases, another 28 times against another 28 stories so now I have 56 references in those test cases - completely unmaintainable.

I think we need the ability to reference the Test Run (the R###) instead of the tests cases (C#####) to be efficient about this and not blow the integration referencing. It’s almost as if the user needs to be able choose their level of reference integration.

We also don’t want to keep copying test cases as that gets cumbersome.

We’re really struggling with this as Testrail is the best alternative we’ve found but a lot of it was hinged on Jira Integration and ease of use for our engineering base.

Perhaps an interactive showing of the problem might help over a WebEx or something?

Thanks,
-Rick


#9

Hi Rick,

Thanks for the detailed explanation, that helps a lot. Associating references with test runs (and suites) as opposed to just cases is something that has been brought up a few times since the JIRA add-on release and we now also have a pretty good understanding of the use cases :slight_smile:

We also think that it makes sense to make this more flexible and allowing to link to references/user stories on multiple levels and it’s on our list of things to look into. That’s a pretty complex feature to get right to be honest (both technically and from an UI perspective), so it requires some planning and time but we will make sure to review this again for a future version.

Thanks again for your feedback.

Cheers,
Tobias


#10

Hi,
Do we have any features to solve this problem?

Thank you,
Zsofi


#11

Hi Zsofi,

Thanks for your reply! We still have plans to look into this for a future version, however we don’t currently have any ETA. I’m also happy to add another vote on your behalf to this request.

Regards,
Marco


#12

Yes, this is CRUCIAL to the reusability aspect of Testrail!!!

Having to create identical test cases in the system every single time an existing feature has new development done on it, is a massive waste of time due to the redundancy. The flip side of that is to work with the Jira integration in it’s current state and that is resulting in a seriously over convoluted and confusing display.

Our entire QA department is protesting even using Testrail correctly at this point, as replacing the Jira aspect of our workflow is not an option.

PLEASE consider moving the priority up on this, I created a gurock account simply to leave this comment - but there are loads of other users silently suffering!


#13

Hi Jennifer,

Thanks for your feedback! We agree that this would be useful to have, and I’ve added your feedback and vote to this feature request as well.

Regards,
Marco


#14

Hi,

Another vote for this feature. Is this already on the backlog and if yes, when will it be implemented ? Thanks.

Regards,
Ronald


#15

Hi Gurock Team,

Our folks are also VERY interested in seeing this feature come to reality. Any thoughts on how this might be progressing? Thank you.