Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Automatic test result according to defect status


#1

Hi,

Is there any way I could update the test result of a test case according to the status of a defect? For example, I would like for all of the test cases that failed and have a defect assigned to them to automatically become re-test. Some kind of automation workflow of this sort would be really nice to have :smile:

Thanks


#2

Hi Guy,

Thanks for your posting! Depending on the tool you integrate with (e.g. JIRA), it might be possible to implement this via a notification feature or plugin. For example, if your tool supports notifications for status changes, you can hook into this and automatically update the status in TestRail (via TestRail’s API) based on the defect status change. That said, we usually recommend changing test statuses manually in TestRail because it doesn’t always make sense to automatically set a test to retest when a defect was resolved (for example, if the testers don’t have access to the latest build/version or something else is blocking a retest).

Regards,
Tobias


#3

Hi Tobias,

I’m looking into this now and was hoping that the TestRail API would bring back linked defects to tests however this doesn’t seem to be the case.

What I was hoping to do was create a JIRA webhook that updated tests that were linked to that defect ID.

How would I identify which test result to update from an external tool?


#4

Hi Paul,

Thanks for your posting! Could you share an example of what you are trying to accomplish? Defects IDs can also be submitted and linked via TestRail’s API and the defect IDs are also returned as part of the API responses.

Cheers,
Tobias


#5

What I wanted to achieve was when a bug in JIRA has been resolved (with a resolution status of fixed) then an API would trigger to set the status of the linked tests (tests with that defect in the “defects” field) set to the status of retest.

Does that make sense? I’m not entirely sure how I’d create that trigger in JIRA though.


#6

Hello Paul,

Thanks for your reply! Yes, that definitely makes sense and thanks for the explanation. It’s already planned to look into this as a built-in feature and we would like to provide options for this in the future. I believe there are already ways to implement this with JIRA’s web-hooks + TestRail’s API. This would be quite complex to implement though but I’m happy to provide more details on this if you are interested.

Cheers,
Tobias


#7

Hi tgurock!

Could you please give more details on this. Implement this with JIRA’s web-hooks + TestRail’s API.


#8

Hi Viktor,

This would require writing a JIRA plugin/add-on and the documentation for JIRA’s web hooks can be found here:

https://developer.atlassian.com/static/connect/docs/latest/modules/common/webhook.html

This is quite complex to be honest and we usually recommend a manual workflow. This has many advantages and you can a) review the tests before making any changes and b) add additional comments for testers.

Cheers,
Tobias


#9

Just wanted to add a “me too” to this feature request. Manual changing of test status in Test Rail does not scale well for large test teams or large releases. I can deal with it for now but would love to have a Fixed issue in Jira automatically set the associated test cases in Test Rail to Retest!


#10

Hi Jeff,

Happy to add this to the list, thanks for your feedback!

Cheers,
Tobias


#11

i think this is the same/similar request: [Feature Request] Mark test as "Retest" when a defect was solved

I tried to do this via JIRA webhooks and the TestRail API, as we are using JIRA webhooks intensively (and they are quite easily to set up)

But when an issue has changed (e.g. resolved), how do I make the connection between the issue and the test ? As far as I remember, JIRA does not send any external information (that is e.g. TestRail link).

So I get information about an issue with ID and the action, but how shall I match that issue id to a test that I can manipulate then ?


#12

Hi Andreas,

The idea is to point the webhooks to a small script of yours that reviews the incoming data from JIRA and can then update test results in TestRail (via TestRail’s API). Please note that we still recommend a manual approach over a fully automated because this makes it much easier to pass additional context details to a tester or decide manually if a test status can really be changed.

Cheers,
Tobias


#13

I know what it is meant and this is exactly what I wanted to setup - the problem remains, that I seem to get no information regarding TestRail when using a JIRA web hook.

As long as you not have any knowledge how to make this script workable, it is not possible to do so, as you implied in [quote=“tgurock, post:6, topic:2112”]
I believe there are already ways to implement this with JIRA’s web-hooks + TestRail’s API. This would be quite complex to implement though but I’m happy to provide more details on this if you are interested.
[/quote]


#14

Hi Andreas,

The webhooks would only be for notification purposes when an issue changes in JIRA, for example. The webhooks themselves would be independent of TestRail and wouldn’t receive any TestRail related data. If you plan to update test statuses in TestRail, you can use TestRail’s API to change the status in your webhooks but this requires some kind of issue <-> test mapping you would need to manage as part of your webhooks (e.g. in a dedicated database). So, this would be largely independent of TestRail and only use TestRail’s API to update test statuses.

Cheers,
Tobias


#15

yeah and with that it does not make sense anymore - why automate this, but manually curate such datasource.?


#16

Hi Andreas,

It’s currently not directly supported to fully automate this so you would need some kind of mapping to implement this. We are happy to look into adding API support for this use case and some generic API functions that would help with this but this is currently not available unfortunately.

Cheers,
Tobias