Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Feature request: Auto-fill test result custom fields by external script


#1

Hi,
We are testing a quite complex product which includes many models and permutations possible.
As a result, many mandatory custom fields were added for better traceability.
Example of collected information: system model, load average, number of background tasks, exact build number, certain aspects of configuration (drive size, number of cores, etc. etc.)

We have means of collecting the necessary information from multiple sources, including shell scripts, REST API queries etc. As for now, we have a single shell function which collects all the information and presents it in a human-readable way - just to copy and paste it into the appropriate custom fields

The overhead, obviously, is quote serious.
What we are looking for is:

  1. Add an auto-fill button in test result dialog
  2. Pressing the button would trigger a dialog asking for a certain value (system IP address/hostname) and then execute a local shell script
  3. Script returns output in any desired format (JSON, CSV whatever) and the system fills appropriate fields with the values.

Could somebody assist? I am not good at JavaScript/PHP programming

The assumptions are:

  • The script may be run either locally on a workstation or
  • The script may be run on external machine, TestRail server should trigger the script by SSH to that machine
  • Better off if it could be implemented on server side, since client could theoretically unable to connect to target system (e.g. VPN)
  • Script could be triggered also after updating a certain result field (system name)

This is my first post, sorry if it is too long. Appreciate your help.
Thanks in advance!


#2

Hello Sergey,

Thanks for your posting! This situation you describe looks like a good fit for using TestRail’s API:

http://docs.gurock.com/testrail-api2/start

You can do most of the commonly used things in TestRail’s UI with TestRail’s API and your use case sounds like a perfect fit to automate some aspects of your testing. For example, you can use the add_result API method to add results to tests:

http://docs.gurock.com/testrail-api2/reference-results

You can also look into using this method to add additional details to manually executed tests as a post-processing step (e.g. your environment and context details). I’m happy to provide more details if this sounds like a possible solution.

Regards,
Tobias


#3

Hello,
Thank you for your response.
The issue is we DO use APIs for submitting automated test results.
However, the problem is with manual tests.

What I am looking for, again, is a way to automatically fill in particular fields on a test result dialog page, DURING test execution, from within TestRail.

Regarding the

You can also look into using this method to add additional details to manually executed tests as a post-processing step

I guess this is not an option really, since there is no API for changing test result, but only adding another result is possible. It neither solves the missing mandatory fields problem or provides any of added value.


#4

Hello Sergey,

Thanks for the additional feedback! A customization like you suggested in the first posting is not really possible inside TestRail as it would require server-side changes as you mentioned.

I would still recommended looking into using the API for this. It’s correct that you can only add new results and can’t update existing results via TestRail’s API. But you can simply add another result with the same status which wouldn’t change the overall status of the test. During manual execution, you would store the script details (server/hostname, etc.) as part of the result. An automated task could then check for new test results, collect the environment details via your script and then add a new test results with all the custom fields.

I hope this helps!

Regards,
Tobias


#5

Tobias hi,
Thank you for suggestions

However, the suggested way of test result submission is somewhat clumsy and just “not right”…
Anyway, just in case, I am asking again: is there a way to add custom button in test result dialog context?


#6

Hello Sergey,

Thanks for your reply. While it would theoretically be possible to implement this, this wouldn’t be easy to do and would require quite a bit of effort to get right and also experience with JS/server-side scripts. I would really recommend looking into the API option as this would be much easier to implement and forward-compatible with future TestRail versions.

Regards,
Tobias