Subscribe Join 180,000+ subscribers and receive articles from our blog about software quality, testing, QA and security. Subscribe

Test Data field for Different Test Run

Hi There,

We are using Test Rail for test automation but seems limitation when we use Test Data field for different test run for 1 test suite. Pl confirm

Scenario:

  1. Created couple of test run for 1 test suite. Testrun1, Testrun2 and Testrun3
  2. When edit test data field for Testrun1, it automatically changes other test run’s test data fields for Testrun2 and Testrun3.

We want to use different test data set for different test run which are created. Could you please help us to do so?

Hi @muditgupta87,
this feature is IMHO not supported right now.

I think you are talking about a feature usually handled by Parameters, either to have multiple different ‘versions’ of the same case (test szenario with different values) or on using the case as a test in a run with different values. In TestRail there are no parameters for it. You can try to simulate it via configurations, but this is more related to multi browser test etc. and not practical for different values.

Changing the case between the run doesn’t mean to change a value only, it is a new version of the case which will be used for every fresh execution of a test.

For manual testing I implemented different workarounds, usually with ‘intelligent’ copies of the cases - usually needed because of reporting issues.

Hi… Thanks for your reply. Seems not supported by Test rail.
Let me brief my requirement.

  1. Assume, I have chrome browser with 2 environment .
  2. Created 2 test run with same test case
  3. we have 2 different environment UAT and PQT
  4. UAT environment has different set of test data and PQT environment has different set of test data
  5. want to execute same test case with different test data for both environment.

If I change test case’s data for UAT env , it automatically change test case’s data for UAT env so my test case got failed.

How to achieve using test rail?

Regards,
Mudit

Hi Mudit,
welcome to the club. A quite common requirement.
Forget about chaning data in the case definition to have two different versions.
Of course you can set everything to execute the tests and then edit the cases to and execute those tests. The executed ones will keep the data from the execution, but that’s nasty.

If it would be just a questionof different environments, the configurations of TestRail yould help you. Good for reporting and easy creation with case mutation.

If you have simple cases (small number of steps if any), probably a small set of vary data you can handle it with manual copies. A custom field indicating the environment will help you to create the runs and finally for the reporting.

For a bigger number of cases having a lot of variants (e.g. more than 15) I implemented a tool for my customer using the the TestRail API.
Starting with a logical case containing the scenario in steps having parameters (my own syntax) and a custom text field wiht a table storing the data sets (multiple parameters for one case).
A separate template offers the field and makes these cases easy to identify.
The tool creates concrete cases per data set, having the parameters replaced with the data, ready to be executed.
Having a clear structure to store them, it is even possibleto update the cases.

Your requirements sounds like something in between…

If the data is only stored in one dedicated field, I see two other ways in addition:

A dirty way might be to store the data in a fresh result for a test before starting.
Create the runs and add a result storing the data for the test e.g. in the comments. Depending on the number possible better done via the API.

The silly way is add both data sets to the case but separate it somehow e.g. using a table. The tester gets instructed which one he has to use.

But going back to your first request: Why do you need to add the data to the test at all, if you’re using automation? Do your tests pick up the data from TestRail?
If not there is IMHO no need to keep them in the case.
If needed for any later reporting, the automation could probably add the used data to the result.

Well, I think at least some points to thing about…

Have fun
Karsten

Hi Karsten,

Thank you for more detail.

As You asked some questions, here is the answer.

Why do you need to add the data to the test at all, if you’re using automation? Do your tests pick up the data from TestRail?

Yes, My tests pick up the data from TestRail as I am using TestRail for automation.

We are fetching data set from Test Data field from TestRail and My test scripts is getting executed based on data set available. I hope, it will clear your doubt.

Any suggestion now?

Regards,
Mudit

Hi Mudit,

if your tests know at runtime which environment is used, I would suggest the following:

  1. Place the different data sets in the same field. Use an easy structure, e.g. a table. Name each set with the environment to use it for (or you need a mapping).

  2. Extend your logic to read the data from TestRail to handle multiple values and pick the needed environment.

  3. Optionally add the used test data and environment to the result to avoid mistakes.

Does this sounds OK for you?

Regards
Karsten

Hi kwirth,

Thanks for your suggestion and seems some extra efforts to use the testdata field for automation with different environment.
I will discuss the same with my team.