Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Test environment summary for each TestRun


#1

Hi All,

I have been using TestRun to post results of Selenium Test Case execution to TestRail using add_result_for_case API.
All works well, except the I also want to post the environment details (application version etc) on test run. I have two options -

  1. Use “comment” and post environment details as comments. But then this information would be duplicated in all test cases since environment info would be duplicated in all test cases.

  2. Use “description” and post this information on the test run level. But then this information would be overwritten with each new test run.

One possibility is to have a new test run for each automated run and use “description” field to post environment info on test run. But then I would end up with many-many test runs and navigating through history of results would be difficult.

What options do others use?


#2

Hi Tarun,

Thanks for your posting. We would recommend using the Description field on the run level in this case. To keep the Test Runs & Results overview manageable, you can look into using test plans to group your runs. E.g. one approach that is often used is to add a new test plan for automated runs every week and then add all runs for that week to the test plan instead of directly to the project.

Cheers,
Tobias


#3

Thanks Tobias,

But I guess there is not going to be an an easy solution since I don’t want to use multiple test runs but one test run and post results on it time and again since it makes it easy to look through test case history. Having different test runs is going to make it laborious to scan through, is not it?

Thanks
Tarun K


#4

Hi Tarun,

Yes, this would make it bit more difficult but you would still have access to it via the Context & History tabs:

An alternative would be to use a single run and it’s probably best to add the context details to the test result then, to make it clear which environment/context was used for which test result.

Cheers,
Tobias


#5

Thanks Tobias, this helps in achieving what I want.
One last question through. In my activity section I see only status, test case summary and tested by columns. How do you get “In Release 1.1”, “Release 1.1” etc message messages printed. I am using add_result_for_case API. Which parameter should I set these values?
If I could set these values (at least one value like - In Release 1.1)

If not this then could I see history of Test Run description? In that case, even if the description is overwritten with each test run then I would be able to see what previous values were.

then it would solve my problem.

Thanks
Tarun K


#6

Hi Tarun,

The activity list can’t show the run description or comments/description of previous results but you can always open the tests in new tabs and this would show the full test result history. If this works better for you, you can also use a single test run and add the results to the same tests and this would make it easier to see the full result history with previous test results.

Cheers,
Tobias


#7

Hi Tobias,

This is how I post description of test environment on test run. Please notice that I don’t post it as test comment since this information is going to be duplicate for all test cases in a test run -

Now description gets overwritten on each new test run. How do I see the history of description ? History of tests is not going to display description since it is not associated with a specific test case, is not it?

Thanks
Tarun K


#8

Hi Tarun,

TestRail wouldn’t show the history or past values for the description and only the most recent content. An option would be to include the previous description when starting a new test run/updating the run description. Would this work for you?

Cheers,
Tobias


#9

Hi Tobias,

including the past description is not feasible as -

  1. description would go very very huge
  2. would be difficult to gauge history of test and corresponding description

Is there a reason why history of description is not maintained?

Thanks
Tarun K


#10

Hi Tarun,

The description is kept in a single database field and there’s no history/revisions for test runs like you have with test cases, for example. So, TestRail will only keep the most recent version of the description and previous versions are no longer in the database and overridden each time you make a change to the run description.

Cheers,
Tobias


#11

Thanks Tobias,

Is there a plan to keep history of description? Could I file this a feature request?

Thanks
Tarun K


#12

Hi Tarun,

There are currently no plans for this and this behavior of keeping only the most recent version of a field would be standard for most web applications. TestRail already keeps track of revisions/field changes for test cases (e.g. to see changes of other users) and also has the result history but runs and other entities are usually not updated frequently and TestRail only keeps the most recent version of each field for those entities.

Cheers,
Tobias


#13

alright Tobias, can live with it for now. btw I am very impressed with TestRail and how well it integrates with Selenium.


#14

Hi Tarun,

That’s great to hear, thanks :slight_smile: Please let me know in case anything else comes up, happy to help.

Cheers,
Tobias