Join 34,000+ subscribers and receive articles from our
blog about software quality, testing, QA and security.

Adding a new test run configuration to an existing entry


#1

I’m having a little trouble with update_plan_entry

I have a test plan that has a test run with a configuration. I would like to add an additional configuration to it. How can I do that? I’m trying to use update_plan_entry, but I’m not getting the expected results.

Thanks,
Don


Add_plan_entry with config_ids says The remote server returned an error: (400) Bad Request
#2

Hello Don,

Thanks for your posting. Manipulating the individual test runs/configurations is currently not supported with update_plan_entry and you can only update the entry-level fields (e.g. Name, Assigned To, etc.). Adding configuration support for this particular function is on our to-do list. You could look into adding multiple entries (add_plan_entry) instead of using one entry with different configurations. Would this also work for you?

Regards,
Tobias


#3

No, and here’s why. We run some Selenium automated tests on our product. When complete, a Perl script is run to parse the output and load the results into TestRail. This has been working fine. However, we’re now running those tests on different configurations … at separate times, so when a result is parsed it’s only for one of the configurations at a time, and it’s not known what the other configurations will be.

But I do have a workaround. The script will work fine if it adds the test results to an existing test run with configurations. So, I can modify the test run in the UI, adding the new configuration to the ones already there, and then run the script to update the tests in the run and their results.


#4

Hello Don,

Thanks for the update and the additional details. Good to hear that you found a good workaround and we will make sure to look into adding configuration support to update_plan_entry in one of the next versions as well.

Regards,
Tobias


#5

Hi Tobias,
I am on the latest version of Testrail and I have the exact same problem. Any idea if the API can be enhanced to add support one entry with multiple configurations.

Unfortunately, manually adjusting configurations is not an option for us.


#6

Hello Ali,

Thanks for your feedback on this. You can use add_plan_entry to create test plan entries with multiple test runs/configurations:

http://docs.gurock.com/testrail-api2/reference-plans#add_plan_entry

It’s just that updating test plan entries (update_plan_entry) currently does not support configurations.

Regards,
Tobias


#7

Hi Tobias,
Are there any plans on adding API support for configurations in update_plan_entry?
The UI is quite unfriendly and looking at results becomes quite hard. This is especially true because the test run are sorted based on the order of entry.

I am attaching some screenshots

http://awesomescreenshot.com/00e2p0zmb4
http://awesomescreenshot.com/0772p0zl63


#8

Hello Ali,

Don’t you have the chance to add all test runs/configurations at once? You can then easily use add_plan_entry which has full support for configurations. The workflow would look as follows:

[code]call add_plan_entry

  • W2003
  • W2008
  • W2008R2
  • W2012

for each test run of the test plan:

  • execute the tests
  • add the results
    [/code]

Also, please note that you can change the view mode for the test plans in the UI to display the test runs/configurations in compact mode (via the yellow icons when you view a test plan).

Regards,
Tobias


#9

Hi Tobias,
We can’t add all the configurations at once because we don’t really know them. Let me describe the use case

We have a lot of jobs in jenkins whose results we are trying to dump into testrail. jobs typically test one of these configurations

win2003
win2008
win2012
linux-centos
linux-debian
linux-redhat

example:
job1-win2003
job1-win20012

job2-linux-centos

job3-linux-centos
job3-linux-redhat
job3-linux-debian

Because of business requirements, we cannot presume that each job will run against all configurations. As such, we cannot add all configurations to the testrun using add_plan_entry. We need a way to add a configuration to an existing run.

The sequence of events will be
jenkins runs job1-win2003 --> this will results in one test run with one config (win2003)
jenkins runs job1-win2008 --> we need to add a new configuration (win2008) to the run above.

This is where it seems like the API does not have support.


#10

Hello Ali,

Thanks for the explanation. Yes, the current API doesn’t support this yet unfortunately. If you don’t have the chance to create all test runs upfront, you would need to create separate test plan entries using add_plan_entry each time you start a new job. It’s planned to add support for configurations to update_plan_entry as well and we already have this on our todo list.

Regards,
Tobias


#11

I am also running into the problem of configurations not supported with update_plan_entry. Look forward to an update.

Thanks,
Kurt


#12

Thanks for your feedback, Kurt. We will make sure to look into this for one of the next versions.

Regards,
Tobias


#13

Hello Tobias,

The ability to add a new test run configuration to an existing entry is important for us since we don’t know which configuration ahead of time. In addition, if I create an entry in the test plan with ALL the possible configurations (see following example):

TestSuite_1
TestSuite_1 (windows_xp)
TestSuite_1 (windows_2K12)
TestSuite_1 (ubuntu_linux)

And afterwards only run TestSuite_1 against Windows XP OS, the test results (Test Plan Summary) will show Pass/Fail results PLUS a large amount of un-tested cases (due to test cases not running in both configurations in the above examples).
The best approach is to the have the ability to ** add new configurations via the API ** using our automation framework whenever we kick off a regression run.

In the message thread, I noticed that you have this feature in your TO DO list. Can you please let me know when do you estimate to have this implemented ??

Thanks a lot,
–Jorge


#14

Hello Jorge,

Thanks for your posting and feedback on this. We currently don’t have an estimate for this feature request and recommend using a separate test plan entry (per configuration) if you need to dynamically create/delete configurations (add_plan_entry). Adding/deleting test runs/configurations to/from test plan entries is currently only supported via the UI. I’ve just added another vote to the API enhancement request, thanks!

Regards,
Tobias


#15

Hello Tobias,

Many thanks for the quick reply! At the moment we are adding the test suites with configuration as SEPARATE entries in the test plan but as you can imagine it is a pain to scroll up and down the test plan to visualize the results (i,e, same test suite running on various configurations).

Anyway, hope this feature will be implemented in Test Rail soon! :slight_smile:

Many Thanks!
–Jorge


#16

Hello Jorge,

Thanks for the update. You can change the view mode for test plans (via the little yellow icons next to the test plan name when you view a test plan). The test runs will then be displayed in a more compact way and require less screen space.

Regards,
Tobias


#17

It’s been about a year and a half since this thread was created, and it’s still a badly needed feature. The various multitude of configurations I have makes information unwieldy.

My workaround is to make a new copy of all the existing configurations and test results, and append the new configuration to this copy. Then I delete the old ones. This is awful. Please end my suffering.


#18

Hi there!

Thanks for your feedback. It’s still planned to look into this and I’m happy to add another vote to this feature request. I currently don’t have a time frame or estimate for this feature but API enhancements (especially configurations) are on our todo list.

Cheers,
Tobias


#19

Yup, this makes using configurations from the API pretty useless unless you know upfront what all configurations you want to test – and that often does not happen when running via an Automated environment.

Is this something that requires much refactoring to implement? Seems like 2 years is a long time to wait for this.


#20

Hi Jim,

Thanks for your feedback. One alternative could be to create runs for all configurations you have, even if you don’t use them but I understand that adding configurations on the fly would be preferred. Instead of using configurations, you could also look into creating plan entries (runs) without configurations (so, one add_plan_entry per automated run). It’s definitely still planned to look into adding support for changing the configuration set also with the API.

Cheers,
Tobias