Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Test results to single plan/run from multiple Jenkins jobs

#1

I have test runs with multiple configurations. Each configuration is run as a separate job in Jenkins. I would like to map all the runs to a single test plan/run. I cannot figure how to use the TR API to do this, since creating a test run/plan with the API returns the ID, and I would need to be able to share this across the jobs. And all jobs run separately.

I see the option to create a single plan and run and keep them open to have the same ID. Then just upload results to that same run ID over the API. But then I would not have access to previous results. I would like to have access to run history.

Any ideas on how to do this best?

I can use the API or any other means, such as the testrail Jenkins plugin.

0 Likes

#2

if you update the same tests over and over, you can still see all the history by clicking on the activity button in the GUI

0 Likes

#3

Hi @uliuli,

Thanks for the post! @tstutson is correct that you would be able to view the result history within the UI when viewing each test.

With the API, you would also be able to use the ‘get_results_for_run’ method. This method would return results for the test run in order by timestamp, with the most recent result first first in the response. This would return up to 250 results per request, so you may need to use limit and offset filters to obtain all results for the run.

Alternatively, you can use a test plan to create separate test runs to match your configurations and use the tests’ corresponding case IDs (for the case in the test suite or project) when submitting results, as these IDs would be static across multiple test runs. The ‘add_results_for_cases’ method would be used in this instance. If you are running tests against different configurations, this would likely be the method which suits your needs, allowing you to reuse case IDs while maintaining separate test runs.

Hope this information helps!

0 Likes

#4

Thanks @tstutson and @jon.reynolds.

I had not noticed the activity history, this seems like it could work. I tried to upload several results for a single run and the activity tab seems to show them all over time.

If I understand this correctly, I can just create a set of test runs for each configuration from a test plan. Then just use these static test case id and static test run ids for results upload, and use the activity tab to view results trends. For these cases, I would not start new test runs, other than for special purposes, but rather just keep using the same one.

Is this correct?

0 Likes

#5

HI @uliuli,

Your interpretation appears correct. You could create a set of runs for each configuration then add results to each run using the case id and run id corresponding to the configuration.

Depending on the frequency in which you submit results, you may want to close your test plan and start a new one from time to time, or at regular intervals such as weekly or monthly.

If you are using TestRail with heavy automation, we also recommend purging older test runs/plans from time to time and this will keep the database and tables at a manageable size and help with overall performance. For example, many teams implement an auto-purge mechanism with get_runs/get_plans (e.g. with the created_before filter) + delete_run/delete_plan for runs/plans that are older than a certain threshold (e.g. a few weeks or months) and this approach is also something we recommend.

0 Likes

#6

Thanks @jon.reynolds,

I tried the static ID approach and so far seems to work.

Do you have some estimates on how big test sets should I start purging? So if I get 100 tests a month, 1k, 10k, 100k, 1 million, …?

0 Likes