Join 34,000+ subscribers and receive articles from our
blog about software quality, testing, QA and security.

[Automation] Jenkins CI automation integration


#6

OK.
There is an issue with the Testrail plugin after every Jenkins upgrade/downgrade.

Here are the steps to fix it:
From JENKINS_HOME directory, delete testrail.testrail.TestRailNotifier.xml file.
Restart Jenkins.
Go to Jenkins Management page --> Configure System. Add Jenkins server configuration credentials as it will be wiped out.
You should be good now.

Hope it helps.


#7

Thanks for your help with this, venkat, that’s appreciated!

Regards,
Tobias


#8

[quote=dgurock]Hi there,

The team at SimplyMeasured has developed and published a Jenkins plugin to automatically submit JUnit-style automated test results to TestRail. You can find the source code for the plugin here:

https://github.com/simplymeasured/testrail-jenkins-plugin

Thanks for publishing this code, I’m sure it’s useful for other customers as well.[/quote]

Hi there,

The URL to Simply Measured github plugin returns a 404 error. Could someone please post the updated URL?

:slight_smile:


#9

Hello Phil,

I believe the Simply Measured plugin/project is now longer publicly available I’m afraid. You could try to contact the author of the plugin.

Regards,
Tobias


#10

Hello Tobias,

Thanks for the reply. One of my collegues actually managed to find a fork of the original Simply Measures plugin project.

You can find it here;
https://github.com/achikin/testrail-jenkins-plugin

We’ve installed it and it works! V Happy:D

Phil


#11

That’s great, Phil, and thanks for sharing this!

Regards,
Tobias


#12

Hi,

We built and deployed the plugin in a sandbox machine and when trying to push results I’m getting the following -

Uploading results to TestRail.
Failed to add results to TestRail.
status: 400
body :
{“error”:“Field :results cannot be empty (one result is required)”}
Build step ‘Publish test results to TestRail’ marked build as failure
Finished: FAILURE

Any idea what we might miss here? Obviously there’s something wrong with missing results but the input is a JUnit xml format file produced by SoapUI.

Thanks!!


#13

Hi acelow

The plugin expects to find a file called “results.xml” with your junit test results in them. You should point the Jenkins plugin to this file using the “Test Report XMLs” plugin variable. This location is relative to where the test is executed. For example if your tests execute in /mnt/data2/jenkins/workspace/fancytest then your XML report variable should be just “results.xml”.

If you have created the result.xml file deeper down then the relative address will be required. For example for a path underneath where your tests are executed, let’s say in testrestults, it should be like this;

testresults/results.xml

Hope this helps.

Phil


#14

A nice option for this would be the ability to create a test run and then kick off a Jenkins job from there (clicking a button), with results coming back to the source test run. I had a Rube Goldberg way of doing this directly from my own codebase, but it was very ugly and maintenance intensive and I have since taken another path.


#15

Thanks sysmech

I’m currently POCing the plugin so my results.xml is “floating” in some arbitrary location and it’s not related to a specific job.
Does it have to be in a Jenkins workspace? Cant I provide an absolute path the the “Test Report XMLs” plugin variable?


#16

Hi acelow

As far as we have tested, the results.xml file has to be in the job workspace area. We are using the plugin “as is” at the moment, however in pulling it apart find out what we had to do with the results we noticed that there is already code in there to handle multiple unnamed result files: the code of which appears not to be implemented.

We’ll be changing the plugin later in the year to make it more flexible. Currently it creates a test run each time it is executed and we wan’t to be able to update an existing run if one already exists.

Cheers,
Phil


How are you managing run of automated tests and results through api
#17

Hi Sysmech,

updating an existing test run with each subsequent run will lose a case history. Don’t you want to keep track of it?
What we’re not able to confirm now (because of the restriction above) is will the plugin create new cases if the ones that ran are not already in testRail or do we need to manually create them in TR before running the tests?
Basically the use case is to gather our Unit tests and other automations results also in TR but we don’t want to write them down explicitly in TR but to have them updated in TR according to the returned XML.


Need a sample xml or template to push the testresults from jenkins to testrails
#18

Hi Sysmech/Acelow,

I am also getting the same error, as mentioned by Acelow, even when i place the results.xml file insde the jenkins workspace folder.
However, when i place the xml file inside the jobs folder ( instead of the workspace folder) , i get the following error:
Failed to add results to TestRail.
status: 400
body :
{“error”:“No test results or comments so far.”}
Build step ‘Publish test results to TestRail’ marked build as failure
Finished: FAILURE

I am not sure if i am missing anything in the Junit style xml report format since i am creating the xml myself.

If any of you can help with the specific format that the TestRail interprets ?
and also, for eg, Jenkins is Building in workspace “D:\testrail-jenkins-plugin-master\work\workspace\TestRail”. So should i place the xml inside the TestRail folder ?

Thanks!


#19

Hi. Where can I find this variable?


#20

Hello sysmech,

I grabbed the plugin from the link you provided and I downloaded the zip file and changed the extension from zip to hpi and then uploaded the plugin via UI.

I see the following :
javax.servlet.ServletException: java.lang.NullPointerException
at hudson.PluginManager.doUploadPlugin(PluginManager.java:895)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
at org.kohsuke.stapler.MetaClass$3.doDispatch(MetaClass.java:183)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)

Please help!!


#21

Hello Shalini,

We’ve not used the Jenkins plugin for quite some time so I’m a bit rusty on its usage.

Are you seeing this error when you try to use the plugin or during installation?

Kind regards,

Phillip Smith
Test, Integration & Deployment Manager
Systems Mechanics Ltd
Email: phil.smith@sysmech.co.ukmailto:phil.smith@sysmech.co.uk
Office: +44(0)1189 332220
Mobile: +44(0)791 739 0252

Twitterhttp://twitter.com/sysmech | LinkedInhttp://linkedin.com/company/systems-mechanics-ltd | Bloghttp://sysmech.co.uk/blog/


#22

Hi,

Thank you for getting back to me!!

The error is seen when I try to install the plugin.

Regards
Shalini


#23

How we can configure test rail plugin with jenkins as it is zip file and jenkins will accept .hpi file.


#24

Hi Shalini,

How you convert zip to hpi file.


#25

Hello,

For Test NG there is a plug-in for Bamboo or Jenkins?