Join 34,000+ subscribers and receive articles from our
blog about software quality, testing, QA and security.

Any successful integration with SmartBear's TestComplete?


#1

I’m trying to submit automation testing results using TestRail’s API but I’m having problems with it.
Do anyone got it working successfully?
The process isn’t very straightforward… can you give steps to get it done?


#2

Hi Leandro,

Thanks for your posting! The idea is to submit test results via TestRail’s add_result or add_result_for_case API methods (or the equivalent bulk versions). You can find a good introduction and overview of other integrations on our blog:

https://blog.gurock.com/testrail-test-automation-test-management/

A good starting point is also the API reference and getting started guide:

http://docs.gurock.com/testrail-api2/start

There are also ready to use API bindings for various programming languages you can already use (.NET, Java, PHP, etc.).

Cheers,
Tobias


#3

Hi Tobias,

Thanks for your reply. None of those integrations happen to be with TestComplete. I’m particularly interested in successful integrations with that tool as it’s the one I can’t make work at the moment.

Thank you,
Leandro


#4

Hi Leandro,

Thanks for your reply. I’m not 100% sure which integration points TestComplete provides (so, if it’s possible to execute custom code after your tests ran, for example). If TestComplete generates a result file (such as XML or CSV), you can always just submit the test results as a post-processing step by going through the result file and submitting the results to TestRail via our API. This is commonly used with other integrations as well (xUnit, Jenkins, etc.) and a good way to implement a loosely-coupled integration that is relatively independent of the actual tool and only requires knowledge of the result file.

Cheers,
Tobias


#5

Hi Tobias,

That sounds great. So far I could only find this forum discussion using what apparently is a successful integration.
TestComplete doesn’t generate files, but it can call API methods being add_result_for_case the one interested in as you mentioned before.

Cheers,
Leandro


#6

Hey Leandro,

Thanks for the update, sounds good! This should be a viable solution but would require some customizations I think because TestComplete wouldn’t really know how to call TestRail’s API by default. You would also need an equivalent test case in TestRail for every test in TestComplete to add test results but this would definitely be an option.

Cheers,
Tobias


#7

Hi Tobias,

That’s exactly the case. The only problem is I cannot get TestComplete to call the methods as described in the forum. Actually it calls the methods but some Json parsing is failing. Not sure how to debug it so I need help from someone who has done it before I guess.

Thank you,
Leandro


#8

Hi Leandro,

Thanks for your reply. It’s likely best to go through this with a developer on your side who has experience with TestComplete. We don’t have direct experience with TestComplete ourselves and only the TestRail side but we are happy to help in case you have any questions on how to call the API.

I hope this helps!

Cheers,
Tobias


#9

Hi Tobias,

Thanks for your help! I just discovered when I refresh my settings in testrail I don’t see the API keys anymore… Are they still valid?

I see in the documentation this:
You can generate multiple API keys for different systems or third-party tools and revoke access by deleting the API key at any time under My Settings.

But I just can get to see the already generated keys…

Cheers,
Leo


#10

Hi Leo,

You would need to generate the key and then save the My Settings page. You would be able to see the full list of keys (names) you’ve added and saved over time on this page but the actual authentication key/password is only shown once when you generate the key (and TestRail only stores a hash of that).

I hope this helps!

Cheers,
Tobias


#11

I see what you mean, but my API keys are cleared out when I refresh my settings page so they’re not being stored for some reason


#12

You would need to click Save Settings at the bottom of the form after adding your keys. If you leave the page without saving, TestRail wouldn’t remember/store the keys permanently.

I hope this helps!

Cheers,
Tobias


#13

hahahah I can’t believe it…
Definitely there must be a better way to get your keys stored.

It’s working now, thank you!


#14

I’ll show you just in case what the error is.
You might have come across something like that before:

Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path ‘’, line 0, position 0.
at Newtonsoft.Json.JsonTextReader.ParseValue() in C:\Development\Releases\Json\Working\Newtonsoft.Json\Working-Signed\Src\Newtonsoft.Json\JsonTextReader.cs:line 1591
at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings) in C:\Development\Releases\Json\Working\Newtonsoft.Json\Working-Signed\Src\Newtonsoft.Json\Linq\JObject.cs:line 372
at Newtonsoft.Json.Linq.JObject.Parse(String json, JsonLoadSettings settings) in C:\Development\Releases\Json\Working\Newtonsoft.Json\Working-Signed\Src\Newtonsoft.Json\Linq\JObject.cs:line 420
at Gurock.TestRail.APIClient.SendRequest(String method, String uri, Object data) in c:\Users\MyUsername\Documents\TestComplete 12 Projects\TestRail Integration\testrail-api-master\dotnet\Gurock\TestRail\APIClient.cs:line 173
at Gurock.TestRail.APIClient.SendPost(String uri, Object data) in c:\Users\MyUsername\Documents\TestComplete 12 Projects\TestRail Integration\testrail-api-master\dotnet\Gurock\TestRail\APIClient.cs:line 90
Error location:
Unit: "Winscribe Desktop V8 - Regression Tests\00 Master\Script\TestRail"
Line: 86 Column: 5.


#15

Hi Leandro,

Sorry, haven’t seen this before. It looks like you either submit the data using an incorrect format (something else than JSON), or you use the incorrect TestRail address. It looks like the web request might return HTML instead of JSON so it’s likely an incorrect TestRail address you configured. You would only need to pass the base address of TestRail to the APIClient, e.g. https://example.testrail.net/.

I hope this helps!

Cheers,
Tobias


#16

Well… that’s something new!!
Now the error is this:
Gurock.TestRail.APIException: TestRail API returned HTTP 400 (“No (active) test found for the run/case combination.”)

I know the values I’m passing are valid… in which case something else is failing but definitely on the right track now.

Thank you Tobias!


#17

Hi Leandro,

It looks like you called add_result_for_case and there’s no test for the test run and case ID you provided. The run ID refers to the R### IDs and the case IDs would be those with the C### numbering. If you have the test IDs (T###), you can use add_result instead.

Cheers,
Tobias


#18

Hi Tobias,

Totally right again!!

Got it working in the end. Thanks a lot!


#19

That’s great to hear, happy to help :slight_smile:

Cheers,
Tobias