Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Advice on TestRail / Automation setup


#1

I’m after some advice on what I imagine should be a pretty basic setup of TestRail with automation feeding test results.

We’re a company that does mainly Waterfall/Interative across numerous applications/projects, we’ve recently started using TestRail and are setting up projects per application to store test cases in with baseline support. We also plan on having TestRail projects per actual project where we move/copy test cases to as required.

We’ve also very recently started automating some test cases via Eclipse(Java, Selenium, SOAPUI, TestNG). We plan on pushing test results per test case back to TestRail. Currently I have a specific automated test case in Eclipse per TestRail test case.

Can someone please advise on how I can map the Eclipse automated test cases to TestRail test cases, initially I’ve named each @Test method call with the TestRail caseID which I now realise probably won’t work as the TestRail test case ID will change if I copy it from the repository project to the development project, I’ve also looked at moving the test cases from the repository project to the development project while a Test Run is initiated but the Test Run loses all Test Results when I copy the cases back to the repository.

How do others handle this? Or is my approach completely wrong?


#2

Hi Daniel,

Thanks for your posting! We would advise against copying test cases and the case IDs would be static in this case. Why do you use separate projects for your repository and development project? Couldn’t you use a single project instead? If your cases change over time, TestRail has advanced versioning features for that and you can simply archive test runs & plans, for example:

http://docs.gurock.com/testrail-userguide/userguide-tips#closing_test_runs
https://blog.gurock.com/test-management-test-case-versioning/

Cheers,
Tobias


#3

Hi Tobias, Thanks for the reply,

We use a project as a repository per application as we’re currently going through automating test cases outside of any development project. This seems to be the biggest thing missing from TestRail that I would’ve expected to be there, nowhere to store a repository of test cases as everything seems geared towards re-working everything for each project.

For example, I’m currently automating testing of a website that I know will be part of a development project in the future, but I’m unsure when or what will change at this point, where do I store the test cases for the current version of code.

How would you suggest we manage this?

Thanks,
Daniel


#4

Hi Daniel,

If you are reusing the same test cases over time, it can make sense to use a single project instead and this would be the reusable, central case repository. Most customers use this approach and you wouldn’t usually need to copy cases in this scenario (except when you use advanced versioning features such as baselines).

Cheers,
Tobias


#5

Thanks Tobias,

Just to clarify you’re suggesting a single project to hold all test cases for all our applications. We then manage development projects (which could be for a single application or across multiple applications) via milestones/test plans etc?


#6

Do you use the same cases for all applications? Can you provide a few more details on testing process and how you currently structure your projects/cases and applications?

Cheers,
Tobias


#7

Hi Tobias,

We have approx 10 applications we test, we generally have 2-4 development projects running at a time, nearly all of our applications interface with each other but the majority of cases are specific to an individual application. All applications only have 1 production version. Atm all our testing is manual but we’re slowly moving into automation.

At a very high level I can see how using a single project could potentially work using milestones/plans for each separate development project, potentially with baseline support as we can have 2 development projects changing similar functionality within a single application at a time. This would ultimately lead to a very large number of test cases 10k+ within a single suite which may be a pain navigating. Are there any gotchas in having a large number of tests in a single project?


#8

Hi Daniel,

For larger test suites and projects, we recommend the view mode with a single section and this scales well to tens of thousands test cases and more:

This shows a single group/section at a time only (with or without sub-groups) and scales well even with very large case repository or runs. This is similar to how Windows Explorer and Mac Finder would show the files and folders and they also wouldn’t show all files on disk in a single view.

I hope this helps!

Cheers,
Tobias