Join 34,000+ subscribers and receive articles from our
blog about software quality, testing, QA and security.

Best practices for automating manual test cases


#1

Hey there, I have worked for a couple companies that have gone through the implementation of automation into TestRail. In each company, there has always been different practices of doing things. I would like to ask the community & gurock what their best practice would be for the following examples:

  • When you automate a manual test case, what do you do with the manual case and what do you do with the automation case?
    ex: Do you keep the manual test and just flag it as automated, if so do you point to the case id in automation to have a link and add any other fields to the testrail case? How do you keep it up to date if the test changes?
    ex2: Do you delete the manual test and import the automated test in its place?
    ex3: Do you keep both tests but flag the manual one as automated?

  • What do you use the reference field for?
    ex: do you use it to link the jira epic/story for the test case? how do you then link to the requirements?
    ex2: do you use it to link to the requirements, if so, what tool do you use?

  • Most automated tests are webdriver, api, etc… but do you include unit tests in testrail?

  • If I have a manual test that’s been automated but not under each platform, what do you do?
    ex: Don’t flag it as automated but make a note in the test stating its automated for xyz platforms (or a separate test case field with automated platforms)
    ex2: Flag it as automated and spin up a new test case for the non-automated platforms
    ex3: Not a concern, if it works on one platform, it should be sufficient

  • If you add automated tests to testrail that do not have a corresponding test case, where do you put them?
    ex: use a section id of the feature to place them where they should logically go
    ex2: use a section id of the feature but then have an ‘automated’ folder where they go
    ex3: have a flag automation folder

  • How do you work around being reliant on testrail ids?
    ex: if you migrate from a cloud solution to onprem… all of your ids break. If each test case is linked to an id, this would be thousands of code changes needed… or if a manual tester moves sections around to reorganize the group tree… it may break automation

I know that’s a lot but I think it’s a worthwhile discussion.

Thank you!


#2

Hey Alias,

Thank you for the post and in depth information.

When you automate a manual test case, what do you do with the manual case and what do you do with the automation case:

Many customers choose to keep the manual test and add a custom field that marks the case as automated. This makes it much easier to maintain your test case repository as you do not have to create an additional test case and you can always turn the automated status on or off if need be.

What do you use the reference field for:

The reference field can be used for a few different things. The most common use of the references field however is for linking a test case to a requirement ticket. i.e., JIRA ticket. When you have integration setup between TestRail and your defect/requirement tracker, TestRail will also allow you to either click and directly load the reference information or display the reference information in a popup modal window within TestRail.

Most automated tests are webdriver, api, etc… but do you include unit tests in testrail:

You can definitely store your unit tests within TestRail. That being said, you should integrate TestRail with your automation/CI tool. You can integrate TestRail with basically any test automation tool and many of our customers integrate their automated tests. You would integrate test automation tools by using TestRail’s API. You can learn more about the API on our website here:

http://docs.gurock.com/testrail-api2/start

While we don’t currently have ready-to-use scripts for specific test automation tools, many of our customers use TestRail together with various test automation tools and frameworks such as Selenium, JUnit, Cucumber, many commercial tools, CI tools like Jenkins etc. It’s also planned to offer more example scripts in the future. We also offer basic API bindings for various programming languages such as Java, .NET, Python, Ruby, PHP etc. via the above link. You might also find this article interesting with details of how other companies like the BBC integrate their automated tests with TestRail:

https://blog.gurock.com/?p=4427

If I have a manual test that’s been automated but not under each platform, what do you do:

In this case, you would need to have multiple versions of the test case, although you could always use configurations to create your test runs. You would need to go in and mark any test cases per configuration that were not automtaed as manual, but this could save you some time in the long term as well as eventual overhead within TestRail.

If you add automated tests to testrail that do not have a corresponding test case, where do you put them:

You would always need to have a corresponding test case within TestRail for each automated test. TestRail has no mechanism for storing testing information without a test case to store it in.

How do you work around being reliant on testrail ids:

TestRail is very reliant on IDs. TestRail uses IDs to help identify which test case is which, etc. You can always create a custom field that you input your own custom ID or marker data in and use that to sort/filter your test cases by.


#3

Thank you for your quick reply Marty, it’s very helpful.

RE: automating a manual test case:

  • So I’d have an extra field in the test case called automated or not, I’d flag that. But then I have a few more questions :slight_smile:
  1. I initially had one for 'how do I know what platform this is automated for) but I think maybe adding a custom field for automation platform, map it to the code and populate it would be best… thoughts?
  2. To link to the manual test case, there seems to be 3 options: use the name, use the test case id or use a custom field. If we use a custom field, I guess we could use a UUID, add it to the test case, and utilize it in the code, that might be the safest option… and it would allow us to find the test case in other baselines (we make a baseline for each release and keep it alive until its end of life support) so we should be able to execute runs of previous releases easier.
    3/ What happens if my test case changes… I’d now have two places to update… How do you propose to deal with changes? I can think of a few options… a) update both, suck it up, b) code your automation to update testrail steps, c) create a new test case with the updated steps and archive the old one.

RE: reference field

  • So the practice from your suggestion would be to store all requirements in the epic of whatever feature, add it to the test cases, then utilize the reference reports in testrail to see coverage for requirements? I suppose we could also add unit tests there too but that would mean we’d have to import them into testrail…

Another thing I didn’t discuss, sections. I have proposed various different ways to structure test cases but I think I’ve agreed on higher level grouping > products > features > test cases, ex: Platforms > Docker > Able to create a container > testcase1, 2, 3… we have a lot of products that tend to melt into other products (like JIRA does), so the rule I’ve come up with is that if it touches more than 1 place: make a separate section for it. Thoughts?

Also what about running automation on previous releases? I guess we’d use the test cases in baselines (if we branch each release) to run against them? + down the road we wouldn’t want irrelevant automated tests inside the master/default suite so what would usually happen to those guys? I assume they’d be removed and the code would no longer reference it…?

And finally… if we went with a custom field, UUID for mapping our automation test cases in testrail… is there a way to query it? My colleagues mentioned that in the API you can get a 1 to 1 mapping of a test case if you search via id, but if you search via UUID, you need to then query every test case then go through each one, one by one, through each field until you find the UUID… making it slower, is this a valid statement? I’m not a dev so I haven’t gone through the limitations of the API personally…


#4

Hey @martylavender, hoping to hear back from you with some additional insights to my follow-up. Thank you for your time!


#5

Hey Alias,

Let me look over these and get back to you. I should have a response by tomorrow to these questions.


#6

Thanks @martylavender :slight_smile:


#7

Hey Alias,

My apologies as I have been out of the office ill. Let me see if I can address these questions for you now! :slight_smile:

  1. I think this is a good idea. You could always have a second custom field that has the platform that is being tested. You can also just separate these automatically in your test plans by using configurations. This would allow you to setup each platform/software system you are testing and TestRail would automatically place your tests within these different configurations as test runs under your test plan.

http://docs.gurock.com/testrail-userguide/userguide-gettingstarted#test_plans_and_configurations

  1. This would work very well that way you can narrow down each type of test case even more.

  2. In this case, using the API may be the best way to do this. Obviously it would take some additional coding/development time on your part but it would definitely save you time overall in the long term.

  3. This is exactly how I would recommend doing this. The references field can actually be a very powerful tool to use when checking these kinds of things.

  4. Sections and subsections within a single large repository is the best way to go when it comes to categorizing/organizing your test cases. This makes it extremely easy to move test cases around, start test runs/plans, etc. Go with this and you will find it scales extremely well.

  5. This really depends on what you want to do with them. I would recommend having some sort of mechanism on your automation side that takes into account that these tests no longer need to be run or need to be included and it would simply eliminate running against them.

  6. The easiest thing to do here is to use the API. You can use say the get_cases method and then query those results for the UUID custom field. You would just be looking for example: custom_uuid_custom_field In this case, the system name of your custom field would be ‘uuid_custom_field’ and you would add custom_ to the front of it. Its actually not as hard to query for a custom field as it initially seems.

http://docs.gurock.com/testrail-api2/start

Let me know if this helps!