Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

How to Organize and Tag Test Cases for Automation Initiative


#1

I am a consultant working with a cloud-based software company. I am quite new to test rail.

The company I work with uses Test Rail to manage about 20,000 test cases across 15 projects. Of those, maybe 4000-6000 test cases across 4 or 5 projects have been run at least once during the past 8 months.

Some of those test cases are well documented (with descriptions of test scenario, test data sets and expected results). Others are simply one line descriptions (which require a domain expert to invoke).

We want to automate those frequently-used test cases and need a process to manage the automation effort.

Across these 4000-6000 test cases (again across 4-5 projects), I would like to create a report showing the following:

“State” of each Test Case

  • Not Yet Automated
  • Don’t Plan to Automate
  • Test Case Update Required
  • Automation Development in Process
  • Automation Verification in Process
  • Test Case Automated

Automation Priority

  • Low Priority
  • Medium Priority
  • High Priority

Who Responsible for Automation

  • Tester A
  • Tester B

I need this report to help prioritize our efforts and tracks who is working on what. Ideally, I would put this into a backlog (and manage it on a Jira agile board). But, that may be wishful thinking.

Does anyone have ideas on how to do this? I want to use as lightweight a process as possible. One key is that it must be easy to apply the above “attributes” to each of the test cases and then update those “attributes” as the test case status changes.

As a note, we are willing to collapse our test rail projects. We just are uncertain how to do this (we don’t want to lose past results associated with a test run). Likewise, we are more than happy to add “custom variables”. I am just uncertain if these can be associated with the test cases themselves.

Thanks in advance for your help.


Copying Custom Properties Values Along with Copying Test Cases
#2

Hi Doug,

Thanks for your posting! I would recommend adding custom fields (of type dropdown) for state/automation-priority/who-responsible on the test case level:

http://docs.gurock.com/testrail-userguide/howto-fields

You can then use the Cases > Property Distribution report from the Reports tab to generate distribution reports for your custom fields (one per field), including the count/percentage per group which gives you a good idea on the migration progress.

Cheers,
Tobias


#3

Tobias,

Thanks for the information. It is quite helpful.

I had some additional questions.

  • Regarding custom fields, how is sorting handled within the Property Distribution reports? It appears you can control the sorting (within the drop-down), but it seems different in the report.

  • When using a custom drop down, does the “number” (which is part of the “number/option” pair) have any significance – other than “TestRail stores the selected value internally as a number”? That is, is it used for sorting (the report)? I ask, because my numbers are already out of sequence (from a display perspective).

  • More importantly, is there an easy way to “batch update” the custom variables across a set of test cases? It’s a bit painful to edit these individually. It would be nice if i could do a search (by say test case id) and say, “Set this group of test cases to have an automation status of ‘x’”.

Thanks in advance for your help!

Doug


#4

Hi Doug,

Thanks for your feedback. Sorting in the Property Distribution report is not based on the display name but on the actual value. This is a bit different from how the field is presented in the rest of TestRail (in the dropdown). The number is mostly used as unique ID/index and doesn’t have any meaning otherwise. You shouldn’t change these values once a field is in use because the cases would still point to the same IDs.

Regarding bulk-edit: yes, you can edit multiple cases on the Test Cases tab by selecting them and then Edit | Edit selected. Or, you can configure a filter and then select Edit | Edit all in filter.

I hope this helps!

Cheers,
Tobias


#5

Tobias,

Thanks for the reply.

Regarding bulk-edit…when I bulk edit multiple test cases, can I selectively update only certain fields? That is, can I set the “Automation Status” (a Custom Field) to “Pending” for a set of Test Cases without impacting the value of other Custom Fields (i.e. attributes) assigned to those same Test Cases?

Also, is it possible to perform a search on “Automation Status” (e.g. find all Test Cases where Automation Status = “Pending”) and then select some of those test cases and perform a bulk edit on those (say, to reset Automation Status = “In Review”)?

Thanks,

Doug


#6

Hi Doug,

Yes, you can update just a single field when editing multiple cases. You can also use the filters on the cases pages and then manually select some cases from this subset and don’t need to include all cases that match the filters. You would then just need to use the Edit | Edit selected option again.

Cheers,
Tobias


#7

Tobias,

I (finally) found the search mechanism. It seems as if I cannot perform a search across all Test Cases within a single project. That I can only search (rather filter) within a Test Suite.

Or…Am I missing something.

As a note, We are running version 5.0.0.3367, built on 9/16/2015.

Thanks,

Doug


#8

Hi Doug,

The search (at the top of the screen) would include all cases of a project (and runs/tests/results/milestones etc.). I assume you mean the filter on the Test Cases tab, is this correct?

The filters operate on the case repository/test suite level. Starting with TR 4.0, we recommend using just a single test suite (case repository) per project and this setup would basically mean that a test suite and the case repository of a project would be the same:

If you have multiple suites, you would need to go to each suite but the filter is remembered and automatically applied every time you open the test suite/case repository.

Cheers,
Tobias