Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Pilot evaluation notes


#1

We have started using TestRail for one of our product releases as a small pilot project evaluation… So I just wanted to share some of our working notes and thoughts.
As the job of any QA is to find bugs, the form of these notes will just be highlighting the ‘bugs’ or ‘feature requests’ that we have encountered while using TestRail:

Feature Requests

  1. It is great that whenever you hover over a reference link, you see the status of that link, but would be even better is if you didnt need to hover over it to see the status. If the status could be indicated by an icon or a colorchange of the link then it is easy to get an overview of which references in a list of test cases, that are closed - ie. ready for testing.
  2. Internal links. So that you can reference test cases to each other instead of 3rd party bugtrackers.
    2.1 Another approach could be ‘Shared Test Cases’ that can be accessed from different projects. And updated from either.
  3. Being able to use Issue links inside the Description field of a Test Plan. So that you can also hover over links in here to see their status.
  4. Being able to deprecate Test Cases instead of just deleting them would be good to enable us to do continous housekeeping without loosing test results that have already been executed and logged.
  5. When batch-editing test cases, it would be nice with an ability to select to use the Steps from case 1 and apply these to the other tests. Right now you need to re-write the Steps you want to apply to a batch of test cases.
  6. When you select test cases for a configuration, it would be nice if the popup window automatically opened in the same section as the testcases selected in the Test Run from which the configuration was drawn.
  7. When you have selected test cases in the selection window, it would be nice if you could just hit Enter on the keyboard to add them instead of having to mouseclick the OK button.
  8. It would be nice to be able to resize the Configurations window. We have a lot of configurations and scrolling can become tedious.
  9. When selecting and configurations and clicking OK to create all the permutations, it would be nice that NO test runs are selected by default. In most cases we only want to run 10%-30% of the possible configuration permutations.

Best Practices

  1. Dont use more than 2 Configuration groups at a time. If you need to make a 3, 4 or many-dimensional configuration, then create a new group where those are concatenated on their own lines.
  2. When adding configurations to a test run, start by deselecting all configurations and then add sequentially the ones you want.
  3. We need to define a common mandatory syntax for Version strings so that we do not confuse things. Ideally the Version string should ALWAYS include the product key.

#2

Hi Paul!

Thanks for your posting and sharing your feedback. I’m happy to answer your questions below:

  1. That’s already for the defects via the new Defects pages and it’s planned to look into this for the References as well.
  2. You can already link to other cases (and other objects) as follows: [C17]. TestRail would replace this a link to the test case with ID 17. Other supported objects are tests, runs/plans, milestones and projects (via T, R, M and P).
  3. It’s already planned to extend the auto-linking (point 2.) to add support for references and defects as well but we currently don’t have a time frame for this.
  4. You can mark test cases as deprecated with a custom field for example (checkbox or dropdown) and then filter those cases when you create test runs or plans on the Select Cases dialog. When you close test runs or plans, you can also safely delete deprecated/no longer needed test cases and the runs/plans would still display the tests and results for those cases: http://docs.gurock.com/testrail-userguide/userguide-tips#closing_test_runs
  5. -9. Thanks for the suggestions! I added them to our list of things to look into and we will make to review them for a future version.

Cheers,
Tobias