Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

ALMOST perfect!


#1

First of all let me congratulate you on one of the better test case management tools I’ve seen so far. It strikes me that many tools out there were seemingly written back in the '80, and really none of them come close to a proper web2.0 application that I am looking for to maintain our test cases. Yours is the first that comes close. That said, there’s one major problem that would stop me from being able to use it, so I thought I’d give you some feedback.

The absolute key to a decent management system for me is the scalability and thus having the ability to use the system for multiple projects, multiple releases to multiple customers, with multiple milestones etc. To achieve this you really need to remove any and all cases where duplication of effort occurs. I’ve already noticed a few comments in this forum that have to do with the ability to clone/move test cases between suites and projects, but I think that is not the solution to the problem, it is merely the solution to the symptom of the problem, which is to be able to run the same test for multiple projects.

By cloning the test cases you exponentially increase the management of said test cases. Assume I have a dozen odd projects, each which has a dozen odd customers each with a handful of releases (alpha, beta, final, etc) then you are already talking about 12123 = 432 configurations that need testing and that’s not even counting difference in hardware/platforms the software is run on.

Assuming I have a couple of thousand test cases per project, I do not want to clone the generic test cases 432 times. Imagine a typo in one of the original test cases!

It is therefor imperative that a test case does not have a parent-child relationship with the test suite. Or for that matter, the test suite should not have a parent-child relationship with the project. Rather, test cases should be associated with one or more suites and suites should be associated to one or more projects; in other words an 1-to-n relationship is required rather than a 1-to-1.

This way I can write (and maintain!!) a single test case in one place and any and all associated projects/testsuites/testruns that “use” this test case will automatically update correctly if/when I change the test case over time.

Although this might sound like a big jump from your current design, I dont think it actually is. From a UI perspective it is a rather straight forward change. Rather than having a dropdown list for suite for a given testcase, you simply have a comma separated text box. And much like how tags are added to anything nowadays, the text box can “suggest” suites as you type. This will allow you to have one test case associated to multiple test suites. In a similar fashion a project should have the ability to be associated to multiple test suites.

From the backend all that would be required is a few more 1-to-n db tables to match up the test cases with suites, and subsequently the suites to the projects.

That simple change should make your testRail a lot more powerful in my opinion. And frankly without it, although I’d love to, I can’t see how we could shoe horn the current parent-child system you have in to our process since it does not even scale to the number of projects we currently have, let alone future extensions.

There are a number of additional things I’d like to see, but I won’t go in to too much detail on these, as the afore mentioned comment is really my biggest concern. Non the less, I’ll list them here in no particular order:

  • ability to import csv and other formats
  • ability to export to csv
  • additional user roles, and security; to allow users only access to some projects (useful to give customers limited/controlled insight to the project)
  • ability to actually tag test cases, so that I can create test suites from all cases in a tag
  • ability to see the number of times a particular test case has been run (and for which project/suite) from the testcase view page
  • when viewing a test within the run (to execute) I can start/stop the timer, but I’d also like to go to the “next test case” from that view, rather than having to go up to the “run” view again first.

Some small defects of the current system I found so far:

  • the %id% is case sensitive in the defect view URL. Although the help text lists it as %Id% it only accepts %id%
  • the ID for a bug is numeric only. This should be free text to integrate better with other defect management systems like Atlassian’s Jira which use IDs like “FOO-23”
  • when I stop the timer on a test execution it allows me to enter the result, which is what I expect. However, if I enter a result manually, the timer does not stop.

I’m sure I’ll have more feedback later, but I’d really like to hear your thoughts on the 1-to-n relationship of test cases first.

/Cheers,
Jelte Liebrand
Quickoffice
Director of Engineering
General Manager UK Office


#2

Hello Jelte,

Thanks a lot for the detailed feedback, we really appreciate it. Please see below for our answers to your points:

Based on the various feedback we’ve received so far, including yours, we understand that we still need to work on configuration support and making TestRail’s structure more flexible. I would like to better understand your need to have a single test case in multiple test suites and the need to have a single suite in multiple projects. It would be really great if you could provide me with a quick example test case (or multiple) which you usually use across projects/test runs, if that’s possible (if you don’t want to post it on this public forum, please email me at dg@gurock.com). This would help us better understand your project/cases and help us understand the request in context.

We are thinking about various ways to improve handling of configurations, but I’m not sure if this would help in your case. We generally recommend to organize cases in configuration dependent and independent suites, while a configuration can be anything a project needs to be tested against (operating systems, web browsers, specific hardware etc.). It is then recommended to start a separate test run for each suite/configuration combination.

We plan to make it more easy to start multiple runs at once and to manage the different configurations within TestRail. We might even add a new entity that allows to organize multiple runs/suite (called Collection, Test Plan or something different, we are not sure yet) and to make it easier to rerun entire run collections. Now, it would be interesting to know if this would solve some of your requirements, even if you would need to rethink/reorganize your test suites. We are still thinking about other ways to make TestRail more flexible and will also think about introducing 1:n relationships as you suggested.

[quote]- ability to import csv and other formats

  • ability to export to csv[/quote]

I agree that this would be useful to have and I’m sure we will provide this eventually.

[quote]- additional user roles, and security; to allow users only access to some projects (useful to give customers limited/controlled insight to the project)
[/quote]

This is also on our feature request list. We decided not to include it in the first version as we wanted to gather more feedback first and release TestRail 1.0 earlier.

[quote]- ability to actually tag test cases, so that I can create test suites from all cases in a tag

  • ability to see the number of times a particular test case has been run (and for which project/suite) from the testcase view page
  • when viewing a test within the run (to execute) I can start/stop the timer, but I’d also like to go to the “next test case” from that view, rather than having to go up to the “run” view again first.
    [/quote]

All great suggestions, thanks. We have thought about providing Next/Previous buttons to tests in the past, but weren’t sure how to handle the fact that users might come from different locations when viewing a test. E.g., Next would mean something different if the user came from her Todo list. The user would then expect that the Next button would link to the next test in her Todo list, making it non-trivial to implement and understand for users. We will think about this again though.

[quote]- the %id% is case sensitive in the defect view URL. Although the help text lists it as %Id% it only accepts %id%

  • the ID for a bug is numeric only. This should be free text to integrate better with other defect management systems like Atlassian’s Jira which use IDs like “FOO-23”
  • when I stop the timer on a test execution it allows me to enter the result, which is what I expect. However, if I enter a result manually, the timer does not stop.
    [/quote]

We’ve already fixed the defect ID problem and limitations in our internal build based on your feedback. We will take a look at the timer issue.

Thanks!

Regards,
Dennis