First of all let me congratulate you on one of the better test case management tools I’ve seen so far. It strikes me that many tools out there were seemingly written back in the '80, and really none of them come close to a proper web2.0 application that I am looking for to maintain our test cases. Yours is the first that comes close. That said, there’s one major problem that would stop me from being able to use it, so I thought I’d give you some feedback.
The absolute key to a decent management system for me is the scalability and thus having the ability to use the system for multiple projects, multiple releases to multiple customers, with multiple milestones etc. To achieve this you really need to remove any and all cases where duplication of effort occurs. I’ve already noticed a few comments in this forum that have to do with the ability to clone/move test cases between suites and projects, but I think that is not the solution to the problem, it is merely the solution to the symptom of the problem, which is to be able to run the same test for multiple projects.
By cloning the test cases you exponentially increase the management of said test cases. Assume I have a dozen odd projects, each which has a dozen odd customers each with a handful of releases (alpha, beta, final, etc) then you are already talking about 12123 = 432 configurations that need testing and that’s not even counting difference in hardware/platforms the software is run on.
Assuming I have a couple of thousand test cases per project, I do not want to clone the generic test cases 432 times. Imagine a typo in one of the original test cases!
It is therefor imperative that a test case does not have a parent-child relationship with the test suite. Or for that matter, the test suite should not have a parent-child relationship with the project. Rather, test cases should be associated with one or more suites and suites should be associated to one or more projects; in other words an 1-to-n relationship is required rather than a 1-to-1.
This way I can write (and maintain!!) a single test case in one place and any and all associated projects/testsuites/testruns that “use” this test case will automatically update correctly if/when I change the test case over time.
Although this might sound like a big jump from your current design, I dont think it actually is. From a UI perspective it is a rather straight forward change. Rather than having a dropdown list for suite for a given testcase, you simply have a comma separated text box. And much like how tags are added to anything nowadays, the text box can “suggest” suites as you type. This will allow you to have one test case associated to multiple test suites. In a similar fashion a project should have the ability to be associated to multiple test suites.
From the backend all that would be required is a few more 1-to-n db tables to match up the test cases with suites, and subsequently the suites to the projects.
That simple change should make your testRail a lot more powerful in my opinion. And frankly without it, although I’d love to, I can’t see how we could shoe horn the current parent-child system you have in to our process since it does not even scale to the number of projects we currently have, let alone future extensions.
There are a number of additional things I’d like to see, but I won’t go in to too much detail on these, as the afore mentioned comment is really my biggest concern. Non the less, I’ll list them here in no particular order:
- ability to import csv and other formats
- ability to export to csv
- additional user roles, and security; to allow users only access to some projects (useful to give customers limited/controlled insight to the project)
- ability to actually tag test cases, so that I can create test suites from all cases in a tag
- ability to see the number of times a particular test case has been run (and for which project/suite) from the testcase view page
- when viewing a test within the run (to execute) I can start/stop the timer, but I’d also like to go to the “next test case” from that view, rather than having to go up to the “run” view again first.
Some small defects of the current system I found so far:
- the %id% is case sensitive in the defect view URL. Although the help text lists it as %Id% it only accepts %id%
- the ID for a bug is numeric only. This should be free text to integrate better with other defect management systems like Atlassian’s Jira which use IDs like “FOO-23”
- when I stop the timer on a test execution it allows me to enter the result, which is what I expect. However, if I enter a result manually, the timer does not stop.
I’m sure I’ll have more feedback later, but I’d really like to hear your thoughts on the 1-to-n relationship of test cases first.
Director of Engineering
General Manager UK Office