I’ve been spending some time playing with TestRail and I really like what I
see – it’s looking really good for such a new product!
However, due to the way our product development is done, I am having a hard
time seeing how to smoothly integrate TestRail into our organization in a
way that is sustainable, particularly as the number of products (and the
number of versions of those products) that we must test increases over
Scanning the forum, it seems that I’m not the only one with these
challenges. Some posts have brought up the need to be able to “tag” test
cases as a way to address their needs. I am not sure I completely follow
how tagging would be implemented, and how well it would map to my
situation, so I guess the best thing to do is to post a description of our
situation, and how I would like TestRail to work with us to meet our needs.
So here goes…
We have a number of different products, each of which normally has several
different versions in the field at any given time, and at least one
not-yet-released version in active development. For example, we might have
the following versions out “in the wild”:
o V1.1 o V1.2 o V2.0
And say that our developers are actively working on a “bugfix” release to
V2.0 (to be called V2.1) as well as development of some serious new
features in what will become V3.0.
Looking at this in terms of test cases, most of them are equally applicable
to all versions of the product; for example, a test case that tests the
ability to login to the product – the login workflow just doesn’t change
from version to version.
However, there are test cases that have more specific version
requirements. For example, they test a feature that is only available in
certain versions (say a new feature that wasn’t in the product previously),
or an old feature that has been retired. It seems that this boils down to
the following general cases:
o Test cases that are applicable to versions up to and including version <X> o Test cases that are applicable to version <X> or later o Test cases that are applicable to only between versions <X> and <Y>
I would like to be able to maintain a single set of test cases for each
product and, for each test case, specify one of the above “applicability
ranges” (applicable to any version, applicable to versions up to and
including , applicable to versions or later, applicable to versions
between and ).
I would then like to be able to instruct TestRail to construct a test run
for, say, our upcoming version 2.x bugfix release (2.1), and have TestRail
include only those test cases that are applicable to this version.
Of course, this test run might not be 100% accurate – there might be some
test cases that are no longer applicable (for example, an old feature that
has been newly retired); in that case, I’d tweak the test case to note the
version at which the test case becomes no longer applicable. Newly-added
features would require new test cases, of course – they would have their
"applicability" set to “version or greater”.
Ideally, I’d like to be able to have this kind of capability for test suite
sections, and entire test suites themselves.
Of course, I might not have the details right – maybe the assembly of the
appropriate test cases is not done as part of the creation of a test run;
maybe there’s a different mechanism that should be used. But I do know
that without this kind of capability, it would be difficult for us to use
TestRail, and would only get more difficult over time.
Is the goal of TestRail to support this kind of usage? If so, in what kind
Thanks for putting up with a long post!