Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Confusion on workflow - very detailed post!

Hi,

I created a trial account, and have been trying to out to figure out how to use it. I have several questions on the workflow/terminology, and would appreciate any help.

Specifically:

  • I created a project
  • went to project’s dashboard->Overview

Question- at this point the workflow in not clear. On the Overview I see “Add Milestone” and “Add Test Run” buttons. ???!!? I do not have test cases so what does it mean to add either of these at this point? I just have a shell project now.

  • decided to click on the tab ‘Test Runs and Results’

  • I see a button on the left "Add Test Run’, and 2 buttons on the right ‘Add Test Run’ and ‘Add Test Plan’
    Again, what is the user to do here with just a shell project being created?

  • decided to click on the tab ‘Test Cases’ - see two buttons ‘Add Section’ and ‘Add Test Case’.
    Question- I have seen many references to ‘Test Suites’- I have gone thru the entire UI, but do not see any "Test Suite’ any where! Am I missing something? Very confusing.

  • added couple of ‘sections’ and some test cases inside these sections. Still do not see any ‘Test Suite’.

  • on the tab ’ Test Runs and Results’, clicked on the button ‘Add Test Plan’ and created one.

  • back on the Overview tab, I now see this "Test Plan’ listed under “Test Runs”. It is also listed under “Activity” section as a “Test Plan”. Very, very confusing! Is TestPlan==TestRun? I have read posts where you say a TestPlan is a set of Test Runs.

  • on the above tab, I also created a Test Run. I see this listed both on the Overview and also on the ‘Test Runs and Results’ tab.
    So, this last tab lists both Test Plans and Test Runs - confusing.

Is the TestRun I created a stand-alone “Test Run” associated with the project, and not a part of any “Test Plan”? So, I can have both “Test Plan” and “Test Run” inside a project, and unrelated to each other?

  • next I added the Test Run to the Test Plan. Now, does the Test Run “belong” to the Test Plan?? Can the Test Run be “executed” independent of a Test Plan??

I am also not clear on “running a test plan or test run”? Is there a “Start Test Run” or “Start Test Plan” button/concept? Where do I see this? I see a “Rerun” option at the top right when I click on a TestRun or TestPlan. When do these “test runs” start running? The moment I create them? Can I pause/stop these runs?

Many more questions, but will post them separately.

Thanks in advance for any clarifications.

Hi Rao,

Thanks for the post! Getting started with a new TestRail can be a bit overwhelming as there are a lot of entities, functions and systems to manage, but I’ll do my best to break it down for you and address your questions.

Typically, once you’ve got a project created, the next best place to focus is on creating your test cases. You mentioned that you’re not seeing the Test Suites option, and that is by design. Starting with TestRail 4.0, we made test suites optional for projects, and the default recommended project type does not include them. For most teams, having all test cases in a single repository organized into sections and sub-sections is a bit easier to manage and allows you to create test runs that encompass the entire scope of the project rather than only the tests within a specific suite. You can still set up projects using test suites, and if you’d like you can convert existing projects to test suite modes as needed. You can read up a bit more about that on our blog here:

https://blog.gurock.com/introducing-testrail-4-0/

Once you’ve got your test cases up and running, you can start building out the structure of your project using milestones, test runs, and test plans.

Milestones are a collection of test runs, and are like a mini-project that allows you to track testing progress within a project for different versions or releases. They aren’t required, but can be a big help in organizing your testing structure for larger scale projects that will be iterated on.

Test runs are a collection of test cases that act as a basic outline for your testing team. Once you’ve got a test run set up, you can assign the full run or certain tests within to members of your team, and they can go through the tests and add results and other details as they complete the outlined steps. You can edit a test run at any time to add or remove test cases if necessary.

Test plans are used to create multiple test runs based on different configurable options in one go. This is very useful when all test cases should be tested in different environments, like different browsers and/or operating systems, for example.

When you choose to create a test run on it’s own, it is not part of the larger test plan. Test plans as an entity within TestRail are treated like a very large multi-faceted test run. It is not currently possible to add an existing test run into a test plan, they are separate from one another.

There is no function to start a test run, the test run begins when the user begins adding results to the tests. In each test case you can set an estimated time for how long the test should take, and when adding a result the tester can use the built in timer to measure the time the test is taking, or manually add how long the test took to complete.

I hope that clarifies things and gives you a better idea of the starting point to working with TestRail. For further information, I would recommend taking a look at some of the documentation on our site. You can find a good (albeit somewhat out-of-date) getting started document that outlines much of what I’ve described here:

http://docs.gurock.com/testrail-userguide/userguide-gettingstarted

We have a lot of additional guides and documents you can also check out through the index page here:

http://docs.gurock.com/testrail-userguide/start

If you have any further questions you can always reach out here through the forums, or you can contact our support team directly at contact@gurock.com.

HI,

Thank you for a detailed explanation.

You mention that you cannot add an existing Test Run to a Test Plan. However, we have to create a Test Run(id =R1) first, and only then can it be added to a Test Plan. And when I do this, I notice that under the tab “Test Runs and Results”, both the Test Plan, and the Test Run are listed separately. On digging a bit deeper, the id’s of the Test Runs inside the Test Plan are different from the ids of the Test Run “outside”. This seems to imply that for this project, I can add test results either to the Test Run (id=R1) that is “outside” of the Test Plan, or to the Test Run(id=R2) that is “inside” a Test Plan. My expectation was that once I create a Test Plan for a project, that is the only entity I should see in the project, e.g. when I go to “Test Runs and Results”. There must be more that I would perhaps grasp once I spend more time experimenting.

Hey Rao,

Thanks for getting back to me. The test run you created outside of the test plan (R1) is not part of the existing test plan, and is a full standalone test run. As I mentioned, test plans are great for teams testing against multiple configurations in tandem, but they are not required, and most teams use test runs independently of the test plans. When you add a new test plan, on the right side you’ll be prompted to add a new test run within the test plan, and it will not affect or include any runs created separately.

You can have test runs and test plans active independently of one another, which is why you’ll see both listed in the “Test Runs and Results” tab. This can be useful if the product you are testing has some features which need to be tested in specific environments (like different browsers and OSes) and other features which are independent of those configurations. In this way, you could set up a test run that tests all features that are outside of those configurations, then set up a test plan to test the features that are browser-specific, as an example. In these cases, using a milestone to track the progress of both the test runs and test plans associated with a given release is very helpful.

I hope that clarifies things further. Please let me know if you have any further questions.