Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Drive CI automated testing with Test Plans


#1

I have a new feature idea for you.

We are trying to design a CI test automation system with the TestRail test plan as the basis. Here’s how it would work:

  1. QA Leads set up a test plan in test rail with the following test plan entries:

a) Smoke
b) New Features
c) Integration
d) System
e) Regression

Each of the test plan entries will have a set of configurations which determine the actual test runs.

  1. The test plan is linked to a milestone
  2. The build system is set up to specify the product + milestone when building
  3. The CI system is set up to wait for a build
  4. Once a build is found, the CI system can find the test plan in TestRail using the product + milestone
  5. The CI system would run test plan entries in order (since they are nicely indexed).
    Example: A smoke test runs would be done, then all new feature test runs, then the integration test runs, then the system test runs, and then the regression test runs. The assumption might be that all of the test runs for an entry must pass before running the next test plan entry.

That would work OK except that each test plan entry can only specify one set of configurations on a single test suite. I’d like to have another level in the test plan for testing stage. Each testing stage could be defined with a set of test plan entries (that each have a set of test runs based upon a set of configurations).

So your test plan might be structured like this:

STAGE: BVT
-> ENTRY: API TESTS
—> TEST RUN: Windows 7
—> TEST RUN: Windows 8
-> ENTRY: UI TESTS
—> TEST RUN: Windows 7, Firefox
—> TEST RUN: Windows 8, Firefox
—> TEST RUN: Windows 7, Chrome
—> TEST RUN: Windows 8, Chrome

STAGE: SMOKE
-> ENTRY: API TESTS
—> TEST RUN: Windows 7
—> TEST RUN: Windows 8
-> ENTRY: UI TESTS
—> TEST RUN: Windows 7, Firefox
—> TEST RUN: Windows 8, Firefox
—> TEST RUN: Windows 7, Chrome
—> TEST RUN: Windows 8, Chrome

STAGE: FEATURE
-> ENTRY: API TESTS
—> TEST RUN: Windows 7
—> TEST RUN: Windows 8
-> ENTRY: UI TESTS
—> TEST RUN: Windows 7, Firefox
—> TEST RUN: Windows 8, Firefox
—> TEST RUN: Windows 7, Chrome
—> TEST RUN: Windows 8, Chrome

STAGE: REGRESSION
-> ENTRY: API TESTS
—> TEST RUN: Windows 7
—> TEST RUN: Windows 8
-> ENTRY: UI TESTS
—> TEST RUN: Windows 7, Firefox
—> TEST RUN: Windows 8, Firefox
—> TEST RUN: Windows 7, Chrome
—> TEST RUN: Windows 8, Chrome

I really think that adding testing stage support would be a nice feather in your cap and allow teams to write CI systems that are fully integrated with TestRail. If the addition of this drove some team (like mine) to create an open source tool to drive automated CI testing from TestRail, it might also help sell your product.


#2

Hi Todd,

Thanks for your posting and feedback! You could look into using multiple test plans (which are also linked to the same milestone/iteration/sprint) instead of stages in one test plan. I believe you could simply replace the multiple stages in a single test plan with multiple, separate test plans and could implement the same staging features/behavior. To implement the indexing or ordering, you can add additional meta details to the test plan name or description, for example. Would this also be an option?

Cheers,
Tobias


#3

We will likely have to go with multiple test plans to allow flexibility when deciding what tests will be run during each CI stage. Either way we go, we would have to use some sort of convention to specify the ordering. It’s not ideal, but it could work.

Does your answer mean that your organization would not consider this feature?


#4

Hi Todd,

Thanks for your reply! We are considering this and I’ve added this to our list of things to look into. It’s just that this feature request came up for the first time and I believe multiple test plans would be a good workaround in this case. I hope this works for you and your team as well. You can either specify the order as part of the test plan name or use the description (I think the name would be more convenient as you can also see this directly in TestRail on the overview pages).

Cheers,
Tobias