This was not how I expected configurations to work and isn’t useful at all. Imagine you don’t test only on Windows, so you create a group for Windows, Linux, Mac. Then you want to have different configurations for each group (like IE, Chrome, Firefox, Safari; or browsers with device below that like Android, iPhone, etc)
Let’s say I want to run Mac/Safari, Windows/IE, and Linux/Chrome in today’s test pass. So I check those boxes.
It creates 9 test runs for:
- Mac for each of: Safari, Chrome (which I didn’t ask for), IE (which just doesn’t make sense)
- Windows for each of: IE, Safari (Doesn’t make sense), Chrome (makes sense but I didn’t intend to run this)
- Linux, similar to above.
Even if I follow your scheme, let’s make an OS group (with Windows, Linux, Mac) and a Browsers group (with IE, Chrome, Firefox, Safari). This still creates 12 total combinations many of which are just crazy.
I’d rather have a hierarchy like test cases with no automatic combinatorial matrix (and maybe just quick copy function) and be able to manage these globally and just have:
- Stock Android browser
Or just give me the matrix and allow me to select what I want.
Combinatorial testing is great and all but there are way too many forms of it for you to automatically decide which combinations make sense.