Hi, Upon creating a run using the api, we use include_all as true, because at runtime, we don’t know which test cases Cucumber executes. Each parallel thread will add results to the test run as scenarios complete. No problems here.
Once parallel threads are finished testing and publishing results, we need to close the run. But at this point, include_all should be false, and only include the tests that executed.
How do you best recommend efficiently fetching all the test case ids in the run that contain statuses Passed or Failed, so I can use that list in update_run with case_ids?
As a workaround, I’ve tried having each parallel thread update the test run description with the list of test case ids it ran, but sometimes test cases get left out due to race conditions between threads, retries on 500 deadlocks due to conflicts posting with other threads, and ensuring all case_ids had made it into the description before changing include_all to false.
A second workaround would be we could fetch all tests for the run, and get_test on every one to compile a list of associated test case_ids. Wouldn’t this add to our http traffic and add more to the rate-limiting problem?
A third workaround I thought of would be to move @wip tests etc into a suite called not ready, and just have include_all always true. This isn’t ideal, but seems it is the most straight forward alternative.
A quicker way would be if you could supply a means to narrow test case scope somehow to only those who executed so we can automatically scope and close the test appropriately. This seems practical with parallel runners everywhere. Could we have this feature request or implement something soon with the api for this?
Do you have any clean suggestions?