Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Can someone confirm the resolution of test results?


#1


The graphic shows a bunch of results for a series of tests that are executed - however the order seems wrong. We’re writing the test results to TestRail using the testRail NPM package.

As you can see from the ordering the orders seem wrong. Is this a ‘resolution’ issue? Were all those are actually recorded as having the same time (11:15) or is it stored in the DB with seconds?

Any ideas?


#2

Colleague and I will actually check what data is stored using the API - will update when I have some more information


#3

the question is - did you add them all with the same time ?


#4

They are all added at roughly the same time - certainly within a minute. We did an API check and the date added appears to be a date serial, and we can see that there are some single digit differences between some of the events. so I think we’ll do some more investigations. I have a suspicion that this is an async writing issue (maybe).

We did a call to get_results_for_case and here are the results

[
{
    "id": 19377,
    "test_id": 166634,
    "status_id": 5,
    "created_by": 5,
    "created_on": 1536146165,
    "assignedto_id": null,
    "comment": "One or more steps failed!",
    "version": null,
    "elapsed": null,
    "defects": null,
    "custom_step_results": null
},
{
    "id": 19373,
    "test_id": 166634,
    "status_id": 5,
    "created_by": 5,
    "created_on": 1536146151,
    "assignedto_id": null,
    "comment": "Test: C4954 Select Service Type failed with the error : An element could not be located on the page using the given search parameters.",
    "version": null,
    "elapsed": null,
    "defects": null,
    "custom_step_results": null
},
{
    "id": 19372,
    "test_id": 166634,
    "status_id": 5,
    "created_by": 5,
    "created_on": 1536146151,
    "assignedto_id": null,
    "comment": "Test: C4954 Enter Consignment Reference failed with the error : the string \"SAMPLE EXCEPTION\" was thrown, throw an Error :)",
    "version": null,
    "elapsed": null,
    "defects": null,
    "custom_step_results": null
},
{
    "id": 19371,
    "test_id": 166634,
    "status_id": 1,
    "created_by": 5,
    "created_on": 1536146150,
    "assignedto_id": null,
    "comment": "Test: C4954 Log on as depot 53 succeeded.",
    "version": null,
    "elapsed": null,
    "defects": null,
    "custom_step_results": null
},
{
    "id": 19370,
    "test_id": 166634,
    "status_id": 5,
    "created_by": 5,
    "created_on": 1536146150,
    "assignedto_id": null,
    "comment": "Test: C4954 Select a shipper account failed with the error : the string \"SAMPLE EXCEPTION\" was thrown, throw an Error :)",
    "version": null,
    "elapsed": null,
    "defects": null,
    "custom_step_results": null
}
]

Notice that the ID’s are sequential, but the created_on are identical - at least for the records that are flipped (1 & 2 for instance) - so I think that this is possibly because the test execute within 1 second - or because of asynchronous writing. Easiest solution is to wait for at least 1 second after writing a record. If that fails then we can look at async.