This all makes sense. So the high level issue is that we cannot pull down more than a days worth of data (i.e. filter tests, test cases, test results, and test runs to created_after today - 1 day). Otherwise, the ETL process takes too long (hours+).
Our goal is to grab any data that has changed across those entities (test cases, test results, tests, and test runs) in the past day - or just since the last ETL took place.
Let’s take a test run for example. If I only grab data for test runs that aren’t complete, then things work fine. I no longer need to filter on create_after as I can limit what I ETL to incomplete test runs.
But how do I identify test runs that have recently been closed and pull back the associated tests and test results? The only way to do this is to pull down ALL test runs, which simply takes too long given the amount of data in our system.