I have a job that gets all runs for a project, and then calls get_results_for_run for each one. The job sleeps 3-5s between each call so I don’t hit the API limit. But, the job will not complete if I try to get a large amount of results.
It runs on average 1.5 hrs and then fails on a 504.
The job works if I filter to a smaller number of runs, like around 50, but does not work when the run count is 1000 or more.
I can filter down my job and run on smaller sets of runs, but if I want to pull them all it becomes very manual. Do you know why the api may be giving up after an hour of a steady drip of calls every 3-5 seconds?