Thanks, I think that will work. I can set the extract app to start at the beginning of the interval when I am expecting the data, and then keep running (with 10 minute pauses) until the data is loaded.
Do you know how resource intensive this is compared to having the app scheduled to run every 10 minutes? Does "sleeping" processes hog resources? I might consider doing this for monthly data as well, but with longer intervals. E.g. I could have the app check every hour, but it might keep going for up to a week.
I donot know how many resources it takes... I suppose that having the application loaded in RAM even if it is sleeping will consume more resources that launching it every 10 minutes... But it would have to be tested... Maybe your initial option of raise an error finally is the best approach... Anyway, please, inform in this thread about the final resolution of your problem... You have my curiosity... ;-)
I will follow your advice, and my plan is to do the following:
1. Schedule reload task at beginning of the time interval where data is expected (10:00)
2. Check for new data every 10 minutes, loop until new data is available
3. Timeout after 3 hours (13:00)
4. Retry up to one time if timeout (13:00-16:00)
4. Reload task will start again (scheduled to run daily)
I will only check for new data until 16:00, since if it has not arrived then we can wait until the next day. I also want to have a retry, in case the first load fails for some other reason than no data being available (e.g. database connection).
Thank you for helping!