Reload only when new data

[#1]

Hi,

Currently my Qlik Sense app is set up to reload at a fixed schedule. To make this more robust in case data is delayed from the source, I would like Qlik Sense to check for new data regularly, and only load when new data is available.

This is the setup I'm considering:

I have created a script in the extract app that checks for new data and this works as intended. If there's no new data, I stop the load with "Exit script;".

My issue is that the visualization app is set up to reload when the extract task completes successfully, and when I use "Exit script" the task is still considered successful, which means that the visualization app will reload even if there is no new data. I would like to avoid that.

One option would be to throw an error instead of using "Exit script;", but throwing errors every 10 minutes is not an elegant solution.

Does anyone have a suggestion for how to approach this?

[hector_munoz · Accepted solution]

Hi Magnus,

A possible solution could be that Extract App slept for 10 minutes (SLEEP 10000;) if it did not find new data in database. I mean, you create a loop that is broken only when you find new data, then Extract App ends and next apps are reloaded... Maybe it serves...

Regards,

H

[#3]

Thanks, I think that will work. I can set the extract app to start at the beginning of the interval when I am expecting the data, and then keep running (with 10 minute pauses) until the data is loaded.

Do you know how resource intensive this is compared to having the app scheduled to run every 10 minutes? Does "sleeping" processes hog resources? I might consider doing this for monthly data as well, but with longer intervals. E.g. I could have the app check every hour, but it might keep going for up to a week.

[hector_munoz]

Hi Magnus,

I donot know how many resources it takes... I suppose that having the application loaded in RAM even if it is sleeping will consume more resources that launching it every 10 minutes... But it would have to be tested... Maybe your initial option of raise an error finally is the best approach... Anyway, please, inform in this thread about the final resolution of your problem... You have my curiosity... 😉

Regards,

H

[#5]

Hi Hector,

I will follow your advice, and my plan is to do the following:

1. Schedule reload task at beginning of the time interval where data is expected (10:00)

2. Check for new data every 10 minutes, loop until new data is available

3. Timeout after 3 hours (13:00)

4. Retry up to one time if timeout (13:00-16:00)

4. Reload task will start again (scheduled to run daily)

I will only check for new data until 16:00, since if it has not arrived then we can wait until the next day. I also want to have a retry, in case the first load fails for some other reason than no data being available (e.g. database connection).

Thank you for helping!

[hector_munoz]

Hey Magnus,

Very good solution! And nice to help you!

Regards,

H

[Chalorme]

Hi,

could you please share your script by which You are checking the latest date from backend table - and depending upon that how you’re holding or triggering the qlik app reload? 
cause I am facing a serious problem , where on some days backend table procedure is still overrunning & our qlik app is getting reloaded in between that , as per schedule - and sometimes due to this refresh timings syncing issue dashboard is populating null records - so how can we avoid it ?

my date field is “update_date”.