5 Replies Latest reply: Mar 29, 2017 4:00 AM by Muñoz Héctor RSS

    Reload only when new data

    Magnus Buseth

      Hi,

       

      Currently my Qlik Sense app is set up to reload at a fixed schedule. To make this more robust in case data is delayed from the source, I would like Qlik Sense to check for new data regularly, and only load when new data is available.

       

      This is the setup I'm considering:

      reload.JPG

      I have created a script in the extract app that checks for new data and this works as intended. If there's no new data, I stop the load with "Exit script;".

       

      My issue is that the visualization app is set up to reload when the extract task completes successfully, and when I use "Exit script" the task is still considered successful, which means that the visualization app will reload even if there is no new data. I would like to avoid that.

       

      One option would be to throw an error instead of using "Exit script;", but throwing errors every 10 minutes is not an elegant solution.

       

      Does anyone have a suggestion for how to approach this?

        • Re: Reload only when new data
          Muñoz Héctor

          Hi Magnus,

          A possible solution could be that Extract App slept for 10 minutes (SLEEP 10000;) if it did not find new data in database. I mean, you create a loop that is broken only when you find new data, then Extract App ends and next apps are reloaded... Maybe it serves...

          Regards,

          H

            • Re: Reload only when new data
              Magnus Buseth

              Thanks, I think that will work. I can set the extract app to start at the beginning of the interval when I am expecting the data, and then keep running (with 10 minute pauses) until the data is loaded.

               

              Do you know how resource intensive this is compared to having the app scheduled to run every 10 minutes? Does "sleeping" processes hog resources? I might consider doing this for monthly data as well, but with longer intervals. E.g. I could have the app check every hour, but it might keep going for up to a week.

                • Re: Reload only when new data
                  Muñoz Héctor

                  Hi Magnus,

                  I donot know how many resources it takes... I suppose that having the application loaded in RAM even if it is sleeping will consume more resources that launching it every 10 minutes... But it would have to be tested... Maybe your initial option of raise an error finally is the best approach... Anyway, please, inform in this thread about the final resolution of your problem... You have my curiosity... ;-)

                  Regards,

                  H

                    • Re: Reload only when new data
                      Magnus Buseth

                      Hi Hector,

                       

                      I will follow your advice, and my plan is to do the following:

                       

                      1. Schedule reload task at beginning of the time interval where data is expected (10:00)

                      2. Check for new data every 10 minutes, loop until new data is available

                      3. Timeout after 3 hours (13:00)

                      4. Retry up to one time if timeout (13:00-16:00)

                      4. Reload task will start again (scheduled to run daily)

                       

                      I will only check for new data until 16:00, since if it has not arrived then we can wait until the next day. I also want to have a retry, in case the first load fails for some other reason than no data being available (e.g. database connection).

                       

                      Thank you for helping!