Qlik Sense schedule incremental load of QVD files and data model refresh table by table

[arixooo123]

Hey Guys,

I have a QVD generator app in which handles the initial load, now I want to add incremental load scripts to that. I just can't figure out where to add my scripts and how to schedule them to run (some tables daily, some weekly) and after each incremental load, refresh the data model.

Unfortunately, couldn't find any related materials or links for doing these things qlik sense.

Thank you so much for your help

[micheledenardi]

Hi Ar,

Here you can find the documentation to implement incremental load in QlikSense (the code is the same of QlikView):

Loading new and updated records with incremental load ‒ Qlik Sense

If you want to manage 2 different update tasks (one daily and one weekly) just put an if statement and run the update every day, something like this:

If weekday(today())=6 then //On sunday

     //your weekly incremental load script

else

     //your daily incremental load script

end if

[arixooo123]

Thanks Michele,

I've read those posts, but that's not actually my concern.

I have 20 QVD files that need to be updated (Some every 15 minutes, Some every 4 hours, some daily and some weekly).

Your script looks like will do it, but I am not sure how to write it for my own situation.

Also, I'll appreciate it if you could suggest how to refresh data model in the most optimal way?

For instance, only refresh the model for updated tables and not the whole model every 15 minutes

[micheledenardi]

When you reload Qlik documents all table are updated (some table could contain updated data(new QVDs), other could contain old data(old QVDs)).

In order to achieve your goal i suggest to create 4 different QVD_Generators.qvw :

- One for 15 minutes update task

- One for 4 hour update task

- One for daily update task

- One for weekly update task

Doing this you can have in the same folder some QVDs with updated data and others with old data.

After this you can set an incremental load according to your purpose (Update, Update/Delete...).

[arixooo123]

That's exactly the point, reloading the whole data model will take 20 minutes, isn't there any optimized way of doing this?

For instance, only update the model based on the QVDs that have new data?

Assume we have 20 QVDs that 15 of them need to be updated once a week and only one of them is updated every 15 minutes.I am wondering if there's a way to only update model for that table and avoid reloading the whole thing

[micheledenardi · Accepted solution]

Yes, there is a possibility to do this.

So... the default qliksense working path is c:\ProgramData\Qlik\Sense\App.

Using the QMC you have to find the ID of your data model app (ex. 03CB7255-DE48-4B16-962D-145D8D9A952B).

In the script of you data model app put a binary load from itself (the app refresh his own data):

BINARY lib://AppsFolder (ssvsa030_adm_qlik)/03CB7255-DE48-4B16-962D-145D8D9A952B;

Now, to setup the script that allow you to refresh only some tables you have to use an external QVD (generated by your QVD Generatos) which contain the information about what type of reloading you are doing (15min, 4h, 1d,1w).

After the binary load you have to read this QVD and use if statement to write your script so:

ReloadTypeTab:

Load

     ReloadType

From ReloadType.Qvd (qvd);

let vReloadType= = peek('ReloadType',0,'ReloadTypeTab');

if vReloadType= '15MinRelaod' then

     //incremental load for table1, table2, table3

elseif vReloadType= '4HourRelaod' then

     //incremental load for table4, table5, table6

elseif vReloadType= 'DailyRelaod' then

     //incremental load for table7, table8, table9

elseif vReloadType= 'WeeklyRelaod' then

      //incremental load for table10, table11, table12

end if

[arixooo123]

Thank you so much for your help

[arixooo123]

Hi Michele,

Just to make sure that I understand your answer, would you please explain what do you mean by:

if vReloadType= '15MinRelaod' then

     //incremental load for table1, table2, table3

In this step, I assume that the QVD is updated and contains the most recent data. now, I want to refresh the model (in order to add the new data to the model)

the next question is, will it be automatically affected in my dashboards too?  I binary loaded the data model qvw into another app called dashboards and started developing charts in that app.