Can you guys please help me incremental load in qlik sense.

[#1]

Can you guys please help me incremental load in qlik sense. as we have around 60-70 lakhs of rows with 200-400 columns in each. So its too much time taking process to load the complete data in qlik sense.

[OmarBenSalem]

I'll guide with theoritically;

What you have to do :

1) Load all the data the first time:

The first day, you'd want to have all the available data till the loading date:

table:

load *

from source;

// then store all of it into a qvd;

store table into table.qvd(qvd);

once you do this, you need to alter your script.

No need to load everything from now on, but only the data of the current day (for example):

so comment the old script and replace with :

2) load new data + concatenate to all existing data + store into SAME qvd:

// load new data

table:

load * from source WHERE DATE=Today();

//concatenate new data with existing data

concatenate

load * from table.qvd(qvd);

//storing the new concatendated data into the same QVD

store table into table.qvd(qvd);

With that, every day you load your app, you just import the data of the day, concatenate it with all the existing data and store the whole thing into the SAME qvd and repeat.

Hope this is helpfull.

Omar BEN SALEM.

[vkish16161]

Omar's suggestion is somewhat correct but you'll use a where not exists (unique_identifier) after the from statement,

otherwise the solution won't work.

[OmarBenSalem]

It will work; what the where not exists(unique_identifier) does is making sure to update the content.

table:

load * from source WHERE DATE=Today();

//concatenate new data with existing data

concatenate

load * from table.qvd(qvd) where not exists (unique_identifier); // this will import all the new data, and will import from the already stored data only the lines where the unique_identifier wasn't imported in the new data; with that, we do the update.

if we don not use this, we'll have duplicated lines (if the same line exists in the loaded sqvd and in the new data)

//storing the new concatendated data into the same QVD

store table into table.qvd(qvd);