SQL partial reload by new date

[Anonymous]

Hi all,

I have searched for something similar but mostly people tend to do what I want with excel or qlikview table files. I'm looking to do a partial load using SQL.

Imagine a table is loaded from a database using SQL:

TestTable:

LOAD

dataField1,

dataField2,

dateField;

SQL SELECT

dataField1,

dataField2,

dateField

FROM testdata;

Now let's say data is added to that table later, obviously with a more recent date. All I want to do is load the data from that more recent date.

So I start by trying:

ADD

DateLoad:

SQL SELECT max(dateField) as maxdate from testdata;

LET vMaxdate = peek('maxdate')

The goal then being to use the variable above in the WHERE clause of the update table. BUT, how would I get that table concatenated with the one above?

[YoussefBelloum]

Hi,

you are trying to do an INCREMENTAL LOAD,

basically, you have 2 tables, the first one is the archived table and the second one is the "recent" table, you will need to CONCATENATE them.

read this https://www.quickintelligence.co.uk/qlikview-incremental-load/

[isingh30]

Did you try incremental load?

Check this - https://www.analyticsvidhya.com/blog/2014/09/qlikview-incremental-load/

Thanks!

[Anonymous]

While the idea is good, this qlik is going to be reloaded on a server which is out of my control. Whether or not they allow for saving qvds... especially ones with an enormous load - we're talking hundreds of millions of records - is something I'd have to investigate.

There is no way to take the existing maximum date for data that's currently loaded, and use that as a variable for the next partial load? fieldDate > 'lastMaxdate' etc.

[YoussefBelloum]

Yes it is possible to pick max date and min date from whatever table you want, but the problem is that you need to reload all the model.. so one part must be derived from an "archived" source, which is the the QVD format (because it is very fast in reading)