Skip to main content
Announcements
Marching toward a simplified navigation! READ ON
cancel
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable

SQL partial reload by new date

Hi all,

I have searched for something similar but mostly people tend to do what I want with excel or qlikview table files. I'm looking to do a partial load using SQL.

Imagine a table is loaded from a database using SQL:

TestTable:

LOAD

dataField1,

dataField2,

dateField;

SQL SELECT

dataField1,

dataField2,

dateField

FROM testdata;

Now let's say data is added to that table later, obviously with a more recent date. All I want to do is load the data from that more recent date.

So I start by trying:

ADD

DateLoad:

SQL SELECT max(dateField) as maxdate from testdata;

LET vMaxdate = peek('maxdate')

The goal then being to use the variable above in the WHERE clause of the update table. BUT, how would I get that table concatenated with the one above?

4 Replies
YoussefBelloum
Champion
Champion

Hi,

you are trying to do an INCREMENTAL LOAD,

basically, you have 2 tables, the first one is the archived table and the second one is the "recent" table, you will need to CONCATENATE them.

read this https://www.quickintelligence.co.uk/qlikview-incremental-load/

isingh30
Specialist
Specialist

Did you try incremental load?

Check this - https://www.analyticsvidhya.com/blog/2014/09/qlikview-incremental-load/

Thanks!

Anonymous
Not applicable
Author

While the idea is good, this qlik is going to be reloaded on a server which is out of my control. Whether or not they allow for saving qvds... especially ones with an enormous load - we're talking hundreds of millions of records - is something I'd have to investigate.

There is no way to take the existing maximum date for data that's currently loaded, and use that as a variable for the next partial load? fieldDate > 'lastMaxdate' etc.

YoussefBelloum
Champion
Champion

Yes it is possible to pick max date and min date from whatever table you want, but the problem is that you need to reload all the model.. so one part must be derived from an "archived" source, which is the the QVD format (because it is very fast in reading)