topic SQL partial reload by new date in QlikView

SQL partial reload by new date

Anonymous — Wed, 25 Nov 2020 16:16:04 GMT

Hi all,

I have searched for something similar but mostly people tend to do what I want with excel or qlikview table files. I'm looking to do a partial load using SQL.

Imagine a table is loaded from a database using SQL:

TestTable:

LOAD

dataField1,

dataField2,

dateField;

SQL SELECT

dataField1,

dataField2,

dateField

FROM testdata;

Now let's say data is added to that table later, obviously with a more recent date. All I want to do is load the data from that more recent date.

So I start by trying:

ADD

DateLoad:

SQL SELECT max(dateField) as maxdate from testdata;

LET vMaxdate = peek('maxdate')

The goal then being to use the variable above in the WHERE clause of the update table. BUT, how would I get that table concatenated with the one above?

Re: SQL partial reload by new date

YoussefBelloum — Wed, 28 Feb 2018 19:39:48 GMT

Hi,

you are trying to do an INCREMENTAL LOAD,

basically, you have 2 tables, the first one is the archived table and the second one is the "recent" table, you will need to CONCATENATE them.

read this https://www.quickintelligence.co.uk/qlikview-incremental-load/

Re: SQL partial reload by new date

isingh30 — Wed, 28 Feb 2018 19:50:54 GMT

Did you try incremental load?

Check this - https://www.analyticsvidhya.com/blog/2014/09/qlikview-incremental-load/

Thanks!

Re: SQL partial reload by new date

Anonymous — Thu, 01 Mar 2018 12:05:06 GMT

While the idea is good, this qlik is going to be reloaded on a server which is out of my control. Whether or not they allow for saving qvds... especially ones with an enormous load - we're talking hundreds of millions of records - is something I'd have to investigate.

There is no way to take the existing maximum date for data that's currently loaded, and use that as a variable for the next partial load? fieldDate > 'lastMaxdate' etc.

Re: SQL partial reload by new date

YoussefBelloum — Thu, 01 Mar 2018 12:45:34 GMT

Yes it is possible to pick max date and min date from whatever table you want, but the problem is that you need to reload all the model.. so one part must be derived from an "archived" source, which is the the QVD format (because it is very fast in reading)