Skip to main content
Announcements
See why Qlik was named a Leader in the 2024 Gartner® Magic Quadrant™ for Data Integration Tools for the ninth year in a row: Get the report
cancel
Showing results for 
Search instead for 
Did you mean: 
etrotter
Creator II
Creator II

Incremental Load Help

Hi Community!

 

I had a quick question involving my script for my incremental load- 

I first load my current QVD: 

QVD:
LOAD
id as key,
id as filter_id
FROM [lib://QVDS\Layer 1\Entry.qvd]
(qvd);

Then I load the new script, and I want it to load only the new ids from each page.

Note: the loop is working but it seems to be ignoring the "Where Not Exists" and when I run this all the data gets pulled from the source: 

[$(vTable)]:
LOAD * INLINE [ID];

For vPageNumber = 1 to (vTotalPages)

CUSTOM CONNECT TO $(vConnectionString1)$(vObjectID)$(vConnectionString2)$(vPageNumber)$(vConnectionString3);

Concatenate ([$(vTable)])
SQL Select
"__KEY_root",

(SELECT
"id",
"id" as "key",
"field_462",
"field_462_raw",
"field_447",
"field_447_raw",
"field_464",
"field_464_raw",
"field_465",
"field_448",
"field_449",
"field_466",
"field_580",
"field_580_raw",
"field_591",
"__FK_records",
"__KEY_records"
FROM "records" PK "__KEY_records" FK "__FK_records")
FROM JSON (wrap on) "root" PK "__KEY_root"

Where not exists (filter_id,id);


Next vPageNumber;

Please let me know if you have any ideas, I would really appreciate the help!

Thanks!!!  Smiley Very HappySmiley HappySmiley Very Happy

Labels (1)
6 Replies
vishsaggi
Champion III
Champion III

Not sure if this works may be try like near where not exists

WHERE not exists(filter_id, "id")
etrotter
Creator II
Creator II
Author

Thanks for relpying, but it is still loading all of the values.
mikaelsc
Specialist
Specialist

can't use an exists statement in an sql query... only in load statements. use a preceding load.
santhiqlik
Creator
Creator

Hi,

I would suggest to store the Select Query Data into separate QVD. And use that QVD with where not exits

vishsaggi
Champion III
Champion III

Oops i missed it. We cannot use Not exists inside sql. As Mika suggested do a full load of your SQL query then use

SQLTable:
YOursqlquery;

NoConcatenate
Final:
LOAD *
Resident SQLTable
WHERE Not exists(filter_id, "id");

Drop Table SQLTable;

Something like this you can try.
etrotter
Creator II
Creator II
Author

Thanks so much for all your help! 

I'm still running into the issue of the load time. My goal for doing an incremental load was to only load the new values from the source cutting down on the load time of the initial pull rather than pulling all the data in then limiting it. Is this possible to do in SQL?