Incremental Load Help

[etrotter]

Hi Community!

I had a quick question involving my script for my incremental load- 

I first load my current QVD: 

QVD:
LOAD
id as key,
id as filter_id
FROM [lib://QVDS\Layer 1\Entry.qvd]
(qvd);

Then I load the new script, and I want it to load only the new ids from each page.

Note: the loop is working but it seems to be ignoring the "Where Not Exists" and when I run this all the data gets pulled from the source: 

[$(vTable)]:
LOAD * INLINE [ID];

For vPageNumber = 1 to (vTotalPages)

CUSTOM CONNECT TO $(vConnectionString1)$(vObjectID)$(vConnectionString2)$(vPageNumber)$(vConnectionString3);

Concatenate ([$(vTable)])
SQL Select
"__KEY_root",

(SELECT
"id",
"id" as "key",
"field_462",
"field_462_raw",
"field_447",
"field_447_raw",
"field_464",
"field_464_raw",
"field_465",
"field_448",
"field_449",
"field_466",
"field_580",
"field_580_raw",
"field_591",
"__FK_records",
"__KEY_records"
FROM "records" PK "__KEY_records" FK "__FK_records")
FROM JSON (wrap on) "root" PK "__KEY_root"

Where not exists (filter_id,id);

Next vPageNumber;

Please let me know if you have any ideas, I would really appreciate the help!

Thanks!!!  

[vishsaggi]

Not sure if this works may be try like near where not exists

WHERE not exists(filter_id, "id")

[etrotter]

Thanks for relpying, but it is still loading all of the values.

[mikaelsc]

can't use an exists statement in an sql query... only in load statements. use a preceding load.

[santhiqlik]

Hi,

I would suggest to store the Select Query Data into separate QVD. And use that QVD with where not exits

[vishsaggi]

Oops i missed it. We cannot use Not exists inside sql. As Mika suggested do a full load of your SQL query then use

SQLTable:
YOursqlquery;

NoConcatenate
Final:
LOAD *
Resident SQLTable
WHERE Not exists(filter_id, "id");

Drop Table SQLTable;

Something like this you can try.

[etrotter]

Thanks so much for all your help! 

I'm still running into the issue of the load time. My goal for doing an incremental load was to only load the new values from the source cutting down on the load time of the initial pull rather than pulling all the data in then limiting it. Is this possible to do in SQL?