Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Below is my incremental loading script:
Test:
Load
*
where primary key>'xxx';
select * FROM CSV (header on, delimiter ",", quote """") "abc"
WITH CONNECTION (Url "$(vUrl1)");
concatenate load
*
from abc.qvd
where not exists(primary key);
This only works for avoiding duplicate calculating for the whole abc.qvd each now I start this app task.
But if there is any way that I can only load latest qvd into app RAM rather than loading the whole year QVD file data into app RAM again? I means if there is a way that app could keep the previous run qvd data in RAM and only load required latest data onto the old data in app RAM?
Test:
Load
*
from abc_2021.02.02.qvd;//2021.02.02 should be defined as latest date
Below is my code to load abc data: But 'Select' time almost cost much less time than loading abc_total.qvd since abc_total.qvd contains all the historical data that is huge.
Let ThisExecDate = Date(now(),'YYYY.MM.DD');
Trace ThisExecDate :$(ThisExecDate );
Total:
LOAD
*
FROM [lib://PROD QVDFiles/abc_$(ThisExecDate).qvd](qvd)
where abcname<>'123456';
Concatenate LOAD
*
FROM [lib://PROD QVDFiles/abc_Total\abc_Total.qvd](qvd)
where not Exists(abcId);
store Total into [lib://PROD QVDFiles/abc_Total\abc_Total.qvd];
drop table Total,Exclude;