I am trying to use incremental load so that tables load faster and are less taxing on our server. The code seems straight forward enough. However, when I run it, no new rows are loaded, even though the DATE_UPDATED field is clearly different between the two sources.
My only thought is that the TO_CHAR function is not working or the date format is incorrect and the variable (vC3Data) is being ignored in the where clause.
Any thoughts?
Script
LastUpdate:
load max(DATE_UPDATED) as Last_Update_Date
FROM [$(LoadPath)C3Data_old.QVD] (QVD);
let vC3Data = date(peek('Last_Update_Date',0,'LoadTime'),'YYYY-MM-DD HH:MM:SS');
drop table LastUpdate;
FromSource:
load max(DATE_UPDATED) as Last_Update_Date;
SQL SELECT DATE_UPDATED
FROM "CARISMA3_TRACKWISE".PR;
let vPR = date(peek('Last_Update_Date',0,'LoadTime'),'YYYY-MM-DD HH:MM:SS');
drop table FromSource;
C3Data:
LOAD
ID,
DATE_UPDATED,
'No' as New;
SQL SELECT *
FROM "CARISMA3_TRACKWISE".PR where (PROJECT_ID = 21 or PROJECT_ID = 22 or PROJECT_ID = 34 or PROJECT_ID = 35 or PROJECT_ID = 24 or PROJECT_ID = 37 or
PROJECT_ID = 23 or PROJECT_ID = 36 or PROJECT_ID = 25 or PROJECT_ID = 38) and TO_CHAR(DATE_UPDATED,'YYYY-MM-DD hh:mm:ss') > '$(vC3Data)';
