Hi,
I am trying to use incremental load so that tables load faster and are less taxing on our server. The code seems straight forward enough. However, when I run it, no new rows are loaded, even though the DATE_UPDATED field is clearly different between the two sources.
My only thought is that the TO_CHAR function is not working or the date format is incorrect and the variable (vC3Data) is being ignored in the where clause.
Any thoughts?
| Script |
|---|
LastUpdate: load max(DATE_UPDATED) as Last_Update_Date FROM [$(LoadPath)C3Data_old.QVD] (QVD); let vC3Data = date(peek('Last_Update_Date',0,'LoadTime'),'YYYY-MM-DD HH:MM:SS'); drop table LastUpdate; FromSource: load max(DATE_UPDATED) as Last_Update_Date; SQL SELECT DATE_UPDATED FROM "CARISMA3_TRACKWISE".PR; let vPR = date(peek('Last_Update_Date',0,'LoadTime'),'YYYY-MM-DD HH:MM:SS'); drop table FromSource; C3Data: LOAD ID, DATE_UPDATED, 'No' as New; SQL SELECT * FROM "CARISMA3_TRACKWISE".PR where (PROJECT_ID = 21 or PROJECT_ID = 22 or PROJECT_ID = 34 or PROJECT_ID = 35 or PROJECT_ID = 24 or PROJECT_ID = 37 or PROJECT_ID = 23 or PROJECT_ID = 36 or PROJECT_ID = 25 or PROJECT_ID = 38) and TO_CHAR(DATE_UPDATED,'YYYY-MM-DD hh:mm:ss') > '$(vC3Data)'; concatenate LOAD ID, "DATE_UPDATED", 'No' as New FROM [$(LoadPath)C3Data_Old.QVD] (QVD) where NOT exists(ID); STORE C3Data into $(StorePath)C3Data_Old.QVD; |
