3 Replies Latest reply: Oct 8, 2014 3:57 PM by Massimo Grossi RSS

    Incremental Load not working

    Drew Collins

      Hi,

       

      I am trying to use incremental load so that tables load faster and are less taxing on our server. The code seems straight forward enough. However, when I run it, no new rows are loaded, even though the DATE_UPDATED field is clearly different between the two sources.

       

      My only thought is that the TO_CHAR function is not working or the date format is incorrect and the variable (vC3Data) is being ignored in the where clause.

       

      Any thoughts?

       

      Script

      LastUpdate:

      load max(DATE_UPDATED) as Last_Update_Date

      FROM [$(LoadPath)C3Data_old.QVD] (QVD);

       

      let vC3Data = date(peek('Last_Update_Date',0,'LoadTime'),'YYYY-MM-DD HH:MM:SS');

         

      drop table LastUpdate;   

       

      FromSource:

      load max(DATE_UPDATED) as Last_Update_Date;

      SQL SELECT DATE_UPDATED

      FROM "CARISMA3_TRACKWISE".PR;

       

      let vPR = date(peek('Last_Update_Date',0,'LoadTime'),'YYYY-MM-DD HH:MM:SS');

         

      drop table FromSource;   

       

      C3Data:

      LOAD

      ID,

      DATE_UPDATED,

      'No' as New;

      SQL SELECT  *

      FROM "CARISMA3_TRACKWISE".PR where (PROJECT_ID = 21 or PROJECT_ID = 22 or PROJECT_ID = 34 or PROJECT_ID = 35 or PROJECT_ID = 24 or PROJECT_ID = 37 or

      PROJECT_ID = 23 or PROJECT_ID = 36 or PROJECT_ID = 25 or PROJECT_ID = 38) and TO_CHAR(DATE_UPDATED,'YYYY-MM-DD hh:mm:ss') > '$(vC3Data)';

       

       

      concatenate LOAD

          ID,

          "DATE_UPDATED",

          'No' as New

          FROM [$(LoadPath)C3Data_Old.QVD] (QVD)

          where NOT exists(ID);

       

      STORE C3Data into $(StorePath)C3Data_Old.QVD;