Incremental load loading duplicate data for latest data on second run using '=' in where condition
Hi Everyone,
I get data for 10 day (1,2,3....10) everyday, I want to store my current day data every day in qvd to use for my project. My requirement is to load '='
I am creating incremental load using below codes it is working fine for first time every day but when i execute it second time in same day it is duplicating rows for latest data. When i put '<' or '>' in where condition of latest data file it is working fine for second run on same day. But when i try to put '=' sign it is loading latest data again after first run.
//*** Loading QVD to pick maximum Date*** Temp_1: LOAD Date Name, Entry, Execution, FROM [lib://folder/History.QVD] (qvd);
Let vNextD = date(date#(Today())+1,'MM/DD/YYYY')
Let vToday = date(date#(Today()),'MM/DD/YYYY')
NoConcatenate
// Selecting Maximum Date from QVD Data
Temp_2:
Load Max(Date) as MaxDate Resident Temp_1;
let vMaxDate = peek('MaxDate',0,'Temp_2'); Drop Temp_2;
NoConcatenate
IncrementalLoad: LOAD Date Name, Entry, Execution FROM [lib://Other/Daily.txt]
where Date = $(vvToday );
// Also used tried below but not working as it is loading duplicate data on second run on same day //where Date >$(vMaxDate) and Date < S(vNextDay);
concatenate
LOAD * resident Temp_1;
// ****Tried below also but not working *****
//where exists (Date);
//where not exists(Date);
Drop table Temp_1;
Store IncrementalLoad into [lib://folder/History.QVD] (qvd);
Drop table IncrementalLoad;
Please help if anyone have any solution or alternate for this.
I would suggest follow the standard approach explained in Qlik help -
You would need to keep track of max date in your variable and use that in your where clause instead of today(), after final update variable gets updated to max date every time so that duplicated records would be excluded in multiple runs on same day.
