Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi Everyone.
I have a SOURCE qvd with following type of records.
ID Name Amount update_date
123 james $300 Feb-01-2014
127 John $234 Feb-01-2014
547 Alex $555 Feb-02-2014
New file from sales comes in daily basis with new and corrected records. Therefore I want to update any corrected records and append new records in above qvd.
for example today's sales file contains old & corrected records with update date.
ID Name Amount update_date
123 james $300 Feb-01-2014 // gets dropped as new entry from same ID but new date exist
123 james $500 Feb-03-2014
127 John $234 Feb-01-2014
547 Alex $555 Feb-02-2014
432 Judy $800 Feb-10-2014 // gets added as new entry.
My source should get updated with all the new records + all the updated records with latest update date.
Final results.
ID Name Amount update_date
123 james $500 Feb-03-2014
127 John $234 Feb-01-2014
547 Alex $555 Feb-02-2014
432 Judy $800 Feb-10-2014
thanks
Either try to load your daily data sorted by ID and Date desc, then use the where not exists (ID) clause,
or load your daily data:
DAILY:
LOAD * INLINE [
ID, Name, Amount, update_date
123, james, $300 , Feb-01-2014
123, james, $500 , Feb-03-2014
127, John , $234 , Feb-01-2014
547, Alex , $555 , Feb-02-2014
432, Judy , $800 , Feb-10-2014
];
and then
LOAD *, ID as TmpID Resident DAILY where not exists(TmpID, ID) order by ID, update_date desc ;
drop table DAILY;
drop field TmpID;
Ufff
I'm not sure but you can try using an applymap with a key (ID&'_'&update-date) with the old information and put a flag to remove this rows:
Map_Remove:
Mapping
Load
ID & '_' & update-date as Remove,
0 as flagtoremove
from oldinformation;
Sales:
Load
ID,
Name,
Amount,
update_date,
applymap('Map_remove', ID & '_' & update-date,1) as flag
from newinformation
where applymap('Map_remove', ID & '_' & update-date,1) = 0;
Above keep only actual information delete old information in second source.
Best regards.
Hi Swuehl
I will try this and let you know, however can you please explain this code.
LOAD *, ID as TmpID Resident DAILY where not exists(TmpID, ID) order by ID, update_date desc ;
how does it work, whats the logic.
thanks
HI Swuehl
so you are saying if I sort the data by ID and Date desc and then use not exist it will only pick the first instance of record and that way get the latest update.
second question how do you sort by ID and Date desc
Can you please explain what this code does.
where Peek(ID) <> ID
order by ID, update_date desc;
I have yet to try your solution. But i will give it a try.
where not exists(ID,ID); // what does second parameter do.
Final:
NoConcatenate load *
Resident Source // 1*
where Peek(ID) <> ID // 2*
order by ID, update_date desc; // 3*
1* read from a previous loaded table (resident=already in memory)
2* and 3* :
order by: I want to read the record sorted by ID and, for same ID, update_desc descending
where Peek(ID) <> ID filter: discards rows if the ID is the same of previus read record
example
ID update_date Peek(ID)
1 31-12-2010 keep
1 10-10-2009 1
1 1-1-2007 1
2 1-1-2010 1 keep
2 1-1-2009 2
2 1-1-2008 2
3 12-10-2014 2 keep
4 10-10-2007 3 keep
result: the 4 rows in bold
from online help
peek(fieldname [ , row [ , tablename ] ] )
Returns the contents of the fieldname in the record specified by row in the internal table tablename. Data are fetched from the associative QlikView database.
Fieldname must be given as a string (e.g. a quoted literal).
Row must be an integer. 0 denotes the first record, 1 the second and so on. Negative numbers indicate order from the end of the table. -1 denotes the last record read.
If no row is stated, -1 is assumed
Hi Massimo
What if my data has 2 dates and the uniq key is made of 3 fields example:
ID&'|'&Name&'|'CreatedDate as UniqID.
Where CreatedDate is used only to find uniq combination but update date will be used to determine what records gets purged or appended.
How do you handle that scenerio?
Source:LOAD * INLINE ; ConcatenateLOAD * INLINE ; Final:NoConcatenate load*Resident Sourcewhere Peek(ID) <> IDorder by ID, update_date desc; drop table Source;
RESULT
IDNameAmountupdate_date123james500feb-03-2014127John234feb-01-2014432Judy800feb-10-2014547Alex555feb-02-2014
I will mark 2 as correct answer, however I will open up new thread as my requirement is now different then what I posted here.
thanks everyone for help.