Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Insert, Update, Delete Incremental Load Without Entire History

So basically we get a nightly feed which includes 7 days worth of updated/new data. Along with that we get a file including all deleted id's. My objective is to do an incremental load, insert and update, using my initial QVD (made form a "catch up" file) and the 7 day file we get nightly. I then need to use the deleted ID file to remove any records from the table I created in the incremental load. Since we don't get a full export I can't utilize the inner load of all IDs most people use.

What would be the best way to go about this?

1 Reply
Gysbert_Wassenaar

Maybe like this:

// load deleted ID's in a temporary table

T1:

LOAD ID FROM deleted_ids_file (txt, ...etc...) ;

// load only the records from the incremental_data qvd that

// are not in the deleted id's file in a temporary table

T2:

LOAD * FROM incremental_data.qvd (qvd)

WHERE NOT EXISTS(ID);

// store the records from the remaining ID's

STORE T2 into incremental_data.qvd (qvd);

// drop tables that are no longer needed

DROP TABLES T1, T2;

// load the updated and new records from the nightly 7 day file

Final:

LOAD * FROM sevendaysupdates.qvd (qvd);

// load only the records from the incremental_data qvd that are

// not already in the nightly 7 day file

concatenate(Final)

LOAD * FROM incremental_data.qvd (qvd)

WHERE NOT EXISTS(ID);

// store the complete set of records in the incremental qvd

STORE T2 into incremental_data.qvd (qvd);


talk is cheap, supply exceeds demand