I am asked to create a script for incremental load of some tables. The production database gets truncated and created all over again everytime so the creators haven't really noticed if any permanent delete happens. They are aware of updates in the same record though.
Best scenario for me is to go the conservative way and choose the "insert, update, delete" method, right?
Wouldn't hurt to, right? I mean, this takes account of every aspect even if a real deletion never happens.
Correct me if I am wrong please. Maybe there's something I didn't grasp.
Incremental load normally refers to the timestamp of qvd created previously. If the table gets truncated, of course the qvd timestamp will be remaining the same. Yes in this case, the load will miss out the data happened prior to last reload date/time. In order to avoid this, it will be better to query the table:
1. Total amount of the record is not zero.
2. The total amount of record prior to last reload based on the timestamp is more than zero.
3. If above conditions are met, then incremental load can start. Otherwise, the full load should be done.
Hope this helps.
Help users find answers! Don't forget to mark a solution that worked for you! If already marked, give it a thumbs up!
Hi @Lisa_Sun , i'm not sure I understand why you say the timestamp in my qvd will be remaining the same. The qvd up to today will have a timestamp let's say timestamp_1 & with the truncation of the database table and the rebuilding of it, my incremental load will still get every id after this timestamp_1. Meanwhile, to come back to my initial question, any change (update or delete) will be checked with the "insert, update, delete" method ( look here ), if i'm correct.
Also- these conditions you suggest (1 & 2) are about the production table?
Thank you in advance 🙂 !!! Really appreciate the help here
