- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Need help on increamental load without date and timestamp fields
Hello,
I have a few tables, which do not have the date/timestamp field to show when was the row modified. the tables are connected through other unique fields. What are the recommended ways to do incremental loads. If the unique fields are strings, then in my opinion, the incremental load will be very slow if we are searching for every new key in the old records to check whether its a new records or old one.
note: need to perform insert,update ,delete operations must(bcz my data should be update for every 10 minutes)
What are the recommended ways to do incremental load in this scenario
Thanks
Prvn
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Inserts and deletes are no problem if you have a primary key in your table. But if you don't have a field that indicates that a record has been updated then you need to calculated hashes over all the fields of the record except the primary key and then compare all the hashes per primary key to identify the changed records. See this blog post for an example: http://www.qlikfix.com/2014/03/11/hash-functions-collisions/
You should really urge the database administrator of the source database to add a modification datetime field to the table that is filled with a trigger on insert and update.
talk is cheap, supply exceeds demand