Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Question: How to do incremental load of QVDs

I have a DB which has many tables in it. Data is inserted, updated and deleted on a daily basis. I have few QVDs setup that pulls data from these tables twice daily. Then there are many QVWs that reads data from these QVDs. My problem is that data pull from DB to QVD (complete load) is taking a long time (about 2 hours now) and its increasing with each day. Is there a way to pull only those records from DB that are inserted, modified and deleted and update the DB incrementally?

Thanks

1 Solution

Accepted Solutions
dirk_konings
Creator III
Creator III

This could help :

View solution in original post

4 Replies
Not applicable
Author

Correction: I meant "update the QVD (not DB) incrementally" in the last line.

Also, every single record in my DB has record creation and last modified time stamp.

dirk_konings
Creator III
Creator III

This could help :

johnw
Champion III
Champion III

Nice presentation. For deletes, I haven't found the inner join of existing primary keys back to the table to be particularly slow. Since the primary key is typically sitting on an index, a database management system typically only needs to read that index, which can be MUCH faster than reading the table data itself. It is also reading the ENTIRE index, which a smart DBMS should be able to do in a highly-optimized way.

Not applicable
Author

Thanks Dirk. I will give this a try and feel that it's going to solve my problem.