Incremental Load on SQL Joined Tables

[sunny_talwar]

Hi Friends -

I have done incremental loads few times before, but this time I have a slightly different scenario. There are these pre-existing application which runs every single day to populate millions of rows. All these application uses some kind of SQL query which joins multiple tables from the data source.

Now what I am trying to do is to implement an incremental reload to this so that we can reduce the overall reload time but in order to do that I have few options

1. Somehow implement the incremental reload to the complicated query, which I will have to to think about because each of the tables will have there UniqueID and might have a different modified date.
2. Create incremental qvds for each of the tables separately and then perform all the joins in QlikView.

Has anybody worked on this kind of scenario before? Or would anybody be able to give advice here?

Thanks,

Sunny

marcowedel‌, swuehl‌, mrkachhiaimp‌, jagan‌, marcus_sommer‌, swuehl‌, maxgro‌, rwunderlich‌, gwassenaar‌, hic‌

[Anonymous]

Hi Sunny,

I have done this for some customers, and I have done this, because I had a large time window on QlikView server, but not in data base server. My logic is:

On one QVD I store the last dates of each DB query.

One QVD with the stored data.

Reload each query where the dates are greather than QVD date.

Join them on QlikView.

Concatenate with QVD content.

Store the new table on the QVD.

Store max dates on QVD dates.

But it all depends on your architecture and whether access to the database has to be fast or not.Complicated queries causes that the database slow down, but avoid the logic of QlikView. If you have a large time window is better than logic is performed in QlikView, but if not, is better to do over the database server.

Regards!!