Qlik Community

New to QlikView

Discussion board where members can get started with QlikView.

Not applicable

Apply Incremental reload to extract 1.5TB of data in few minutes..

Hi!

I want to extract approx. 1.5TB of data from database. How can i apply incremental reload to extract such a massive volume of data in few minutes? Also how i identify the change data in the database without using a date condition ? Does SQL server change data capture works here ?

Please help on this..

Thanks in advance!

3 Replies
MVP
MVP

Re: Apply Incremental reload to extract 1.5TB of data in few minutes..

You need to have some primary key or date to use incremental load....

Re: Apply Incremental reload to extract 1.5TB of data in few minutes..

First time you have to do Full Data load and make a QVD then find the

1. Primary key in the data base.

2. If primary key is not available then use Composite key combination.

3. Find any updation date and compare it with last Reload time of the application.

On the basis of this points you can identify the scenario for the incremental load. I Suggesting you to find the best way is Incremental load like on the above points find the way to do only one day before updated records that is mean not load all rows daily it decreases your application reload time and save time also.

MVP
MVP

Re: Apply Incremental reload to extract 1.5TB of data in few minutes..

there is no a single answer, it depends on

- dml on your db table (insert and/or update and/or delete, delete is worst case)

- field (usually date, primary key) to detect change in your table

to better understand the different possibility in the online help look for

Using qvd file for incremental load