data loading is slow

[#1]

Team

  I am having millions of records in the table , I tried to load and save the data in the qlikview, it is taking 6 to 7 hours to load and save the data in qlikview. any suggestions how to minimize the data load time.

Thanks

Srini

[trdandamudi]

It will be great if you share a sample of your app. So that we can take a look and see if we can help. With out seeing the code it is very difficult to suggest.

[#3]

Tirumala

   I am just using the loading script like

load * from tablename where

date = (giving five years of data)

,  Each country is having more than millions of data, I am working with the insurance hence the data are huge.

it is taking for one country 15 minutes, we have more than 25 countries. client fields these data loading takes longer...

[engishfaque]

Dear Srini,

I would suggest you to load your all the data first time, after that you must implement incremental load for loading updated or newly created data and rest of the (old data) will be same, if there is no change.

Incremental load will reduce you load time.

Here is the way to implement Incremental load in QlikView:

Incremental Load in Qlikview - Sources

Incremental Load

Kind regards,

Ishfaque Ahmed

[sujeetsingh]

Well using incremental load and buffer load are best options where as you can try for the optimizations of logic in the script too.

[trdandamudi]

It looks like very simple load script. You haven't mentioned about your system configuration...You can try one of the below and see if it helps:

1) Pull the data by year for all the countries into Qlikview and then save it as a QVD. (You will get 5 QVD;s). Then you can load all the QVD's into Qlikview.

2) Split the countries into groups and then run by year into Qlikview and then save it as QVD, Then you can load all the QVD's into Qlikview.

3) See if you can use "Incremental Load".

Hope someone will respond with their view....

[Anonymous]

Hi

Agree with the guys regarding using incremental loads and qvd's to reduce loading times but also it may be worth looking at any redundant fields you are loading but not using in your application. If possible remove any unused fields from your load script and hopefully together with qvd and incremental loads this should improve your load times.

Kindest Regards

Brian

[stabben23]

If you load from a sql datawarehouse, make sure Your index is correct.

[HirisH_V7]

Hi,

What kind of database is it?

• Daily new records which are not changed or updated once created.

or

• Records Periodically Keep on changing even the initially created first year records are changing based on any criteria.

For example : Customer Tickets database may close the ticket (Record) which is created in the initial year at any point of time once the service is done.

-Hirish