Reloading large data

[#1]

Hello everybody,

I have just created a Qlikview report and deployed it on the QV Server. I'm able to access the report over a browser.

1) 4 of the 12 tables are very large with about 1.5 million rows (3-4 columns). The qvw file size is 51 MB

2) The initial data load took me about an hour.

I'd like to know what is the optimal method of reloading this data. Considering the time it took, I think I can reload only twice or thrice a day. Even that might affect the performance of the production OLTP DB server.

I'd only be building more reports and that is going to add more tables over a period of time. I'm relaly worried about the reload.

We are in the business of managing millions of devices and the activity data is captured on these devices/customers every day and that data definitely needs to be captured into the BI system for trending purposes.

Does qlikview have an option to track only incremental / changed data ?

Kinldy thow some light on How I should go about designing this BI system.

Regards,

Anton

[Clever_Anjos]

Yes, read the manual under the section Incremental Reload, P.540 in my case

[#3]

Hi Thanks for the response. What is "My Case" document. Where can I access it.

[Clever_Anjos]

Here:

http://d1cf4w4kkla6tb.cloudfront.net/documentation/10.00/latest/Qv/English/QlikView%20Reference%20Ma...

[johnw]

Expanding a bit on this, you will probably want to create separate QVWs to handle the incremental reloads, one for each table that you're incremental loading.  Those should create QVDs with the data they've loaded (necessary for the incremental reload in any case).  Then your real application would read that data from the QVDs instead of from the databases themselves.

Loading quickly from large QVDs can also take a little extra effort.  If your load is simple enough (no field manipulation other than optional renames, no where conditions except a single optional exists()), the QVD load will go MUCH faster.  It can sometimes be worth setting up your condition(s) as an exists(), doing some conditions with an inner join after the load, field manipulation after the load and so on.  But before we get into all of that, it's quite possible that simply implementing some incremental loads will solve the problem to your satisfaction, so definitely start there.