topic Load with QVD store Qs in QlikView

Load with QVD store Qs

Mon, 28 Jun 2010 22:58:53 GMT

Below QlikView datamodel has tables tblPnl which contains around 200 mil rows, tblPosition around 1 mil rows and tblHrchy around 2500.

tblPnl	tblPosition	tblHrchy
POS_ID	POS_ID	Book
CCY	CUSIP	[Business Line]
SCEN_DATE	CUST_ID	Desk
TOTAL	Book	Portfolio

I want to create one QVD file for each portfolio, so that in case I have any updates to tblPosition/tblPnl only specific portfolio QVD file can be reloaded without entire reload.

I would really appreciate any pointer/views/ideas on how to go about doing this.

Load with QVD store Qs

Tue, 29 Jun 2010 00:44:38 GMT

I'm not sure that it makes sense to do that. The load script would not be pretty... or maintainable.

QVD loading is pretty optimized. It takes about a minute to load 200 million records, in my experience.

One of my client databases has in excess of 250 million rows in a single table; i just reload it during off-peak hours. The data is slightly out of date as a result, but it suits the purpose for this client.

If that isn't enough, you may need to look into doing a real-time data integration. I've seen a demonstration of the capabilities, but not really looked under the hood.

I don't think that kludging a solution using complex load scripting is the wisest approach. I think you might need to take a step back from the solution you've come up with, and try another approach to the problem.

(what is the problem, by the way? time to reload?)

Load with QVD store Qs

Thu, 01 Jul 2010 17:48:56 GMT

Thank you! Ryan.

You are right, the real issue is - time to reload. I am fairly new to QlikView, I was thinking about some other options :

1. When ever there is update to certain slice of data, I still load entire data into another QVW and then rename the it to production QVW file, don't know if it is possible while users are connected to the QVW ?

2. Split the files into smaller files and load them up in parallel.

I will also look into 'real-time data integration'.

Thanks again.