Load with QVD store Qs

[#1]

Below QlikView datamodel has tables tblPnl which contains around 200 mil rows, tblPosition around 1 mil rows and tblHrchy around 2500.

tblPnl		tblPosition		tblHrchy
POS_ID		POS_ID		Book
CCY		CUSIP		[Business Line]
SCEN_DATE		CUST_ID		Desk
TOTAL		Book		Portfolio

I want to create one QVD file for each portfolio, so that in case I have any updates to tblPosition/tblPnl only specific portfolio QVD file can be reloaded without entire reload.

I would really appreciate any pointer/views/ideas on how to go about doing this.

[#2]

I'm not sure that it makes sense to do that. The load script would not be pretty... or maintainable.

QVD loading is pretty optimized. It takes about a minute to load 200 million records, in my experience.

One of my client databases has in excess of 250 million rows in a single table; i just reload it during off-peak hours. The data is slightly out of date as a result, but it suits the purpose for this client.

If that isn't enough, you may need to look into doing a real-time data integration. I've seen a demonstration of the capabilities, but not really looked under the hood.

I don't think that kludging a solution using complex load scripting is the wisest approach. I think you might need to take a step back from the solution you've come up with, and try another approach to the problem.

(what is the problem, by the way? time to reload?)

[#3]

Thank you! Ryan.

You are right, the real issue is - time to reload. I am fairly new to QlikView, I was thinking about some other options :

1. When ever there is update to certain slice of data, I still load entire data into another QVW and then rename the it to production QVW file, don't know if it is possible while users are connected to the QVW ?

2. Split the files into smaller files and load them up in parallel.

I will also look into 'real-time data integration'.

Thanks again.