I'm not sure that it makes sense to do that. The load script would not be pretty... or maintainable.
QVD loading is pretty optimized. It takes about a minute to load 200 million records, in my experience.
One of my client databases has in excess of 250 million rows in a single table; i just reload it during off-peak hours. The data is slightly out of date as a result, but it suits the purpose for this client.
If that isn't enough, you may need to look into doing a real-time data integration. I've seen a demonstration of the capabilities, but not really looked under the hood.
I don't think that kludging a solution using complex load scripting is the wisest approach. I think you might need to take a step back from the solution you've come up with, and try another approach to the problem.
(what is the problem, by the way? time to reload?)
Thank you! Ryan.
You are right, the real issue is - time to reload. I am fairly new to QlikView, I was thinking about some other options :
1. When ever there is update to certain slice of data, I still load entire data into another QVW and then rename the it to production QVW file, don't know if it is possible while users are connected to the QVW ?
2. Split the files into smaller files and load them up in parallel.
I will also look into 'real-time data integration'.