I would like to use Server/Publisher to automatically produce a qvw document containing a subset of the total data.
Each qvw created, one for each customer, would contain only the data for that customer but I would like to do this without using loop and reduce.
We have a system where many customers feed large amounts of data into one database.
We need to automatically create separate copies of the same Qlikview qvs document, one for each customer which only conatins the data for that customer.
Using loop and reduce is not an option because loading a single qvw containing all of the data would use too many resources.
Currently the server uses about 16GB of memory for about 15 or 20 minutes when loading the data for one customer and at different installations we have around 10, 20 or more customers each requiring their own copy of the qvw.
We cannot keep throwing more memory at the server to cope with all the customers. Setting up a server with over 240GB of memory so that we can do a loop and reduce is an expensive solution. Doing the load one customer at a time on a server with around 24GB of memory is a much more cost effective option.
Consequently I have two problems to solve.
1. Creating the qvd's without having all the huge amounts of data in memory at once.
2. Automatically creating the qvw documents for each customer without having to create a qvw document containing all the data and then doing a reduce.
Ideally I would like to;
Perform incremental loads a few times a day where I select just the changes from the database and then apply these changes to the existing qvd without having to load the whole existing qvd into memory first.
When the qvd's have been updated then loop through the values in the customer table and automatically create a copy of the master qvw, one for each customer.
On each copy of the qvw perform a reload having passed in a variable containing the customer id so that the load can know to only load the data for that customer from the qvd's.
Is this possible? or do I have to manually create a new qvw for each customer and schedule each one to load in a specific order?