How to incremental load new transaction data into QVF
I had quote and trade data updated on transaction level which reload and stored as QVD from oracle database. I also archive the quote data and trade data weekly as trade_week1_2019.qvd/trade_week2_2019.qvd ...
I created a separate app to reload all the data from 2018-01-01 to latest for visualization. The transaction volume is around 20 million and QVF size is around 1.2GB. It takes 25 mins to finish the load script. So I am thinking if the visualization QVF app can only reload new data but not from 2018-01-01 every time.
I tried Buffer load but didn't work. Am I using Buffer load wrongly ?
Quote: buffer load quoteId as tradeId,
FROM [lib://PROD QVDFiles/Trade_His\quote*.qvd] (qvd);
Trade: buffer load tradeId,
FROM [lib://PROD QVDFiles/Trade_His\Trade*.qvd] (qvd);
Re: How to incremental load new transaction data into QVF
It generally doesn't make sense to use Buffer when loading from QVDs. Buffer creates a QVD - in this case one each for Trade and Quote. It's just extra work as you are already loading from QVDs.
You'll want to take a look at your QVD loads and make the, optimized loads. When you add a field like:
*, quoteId as tradeId,
You are un-optimizing the load. Ideally you would just "Load *".
In your Trade load you show "Load tradeId, *" which I believe would result in a script error.
You could possible speed things up by binary loading a qvf with history and then only loading the new QVDs. Binary load is not supported in QS Server, so it would be best to export the historical app to a qvf and binary load from there.
I would start by trying to optimize the QVD loads and then move to binary if that was not fast enough or I couldn't optimize.