How to incremental load new transaction data into QVF

[lolalichang]

I had  quote and trade data updated on transaction level which reload and stored as QVD from oracle database.  I also archive the quote data and trade data weekly as trade_week1_2019.qvd/trade_week2_2019.qvd ...

I created a separate app to reload all the data from 2018-01-01 to latest for visualization.  The transaction volume is around 20 million and QVF size is around 1.2GB.  It takes 25 mins to finish the load script. So I am thinking if the visualization QVF app can only reload new data but not from 2018-01-01 every time.

 I tried Buffer load  but didn't work.  Am I using Buffer load wrongly ? 

Quote:
buffer load
quoteId as tradeId,

*

FROM [lib://PROD QVDFiles/Trade_His\quote*.qvd]
(qvd);

Trade:
buffer load
tradeId,

*

FROM [lib://PROD QVDFiles/Trade_His\Trade*.qvd]
(qvd);

[rwunderlich]

It generally doesn't make sense to use Buffer when loading from QVDs. Buffer creates a QVD - in this case one each for Trade and Quote.  It's just extra work as you are already loading from QVDs. 

You'll want to take a look at your QVD loads and make the, optimized loads.  When you add a field like:

*, quoteId as tradeId,

You are un-optimizing the load.  Ideally you would just "Load *". 

In your Trade load you show "Load tradeId, *" which I believe would result in a script error. 

You could possible speed things up by binary loading a qvf with history and then only loading the new QVDs. Binary load is not supported in QS Server, so it would be best to export the historical app to a qvf and binary load from there.

I would start by trying to optimize the QVD loads and then move to binary if that was not fast enough or I couldn't optimize. 

-Rob