topic Re: How to incremental load new transaction data into QVF in App Development

How to incremental load new transaction data into QVF

lolalichang — Sat, 16 Nov 2024 05:34:37 GMT

I had quote and trade data updated on transaction level which reload and stored as QVD from oracle database. I also archive the quote data and trade data weekly as trade_week1_2019.qvd/trade_week2_2019.qvd ...

I created a separate app to reload all the data from 2018-01-01 to latest for visualization. The transaction volume is around 20 million and QVF size is around 1.2GB. It takes 25 mins to finish the load script. So I am thinking if the visualization QVF app can only reload new data but not from 2018-01-01 every time.

I tried Buffer load but didn't work. Am I using Buffer load wrongly ?

Quote:
buffer load
quoteId as tradeId,

FROM [lib://PROD QVDFiles/Trade_His\quote*.qvd]
(qvd);

Trade:
buffer load
tradeId,

FROM [lib://PROD QVDFiles/Trade_His\Trade*.qvd]
(qvd);

Re: How to incremental load new transaction data into QVF

rwunderlich — Mon, 17 Jun 2019 21:05:33 GMT

It generally doesn't make sense to use Buffer when loading from QVDs. Buffer creates a QVD - in this case one each for Trade and Quote. It's just extra work as you are already loading from QVDs.

You'll want to take a look at your QVD loads and make the, optimized loads. When you add a field like:

*, quoteId as tradeId,

You are un-optimizing the load. Ideally you would just "Load *".

In your Trade load you show "Load tradeId, *" which I believe would result in a script error.

You could possible speed things up by binary loading a qvf with history and then only loading the new QVDs. Binary load is not supported in QS Server, so it would be best to export the historical app to a qvf and binary load from there.

I would start by trying to optimize the QVD loads and then move to binary if that was not fast enough or I couldn't optimize.

-Rob