Qlik Community

Qlik Sense App Development

Discussion board where members can learn more about Qlik Sense App Development and Usage.

lolalichang
New Contributor

How to incremental load new transaction data into QVF

I had  quote and trade data updated on transaction level which reload and stored as QVD from oracle database.  I also archive the quote data and trade data weekly as trade_week1_2019.qvd/trade_week2_2019.qvd ...

I created a separate app to reload all the data from 2018-01-01 to latest for visualization.  The transaction volume is around 20 million and QVF size is around 1.2GB.  It takes 25 mins to finish the load script. So I am thinking if the visualization QVF app can only reload new data but not from 2018-01-01 every time.

 I tried Buffer load  but didn't work.  Am I using Buffer load wrongly ? 

 

Quote:
buffer load
quoteId as tradeId,

*

FROM [lib://PROD QVDFiles/Trade_His\quote*.qvd]
(qvd);

 

Trade:
buffer load
tradeId,

*

FROM [lib://PROD QVDFiles/Trade_His\Trade*.qvd]
(qvd);

 

 

Labels (3)
1 Reply
MVP & Luminary
MVP & Luminary

Re: How to incremental load new transaction data into QVF

It generally doesn't make sense to use Buffer when loading from QVDs. Buffer creates a QVD - in this case one each for Trade and Quote.  It's just extra work as you are already loading from QVDs. 

You'll want to take a look at your QVD loads and make the, optimized loads.  When you add a field like:

*, quoteId as tradeId,

You are un-optimizing the load.  Ideally you would just "Load *". 

In your Trade load you show "Load tradeId, *" which I believe would result in a script error. 

You could possible speed things up by binary loading a qvf with history and then only loading the new QVDs. Binary load is not supported in QS Server, so it would be best to export the historical app to a qvf and binary load from there.

I would start by trying to optimize the QVD loads and then move to binary if that was not fast enough or I couldn't optimize. 

-Rob