Optimizing Data Reload Performance

[theoat]

Hello,

I need to reload a table daily that contains 60 million rows and 125 columns. I considered an incremental reload, but it doesn’t seem feasible since this is a snapshot of the previous day's stock levels.

Do you have any recommendations to speed up this reload on Qlik Cloud ? Previously, we performed this operation on QlikView , where the loading times were significantly shorter.

Thank you in advance for your help.

Enjoy your Qlik.

Kind regards,
Théo ATRAGIE.

[rubenmarin]

Hi, do you really need all 125 columns? Retrieveing only the needed columns may reduce load times.

And the 60 millions rows is for the last day or it has historical data?

Also stocks with zero units exists on the table?

[marksouzacosta]

Hi @theoat,

For large Data Sets I use a mix of Incremental Load and Data Partition. Since Incremental Load is not an option you could try Data Partition. The concept is simple. You pick a field (or fields) from your data set that does not change over time and that makes sense to group by. Often, I group my data by Year/Month. My QVDs/Parquets will look like:

...
stock_202412.qvd
stock_202501.qvd
stock_202502.qvd
stock_202503.qvd
...

You can even create a logic where you only process the current month plus the previous X months reducing drastically the extraction times.

Regards,

Mark Costa

[rwunderlich]

To add to what @marksouzacosta suggested, here's an article on segmenting QVDs with some code samples.

https://qlikviewcookbook.com/2022/03/how-to-segment-qvd-files/

-Rob