topic Re: Parquet vs QVD: A Performance and Size Comparison for Qlik Developers in App Development

Parquet vs QVD: A Performance and Size Comparison for Qlik Developers

marksouzacosta — Mon, 17 Feb 2025 15:40:30 GMT

QVD files have long been the gold standard for Qlik developers, but is Parquet the future?
In this deep dive, we compare file size, load performance, and efficiency.
Discover how Parquet challenges QVD’s dominance and why Qlik’s recent Upsolver acquisition signals a shift toward open data formats. Read more at:
https://www.datavoyagers.net/post/parquet-vs-qvd-a-performance-and-size-comparison-for-qlik-developers

Regards,

Mark Costa

Re: Parquet vs QVD: A Performance and Size Comparison for Qlik Developers

theoat — Wed, 19 Feb 2025 13:28:44 GMT

Hello !

Very interesting! Thank you so much for the tests you conducted! I have a quick question, though: how are your percentages calculated? For example, with the 'Region' table, I find that the Parquet (Brotli) format is about 61.46% smaller than the QVD format (using QVD as a reference). Also, I think it’s more meaningful to say that Parquet is roughly 2.5 times smaller than the QVD format. The work you’ve done is excellent, and I will share it with my teams!

Enjoy your Qlik.

Kind regards,
Théo ATRAGIE.

Re: Parquet vs QVD: A Performance and Size Comparison for Qlik Developers

MindaugasBacius — Wed, 19 Feb 2025 14:07:11 GMT

Does your summary apply to the on-prem solution?

Re: Parquet vs QVD: A Performance and Size Comparison for Qlik Developers

marksouzacosta — Wed, 19 Feb 2025 14:32:54 GMT

No, only for Cloud solutions.

Regards,

Mark Costa

Re: Parquet vs QVD: A Performance and Size Comparison for Qlik Developers

marksouzacosta — Wed, 19 Feb 2025 14:35:23 GMT

Great suggestion @theoat,

I'm working on a v2 of these tests, and I'll include your idea.
Also, I'll probably make it public one day on Git so people can see what I did to do the tests and make their own versions based on it.

Regards,

Mark Costa

Re: Parquet vs QVD: A Performance and Size Comparison for Qlik Developers

pljsoftware — Mon, 24 Feb 2025 08:44:47 GMT

Hi Mark,

I read your article very carefully and I think it's fantastic. In the various communities, there was a lack of an article that analyzed the QVD and Parquet formats in detail.
I would like to see the code you wrote for making the comparisons so I can run some tests on my own, particularly loading from QVD with the Where Exists clause.
Have you done this type of test? What results did you get?
Could you share the code?

Thanks for sharing your experience.

Regards
Luca Jonathan Panetta

Re: Parquet vs QVD: A Performance and Size Comparison for Qlik Developers

marksouzacosta — Mon, 24 Feb 2025 13:53:05 GMT

Hi @pljsoftware, thank you!

Sure, I'll eventually share it. I'm working on a v2 to make the tests more flexible, and I'll make it available in GitHub - I just need more one or two weeks.

Regarding the where exists, I did some tests and Parquet files are still faster! I did not want to believe it so, I stopped the tests for a while to review this part of the tests.

Regards,

Mark Costa

Re: Parquet vs QVD: A Performance and Size Comparison for Qlik Developers

F_B — Mon, 24 Feb 2025 14:37:37 GMT

Waiting for test V2! Thanks for sharing

Re: Parquet vs QVD: A Performance and Size Comparison for Qlik Developers

parkera — Tue, 11 Mar 2025 20:14:46 GMT

Mark,

I opened an ideation case to improve Parquet compression much more, by adding Run Length Encoding option to the file prior to storage. It is sat in ideation and with backing could be delivered into the product to save 50-90% more space and COST to all of Qlik's customers. Please give it the thumbs up and back this request.

https://ideation.qlik.com/app/#/case/460857

Re: Parquet vs QVD: A Performance and Size Comparison for Qlik Developers

parkera — Tue, 11 Mar 2025 20:16:45 GMT

Hi,

We did a test today on a customer log file and Parquet was 100X smaller than QVD.

Re: Parquet vs QVD: A Performance and Size Comparison for Qlik Developers

parkera — Thu, 20 Mar 2025 19:41:46 GMT

Slight segway: If anyone wants a an embedded Qlik DuckDB extension to read Parquet files natively in Qlik Cloud you can get it here: https://differentia.app.box.com/s/374ohip3psd755dq37m2d1egd7sicya6

The extension lets you map a field to your Qlik App and use that to drive the filter on the Parquet file. The file can be refreshed anytime, eg if streamed to, so that you can look at the most recent Parquet file transactions in Qlik. Putting the 'I' in 'BI'.

Use cases might include IoT, Retail basket analysis etc. Great if you have very large apps in QlikView and want to port to Qlik Cloud. It gives you another way to do this. Best of all it lets you store the data files cheaply in S3.

Note: Extension if free to use, and thus not supported. All software is open source. (WASM, Arrow, DuckDB)

Do share how you use it.

Re: Parquet vs QVD: A Performance and Size Comparison for Qlik Developers

marksouzacosta — Thu, 20 Mar 2025 15:17:04 GMT

This is quite interesting @parkera. I'm hearing a lot about DuckDB these days...
By the way, the link you sent is pointing to a login page.

Regards,

Mark Costa

Re: Parquet vs QVD: A Performance and Size Comparison for Qlik Developers

parkera — Thu, 20 Mar 2025 19:42:17 GMT

Hi Mark,

Linked updated: https://differentia.app.box.com/s/374ohip3psd755dq37m2d1egd7sicya6 - should work now