Parquet vs QVD: A Performance and Size Comparison for Qlik Developers

[marksouzacosta]

QVD files have long been the gold standard for Qlik developers, but is Parquet the future?
In this deep dive, we compare file size, load performance, and efficiency.
Discover how Parquet challenges QVD’s dominance and why Qlik’s recent Upsolver acquisition signals a shift toward open data formats. Read more at: 
https://www.datavoyagers.net/post/parquet-vs-qvd-a-performance-and-size-comparison-for-qlik-develope...

Regards,

Mark Costa

[theoat]

Hello !

Very interesting! Thank you so much for the tests you conducted! I have a quick question, though: how are your percentages calculated? For example, with the 'Region' table, I find that the Parquet (Brotli) format is about 61.46% smaller than the QVD format (using QVD as a reference). Also, I think it’s more meaningful to say that Parquet is roughly 2.5 times smaller than the QVD format. The work you’ve done is excellent, and I will share it with my teams!

Enjoy your Qlik.

Kind regards,
Théo ATRAGIE.

[MindaugasBacius]

Does your summary apply to the on-prem solution? 

[marksouzacosta]

No, only for Cloud solutions. 

Regards,

Mark Costa

[marksouzacosta]

Great suggestion @theoat,

I'm working on a v2 of these tests, and I'll include your idea.
Also, I'll probably make it public one day on Git so people can see what I did to do the tests and make their own versions based on it.

Regards,

Mark Costa

[pljsoftware]

Hi Mark,

I read your article very carefully and I think it's fantastic. In the various communities, there was a lack of an article that analyzed the QVD and Parquet formats in detail.
I would like to see the code you wrote for making the comparisons so I can run some tests on my own, particularly loading from QVD with the Where Exists clause.
Have you done this type of test? What results did you get?
Could you share the code?

Thanks for sharing your experience.

Regards
Luca Jonathan Panetta

[marksouzacosta]

Hi @pljsoftware, thank you!

Sure, I'll eventually share it. I'm working on a v2 to make the tests more flexible, and I'll make it available in GitHub - I just need more one or two weeks.

Regarding the where exists, I did some tests and Parquet files are still faster! I did not want to believe it so, I stopped the tests for a while to review this part of the tests.

Regards,

Mark Costa

[F_B]

Waiting for test V2! Thanks for sharing

[parkera]

Mark,

I opened an ideation case to improve Parquet compression much more, by adding Run Length Encoding option to the file prior to storage. It is sat in ideation and with backing could be delivered into the product to save 50-90% more space and COST to all of Qlik's customers. Please give it the thumbs up and back this request.

https://ideation.qlik.com/app/#/case/460857

[parkera]

Hi, 

We did a test today on a customer log file and Parquet was 100X smaller than QVD.