Binary Load vs Optimized QVD Load Speed? Who is the King of Speed?

[Sazabi]

Salutations Qlik Qomrades!

Question about using a Binary Load: Is it faster than Optimized QVD Loads?

In my scenario, I have 13 months of data in my app. Of those 13, 12 are considered stable and will not change. the 13th month (incrementally built because that is latest data) can still change.

If I have a QVF that contains the stable months and just binary load that into the User App + load in the 13th month QVD regularly every day, would that be faster than Optimized Loading 13 QVD files every day?

Keep in mind my current QVF size is 25 GB. App reload time is 1.5 to 2 hours. I would like to shrink it to under an hour.

Regards,

S 

[rwunderlich]

I would guess that binary load would be faster, but why not test and see?

The 1.5 hours load time you currently have -- is that to load 13 QVDs? That sounds like a long time. Are the loads optimized?

-Rob
http://masterssummit.com
http://qlikviewcookbook.com
http://www.easyqlik.com

[Sazabi]

Sure, I'll test out Binary Load and report back 😎

---

@rwunderlich wrote:

Are the loads optimized?

Yep, I am doing an optimized load.

Each of the QVD files being loaded in are ~9 GB big.

Depending on how well the server is feeling, it can take anywhere from 2 to 15 minutes to completely optimize load in a single QVD. Then there are some other processes that occur, but it is very small in terms of run time. 

S

[marcus_sommer]

Beside your origin question it might be worth to check if the data within the qvd's could be reduced in some way. For instant by splitting fields with a high cardinality into lower ones like - a good example for it is the split of timestamps into a date- and a time-field.

Also the format of fields could have a bigger impact. And of course the removing of all not in the app used fields - especially if there are record-ID's which are usually only useful by validating the data but not in the daily business.

- Marcus

[Sazabi · Accepted solution]

Thanks Marcus, I will keep that in mind as well!

@rwunderlich 

I tested out the binary load, and I was able to binary load a 22 GB QVF containing just the 12 stable months' data in a little over 5 minutes. 

Overall, my app reload for a 25 GB app is now around 30 minutes.

Regards,

S