topic Pack Distinct String Values for better performance - Any Ideas? in QlikView

Pack Distinct String Values for better performance - Any Ideas?

Wed, 14 Apr 2010 20:16:38 GMT

Hi,

we are planning to use QlikView to analyze our product data. I did already some tests and I've figured out, that we have got one main problem.

Our serial number before 2010 starts with a 0. Examples are 091212121234 or 08010101122342 or A24323454. We have got already approx. 25 mio SNs in a lot of different databases. While doing the tests, I have recognized, that those distinct String Values consume a lot of memory and performance.

Does anybody has got experience how to store this as number and show it, like reading it? Is there a way with dual or / and autonumber?

Regards

Christian

Pack Distinct String Values for better performance - Any Ideas?

Anonymous — Thu, 15 Apr 2010 01:41:39 GMT

Christian,

You can use autonumberhash128 (or 256) to reduce the memory used by SN. But in this case you cannot show them as is anymore - they will be replaced with numbers. I think that making it "dual" will not help because you're adding short data to long, not repalcing it.

Frankly, this sort of data should not be a problem unless you have tens of millions of them.

Pack Distinct String Values for better performance - Any Ideas?

Thu, 15 Apr 2010 21:33:57 GMT

Hi,

autonumberhash is a good idea for the key. Thanks! This is really looking good.

There are tens of million of distinct values, so it is a problem. Certainly not for a well suited server, but my 32bit notebook with 4gb of memory comes to its limits, when loading 8 million rows with some more attributes. As this is quite unuasual (I've loaded already 20 million rows with a much older notebook with 1.5gb), I am continuing to find ways to optimize memory consumption.

Regards

Christian