Pack Distinct String Values for better performance - Any Ideas?
Hi,
we are planning to use QlikView to analyze our product data. I did already some tests and I've figured out, that we have got one main problem.
Our serial number before 2010 starts with a 0. Examples are 091212121234 or 08010101122342 or A24323454. We have got already approx. 25 mio SNs in a lot of different databases. While doing the tests, I have recognized, that those distinct String Values consume a lot of memory and performance.
Does anybody has got experience how to store this as number and show it, like reading it? Is there a way with dual or / and autonumber?
You can use autonumberhash128 (or 256) to reduce the memory used by SN. But in this case you cannot show them as is anymore - they will be replaced with numbers. I think that making it "dual" will not help because you're adding short data to long, not repalcing it.
Frankly, this sort of data should not be a problem unless you have tens of millions of them.
autonumberhash is a good idea for the key. Thanks! This is really looking good.
There are tens of million of distinct values, so it is a problem. Certainly not for a well suited server, but my 32bit notebook with 4gb of memory comes to its limits, when loading 8 million rows with some more attributes. As this is quite unuasual (I've loaded already 20 million rows with a much older notebook with 1.5gb), I am continuing to find ways to optimize memory consumption.
