Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
hi all,
so as I understand I cannot use AutonumberHash if the data gets reloaded in 2 separate reloads. What I've seen from Help files is that I should use Hash128 etc, but that returns fields that are very long...
What do you do guys generally do to get a persistent value across loads?
Thanks
Your dataset seems quite large. More important as the number of records is the number of distinct field-values. Means if there are just a few thousands distinct values by billions of rows your intended optimization won't be really significantly. But if there are quite many distinct values it would be different (just keep in mind that you could have max. 2 billions of distinct field-values within a single field).
I think I would in your case investigate if there is any logic within the values included so that you may capable to left the first or last numbers and/or replace/change other parts. If this isn't possible I would split the value into two pure numeric fields. In the target-application you could string-concat both parts within an autonumber() to get a valid key and to keep the RAM as small as possible.
- Marcus
What's the aim for it? Why not just keeping the origin values - especially if they should be used within several applications?
- Marcus
Hi Marcus,
The key fields may potentially be made of of huge values - like '155.98.1237.263.178.32'. I though using autonumberhash might make things better if I could give a numeric value, say 9 for my example, instead of Qlik having to evaluate that terrible long value.
Thanks
so you have to load the datas in the same script, create an autonumberfield and store them into qvd. Then you can load the datas into your apps.
Regards
So the topic is not to use this (autonumber) key across multiple applications else you want to keep the qvd's smaller? How many records and distinct field-values are there? Does all field-values look like your example or are there more differences in length, numeric vs. alpha-numeric or strings, blanks, NULL's? Are any logics included?
- Marcus
Hi Marcus
I want to be able to reduce the size of the QVD's essentially by using Autonumber on those long key fields. Also make linking between tables that both contain these values, quicker.
The field values will always look like the example I gave ('155.98.1237.263.178.32') but will be different for every single key value of the Table - and we're talking billions of rows.
Your dataset seems quite large. More important as the number of records is the number of distinct field-values. Means if there are just a few thousands distinct values by billions of rows your intended optimization won't be really significantly. But if there are quite many distinct values it would be different (just keep in mind that you could have max. 2 billions of distinct field-values within a single field).
I think I would in your case investigate if there is any logic within the values included so that you may capable to left the first or last numbers and/or replace/change other parts. If this isn't possible I would split the value into two pure numeric fields. In the target-application you could string-concat both parts within an autonumber() to get a valid key and to keep the RAM as small as possible.
- Marcus
Thank you Marcus, your advice has been great.
just to be sure I understand you correctly, if the number of distinct values where more, the Autonumber would have a significant influence?
thanks