Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
In Qlik sense:
All of the data loaded into Qlik Sense is available in two representations: as a string and as a number.
When I test the following two loading with 100,000,000 rows. Why the first one generates such a big size Qlik app? Why the Qlik app size of the second one is not so big?
Just because of data type? number, string?
Thanks!
QlikAppSize1:
Load
RowNo() as ID,
'text1' as text1
AutoGenerate 100000000;
QlikAppSize2:
Load
'RowNo()' as ID,
'text1' as text1
AutoGenerate 100000000;
Hi Haikuo,
The QlikAppSize1 is larger and probably took longer to run, in this case, Qlik has to allocate memory to store 100,000,000 different ID values and one text1 column with a unique value of 'text1', while QlikAppSize2 ran faster because Qlik only has to allocate memory for two text columns ID with a value 'RowNo()' and text1 with a value 'text1'.
Qlik Sense optimizes memory allocation, if a value in your data happens more than once (like 'RowNo()' or 'text1' which happened 100000000 times) it keeps one copy of each one, so if we were able to look at the memory allocation for the second Application, we will find 'RowNo()' and 'text1' just once!
Hope this helps,
Hi Arnaldo, thanks so much for your help!
Your answer helps clean my question.
Previously, I thought that number (float/double) may take more memory than text (string).
When tested again with following loading, it even generates a Qlik app with bigger size.
My question is how to reduce/optimize the Qlik sense app size, if the table for Qlik sense has 100 million rows.
QlikAppSizeText:
Load
Text(RowNo()) as ID,
'text1' as text1
AutoGenerate 100000000;
Hi,
try this...
QlikAppSizeText:
Load
AUTONUMBER( Text(RowNo()) ) as ID,
AUTONUMBER( 'text1' ) as text1
AutoGenerate 100000000;
Best regards,
German
Hi Haikuo,
Qlik already has a very efficient size optimization algorithm, that will take care of many waste of memory. If you are dealing with very large tables, you still have options driven by the following line of thinking:
Does the Qlik Application really need those 100 million records? We could be talking about data expanding several years, device monitoring data (where the data is collected every 30 seconds or less); your options are:
Hope this helps,