Limitation on Max Rows - Qlik Sense

[raajeshn]

Hi Experts,

Request your help on this situation - I am trying to load around 2.97 billion rows of data in my Qlik Sense Application and I am not able to load them. Please find below the list of steps I have followed and other information required.

- Tried loading 2.97 Billion Rows of Data consisting of three columns (all of them are ids)

- Tried loading them from back-end (Amazon RedShift) first and realized that it fails after 2.15 - 2.16 Billion Rows. Query used was a simple select query (like Select column1, column2, column3 from table)

- Guessing if it is an issue with back-end - tried creating three QVDs (of size 740 Millions - 4.5 GB, 970 Millions - 5.5 GB and 1.26 Billions - 8 GB) with different where clauses (essentially splitting the data into 3 QVDs). After splitting, when I tried merging - it exactly failed in the same 2.16 billion. Further, tried changing the order of the QVDs and it fails around 2.16 billion rows

- Hence, with the above info - load is exactly failing at 2.16 whether reading from QVDs or from back-end.

- Some environment info - Qlik Sense 2.0 with 4 Processors and 128 GB RAM. During the load resource utilization (both RAM and CPU) is under 30%.

Has any of you come across the above situation? Also, just wanted to check if I am missing any basics. Request your views on this and appreciate your support.

Thanks & Regards,

Raajesh N

[Jason_Michaelides · Accepted solution]

See this post - Re: SENSE LIMITATION

The QIX engine can only handle 2.14bn distinct values in a field.

I'd also question why you want 3bn rows in a single application....

[raajeshn]

Hi Jason,

Thanks for the swift response.  Got some handle now. Initially, we were planning about one consolidated application. Guess, we will have to think about segmentation now.

Thanks & Regards,

Raajesh N