Qlik Community

Scalability

Announcements
Now Live: Qlik Sense SaaS Simplified Authoring – Analytics Creation for Everyone: READ DETAILS
cancel
Showing results for 
Search instead for 
Did you mean: 
raajeshn
Partner - Creator
Partner - Creator

Limitation on Max Rows - Qlik Sense

Hi Experts,

Request your help on this situation - I am trying to load around 2.97 billion rows of data in my Qlik Sense Application and I am not able to load them. Please find below the list of steps I have followed and other information required.

- Tried loading 2.97 Billion Rows of Data consisting of three columns (all of them are ids)

- Tried loading them from back-end (Amazon RedShift) first and realized that it fails after 2.15 - 2.16 Billion Rows. Query used was a simple select query (like Select column1, column2, column3 from table)

- Guessing if it is an issue with back-end - tried creating three QVDs (of size 740 Millions - 4.5 GB, 970 Millions - 5.5 GB and 1.26 Billions - 8 GB) with different where clauses (essentially splitting the data into 3 QVDs). After splitting, when I tried merging - it exactly failed in the same 2.16 billion. Further, tried changing the order of the QVDs and it fails around 2.16 billion rows

- Hence, with the above info - load is exactly failing at 2.16 whether reading from QVDs or from back-end.

- Some environment info - Qlik Sense 2.0 with 4 Processors and 128 GB RAM. During the load resource utilization (both RAM and CPU) is under 30%.

Has any of you come across the above situation? Also, just wanted to check if I am missing any basics. Request your views on this and appreciate your support.

Thanks & Regards,

Raajesh N

1 Solution

Accepted Solutions
Jason_Michaelides
Luminary Alumni
Luminary Alumni

See this post - Re: SENSE LIMITATION

The QIX engine can only handle 2.14bn distinct values in a field.

I'd also question why you want 3bn rows in a single application....

View solution in original post

2 Replies
Jason_Michaelides
Luminary Alumni
Luminary Alumni

See this post - Re: SENSE LIMITATION

The QIX engine can only handle 2.14bn distinct values in a field.

I'd also question why you want 3bn rows in a single application....

raajeshn
Partner - Creator
Partner - Creator
Author

Hi Jason,

Thanks for the swift response.  Got some handle now. Initially, we were planning about one consolidated application. Guess, we will have to think about segmentation now.

Thanks & Regards,

Raajesh N