Qlik Community

Qlik Scalability

A group dedicated to advancing QlikView's scalability and performance. In this group we will share knowledge on how to perform load tests of Qlikview.

Highlighted
raajeshn
Contributor

Limitation on Max Rows - Qlik Sense

Hi Experts,

Request your help on this situation - I am trying to load around 2.97 billion rows of data in my Qlik Sense Application and I am not able to load them. Please find below the list of steps I have followed and other information required.

- Tried loading 2.97 Billion Rows of Data consisting of three columns (all of them are ids)

- Tried loading them from back-end (Amazon RedShift) first and realized that it fails after 2.15 - 2.16 Billion Rows. Query used was a simple select query (like Select column1, column2, column3 from table)

- Guessing if it is an issue with back-end - tried creating three QVDs (of size 740 Millions - 4.5 GB, 970 Millions - 5.5 GB and 1.26 Billions - 8 GB) with different where clauses (essentially splitting the data into 3 QVDs). After splitting, when I tried merging - it exactly failed in the same 2.16 billion. Further, tried changing the order of the QVDs and it fails around 2.16 billion rows

- Hence, with the above info - load is exactly failing at 2.16 whether reading from QVDs or from back-end.

- Some environment info - Qlik Sense 2.0 with 4 Processors and 128 GB RAM. During the load resource utilization (both RAM and CPU) is under 30%.

Has any of you come across the above situation? Also, just wanted to check if I am missing any basics. Request your views on this and appreciate your support.

Thanks & Regards,

Raajesh N

1 Solution

Accepted Solutions
jason_michaelid
Honored Contributor II

Re: Limitation on Max Rows - Qlik Sense

See this post - Re: SENSE LIMITATION

The QIX engine can only handle 2.14bn distinct values in a field.

I'd also question why you want 3bn rows in a single application....

2 Replies
jason_michaelid
Honored Contributor II

Re: Limitation on Max Rows - Qlik Sense

See this post - Re: SENSE LIMITATION

The QIX engine can only handle 2.14bn distinct values in a field.

I'd also question why you want 3bn rows in a single application....

raajeshn
Contributor

Re: Limitation on Max Rows - Qlik Sense

Hi Jason,

Thanks for the swift response.  Got some handle now. Initially, we were planning about one consolidated application. Guess, we will have to think about segmentation now.

Thanks & Regards,

Raajesh N