Request your help on this situation - I am trying to load around 2.97 billion rows of data in my Qlik Sense Application and I am not able to load them. Please find below the list of steps I have followed and other information required.
- Tried loading 2.97 Billion Rows of Data consisting of three columns (all of them are ids)
- Tried loading them from back-end (Amazon RedShift) first and realized that it fails after 2.15 - 2.16 Billion Rows. Query used was a simple select query (like Select column1, column2, column3 from table)
- Guessing if it is an issue with back-end - tried creating three QVDs (of size 740 Millions - 4.5 GB, 970 Millions - 5.5 GB and 1.26 Billions - 8 GB) with different where clauses (essentially splitting the data into 3 QVDs). After splitting, when I tried merging - it exactly failed in the same 2.16 billion. Further, tried changing the order of the QVDs and it fails around 2.16 billion rows
- Hence, with the above info - load is exactly failing at 2.16 whether reading from QVDs or from back-end.
- Some environment info - Qlik Sense 2.0 with 4 Processors and 128 GB RAM. During the load resource utilization (both RAM and CPU) is under 30%.
Has any of you come across the above situation? Also, just wanted to check if I am missing any basics. Request your views on this and appreciate your support.