Qlik Community

Ask a Question

App Development

Discussion board where members can learn more about Qlik Sense App Development and Usage.

Announcements
Support Cases coming to Qlik Community Oct. 4! Start chats, open cases, explore resources: READ DETAILS
cancel
Showing results for 
Search instead for 
Did you mean: 
ioannagr
Creator III
Creator III

Is system ram of 64gb too little for 500 million rows?

Hello all,

So i'm trying to read from the database, my local ram (laptop's) is 8gb and at around 400 million records, I have the message "Object ran out of memory" and it fails.

Thing is, in a try to load using the system ram (64gb) it also crashes for almost a billion rows.

Is it too little for this data?

7 Replies
Gysbert_Wassenaar

Depends on how much data each row has and the cardinality of the values. 500 million rows consisting of two numeric fields that both only contain integer values between 0 and 7 won't take up much memory. Records with two hundred columns with millions of distinct values in each column will take up lots and lots of memory.


talk is cheap, supply exceeds demand
ioannagr
Creator III
Creator III
Author

Okay let me see if i can deliver the info well. This table originally has around 30 fields but i am only calling a foreign key to another table and  another one field called amount. The foreign key can be found several times in that table. The foreign key is an integer but of  2-12 digits and field amount is also an integer of 9-12 digits. What do you think @Gysbert_Wassenaar ?

Gysbert_Wassenaar

I think you should load 1 million rows and then store that into a qvd. Then use QViewer (https://www.easyqlik.com/) to take a look at the table metadata to see more information about each field like number of unique values and storage required. Then extrapolate that to 500 million rows.


talk is cheap, supply exceeds demand
prasanthan_ravindran
Contributor II
Contributor II

Hi Loannagr,

This is how I have given the suggesstion to one of our company's client. Always to be on the safer side we have proposed them a quite higher range. Hope thism ight be helpful.

 

Thanks

Prasanthan

ioannagr
Creator III
Creator III
Author

Hello @Gysbert_Wassenaar thank you. Is it for free?

ioannagr
Creator III
Creator III
Author

@prasanthan_ravindran Thank you, definitely useful! Will forward this one for sure!

Gysbert_Wassenaar

There is a free version.


talk is cheap, supply exceeds demand