Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable

1.5 TB table into 256 GB Ram Qlik Server

I have a table 1.5 TB in size and we have a Qlik Senser server installed on a 256 GB RAM Server. Is it possible for me to work with this type of data in this Qlik Sense Server?

Note: When I try to load the table I get an error.

1 Solution

Accepted Solutions
Gysbert_Wassenaar

If you get an error then your machine probably ran out of memory. Qlik Sense loads all the data into memory. 1.5 TB is larger than 256 GB. So unless the data compresses well it will not fit. Try leaving out data that you don't actually need by using WHERE clauses to filter out unnecessary records and leave out fields you won't use.


talk is cheap, supply exceeds demand

View solution in original post

6 Replies
Gysbert_Wassenaar

If you get an error then your machine probably ran out of memory. Qlik Sense loads all the data into memory. 1.5 TB is larger than 256 GB. So unless the data compresses well it will not fit. Try leaving out data that you don't actually need by using WHERE clauses to filter out unnecessary records and leave out fields you won't use.


talk is cheap, supply exceeds demand
pascal_theurot
Partner - Creator II
Partner - Creator II

Hello,

You can try to evaluate the theorical consumption memory by loading a part of your file (use fist(xxx) before load).

Try to limit at the minimum the fields needed as suggested by gysbert.

The compression will depend on your data so try to load a significant amount of lines.

Then measure the memory needed for this extract and calculate the memory your complete file would need in memory using the full number of records compared to the number loaded.

This is a very empirical method but it gave me a good estimation for some clients. Don't forget your existing applications on your server to evaluate the full memory needed.

Pascal
afurtado
Partner Ambassador/MVP
Partner Ambassador/MVP

also put in your select just the columns that you need.....

avoid

select * from Orders

use like this....

SELECT Code,Value,Client,Item from Orders

Qlik will store unique value. Take care about date (datetime in SQL Server) that create many records instead one.

furtado@farolbi.com.br
afurtado
Partner Ambassador/MVP
Partner Ambassador/MVP

see also

Symbol Tables and Bit-Stuffed Pointers

furtado@farolbi.com.br
afurtado
Partner Ambassador/MVP
Partner Ambassador/MVP

What I mean is

LOAD just what you need......

and what you LOAD take care about unique value to better compression in Qlik.

furtado@farolbi.com.br
rupamjyotidas
Specialist
Specialist

This might help you on your analysis

Qlik Sense Scalability Tools