Qlik Community

New to Qlik Sense

Discussion board where members can get started with Qlik Sense.

lyrklaunavan
New Contributor

1.5 TB table into 256 GB Ram Qlik Server

I have a table 1.5 TB in size and we have a Qlik Senser server installed on a 256 GB RAM Server. Is it possible for me to work with this type of data in this Qlik Sense Server?

Note: When I try to load the table I get an error.

Tags (1)
1 Solution

Accepted Solutions

Re: 1.5 TB table into 256 GB Ram Qlik Server

If you get an error then your machine probably ran out of memory. Qlik Sense loads all the data into memory. 1.5 TB is larger than 256 GB. So unless the data compresses well it will not fit. Try leaving out data that you don't actually need by using WHERE clauses to filter out unnecessary records and leave out fields you won't use.


talk is cheap, supply exceeds demand
6 Replies

Re: 1.5 TB table into 256 GB Ram Qlik Server

If you get an error then your machine probably ran out of memory. Qlik Sense loads all the data into memory. 1.5 TB is larger than 256 GB. So unless the data compresses well it will not fit. Try leaving out data that you don't actually need by using WHERE clauses to filter out unnecessary records and leave out fields you won't use.


talk is cheap, supply exceeds demand
pascal_theurot
Contributor II

Re: 1.5 TB table into 256 GB Ram Qlik Server

Hello,

You can try to evaluate the theorical consumption memory by loading a part of your file (use fist(xxx) before load).

Try to limit at the minimum the fields needed as suggested by gysbert.

The compression will depend on your data so try to load a significant amount of lines.

Then measure the memory needed for this extract and calculate the memory your complete file would need in memory using the full number of records compared to the number loaded.

This is a very empirical method but it gave me a good estimation for some clients. Don't forget your existing applications on your server to evaluate the full memory needed.

Pascal
MVP
MVP

Re: 1.5 TB table into 256 GB Ram Qlik Server

also put in your select just the columns that you need.....

avoid

select * from Orders

use like this....

SELECT Code,Value,Client,Item from Orders

Qlik will store unique value. Take care about date (datetime in SQL Server) that create many records instead one.

MVP
MVP

Re: 1.5 TB table into 256 GB Ram Qlik Server

MVP
MVP

Re: 1.5 TB table into 256 GB Ram Qlik Server

What I mean is

LOAD just what you need......

and what you LOAD take care about unique value to better compression in Qlik.

rupamjyotidas
Valued Contributor

Re: 1.5 TB table into 256 GB Ram Qlik Server

This might help you on your analysis

Qlik Sense Scalability Tools

Community Browser