Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
I have a table 1.5 TB in size and we have a Qlik Senser server installed on a 256 GB RAM Server. Is it possible for me to work with this type of data in this Qlik Sense Server?
Note: When I try to load the table I get an error.
If you get an error then your machine probably ran out of memory. Qlik Sense loads all the data into memory. 1.5 TB is larger than 256 GB. So unless the data compresses well it will not fit. Try leaving out data that you don't actually need by using WHERE clauses to filter out unnecessary records and leave out fields you won't use.
If you get an error then your machine probably ran out of memory. Qlik Sense loads all the data into memory. 1.5 TB is larger than 256 GB. So unless the data compresses well it will not fit. Try leaving out data that you don't actually need by using WHERE clauses to filter out unnecessary records and leave out fields you won't use.
Hello,
You can try to evaluate the theorical consumption memory by loading a part of your file (use fist(xxx) before load).
Try to limit at the minimum the fields needed as suggested by gysbert.
The compression will depend on your data so try to load a significant amount of lines.
Then measure the memory needed for this extract and calculate the memory your complete file would need in memory using the full number of records compared to the number loaded.
This is a very empirical method but it gave me a good estimation for some clients. Don't forget your existing applications on your server to evaluate the full memory needed.
also put in your select just the columns that you need.....
avoid
select * from Orders
use like this....
SELECT Code,Value,Client,Item from Orders
Qlik will store unique value. Take care about date (datetime in SQL Server) that create many records instead one.
What I mean is
LOAD just what you need......
and what you LOAD take care about unique value to better compression in Qlik.
This might help you on your analysis