How does Qlik Sense handle a dataset of 40 mil item lines?

[#1]

Hi all,

I'm quite new in Qlik Sense and I was just wondering if you have any experience when it comes to creating a dashboard out of 40 mil item lines, every entry with round about 20 dimensions. I tried yesterday to load it but it crashed after 5 hours, could it bit possible that the application can't handle this amount of data?

The data entries are split in 4 years. Is there a chance to duplicate the app for the four years, run it and then merge them into one?

Please feel free to suggest other approaches that you would use

Thank you!

Andreea

[marcus_sommer]

I assume that your pc hadn't enough RAM - you will probably need at least 16 GB RAM for this maybe more. Have a look into the taskmanager if you loaded the data. If the RAM consumption goes to 100% and stayed there for long you will need more RAM or try to optimize your data (only the needed fields, no fields with high amounts of distinct values ...).

- Marcus

[rbecher]

I suggest to limit the amount of rows to some thousand (eg. FIRST 10000 LOAD ...) to see if the application itself loads in a proper way and the data model fits to your requirements. If you have multiple tables you should then check if synthetic keys are created which can consume a lot of memory. Tables should be linked by one field only..

[ToniKautto]

That amount of data is not a problem for the Qlik Sense engine. As pointed out above you are more likely experiencing a logical issue in your reload.

Could you perhaps share the QVF file that you are trying to reload? This would make it possible to see if there is anything obviously incorrect in your script. The QVF file can be found in your document folder under Qlik:

C:\Users\<USERNAME>\Documents\Qlik\Sense\Apps

I am also a bit curious of the crash that you mention. What is crashing, Qlik Sense or Windows? What is the symptom of this crash, do you see nay error message or what happens?