We are having a large app in our Qlik Sense environment (around 95-96 gb app size) which contains billions of records. While opening app, the visualizations are loading too slow. There is nothing to optimize as such in the app as it fetched records directly from database sql query without any group by or major data modelling.
Also the frontend does not contain any major set analysis expression, just simple sum() expressions.
I tried using preload functionality but the issue I am facing here, is after preload, when I try to open the app, the engine service fails and I am presented with "Engine not found" error. Even after disabling the preload and trying to manually open the app for cache warming, the memory and cpu usage spikes to 100% and the engine service fails.
Server configurations he have:
Nodes : 2 engine nodes which are used by users to open the Apps (Note : when a user opens an app, he is redirected to one of the engine nodes) . 2 Scheduler nodes which manages reload
Ram : 768 GB on each of the 4 nodes
CPU : 12 core (can provide additional details if required)
Any suggestions/ways to atleast make the UI faster and responsive apart from scaling the infra ??
Could mashups be of any help here ? Any suggestions regarding the same are welcome !!
I don't know if there are any compression-logic in Sense like it is possible in QlikView. If so, the real RAM footprint might be much larger as the storage-size. With or without any compression you may hit any configuration-limit in Sense as well as within the OS and the machine/VM. Such stuff could become quite complex - a starting point could be to monitor the resource-consumption and checking the various log-files.
Further useful may be to load only the half/quarter of the data-set - are there significantly differences respectively are the differences rather sequentially or exponentially?
Beside the above there may further some potential to optimize the data and/or the data-model, for example avoiding link-tables, row-level formatting, high cardinality fields (splitting timestamps into dates and times and similar stuff), ...
Qlik can handle billions of rows, but my experience is that it requires specialized help to do so. I can recommend my colleague @Oleg_Troyansky or you might reach out to Qlik Professional Services to see if they have a specialist available.
@marcus_sommer , reducing the load to half the data entries reduces the loading times by approx 40%. With complete data set, the dashboard takes around 1 hr to completely load all the charts/ visualizations if I get lucky. Most of the times, I encounter engine not found error.
And as I said, I don't have any link tables or any transformation logic applied in Qlik Side. All the calculation is being done in the SQL query.
That the essential transformations aren't done within the UI doesn't mean that there is no potential to optimize the data-set and the data-model. Like hinted have the formatting and the cardenality of the fields a significantly impact to the sizing respectively the RAM consumption. Take especially a closer look on timestamps, composite keys, sql-table primary keys (aren't necessary within the UI), ... and of course if really all fields/records (not all periods may require an atomic level and may partly aggregated within a mixed fact-table) are needed ...
Beside this I would double the app by removing the entire UI and building a new one - no layout stuff and starting with a simple pivot with one or two dimensions and a measure + a few list-boxes. What happens? How fast opened the app now (with and without certain stored selections) and how is the responsibility of the chart by selections? If this is already quite slow you shouldn't attempt to add dozens of KPI's or other charts. And step by step more complexity could be added (the aim is not a rebuilding else only to detect any bottleneck).
Further take a look on possible compression and any applied section access and/or OnOpen actions - if there is anything it should be deactivated (at least for a testing).
I'd say the first thing to do in cases like this is to understand where the bottlenecks are. And with an app as large as this, I would expect the time to open the app would be a major contributor. I suggest you have a look at the progress information for the app open call to understand what it is that takes time in this step and how much time is spent on individual steps. You can use this tool to get the information (see last screen shot on page regarding progress information):
You're "Engine not found" error might indicate that the engine has crashed. Perhaps from an out-of-memory condition? Fields and tables are compressed in the app on disk, so it might consume a lot more memory when loaded compared to it's disk size.
@marcus_sommer, I tried removing all objects and just added a plain straight table with a few necessary filters, the chart loading time reduced a bit but not so much that it will make a major impact. But the process of applying filters was still sluggish.
I also wanted to ask, if building mashups through enigma.js would help in reducing chart loading times ?
@Øystein_Kolsrud , thanks for recommending the tool will surely take a look. Meanwhile, the users have asked if I can provide an indicator showing the ETA to open the dashboard. Say for eg, my screen is stuck at "Opening the App" loading message, is there any way to show the ETA by which I will be able to atleast see the charts on my dashboard ?
What version of Qlik Sense are you on? As of May 2025, that progress information is available in the client when opening apps, but you have to turn it on. Instructions or doing so can be found here:
