Auto Chart causes server crash in very large Qliksense app

[nsm1234567]

Hi there,

We recently upgraded our Qliksense environment to "May 2022 Patch 11" (we are using on-prem Qliksense).

We have a very large app which is over 100gb's in memory and the main fact table contains 2 billion + rows.  After the upgrade, we consistently had crashes of the Engine service.  The behaviour was that memory would increase very quickly until Qliksense was forced to recover by restarting the engine service.

Upon investigation, I traced one of the crashes to a sheet a user created.  When I went to the sheet, the app started hanging immediately and I was able to replicate the crash.  When I took a closer look, there was an "auto-chart" object that wasn't resolving.  i.e. the user had dragged a simple dimension onto the sheet and the attempt to determine which chart to use seemed to crash the server.

When I reduced the size of the app to contain only one week of data, the object resolved and there were no crashes.  It seems then that there is a bug in Qliksense where auto-chart can not resolve and in fact crash the Engine when the app is very large.  I'm currently trying to workaround this by using SET DISABLE_AUTO_CHART=1; in the load script, but was wondering if anyone had experienced anything similar or if there was a way to report this as a potential issue?

[Albert_Candelario]

Hello @nsm1234567 ,

It would be interesting to change your approach and instead of analyzing a such huge app (100GB memory footprint) tackle it via the ODAG functionality.

On-demand apps enable you to load and analyze big data sources in Qlik Sense. Trying to analyze an entire big data store at one time is highly inefficient. Nevertheless, to make representative visualizations, all the data must be discoverable. Qlik Sense on-demand apps give users aggregate views of big data stores and allow them to identify and load relevant subsets of the data for detailed analysis.

Managing big data with on-demand apps | Qlik Sense on Windows Help

I hope this helps you.

Cheers,

Albert

[nsm1234567]

Hi there,

This is an old app in our environment.  It was built in Qliksense 3.1.  The very large server it runs on which has 1TB of memory was purchased with the specific purpose of running this app in this way.  The app has been very carefully optimised and runs well for a large number of users who have created many sheets in the app.  The only issue is that, after upgrading, Qliksense is now extremely inefficient when trying to do whatever it is it does to generate the auto-chart (as after disabling auto-chart everything seems fine).  I don't think Sense should crash our server profiling a dimension that contains two distinct values.

We've previously experimented with ODAG (not with this app, but for BigQuery connections) and our users found it clunky to use and it was difficult to administer all of the generated apps (this may have changed over the years).  Qliksense is also currently considered a legacy tool in our environment, so it's unlikely that we'd get buy-in to re-architect things.

[Albert_Candelario]

Hello @nsm1234567,

Thanks for sharing your use case.

Since your apps uses 100GB in memory I was wondering how much is on disk. Also interested to know why you mentioned is considered a legacy tool, how you explore the option of migration to Qlik Cloud that could bring opportunities to refactor the current status of the app.

Cheers.

Albert