Scaling QlikView Apps on Legacy Hardware: Latency vs. Capacity Bottlenecks

[xosat91447]

Hi everyone,

I’ve been a long-time advocate for QlikView, and even with the push toward newer platforms, I still find the Associative Engine in classic QlikView to be incredibly snappy for specific types of complex, local data discovery. I’m currently working on a project where I’m trying to optimize a fairly large .qvw for a small non-profit that is running their infrastructure on some older, repurposed enterprise hardware.

As someone who genuinely enjoys the challenge of "hardware recycling"—the idea of squeezing every bit of utility out of older machines rather than just buying into the latest cloud subscription—I’ve hit a bit of a performance wall that I'm hoping the veterans here can help me navigate.

The specific point I’m struggling with is the calculation time for objects using heavy Set Analysis and nested Aggr() functions. On my development machine, everything is fluid, but on the production host—an older workstation we’ve beefed up—the lag is becoming noticeable to the end users.

To give you some context on the environment, the server is running on a legacy DDR3 architecture. I’ve recently maxed out the board using PC & Server memory I had in my spare parts bin: specifically 8GB DDR3 1333MHz (PC3-10600) modules. I’ve managed to get the total capacity up to 64GB, which theoretically should be plenty of room for our 2GB (compressed) data model once it expands in-memory.

My personal insight from years of tinkering is that QlikView is almost entirely dependent on RAM throughput. However, I’m starting to wonder if the lower frequency of the PC3-10600 (1333MHz) is creating a latency bottleneck that no amount of capacity can fix. I’m seeing high CPU wait times during selections, even though the RAM isn't being fully utilized and the CPU usage isn't hitting 100%.

Specifically, I’m curious if anyone has experience deploying QlikView apps on older 1333MHz RAM. Does the Associative Engine suffer significantly from the lower bus speed of DDR3 compared to DDR4, or should I be looking more closely at my data model’s "Density" and "Cardinality" in the Document Analyzer? I’m trying to determine if it’s worth hunting down slightly faster RAM for this old motherboard, or if I just need to get more aggressive with script-level optimizations to reduce the reliance on complex UI expressions.

Has anyone else noticed a "stutter" in selection response times specifically tied to memory clock speeds rather than just raw volume?

Looking forward to hearing your thoughts on whether I should keep tuning the app or if the hardware has finally met its match.

[rwunderlich]

Take a look at this old document https://community.qlik.com/cyjdu72974/attachments/cyjdu72974/japan-group/2217/1/QV_Scalability_Cente.... It has some info on optimal hardware configurations.  

Check the server log for performance related messages that get generated during calc. You may find some hints there. 

-Rob

[marcus_sommer]

I doubt that the RAM speed is the biggest bottleck in your scenario else more important is the CPU performance. IMO it's very likely that a modern laptop is significantly faster as a 10 years or more elder server. As far as my local machine could provide enough RAM it has mostly beaten our server.

Like hinted in link from Rob the architecture could have a big impact because a very long time only Intel CPU's were officially recommended. Especially the AMD Opterons had had a problem with the internal core-communication which could become heavier as the real output and restricting a server to only 4 or 8 cores could be running faster as 64 ones.

Beside this I wouldn't use compression because it saves only disc-space and consumed resources to de-compress the application which is then in full size within the RAM.

The next is to avoid nested aggr() because it means that on the virtual table underlying a chart further virtual tables are calculated - which means to create the dimensional context on which the expressions are calculated.

This dimension-building is the bottleneck because it's a single-threaded measurements and takes usually the most of the calculation-time. The various aggregations later are multi-threaded. The set analysis stuff is usually not really relevant - it are just selections to reduce/adjust the available data-set for the aggregations.

If possible you may try to run some popular benchmarks on the server and your laptop and I could imagine that your laptop wins especially the single-threaded part very clearly.