Differences in reload-times between Qlikview and Sense on the same script and data

[chris_andersson]

Hi,

We've recently migrated our Qlikview environment to Qlik Sense Enterprise on Windows.

We are experiencing huge differences in load times between Qlikview and Sense. In Sense, the reloads take about four times longer to execute. The apps in Sense are just converted from Qlikview, and loads the same exact data with the same exact script.

In the script-log we can clearly tell that it's the QVD-loads that take time. Granted, there are some un-optimized QVD-loads involved, especially in concatenating some tables, but they shouldn't take 4 times longer on Sense which resides on a more powerful server.

It's a single node setup with decent performance - for sure better than the server that runs Qlikview and the QDS. The files resides on the server as well, so no network issues involved with the QVD-loads. 

From what we've found, there is no particular Windows Server setting we have to choose to allow the QVD-files to unpack faster. Is there a setting in Sense we are missing?

Kr

Christoffer

[marcus_sommer]

Before looking for any settings or technically issues you should check if (all) your tables and qvd's contain the same data and number of records as within QlikView. Already very slight differences between the environments like other data-interpretations may lead to quite different results, for example that any join-key's or where exists() clause don't work like expected and amount of data is now much higher.

- Marcus

[chris_andersson]

Hello Marcus and thanks for your feedback.

The underlying data is identical. We connect Qlikview and Sense to the exact same database and tables/views using the same ODBC-connection. (We've tried OLE DB as well, but the ODBC for some reason is faster). 

This of course results in the same exact amount of lines fetched, which I've triple checked in the script logs (and the qvds are exactly the same size). We have a minimal but acceptable difference in read times among the database tables/views - it's the read from the .qvds that takes much, much longer time in Sense. 

We've now re-installed the whole server meaning we nuked the VM-server that Sense was running on. Complete fresh installation of Windows, Oracle client, .NET framework, Sense and other tidbits.  Other than that the Qlikview and Sense server are almost identical in their setup and performance, where the Sense-server is slightly faster overall (Few more cores, a bit more RAM). 

Unfortunately, the result is the same. We're scratching our heads here. 

[marcus_sommer]

It may a bit depend from which release of QlikView you switched to Qlik Sense. Between the big major-release there were some changes to the engine and this relates also to the way how qvd's could be (optimized) loaded respectively how the data stored within the file. AFAIK this leads not mandatory to a faster loading but a significantly slow-down shouldn't happens.

I don't have Sense in a productive use and have therefore no own experience with loading qvd's within a server-environment but I could imagine that there are some new settings respectively in regard to their defaults changed ones. I believe there is now some indexing possible or encryption or similar features which might have a bigger impact on the performance. Maybe you find here some related postings or the release notes hints to any of such changes or other dependencies.  

- Marcus

[chris_djih]

"un-optimized QVD-loads involved" these kind of things are painful. if you have such un-optimized QVD load in Qlik-Sense, you loose all of the power of QVDs. You must definitely try to load all QVDs optimized!

In most scenarios this leads to a subsequent resident load. Makes the script a bit more ugly, but increases the load time 100x.

[marcus_sommer]

Of course un-optimized qvd's load much slower as optimized ones but should there be a big difference between View and Sense if both use the identically load-mode? Rather not ...

- Marcus

[chris_andersson]

Hi again,

Good idea, I'll check the release notes to see if any changes where made to the engine regarding indexing and/or encryption. Maybe it's a feature that the QVDs takes a longer time now, and not a bug 🙂 

I've tried to convert the un-optimized loads to QVD-optimized and that load is lightning fast. I'm kind of hoping not having to change all the extracts but at this time it doesn't look too bright. 

Thank you again Marcus. I appreciate the answer but I can't mark your answer as a solution I'm afraid.

[marcus_sommer]

Just another approach to check if the theoretically better performing Sense environment is really more performant would be to monitor the workload of CPU + RAM + network/storage within the task-manager during the reload in both environments. Maybe there is anything noticeable.

Further a look on the applied security tools/measures may be helpful, for example the exception list from the anti-virus, any proxy-server and similar stuff ...

- Marcus 

[chris_djih]

As you said, there was some major change. I remember that i had to update some of my old QlikView scripts to boost performance. Nevertheless If the sayings are true, that tehy share the same engine, I definitely agree with you that there should not be a big difference.
Furthermore sharing the same code but performing on different technical infrastructure may also lead to differences. I think only a experienced qlik technician could give more valuable answer to this topic.

My point was: You should make all your QVD load optimized! it will Increase performance a lot no matter how the rest of the solution of this thread turns out.

[marcus_sommer]

In most cases it will be exactly like you say. But the performance will always depend on the biggest bottleneck and not each time the optimized mode will lead to a run-time benefit. For example some years ago we had a scenario to load data over a shared Ethernet network and CPU + RAM had to wait that the data come in ... 

- Marcus