Qlikview vs QlikSense QVD builder load time

[pbriggs11]

Hi All - our company has finally began to the move from QlikView to QlikSense...started to put together a QVD Builder for apps to consume QVD's and noticed on the first attempt a significantly longer load time for QlikSense of over 6 hours for the same data when compared to an under 20 minute load time from QlikView.  Clearly I must being doing something wrong here?

QlikSense load script to create QVDQlikView load script to create QVD

[JHuis]

I dont know, but are you doing incremental load in QlikView and not in QlikSense?

[pbriggs11]

Not doing incremental loads at all...it's been on my list of To Do's to learn but with reasonable load times in QlikView never really had the need to.  Guess I'll be looking into incremental loads here soon.

I did forget to share in the original post that the QlikView loads are performed on a Remote Desktop Connection, while Qlik Sense is on local machine (as far as I could tell Qlik Sense Desktop can only be installed on local machine?).  I assume this could be impacting performance?  Just not sure to what degree.

[Or]

Depending on where you are relative to the server, connection quality/bandwidth can make a huge difference to the speed of loading data.

Are you using the same connection definition? Or did you perhaps change from OLEDB to ODBC or otherwise make changes? That might be another thing to consider.

I wouldn't suggest using Qlik Sense Desktop for, well, anything really. Assuming you have a Qlik Sense server set up (or are using SaaS), you should be able to use that to load data without the need for Desktop.

[marcus_sommer]

Beside the already mentioned network-stuff to pull the data it's very likely that the available resources of CPU + RAM are probably quite different between the remote server and the local machine. Especially if there isn't enough RAM available the virtual RAM on the HDD/SSD would be used and this will always cause a very significantly performance decrease.

Further it's not the best choice to put so many reloads within a single application - even if you really drop the loaded tables after the qvd-creation which would release their RAM consumption. Dividing the task into n logically / sizeable chunks would simplify the development and maintaining - especially if also incremental approaches are applied, making everything more robust in regard to any troubleshooting and reloads could be run in parallel and/or within other time-frames or with a different frequency. Of course such measures will increase the overhead to administer all the ETL stuff but in the most scenarios the benefit is much bigger as the needed efforts.

[pbriggs11]

Thanks Or, connection definition is same, literal copy/paste from QV script to QS script (no change to OLEDB/ODBC).

Thanks for the Qlik Sense Desktop comment, you are correct we have Qlik Sense server but not yet set up since we just started the migration last week and our IT team is still currently in the works to configure all the connections.  I'm just a Finance guy who saw value in QV when IT purchased half a decade ago due to our company having no tools at the time, so I'm hacking around in the desktop version to become familiar with Qlik Sense in the meantime. 

Appreciate you taking the time to respond and provide input.

[pbriggs11]

Thank You Marcus.  I will take your recommendation about so many reloads as an action item...time for me to learn incremental loading and like you said figure out how to divide into logical/sizeable chunks.  One of the tables with 80+ million records has about 5 years of customer invoice data, so likely need to trigger of dates/years in the loading process to make it more palatable.