Reload only new data in qlikview application

[Nasibe]

Hello Everyone!

I am having trouble reloading one app every day.

I have to reload 2 different qvds in app,one for 2previous years that is without any changes.the other qvd includes current year data that may change every day.

My app size is about 9.5 gigabytes,it is scheduled to reload in qlik console everyday.it takes about 45 minutes.most of the time reloading failed after 2 minutes.

I wonder if there is any way to solve this problem.

Thank You in advance.

[Or]

Since we have no way of knowing why it fails, it's hard to try and help...

Generally, check that there's enough server memory to handle the data, check that the source you're reading from is stable, and make sure there's nothing wrong with your data model. Also, make sure your QVD load is optimized.

[Nasibe]

Thanks for your reply
My great concern is, can we store old data in app and not reload it everyday.

[Or]

You can look into using Partial Reload:

https://help.qlik.com/en-US/qlikview/May2023/Subsystems/Client/Content/QV_QlikView/Scripting/ScriptP...

https://community.qlik.com/t5/Official-Support-Articles/How-to-use-Partial-reload-in-QlikView/ta-p/1...

You might also look into loading the first QVD into a separate app and then binary loading that before adding the second, but I'm not sure if there'd be any particular advantage to that since you'd still be loading the data every time, just not from QVD.

[marcus_sommer · Accepted solution]

In a quite strict interpretation you couldn't store old data into Qlik because old and new data will have an impact on each other. In a more wider interpretation the from @Or suggested approaches of using partial loading and/or a binary loading are going in this direction.

But most important will be to keep the loading optimized - which means that the load is just a transfer of the data from the file into the RAM with nearly nothing of extra processing. Therefore only the performance of your network/storage and the RAM are essentially.

If both load-parts (old and new) are optimized I would by a data-set of around 10 GB assume a qmc task-runtime from about 1 minute, rather less. If your successful loading needs about 45 minutes it indicates that either the network/storage performance is very low and/or that not enough RAM is available and/or that the loads aren't optimized (to get them optimized you may need more layers within your data-architecture).

Beside this I suggest to review the data-model itself to load only relevant columns and rows and to ensure that the cardinality of the fields is optimized (removing record-id's, splitting timestamps into dates and times and similar measures).

[Nasibe]

Thank you for your reply Marcus
I understand from your answer,it's not possible that store old data (not changeable) data in app and add new data every day.
I didn't know that running task duration in 1 min is feasible for 10 GB size app!!!😳
I attached structure and log of my app,would be kind enough to look at it.
Thanks

[Nasibe]

available RAM for Qlikview is 250 GB,
Total RAM is 500 GB,half of it is dedicated  to SQL jobs and other half is for Qlik jobs.

[marcus_sommer · Accepted solution]

Not enough RAM seems rather unlikely but it looked that QlikView and a SQL Server are sharing the resources. Using such a constellation indicates further that probably all QlikView services are running on the same machine and are competitive against each other - especially the qvs.exe won't release RAM below the min-settings and also the max-setting might be too high to ensure that the distribution service could perform always all tasks without of being impact.

Beside of this it seems that nearly no qvd-load is optimized. To get a load optimized nearly all processing needs to be removed respectively to be done in beforehand - which may require one or several extra layers within the data-architecture. There seems not very much of transformation so the efforts should be reasonable, for example:

• extracting the max. dates/years from the previous layer and storing these information as extra table or within variables
• performing NULL checks within the qvd-creation step
• the same with adjusting the value-direction (positive/negative)
• creating extra exists-tables

The last means something like:

Dummy: load 'MyExcludingValue' as MyFilterField autogenerate 1;

which may then applied per:

Final: load * from QVD where not exists(MyFilterField); // this is an optimized load !

All other kinds of where clauses will leas to un-optimized loads.