topic Huge data in qvd files in QlikView

Huge data in qvd files

Anonymous — Wed, 25 Nov 2020 16:16:04 GMT

We have huge data(more than 2 GB) in 3/4 qvd files.

Qlikview dashboard is built (with inline script using) this qvd files.The dash board has 4-5 list box and 2 pivots.

Since the qvd are huge the qlikview dashboard has also became huge.

Now if we want to make any changes (in script or to add additional list box) it takes long time to open, some time it crashes.

Also, since the dashboard is huge user facing issues to use the dashboard.

Is there any way we can open the dashboard easily for editing.

Also, is there any way we can solve the problem of the user to access the dashboard.

What are the best practices for such scenarios

Re: Huge data in qvd files

shiveshsingh — Sat, 17 Feb 2018 04:43:00 GMT

You can use reduction of application based on user( User Names or territory)

Re: Huge data in qvd files

Anonymous — Sat, 17 Feb 2018 07:29:30 GMT

We cannot use the reduction

Re: Huge data in qvd files

shiveshsingh — Sat, 17 Feb 2018 10:33:46 GMT

But you can remove the loaded data(keep possible values), do the changes and reload again.

Re: Huge data in qvd files

Anonymous — Sat, 17 Feb 2018 15:18:01 GMT

Hi Shivesh,

After keeping the required columns our QVD files is more than 2 GB.

Re: Huge data in qvd files

vishsaggi — Sat, 17 Feb 2018 16:30:44 GMT

Did you check your RAM how much it is when you open this dashboard. ?

Does your inline scripts has any transformations happening while loading your QVD files.

What happens when you just load these QVD files without any conditions in the script?

Re: Huge data in qvd files

Anonymous — Sat, 17 Feb 2018 17:06:20 GMT

When I open this dash board maximum RAM of machine is used.

The dash board has 4-5 list box and 2 pivots.

And now I want to make some changes in this dashboard along with the optimization (minimize the size of .qvw file) so that next time any changes needs to be done then it can be made easily.

Re: Huge data in qvd files

Anonymous — Fri, 02 Mar 2018 02:24:09 GMT

Can any one suggest on my query/issue

Re: Huge data in qvd files

ziadm — Fri, 02 Mar 2018 04:29:30 GMT

2GB is such a big dashboard and wonder what would be the size in a year time .. you have to optimize the dashboard and follow best practice of QVW development

Performance and Optimization Best Practices in QlikView & Qlik Sense

Re: Huge data in qvd files

krishna20 — Fri, 02 Mar 2018 07:38:37 GMT

Hi Amar,

As i understood. your qvw application size is huge and it requires more RAM than which you are using now.

Please find the attached .xls and calculate how much RAM size could be required to your .Qvw file and Change your system configuration accordingly .

RAM Calculation in QV Scalability

Thanks

Krishna

Re: Huge data in qvd files

jonathandienst — Fri, 02 Mar 2018 08:47:02 GMT

>>The dash board has 4-5 list box and 2 pivots.

And the data size is >2GB? Sounds like you could achieve a size reduction by reducing the granularity of the data. Do you really need all the detail? I would investigate how to consolidate the data to the level of detail required.

Having said that, 2GB is not an exceptionally large amount of data for QV application, but you would need to have enough RAM. If your users are never going to drill to the lowest level in tables of hundreds of thousands to millions of rows, then an investment in extra RAM could be avoided.

Re: Huge data in qvd files

Peter_Cammaert — Fri, 02 Mar 2018 09:57:37 GMT

Jonathan is right. This is not a huge dashboard. The problem is that any dashboard will feel like being monstrous and sluggish if there isn't an adequate server available to host it.

QlikView has one exceptional quality aside from the associative logic: it loads all data in RAM so that you and your users can have all data at your fingertips with lightning-fast response times (compared to querying a DB or another data source). But that won't work for one inch if the machine that is hosting your document(s) isn't really able to.

Installing sufficient RAM is the first and cheapest step in the scalability trajectory that QlikView supports. The next one may be adding Cores/CPUs. Or the other way around, depending on which resource is feeling an inordinate amount of pressure.

Re: Huge data in qvd files

mellerbeck — Thu, 12 Apr 2018 04:44:30 GMT

Are you displaying all the data in that pivot table... that will make it take 10 minutes to open. (To troubleshoot, delete all your sheet objects. It should open a LOT quicker.

Calculation condition is your friend in this case.