topic Re: What is limit of QVD size & how we can optimize the over sized QVD in QlikView

What is limit of QVD size & how we can optimize the over sized QVD

Fri, 04 Oct 2013 17:57:26 GMT

What is limit of QVD size & how we can optimize the over sized QVD

Re: What is limit of QVD size & how we can optimize the over sized QVD

Anonymous — Fri, 04 Oct 2013 18:10:02 GMT

Raju, have a look at this post:

http://community.qlik.com/thread/35687

Re: What is limit of QVD size & how we can optimize the over sized QVD

stevedark — Sun, 06 Oct 2013 21:30:41 GMT

Hi Raju,

As far as I am aware there is no physical limit on the size of your QVD. In order to create a QVD though the data first has to be in memory - so the amount of RAM you have is a limiting factor. Given the excellent compression that QlikView brings though it is generally not an issue.

Recently I was working on a QVD with 180 million rows, and forty odd dimensional columns.

Regarding optimising QVD's, I have written a blog post that covers this very topic: http://bit.ly/YnAMqT

Hope that is of use.

Regards,

Steve

http://www.quickintelligence.co.uk/

Re: What is limit of QVD size & how we can optimize the over sized QVD

Mon, 07 Oct 2013 04:59:30 GMT

As per my knowledge there no limit on QVD Size, That is depending on the Server Memory and RAW.

Re: What is limit of QVD size & how we can optimize the over sized QVD

ngulliver — Mon, 07 Oct 2013 07:53:59 GMT

Hi, Raju.

If you need any pointers on optimisation, have a look at this blog.

http://www.quickintelligence.co.uk/qlikview-optimised-qvd-loads/

Cheers,

Neil

Re: What is limit of QVD size & how we can optimize the over sized QVD

jaimeaguilar — Wed, 16 Oct 2013 15:43:13 GMT

According to release notes, if you try load more than 2,147,483,647 unique values into QV, it won't work properly. So it can be a parameter, even if a QVD file could store more records than this, it won't be possible to load them completely in QlikView. Not to mention that 2,147,483,647 is quite a lot of data.

regards

Re: What is limit of QVD size & how we can optimize the over sized QVD

Thu, 17 Jul 2014 05:05:08 GMT

could you please tell me what are dashboard functionalaties

Re: What is limit of QVD size & how we can optimize the over sized QVD

saikatghosh — Thu, 17 Jul 2014 05:13:51 GMT

Hi Raju,

Please note that as per my knowledge there is no limit of qvd. its depends how you optimize the data.

Regards

Saikat

Re: What is limit of QVD size & how we can optimize the over sized QVD

Thu, 17 Jul 2014 05:34:34 GMT

No limit size. Big enough that will still fit in your server's memory.

Re: What is limit of QVD size & how we can optimize the over sized QVD

ngulliver — Thu, 17 Jul 2014 09:09:32 GMT

Hi, Naresh.

Please create a separate discussion for your question or you are not likely to get the answer you need.

Thanks,

Neil

Re: What is limit of QVD size & how we can optimize the over sized QVD

Sun, 14 Sep 2014 15:29:37 GMT

Hi Jaime,

I have 4,590,445,300 unique entries in my qvd, and it fails every time i try to load it in QVW. QV throws an error msg saying - 'Execution of script failed, do you want to reload old data'

It would be great if you can help me out in this.

Thanks a lot in Advance

Regards,

Sagar Gupta

Re: What is limit of QVD size & how we can optimize the over sized QVD

stevedark — Sun, 14 Sep 2014 15:44:28 GMT

The most likely issue here is that you are trying to write to a folder that doesn't exist, a folder you don't have access to or a file that is locked.

Double check the folders and permissions. If you are overwriting an existing QVD, try renaming it before the load. If the file is locked it will prevent you doing this. You may need to reboot your machine to free the lock - but first check if you have any other QlikView instances open that may be holding a lock.

Hope that helps,

Steve

Re: What is limit of QVD size & how we can optimize the over sized QVD

Sun, 14 Sep 2014 16:06:30 GMT

Hi Steve,

Thanks a lot for your reply

I have access to all the folders and files and none of them are locked. Below i will try to explain what I am trying to do:

I have 8 txt files of 2GB each, and the task is to create 800GB data out of it. Here is the approach which I follwed:

Step1 : Create qvd out of existing 8 files by concatenating them - Successful

Step2: Concatenate/append the resulting qvd 50 times to create a final qvd - Successful (resulting qvd is of 180GB with 4,590,445,300 records)

Step3: Load final qvd in QVW such that each record is unique. In order to do this I have created a new column in QV script ----- RowNo() as NewColumn. Now, when I execute this step it runs for around 18hrs and for some reason it never takes in the entire 4,590,445,300 records.

The moment i hit OK it throws an error msg 'Execution of script failed, do you want to reload old data'

Note: I have tried loading the same final qvd by limiting the number of records to 5000 to confirm that there is no Issue in final qvd

Any sort of help will be highly appreciated

Much thanks in Advance Steve Dark

Regards,

Sagar Gupta

Re: What is limit of QVD size & how we can optimize the over sized QVD

stevedark — Sun, 14 Sep 2014 22:28:36 GMT

With those kinds of row counts you need to ensure that everything is as optimised as it possibly can be. Ensure you are not taking any values you don't actually need - redundant join keys for example. Round all your numbers where possible to get to having fewer unique values.

Is there no way that you can pre-aggregate the values so that you have fewer rows?

What is the memory and CPU on the box doing during the reload? Are you simply running out of memory.

By default you will not be able to perform this reload via QlikView Server, as I seem to recall there is a four hour limit on reloads.

Hope that helps,

Steve

Re: What is limit of QVD size & how we can optimize the over sized QVD

Mon, 15 Sep 2014 08:20:34 GMT

Hi Steve,

Thanks a lot for your quick response

Here is the machine specs :

Core: 24

Ram: 256GB

Ram utilization goes till 196GB at the time of reload but the same issue - It doesn't load the entire record

It will be of great help if you can provide your valuable inputs to optimize the QVD

Note: This data is at transaction level and we can't roll it up or group by on any field. We are also trying Direct Discovery with RedShift but there also we are facing lot of issues when it comes to performance

Much thanks in advance Steve Dark

Regards,

Sagar Gupta

1. 800GB data creation - Successful

2. Load data in QlikView - Failed

3. Load data in QlikView(Second attempt) - Failed

4. Error message

Re: What is limit of QVD size & how we can optimize the over sized QVD

Fri, 19 Sep 2014 08:21:57 GMT

As far We have the Knowledge that QVD dose not have any Physical limitation. For crating QVD data must be in Memory so RAM is a limiting factor. For more Visit http://intellipaat.com/blog/qlikview-training-business-intelligence-software/