Dealing with Large Volume of data in Qlikview

[Anonymous]

Hello everyone,

I want to load 3-6 months of data from a table of information broken down by a Date field and Category field.Each Date has about 1000 distinct values of Category values, and about 20 million rows per DATE. From the Dashboard level, When a single date and and a single Category value is picked, We need to see only the data that belongs to this selection 

Can anyone suggest a best solution to implement this?

Thanks in advance

[tresesco]

You mean per date and category combination single record (not at detailed level)? If so, you can aggregate the data at the scipt something like:

Load

          Date,

          Category,

          Sum(Amount) as Amount

From <> Group By Category, Date;

[Anonymous]

Hi Tresesco,

Thanks for ur Reply.

No i require at a detailed level,i.e all the rows that fall under the date and category which is selected.

the case is how do i maintain the 3 to 6 months of data in my QVW, eventhough for analysis i am using only one date and one category at a time.

[tresesco]

20 million rows per date is very big and unusual. See if all records are really needed to users or can put some filter on it, else, consider Direct Discovery.

[Anonymous]

Yes all the records required for the users.

I also tried with Direct Discovery, sometimes it gets failed due to memory issues and sometimes it throws a time out error and also it is very slow.

Would it be better to maintain seperate QVW's on daily/weekly/monthly basis?

[tresesco]

That could be an option which is not a good one. Are you using the latest version of qv? Direct Discovery is much improved now. Have you considered improving hardware capability? Have a look at the attached doc as well.

[Anonymous]

Yes, I m using QV 11 SR6.

We have not considered any improvement in the hardware capability.

Are you suggesting to use Hadoop technology for this.

We are using DB2 has the database.