Skip to main content
Announcements
Qlik Introduces a New Era of Visualization! READ ALL ABOUT IT
cancel
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable

Dealing with Large Volume of data in Qlikview

Hello everyone,

I want to load 3-6 months of data from a table of information broken down by a Date field and Category field.Each Date has about 1000 distinct values of Category values, and about 20 million rows per DATE. From the Dashboard level, When a single date and and a single Category value is picked, We need to see only the data that belongs to this selection 

Can anyone suggest a best solution to implement this?

Thanks in advance

6 Replies
tresesco
MVP
MVP

You mean per date and category combination single record (not at detailed level)? If so, you can aggregate the data at the scipt something like:

Load

          Date,

          Category,

          Sum(Amount) as Amount

From <> Group By Category, Date;

Anonymous
Not applicable
Author

Hi Tresesco,

Thanks for ur Reply.

No i require at a detailed level,i.e all the rows that fall under the date and category which is selected.

the case is how do i maintain the 3 to 6 months of data in my QVW, eventhough for analysis i am using only one date and one category at a time.

tresesco
MVP
MVP

20 million rows per date is very big and unusual. See if all records are really needed to users or can put some filter on it, else, consider Direct Discovery.

Anonymous
Not applicable
Author

Yes all the records required for the users.

I also tried with Direct Discovery, sometimes it gets failed due to memory issues and sometimes it throws a time out error and also it is very slow.


Would it be better to maintain seperate QVW's on daily/weekly/monthly basis?

tresesco
MVP
MVP

That could be an option which is not a good one. Are you using the latest version of qv? Direct Discovery is much improved now. Have you considered improving hardware capability? Have a look at the attached doc as well.

Anonymous
Not applicable
Author

Yes, I m using QV 11 SR6.

We have not considered any improvement in the hardware capability.

Are you suggesting to use Hadoop technology for this.

We are using DB2 has the database.