Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

How to optimize exporting data from database(Teradata)to qlikview

Hi,

i have a large database which   having 20 crores records for every sales org, So i have 3500 sales org ID like this and i'm fetching data for each sales org wise and making Qvd's  so for every Qvd  for a sales org its taking 6-8 hours , So overall its taking 18 days to reload the full Application and sometimes Reload task has been failing because of more time taking  and have to reload that again.

What should i do in that case? how can i optimize it?

1 Solution

Accepted Solutions
avinashelite

Hi Yogita,

18days is a very big deal. I suggest you to try the direct discovery concept of Qlikview i.e no need to load all the data to in-memory instead bring only a required set and rest take it on fly.

Check with this link.

QlikTech and Teradata Partner To Deliver QlikView Direct Discovery for Big Data Analytics | Qlik

OR

To reduce the reload time, try like this:

1.Instead of pulling all the data, pull only the required fields.

2.Do all the aggregation and store only necessary set of data.

3.If it taking so long time then divide the task between multiple qvw's that one to generate sales, one to organisation ,other etc

View solution in original post

9 Replies
avinashelite

Hi Yogita,

18days is a very big deal. I suggest you to try the direct discovery concept of Qlikview i.e no need to load all the data to in-memory instead bring only a required set and rest take it on fly.

Check with this link.

QlikTech and Teradata Partner To Deliver QlikView Direct Discovery for Big Data Analytics | Qlik

OR

To reduce the reload time, try like this:

1.Instead of pulling all the data, pull only the required fields.

2.Do all the aggregation and store only necessary set of data.

3.If it taking so long time then divide the task between multiple qvw's that one to generate sales, one to organisation ,other etc

Not applicable
Author

Hi ,

I'm using all these concepts .

i have divided  task in to multiple tasks and all aggergation at the backend .

avinashelite

You need all the data from DB??? what's your RAM size?? because it also depends on the about of RAM size too.

share your script let me try to add any points on it.

Not applicable
Author

what Direct Discovery is All about ?

avinashelite

Please refer this documents.

Not applicable
Author

Thanks for the reply, i just have  one doubt  if direct discovery will fetch data directly from data source then  application performance will go down as user selection response will go to data source everytime.

avinashelite

no, because you will pull all the dimension's into Qlikview(in-memory) just the measure calculation will be pushed to the DB so it won't consume much of the time.

NOTE:The dimension will get refreshed only when the application is refreshed.

Not applicable
Author

Thanks  Avinash , but is there any other way then Direct discovery.

avinashelite

we have some third party tools which will connect for the DB and generate qvd's for you e.g Qlikview Ex-presser try with that.