Qlik Community

QlikView Deployment

Discussion Board for collaboration related to QlikView Deployment.

Not applicable

How to optimize exporting data from database(Teradata)to qlikview

Hi,

i have a large database which   having 20 crores records for every sales org, So i have 3500 sales org ID like this and i'm fetching data for each sales org wise and making Qvd's  so for every Qvd  for a sales org its taking 6-8 hours , So overall its taking 18 days to reload the full Application and sometimes Reload task has been failing because of more time taking  and have to reload that again.

What should i do in that case? how can i optimize it?

1 Solution

Accepted Solutions

Re: How to optimize exporting data from database(Teradata)to qlikview

Hi Yogita,

18days is a very big deal. I suggest you to try the direct discovery concept of Qlikview i.e no need to load all the data to in-memory instead bring only a required set and rest take it on fly.

Check with this link.

QlikTech and Teradata Partner To Deliver QlikView Direct Discovery for Big Data Analytics | Qlik

OR

To reduce the reload time, try like this:

1.Instead of pulling all the data, pull only the required fields.

2.Do all the aggregation and store only necessary set of data.

3.If it taking so long time then divide the task between multiple qvw's that one to generate sales, one to organisation ,other etc

9 Replies

Re: How to optimize exporting data from database(Teradata)to qlikview

Hi Yogita,

18days is a very big deal. I suggest you to try the direct discovery concept of Qlikview i.e no need to load all the data to in-memory instead bring only a required set and rest take it on fly.

Check with this link.

QlikTech and Teradata Partner To Deliver QlikView Direct Discovery for Big Data Analytics | Qlik

OR

To reduce the reload time, try like this:

1.Instead of pulling all the data, pull only the required fields.

2.Do all the aggregation and store only necessary set of data.

3.If it taking so long time then divide the task between multiple qvw's that one to generate sales, one to organisation ,other etc

Not applicable

Re: How to optimize exporting data from database(Teradata)to qlikview

Hi ,

I'm using all these concepts .

i have divided  task in to multiple tasks and all aggergation at the backend .

Re: How to optimize exporting data from database(Teradata)to qlikview

You need all the data from DB??? what's your RAM size?? because it also depends on the about of RAM size too.

share your script let me try to add any points on it.

Not applicable

Re: How to optimize exporting data from database(Teradata)to qlikview

what Direct Discovery is All about ?

Re: How to optimize exporting data from database(Teradata)to qlikview

Please refer this documents.

Not applicable

Re: How to optimize exporting data from database(Teradata)to qlikview

Thanks for the reply, i just have  one doubt  if direct discovery will fetch data directly from data source then  application performance will go down as user selection response will go to data source everytime.

Re: How to optimize exporting data from database(Teradata)to qlikview

no, because you will pull all the dimension's into Qlikview(in-memory) just the measure calculation will be pushed to the DB so it won't consume much of the time.

NOTE:The dimension will get refreshed only when the application is refreshed.

Not applicable

Re: How to optimize exporting data from database(Teradata)to qlikview

Thanks  Avinash , but is there any other way then Direct discovery.

Re: How to optimize exporting data from database(Teradata)to qlikview

we have some third party tools which will connect for the DB and generate qvd's for you e.g Qlikview Ex-presser try with that. 

Community Browser