topic Re: Reload task takes too much time in QlikView

Reload task takes too much time

master_student — Fri, 17 Jun 2016 08:44:56 GMT

Hello people,

Hope this finds you well.

What do you do if you have a table that contains millions of rows and the reload of data takes more than 30 minutes?

Thanks

Re: Reload task takes too much time

ThornOfCrowns — Fri, 17 Jun 2016 08:49:39 GMT

Is this from a QVD or a data pull from a server, for example?

Re: Reload task takes too much time

migueldelval — Fri, 17 Jun 2016 08:51:03 GMT

Hi Master_student,

Have you try to use QVDs? Maybe you can make QVDs with part of the information (years).

And

Always, in all my projects OLEDB it´s quick than ODBC.

ODBC Vs OLEDB

Regards

Miguel del Valle

Re: Reload task takes too much time

master_student — Fri, 17 Jun 2016 08:54:16 GMT

data from a server. I am not using a QVD

Re: Reload task takes too much time

migueldelval — Fri, 17 Jun 2016 08:55:24 GMT

More information Master_student,

Try this

Regards

Miguel del Valle

Re: Reload task takes too much time

ThornOfCrowns — Fri, 17 Jun 2016 08:56:34 GMT

First question to yourself is do you need to pull the data from the server all the time? Can you save your information to QVD(s) and pull only new incremental data from the server? This depends on how dynamic your data is, of course.

Re: Reload task takes too much time

master_student — Fri, 17 Jun 2016 08:59:53 GMT

Thanks James. do you have a link on how to pull incremental data.

Re: Reload task takes too much time

syukyo_zhu — Fri, 17 Jun 2016 09:01:25 GMT

Do your database be on the same server with your qvw?

and do you do some transformation of your data?

Re: Reload task takes too much time

master_student — Fri, 17 Jun 2016 09:28:03 GMT

Thanks Miguel for the example. will take a look and let you know in case I have questions.

Re: Reload task takes too much time

master_student — Fri, 17 Jun 2016 10:41:00 GMT

No, they are not on the same server.

no transformation needed of my Data.

Thanks

Re: Reload task takes too much time

sasiparupudi1 — Fri, 17 Jun 2016 10:48:11 GMT

Use partial load.. below script appends the modified records to the previously loaded records

Set vTarget='MyTable';

Set vTargetFolder='C:\';

Set dbPrefix ='dbo.';

//if not IsPartialReload() then

if IsNull(FileSize( '$(vTargetFolder)$(vTarget).qvd' )) then

$(vTarget):

SQL Select *

FROM $(dbPrefix)YourTable(nolock) ;

else

let LastExecTime=QvdCreateTime('$(vTargetFolder)$(vTarget).qvd');

$(vTarget):

SQL Select *,

UpdatedDate

FROM $(dbPrefix)YourTable

WHERE UpdatedDate>'$(LastExecTime)';

concatenate add load * from $(vTargetFolder)$(vTarget).qvd(qvd)

where not Exists(YourPrimaryKey);

end if;

STORE $(vTarget) INTO $(vTargetFolder)$(vTarget).qvd (qvd);