topic Re: RE: Loading data in QlikView

RE: Loading data

Anonymous — Wed, 25 Nov 2020 16:16:04 GMT

Hi All

I am trying to load the data from data base as it contains huge data the loading time is taking 1 hour to load . but my requirement is that i want it to be loaded with 15 to 20 min. And i don't want to use QVD. Is their any other way to reduce the Loading time

Thanks in advance !

Re: RE: Loading data

Anonymous — Fri, 15 Jul 2016 08:03:22 GMT

Hi Mahesh,

Have you considered using an incremental load?

Incremental Load using Qlikview Components | Qlikview Cookbook

http://www.quickintelligence.co.uk/qlikview-incremental-load/

Charlie

Re: RE: Loading data

vikramv — Fri, 15 Jul 2016 08:07:01 GMT

Try using incremental load if you have date kind of field...

http://www.analyticsvidhya.com/blog/2014/09/qlikview-incremental-load/

Re: RE: Loading data

Mark_Little — Fri, 15 Jul 2016 08:28:09 GMT

HI,

I would have to say probably not, As above QVD's and incremental loads would be the way forward. The only other thing i can suggest is you make sure you are only loading the data you required, both tables and fields i.e. no Load * from, even in the SQL statements.

Mark

Re: RE: Loading data

Anonymous — Fri, 15 Jul 2016 08:59:28 GMT

Hi Mark

Thanks .. Is their any other way than the incremental load

Re: RE: Loading data

Anonymous — Fri, 15 Jul 2016 08:59:45 GMT

Thanks .. Is their any other way than the incremental load

Re: RE: Loading data

Anonymous — Fri, 15 Jul 2016 09:00:01 GMT

Thanks .. Is their any other way than the incremental load

Re: RE: Loading data

Mark_Little — Fri, 15 Jul 2016 09:09:43 GMT

Hi,

Like i mentioned above, not really.

Other than making sure you are only load the required data and fields.

Mark

Re: RE: Loading data

jonathandienst — Fri, 15 Jul 2016 09:11:13 GMT

Assuming you do not want to change the way the load is performed (incremental load, offline source etc), then the load time is a function of the size of the record set, the SQL query, your database server, your network and the QV machine performing the reload. You would need to work out what is limiting the load speed to determine where to spend time/money.

Does the SQL query have a complex set of joins, including very large tables? How long does the query take in SQL MS (SQL Server) or Toad (Oracle etc)?

Can the SQL extract be performed overnight or at a set time on a schedule? Create a QVD generator that saves this extract to a QVD; then your model reload can be from this QVD, which could be 5-10x (or more) faster than the SQL query.

The load architecture has the largest impact on load times, so if you don't want to change that, I think you are not going to have much success.