Reducing QVD creation

[Anonymous]

Hey all,

I've been working with Qlik fulltime for some time now and I've been trying to reduce the reload time as much as possible.

However there is one piece of code that is taking up 35 minutes and I have no idea how to reduce its time.

It's about the creation of the QVD files at the end of the script.

There are 35 tables with quite some data. The QVD files in total are about 11 GB.

Code:

SET vQvd='lib://Direction/Here';

FOR i=0 to NoOfTables()-1

                LET vTable = TableName(i);

                STORE [$(vTable)] INTO [$(vQvd)\$(vTable).qvd] (qvd);

NEXT i

The last reload time of this piece of code was 33 minutes and 31 seconds.

I'm wondering is it's possible to reduce this time or if it's just because we have a lot of data.

Sincerely,

Jens

[tresesco]

Yes, bigger file takes longer to be stored on the disk. However, there is a possibility to reduce that time. Check your RAM spike during the write/store operation, if it's very high or may be touching the peak, increasing RAM might help.

[Anonymous]

Thanks for the reply.

When I look at the performance dashboard for the last week I see the RAM Committed during the reload of the task between 25-45%.

I've witnessed 1 peak of 52% a few weeks ago, but nothing concerning.

The MAX CPU is peaking between 55-70% with 1 peak to 82%. Is this going to be a problem?

Sincerely,

[tresesco · Accepted solution]

Should not be a problem. When you say,

The last reload time of this piece of code was 33 minutes and 31 seconds.

does that mean, only this STORE command section taking this much of time? Or, your app has some other transformation script as well? If so, you might have to look into that section of script rather than this snippet, because I don't see much of scope here to improve as code. And yes, one more aspect to check - where is this qvd being stored, in local or network path? Network path would always be slower.

[Anonymous]

Only this small part of the STORE procedure takes up the entire 33 minutes and 31 seconds.

The reload of today just finished and now it took 35 minutes and 45 seconds to create all the QVDs.

Look at the time difference for each reload (again 2 minute difference). It's very inconsistent.

The total reload time is 2 hours 50 minutes on average. But this part is the one that takes the longest at this moment. So we try to tackle the biggest ones first.

And it's stored with a network path.I'll try to change it to a local path and look how it influences the reload time.

Thanks for the tip.

Sincerely,