Known problems with parallel loads in QlikView Server?

[Anonymous]

Hello everyone!

For a customer we are administrating a QlikView Server 11.2 SR12 Small Business Edition. Currently, we have the task chain setup for all tasks to run in sequence. We have set it up that way, because at some point we were experiencing problems with the tasks (while still loading in parallel) and our suspicion was that concurrent QVD access (read access only) was the cause of it.

Now we are eager to speed up the nightly reload and use the servers parallel load abilities again. Did anyone ever have issues due to apps READING from the same QVD file at the same time? Would there be any reason for this to be problematic?

Thanks!

Sandro

[Bill_Britt]

HI,

This will be an issue, because of the QVD being locked.

Bill

[Peter_Cammaert]

IMHO a locked file does not block other processes from opening it for Reading (on condition that it's not an Exclusive Lock) It's the Writing combined with attempts to Read the same file that causes problems.

Just did multiple tests with 4 reloads running on the four cores of my laptop (along the way locking up Windows as the reloads kicked the cpu into 100% load) all reading the same 3Gig QVD with lots of WHERE conditions. All went fine, even the one where the QDS mistakenly started four reloads on a single core (no queueing).

Bill, would you advise never to open a single QVD for reading in two or more reload processes at the same time?

Peter

[Bill_Britt]

Hi Peter,

This is an know issue when using publisher to reload task. So, if they try and task start to fail it is because of the locking issue. I think the issues comes in on the initial opening of the file and task stepping on each other.

Bill

[Peter_Cammaert]

So it is a timing issue? If tasks all try to open a single QVD at the exact same time, they may get errors? A simple retry (optionally with a delay of a few seconds) may fix this?

Peter

[Bill_Britt]

Hi Peter,

You are correct and I should have posted that.

Bill

[Peter_Cammaert · Accepted solution]

Hmm. Should we add some nuances to Ian's comment (case 1) in QVD read and write at the same time and consequences?

Also, Marcel could add a similar technique to failing LOADs as he did in a smart way to failing STOREs in Avoid General Script Error in STORE command

[Anonymous]

Alright guys, thank you for your inputs.

In general I would assume, that it works like Ian wrote in the other thread or as Peter wrote about from his tests (thanks for that). What Bill says is what I was afraid can happen.

Now:

Bill explicitly mentions the Publisher. Just to clarify: does the same hold true for the QlikView Server?

If even two reading tasks could step on each others toes with bad timing, what's the conclusion?

Is the parallel load feature in QlikView Server useless?

Does that add a lot of complexity because you have to make sure that the tasks, which load in parallel do not access the same files at the same time?

Or would it be necessary to wrap all read statements into a loop as described in the "Avoid General Script Error" thread?

How do you guys handle this?

Thank you, very productive this thread 🙂

Sandro

[Peter_Cammaert]

Some quick feedback on one of your questions: you do have a point there. Bill explicitly mentioned the publisher while I did my tests on a QlikView server without publisher.

Now that I think of it: even in the high-volume publisher-based environments that I visit/manage, I've never encountered QVD errors caused by simultaneous Reads. On the other hand, STORE errors do happen occasionally. Of course, that doesn't prove that simultaneous reads are absolutely safe, which is your main concern.

BTW the reload task setup for a QVS without publisher and a QVS with publisher isn't that different. Both environments use a QDS-QVB combo to track schedules, trigger tasks and reload documents. AFAIK the one doing the reloads (and the QVD reading), QVB.exe is exactly the same whether you use a full publisher or a limited one...

Peter