topic Re: Several jobs loading the same QVD at the same time in QlikView

Several jobs loading the same QVD at the same time

jonathanciti — Mon, 22 Oct 2018 20:03:42 GMT

I have many very frequent jobs (100 jobs run every 2 minutes)

I need a reference file to be loaded as a lookup into the load script. There is a high possibility that several jobs will attempt to load the same qvd at the same time.

I fear that one of these jobs will fail due to a file LOCK on the qvd because it is being run by another job.

How does Qlik Sense handle such a scenario? Will a job lock a qvd file until it is done with it? If so, is there anyway to avoid this other than attempting to ensure that the loads do not happen at the same time (which will be very difficult)?

Re: Several jobs loading the same QVD at the same time

marcus_sommer — Tue, 23 Oct 2018 11:30:50 GMT

It's quite rarely but I had already had the case that a multiple access at the same time of a certain qvd's led to lock-errors. It only happened if I played manually with several parallel instances because all my regular tasks are neatly chained. Therefore I wouldn't be surprised if it would fail randomly especially as you couldn't really control all the input/output to your storage.

One possibility to avoid errors might be to include an error-logic with ERRORMODE within the script maybe within a loop of 10 trials each and if an error occured just wait (sleep) n seconds and try it again before skipping this task or running into an error or doing anything else.

Another approach might be to set this file to READONLY within the filesystem or something similar - I never worked with these attributes but I could imagine that the OS behaviour of handling a file-access might be adjustable.

Also possible would be just to store n (parallel) copies of these qvd and each task which couldn't chained to another get its own copy.

- Marcus

Re: Several jobs loading the same QVD at the same time

jonathanciti — Tue, 23 Oct 2018 12:02:54 GMT

Thanks, I'm going to play with the ERRORMODE option since we will have quite a few apps that will access this lookup.

I'm wondering if there's another file format that might better allow for multiple concurrent reads, possibly a text file or simply put this in a database?

Re: Several jobs loading the same QVD at the same time

marcus_sommer — Tue, 23 Oct 2018 12:08:42 GMT

AFAIK another fileformat won't change anything because the file-locking is a feature of the OS but putting this lookup-table into a database would be an alternatively because they are designed for such use-cases.

- Marcus