QVD's getting locked during Job Reloads

[AnjaliRai]

Hello All,

Hope can get  a help here!

We are facing an issue in Qlik Sense Platform where few of the jobs are failing as the QVD files are getting locked.
Few observations:
1. The jobs are failing with an error (Failed to open file in write mode). When checked from Server, the QVD files are opened in READ mode only, still job fails.
2. The jobs fails with an error (Sharing Violation) sometimes. When checked from server, the files are open in WRITE mode but no other app/tool is using the files. None of our apps write back to QVD's, only reads.
3. No other jobs are using those QVD files, hence cannot be in use/locked by other applications or any tool.
4. When 'Qlik Engine service' is restarted, jobs are re-triggered. The issue is fixed.
5. Antivirus exclusion is in place, as suggested by Qlik.

This issue is quite frequent with our platform now a days.

Work around: We kill the the locked file session from server and re-trigger the job again. Or, we need to restart the Qlik Engine service to release the locking.

Qlik Sense Version: April 2020-13.72.3

Please suggest !

[marcus_sommer]

Maybe your storage / OS doesn't update the file-state always properly for example by having a heavy workload or if the storage isn't NTFS else any kind of NAS.

Another possibility are any system / third party tools/services which may access these files, like the windows shadow copies or similar.

- Marcus

[Brett_Bleess]

In addition to what Marcus said, I would also go get the Microsoft SysInternals tools, Process Monitor I believe is what you are going to want to check that file when things fail to see what process has the lock on it, as us opening in Read should not take an exclusive lock, so we need to find out what process is locking things.  One thing to check is something like Diskeeper, or other utilities that Marcus mentioned, may have added something new to the environment and that is doing things behind the scenes etc.  The Windows Event Viewer logs might have something to say too, System and Application I would think.  About the best I have to try to help you gather some further information that may help point to root cause.  Something has to be opening in write mode though which will keep us from reading, as I know in QlikView the same thing will happen, multiple reads from same file, no issue, but if something has the write lock, then reads will fail and vice versa.  Sorry I do not have something better but hopefully you can track down some further info.  The following article may help: https://support.qlik.com/articles/000039042

Regards,
Brett

[AnjaliRai]

Hi Marcus,

Yes, we had a meeting with our Server Team and Backup Team to see in case any locking issue at that end. But, unfortunately , we could not captured anything running as such application/tool or something which we suspect might lock the files.

We ran Process monitor as well, but no luck. Nothing strange can be captured from any tools we ran as part of investigation. 

My suspect is - any possibility Qlik is locking the files as every time we restart the Engine service, the issue is resolved.

How we proceed to resolve the issue, any suggestion would be very helpful.

[AnjaliRai]

Hi Brett,

Thank you!

Yes we looked into from that perspective as well as you have suggested. Even I am not able to figure out how files are locked in WRITE mode, as i am very sure none of applications open any QVD's in WRITE mode. They just READ QVD files while reloading.

To highlight, the job is failed but still the QVD file (which the job was using while running) can be seen open at server end in READ mode. When checked the log file, it says 'Cannot open file error' and hence failed.

We ran process monitor to check if any third party tool is locking but no luck. All folders are already excluded from Antivirus so we can rule out this option.

Kind of stuck not sure what can be exactly the root cause for the issue.

Please suggest!

[marcus_sommer]

Which kind of storage is in use? In the earlier days only NTFS storages were official supported - nowadays I don't know but I assume it hasn't changed. If NTFS  is in use it might be helpful to take a look on the changes within the NTFS metadata-journaling to find more about which and when any process has changed anything.

- Marcus

[ArnadoSandoval]

Hi @AnjaliRai 

I will add my two cents contributions after reading the replies and your answers, with some comments and questions:

• You wrote:

This issue is quite frequent with our platform now a days. It seems the frequency of the issue is increasing as well as there was a time when the issue never happen, so, Is it possible for you to idenfity a precise starting date for the issue? if that is possible, then Is it possible to know what enhancements were added to the Qlik Environment (even server environment)

We kill the the locked file session from server and re-trigger the job again.  Does it work? it won't make much sense if it does, but it will help us to know what actually happen when killing-restarting the offending job.

we need to restart the Qlik Engine service to release the locking. This is the most logical approach, but restarting the Qlik Engine service you are clearing any lock, 100% guarantee.

Facts:

• Those QVDs are updated at least once a day, in that case, some jobs wrote to them;
• Regarding your crashing (the one crashing due to the lock) applications; Do you know the offending locked QVD file triggering the crash? (by name, by example: Customer.qvd) Do you know all the QVDs they are reading; if you know all the read-only QVDs involved, you should also identify all the applications writing to them (most likely one application should be doing so). 

 I suspect one of the applications writing to these QVDs is crashing out, or entering a loop holding the QVD open; you should identify these applications and review their logic. (If you know the date when this issue started, and keep track of the Qlik Application changes -change log-, you may closely identify a culprit).

 How often are the affected applications crashing? once a day out of 10 daily runs, or all the time.

Once you estart the Qlik Engine service when the error happen again, the same day or the next day?

I had a similar issue about two years ago, our implementation had jobs running hourly, and some of them ran other jobs in a chain; the issue was related to two jobs compiting to write to the same QVD; We ended up implementing a sort of semaphore, but we need to understand your scenario better before findout the best course of action; right know we are trying to diagnostic the issue and understand your diagnosis as well.

Hope this helps,  

[Brett_Bleess]

I almost screwed you up there, in that I forgot this was Sense, I usually stay in the QlikView side for the most part, as I am not too good with the Sense side of things, at least not as good as I should be.  Only other thing of which I can think would be to be sure you have all the available patches applied for the release you are running, if you are missing any of those, would recommend applying to see if that may correct things.  I cannot recall if we had issues with QVD files getting stuck or not in Sense, apologies.  This also explains you having a hard time figuring out the file locks etc. too, sorry, my brain was stuck on how things work in QlikView, my bad.  I would suspect something is getting hung up in the Repository or something, and I honestly am not sure which log to even point you to as there are so many in Sense! 🙂  If you have not opened a support case on this one, you may want to do that to see if the techs can maybe have a look at the logs to see if they can spot something there, or if they are aware of any issues in the release you are running.  Again, apologies I am not more help, I really need to get get up-to-speed on Sense one of these days, it is just hard to keep both Sense and View information straight, which is why I have not been able to as yet.  

Regards,
Brett

[AnjaliRai]

Hi Marcus,

The storage in use is NTFS only. I had a chat with my backup and server team, they confirmed no recent changes were done from their end very recent. Though the issue we are facing is not daily, but happening all sudden ( can't say when it will happen next). Previously it started with apps related with only one stream but now, its happening for many apps present in other streams.

Just my observation: All the affected jobs loads data for countries ( like 20 countries or some), and the schedule is set as, based on one app success, the other runs, etc.

Regards,
Anjali Rai

[marcus_sommer]

I don't know how the server-reloads are implemented in Sense but in View each task runs an own instance of the qvb.exe. I assume in Sense it's quite similar. If really Qlik is locking the qvd's after the task has successfully finished those instances might be further active and this should be visible within the taskmanager.

- Marcus