Qlik Community

Ask a Question

App Development

Discussion board where members can learn more about Qlik Sense App Development and Usage.

Announcements
April 22, 2PM EST: Learn about GeoOperations in Qlik Sense SaaS READ MORE
cancel
Showing results for 
Search instead for 
Did you mean: 
AnjaliRai
Contributor III
Contributor III

QVD's getting locked during Job Reloads

Hello All,

Hope can get  a help here!

We are facing an issue in Qlik Sense Platform where few of the jobs are failing as the QVD files are getting locked.
Few observations:
1. The jobs are failing with an error (Failed to open file in write mode). When checked from Server, the QVD files are opened in READ mode only, still job fails.
2. The jobs fails with an error (Sharing Violation) sometimes. When checked from server, the files are open in WRITE mode but no other app/tool is using the files. None of our apps write back to QVD's, only reads.
3. No other jobs are using those QVD files, hence cannot be in use/locked by other applications or any tool.
4. When 'Qlik Engine service' is restarted, jobs are re-triggered. The issue is fixed.
5. Antivirus exclusion is in place, as suggested by Qlik.

This issue is quite frequent with our platform now a days.

Work around: We kill the the locked file session from server and re-trigger the job again. Or, we need to restart the Qlik Engine service to release the locking.

Qlik Sense Version: April 2020-13.72.3

Please suggest !

Anjali Rai
Labels (1)
1 Solution

Accepted Solutions
AnjaliRai
Contributor III
Contributor III
Author

Hi @ArnadoSandoval ,

The issue reported is a bug in Qlik. Received an update from Qlik Support Team recently ,Bug ID- QLIK-102376.

Though there is a work around which I have already applied on our servers. As per Qlik, the work around is to introduce the below settings over server that will turn off this optimization to alleviate the QVD Locking error message.

[Settings 7]
EnableQvdAsyncWrite=0

Technote: https://support.qlik.com/articles/000111886?_ga=2.192177130.1302063410.1600674128-1149973368.1591644...

--Anjali Rai

Anjali Rai

View solution in original post

28 Replies
marcus_sommer
MVP & Luminary
MVP & Luminary

Maybe your storage / OS doesn't update the file-state always properly for example by having a heavy workload or if the storage isn't NTFS else any kind of NAS.

Another possibility are any system / third party tools/services which may access these files, like the windows shadow copies or similar.

- Marcus

Brett_Bleess
Support (Former)
Support (Former)

In addition to what Marcus said, I would also go get the Microsoft SysInternals tools, Process Monitor I believe is what you are going to want to check that file when things fail to see what process has the lock on it, as us opening in Read should not take an exclusive lock, so we need to find out what process is locking things.  One thing to check is something like Diskeeper, or other utilities that Marcus mentioned, may have added something new to the environment and that is doing things behind the scenes etc.  The Windows Event Viewer logs might have something to say too, System and Application I would think.  About the best I have to try to help you gather some further information that may help point to root cause.  Something has to be opening in write mode though which will keep us from reading, as I know in QlikView the same thing will happen, multiple reads from same file, no issue, but if something has the write lock, then reads will fail and vice versa.  Sorry I do not have something better but hopefully you can track down some further info.  The following article may help: https://support.qlik.com/articles/000039042

Regards,
Brett

To help users find verified answers, please do not forget to use the "Accept as Solution" button on any post(s) that helped you resolve your problem or question.
I now work a compressed schedule, Tuesday, Wednesday and Thursday, so those will be the days I will reply to any follow-up posts.
AnjaliRai
Contributor III
Contributor III
Author

Hi Marcus,

Yes, we had a meeting with our Server Team and Backup Team to see in case any locking issue at that end. But, unfortunately , we could not captured anything running as such application/tool or something which we suspect might lock the files.

We ran Process monitor as well, but no luck. Nothing strange can be captured from any tools we ran as part of investigation. 

My suspect is - any possibility Qlik is locking the files as every time we restart the Engine service, the issue is resolved.

How we proceed to resolve the issue, any suggestion would be very helpful.

Anjali Rai
AnjaliRai
Contributor III
Contributor III
Author

Hi Brett,

Thank you!

Yes we looked into from that perspective as well as you have suggested. Even I am not able to figure out how files are locked in WRITE mode, as i am very sure none of applications open any QVD's in WRITE mode. They just READ QVD files while reloading.

To highlight, the job is failed but still the QVD file (which the job was using while running) can be seen open at server end in READ mode. When checked the log file, it says 'Cannot open file error' and hence failed.

We ran process monitor to check if any third party tool is locking but no luck. All folders are already excluded from Antivirus so we can rule out this option.

Kind of stuck not sure what can be exactly the root cause for the issue.

Please suggest!

Anjali Rai
marcus_sommer
MVP & Luminary
MVP & Luminary

Which kind of storage is in use? In the earlier days only NTFS storages were official supported - nowadays I don't know but I assume it hasn't changed. If NTFS  is in use it might be helpful to take a look on the changes within the NTFS metadata-journaling to find more about which and when any process has changed anything.

- Marcus

ArnadoSandoval
Specialist II
Specialist II

Hi @AnjaliRai 

I will add my two cents contributions after reading the replies and your answers, with some comments and questions:

  • You wrote:

This issue is quite frequent with our platform now a days. It seems the frequency of the issue is increasing as well as there was a time when the issue never happen, so, Is it possible for you to idenfity a precise starting date for the issue? if that is possible, then Is it possible to know what enhancements were added to the Qlik Environment (even server environment)

We kill the the locked file session from server and re-trigger the job again.  Does it work? it won't make much sense if it does, but it will help us to know what actually happen when killing-restarting the offending job.

we need to restart the Qlik Engine service to release the locking. This is the most logical approach, but restarting the Qlik Engine service you are clearing any lock, 100% guarantee.

Facts:

  • Those QVDs are updated at least once a day, in that case, some jobs wrote to them;
  • Regarding your crashing (the one crashing due to the lock) applications; Do you know the offending locked QVD file triggering the crash? (by name, by example: Customer.qvd) Do you know all the QVDs they are reading; if you know all the read-only QVDs involved, you should also identify all the applications writing to them (most likely one application should be doing so). 

 I suspect one of the applications writing to these QVDs is crashing out, or entering a loop holding the QVD open; you should identify these applications and review their logic. (If you know the date when this issue started, and keep track of the Qlik Application changes -change log-, you may closely identify a culprit).

 How often are the affected applications crashing? once a day out of 10 daily runs, or all the time.

Once you estart the Qlik Engine service when the error happen again, the same day or the next day?

I had a similar issue about two years ago, our implementation had jobs running hourly, and some of them ran other jobs in a chain; the issue was related to two jobs compiting to write to the same QVD; We ended up implementing a sort of semaphore, but we need to understand your scenario better before findout the best course of action; right know we are trying to diagnostic the issue and understand your diagnosis as well.

Hope this helps,  

 

Arnaldo Sandoval
A journey of a thousand miles begins with a single step.
Brett_Bleess
Support (Former)
Support (Former)

I almost screwed you up there, in that I forgot this was Sense, I usually stay in the QlikView side for the most part, as I am not too good with the Sense side of things, at least not as good as I should be.  Only other thing of which I can think would be to be sure you have all the available patches applied for the release you are running, if you are missing any of those, would recommend applying to see if that may correct things.  I cannot recall if we had issues with QVD files getting stuck or not in Sense, apologies.  This also explains you having a hard time figuring out the file locks etc. too, sorry, my brain was stuck on how things work in QlikView, my bad.  I would suspect something is getting hung up in the Repository or something, and I honestly am not sure which log to even point you to as there are so many in Sense! 🙂  If you have not opened a support case on this one, you may want to do that to see if the techs can maybe have a look at the logs to see if they can spot something there, or if they are aware of any issues in the release you are running.  Again, apologies I am not more help, I really need to get get up-to-speed on Sense one of these days, it is just hard to keep both Sense and View information straight, which is why I have not been able to as yet.  

Regards,
Brett

To help users find verified answers, please do not forget to use the "Accept as Solution" button on any post(s) that helped you resolve your problem or question.
I now work a compressed schedule, Tuesday, Wednesday and Thursday, so those will be the days I will reply to any follow-up posts.
AnjaliRai
Contributor III
Contributor III
Author

Hi Marcus,

The storage in use is NTFS only. I had a chat with my backup and server team, they confirmed no recent changes were done from their end very recent. Though the issue we are facing is not daily, but happening all sudden ( can't say when it will happen next). Previously it started with apps related with only one stream but now, its happening for many apps present in other streams.

Just my observation: All the affected jobs loads data for countries ( like 20 countries or some), and the schedule is set as, based on one app success, the other runs, etc.

Regards,
Anjali Rai

Anjali Rai
marcus_sommer
MVP & Luminary
MVP & Luminary

I don't know how the server-reloads are implemented in Sense but in View each task runs an own instance of the qvb.exe. I assume in Sense it's quite similar. If really Qlik is locking the qvd's after the task has successfully finished those instances might be further active and this should be visible within the taskmanager.

- Marcus