Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
andre_ficken
Partner - Creator
Partner - Creator

Services dropped. No restart possible, error 1607

LS,

Last Sunday we encoutered an error situation on a production Qlik server.

The Distribution service went down and would not start anymore continuously reporting error1607.

After some internet searching I found a possible solution that suggested shutting down the services, rename the Distribution service folder under ProgramData\Qliktech and after that restart the services.

We had to set the Qlik services to manual start first and restart the server in order to establish a controlled environment. After the rename the distribution service  started, but the same error now popped up on the Management service. We repeated the same steps as we took on the distribution service, stop all services, rename ManagementService folder and restart service. After this rename and restart all services kept running and the management console was accessible again.

All licensing info as well as NAMED and DOC CAL assignments were kept save, but we lost the entire reload schedule that we had constructed on the server (we are not using publisher). It took us about 45 minutes to rebuild it and after that all worked as a treat and kept working.

I was wondering how to find the cause of the service shutdown. We found some info in the event viewer when the service was re-started in the error situation on sunday:

shutting down the service since there are data with invalid encryption. To continue, activate the 'EraseUndecryptableData' config flag by setting it to true in the config file to erase the invalid data.

Found 52 item(s) with invalid encryption. Follows here in the log file:

Exception while decrypting. EncryptedText length=16 chars Exception=System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.

||    at System.Security.Cryptography.CapiSymmetricAlgorithm.DepadBlock(Byte[] block, Int32 offset, Int32 count) ||  

at System.Security.Cryptography.CapiSymmetricAlgorithm.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount) ||  

at System.Security.Cryptography.CryptoStream.FlushFinalBlock() ||    at System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing) ||  

at System.IO.Stream.Close() ||    at SolutionGlobal.Security.CryptoAlgorithm.Decrypt(EncryptedSecret encryptedSecret, Boolean isCurrentAlgorithm)

The other thing I would like to know is how we can prevent the reload schedule to be lost again. We kept the renamed folders for Distribution and Management service so the files are still accessible for inspection and analysis. Any ideas are welkcome and appreciated.

3 Replies
Chris_Rice
Support
Support

Your best bet here is to open a case with Qlik Support and ask for a Root Cause Analysis, especially if you still have all of the files/folders/logs associated with this incident. 

https://support.qlik.com/QS_CoveoCaseWizard#t=All&sort=relevancy

Based on what you've written, it sounds like file corruption (which re-creating the Distribution Service folder fixed). Continued file corruption caused by an anti-virus or backup utility would explain why this happened a second time.

As for keeping your reload schedule: I don't remember off the top of my head where that is stored in a non-Publisher environment, but it should be fairly trivial to find by creating a task and seeing which file/folder gets updated inside of ProgramData. As long as you back that up, you should be able to replace it in events of emergency.

Sonja_Bauernfeind
Digital Support
Digital Support

@Chris_Rice The reload schedule is stored in the QVPR, so C:\ProgramData\QlikTech\ManagementService\QVPR

The file is called DocumentTask.xml and it stores all types of tasks.

They look like this:

 <DocumentTask Name="File Name.qvw" DistributionGroup="" Enabled="true" Description="" NameIsAutoGenerated="false" AllowPluginClient="true" AllowMobileClient="false" etc etc etc />

Don't forget to Like posts and use the "Accept as Solution" button on content that answered your question! Thanks 🙂
Brett_Bleess
Former Employee
Former Employee

I just wanted to chime in on this one and note the most likely cause of things potentially given what may have happened given weekend.  I suspect there may have been reboots of some of the servers, and if so, there is a key thing a lot of customers forget, and that is if you are using file shares for the Application Data folders for the Qlik Services, we need to be sure that File Server is the last server down and first server back up to ensure the services shutdown properly when their servers are taking down, and they start properly on the restart.  Hopefully this may save some folks some major pains.  The first time I ran into this, it took us a number of weeks to figure out what was happening, so again, in summary, if you have the Qlik Services pointed to a file share for their Application Data folder locations, or have modified the exe.config file and the AppData path there to a file share, you need to be sure the folks doing maintenance on the servers know to take the file server down last and be sure it is back up before they try to bring the other Qlik servers back online.

Oh, and recreating the ManagementService folder was ok, but what you needed to do was restore your QVPR folder from a backup zip file, those are located in C:\ProgramData\QlikTech\ManagementService\QVPR\Backups by default, location can be changed in the QMC\System\Setup\Management Service resource\Repository tab settings, or if you modified the exe.config AppData path, that would be the correct place to likely find things.  Hopefully this clarification may help someone else.

Regards,
Brett

To help users find verified answers, please do not forget to use the "Accept as Solution" button on any post(s) that helped you resolve your problem or question.
I now work a compressed schedule, Tuesday, Wednesday and Thursday, so those will be the days I will reply to any follow-up posts.