Rim Node reporting 4 of 5 services unreachable in QMC

[montanablue2]

I was wondering if anyone has seen this issue before. I had to reissue qlik client cert and local computer Qlik cert for a rim node to get it to re-register with QMC central node but now QMC can only see the Repository service on that rim node. This is for Qlik Sense May 2021 edition.

I don't see any errors in the Proxy logs or Engine logs on rim node. The Repository logs on rim node do have some errors, specifically these errors:

19	20231205T003734.171-0700	ERROR	us2-prd-qsp-a02	Security.Repository.Qlik.Sense.Common.Security.Cryptography.CryptoKey	40	ed276945-f100-4b31-be4b-1c6b9abf00b7	CLOUD\svc-us2-prd-qlik	When decrypting secrets: 	Checksum error	   at Qlik.Sense.Common.Security.Cryptography.CryptoKey.StripAndVerifyChecksum(Byte[] input)↵↓   at Qlik.Sense.Common.Security.Cryptography.CryptoKey.ProcessBytes(Byte[] input, Byte[]& initializationVector, Boolean encrypt)		ed276945-f100-4b31-be4b-1c6b9abf00b7
20	20231205T003734.171-0700	ERROR	us2-prd-qsp-a02	Security.Repository.Repository.Domain.Util.EncryptionHelper	40	7473ed09-d728-42b9-b889-3ecc254a16ed	CLOUD\svc-us2-prd-qlik	Not possible to decrypt encrypted string in database; this is probably due to the secrets key on server certificate has been changed and the old database value hasn't; sending back empty result and leaving the value unchanged in database	Checksum error	   at Qlik.Sense.Common.Security.Cryptography.CryptoKey.StripAndVerifyChecksum(Byte[] input)↵↓   at Qlik.Sense.Common.Security.Cryptography.CryptoKey.ProcessBytes(Byte[] input, Byte[]& initializationVector, Boolean encrypt)↵↓   at Qlik.Sense.Common.Security.Cryptography.CryptoKey.DecryptToBytes(Byte[] bytes)↵↓   at Qlik.Sense.Common.Security.Cryptography.CryptoKey.Decrypt(Byte[] bytes)↵↓   at Repository.Domain.Util.EncryptionHelper.TryDecryptTo[T](Byte[] encryptedBytes, T emptyResult, Func`2 decryptToT)		7473ed09-d728-42b9-b889-3ecc254a16ed

2581	20231205T145039.470-0700	INFO	us2-prd-qsp-a02	System.Repository.Repository.Core.Repository.Common.ModelRepository`1[[Repository.Domain.Model.ServerNodeHeartbeat, Repository.Domain, Version=28.0.6.0, Culture=neutral, PublicKeyToken=null]]	137	52d0c53a-91b0-4e4c-8873-1cde642b6140	CLOUD\svc-us2-prd-qlik	Database object has newer timestamp than object being modified; pim4: 63837409831846, itms: 63837409824458, ityp: Repository.Domain.Model.ServerNodeHeartbeat				52d0c53a-91b0-4e4c-8873-1cde642b6140
2582	20231205T145039.470-0700	ERROR	us2-prd-qsp-a02	System.Repository.Repository.Core.Status.HeartbeatUpdater	137	0b6b4b46-2446-40f0-8b46-dec31a80bdaf	CLOUD\svc-us2-prd-qlik	Failed to update heartbeat for Repository service status on node us2-prd-qsc-a01.diamond-prd-bi.prd.diamond.cloud.kyriba.com		   at Repository.Core.Repository.Common.AbstractModelRepository.CheckConflicts(IRepositoryObject persistedItem, DateTime itemModifiedDate)↵↓   at Repository.Core.Repository.Common.ModelRepository`1.UpdateMany(IEnumerable`1 items, Boolean appendPrivileges, Int64 privilegesFilter)↵↓   at Repository.Core.Repository.Common.ModelRepository`1.Update(T item, Boolean appendPrivileges, Int64 privilegesFilter)↵↓   at Repository.Core.Repository.Common.SecurityAwareRepository.RunWithElevatedOrNoSecurity[TResult](Func`1 func)↵↓   at Repository.Core.Status.HeartbeatUpdater.<PerformExecution>b__6_0(ISecurityAwareRepository repository)		0b6b4b46-2446-40f0-8b46-dec31a80bdaf
2583	20231205T145124.495-0700	INFO	us2-prd-qsp-a02	System.Repository.Repository.Core.Repository.Common.ModelRepository`1[[Repository.Domain.Model.ServerNodeHeartbeat, Repository.Domain, Version=28.0.6.0, Culture=neutral, PublicKeyToken=null]]	134	df668bca-d812-4d1b-809e-e239005a1058	CLOUD\svc-us2-prd-qlik	Database object has newer timestamp than object being modified; pim4: 63837409876860, itms: 63837409869479, ityp: Repository.Domain.Model.ServerNodeHeartbeat				df668bca-d812-4d1b-809e-e239005a1058
2584	20231205T145124.495-0700	ERROR	us2-prd-qsp-a02	System.Repository.Repository.Core.Status.HeartbeatUpdater	134	bb42892d-d692-4158-a532-8f634b7e8e43	CLOUD\svc-us2-prd-qlik	Failed to update heartbeat for Repository service status on node us2-prd-qsc-a01.diamond-prd-bi.prd.diamond.cloud.kyriba.com		   at Repository.Core.Repository.Common.AbstractModelRepository.CheckConflicts(IRepositoryObject persistedItem, DateTime itemModifiedDate)↵↓   at Repository.Core.Repository.Common.ModelRepository`1.UpdateMany(IEnumerable`1 items, Boolean appendPrivileges, Int64 privilegesFilter)↵↓   at Repository.Core.Repository.Common.ModelRepository`1.Update(T item, Boolean appendPrivileges, Int64 privilegesFilter)↵↓   at Repository.Core.Repository.Common.SecurityAwareRepository.RunWithElevatedOrNoSecurity[TResult](Func`1 func)↵↓   at Repository.Core.Status.HeartbeatUpdater.<PerformExecution>b__6_0(ISecurityAwareRepository repository)		bb42892d-d692-4158-a532-8f634b7e8e43

[Maria_Halley · Accepted solution]

@montanablue2 

When you re-created the certificates did you follow these instructions? 

https://community.qlik.com/t5/Official-Support-Articles/How-to-recreate-or-just-delete-certificates-...

[montanablue2]

Hi @Maria_Halley thank you for responding so quickly! Since it was a Rim node having the issue I followed a process similar to the process https://community.qlik.com/t5/Official-Support-Articles/Rim-node-not-communicating-with-central-node... that is also referenced in the link you sent. I may not have done it exactly similar so I will try to go through the process in that link and see what happens

[montanablue2 · Accepted solution]

I followed the process as in https://community.qlik.com/t5/Official-Support-Articles/Rim-node-not-communicating-with-central-node...  and it successfully redistributes the certs and appears to have resolved the issue. Thank you for your guidance!