Failed Qlik Sense Enterprise on Windows August 2021 and May 2021 - Installing AWS AMI: QSE_August/May_2021_BYOL

[twales]

Concerns failures on both August and May 2021 AMI's

August 2021 Update 2:

QSE_August_2021_BYOL_Upd2_1634038484-590d28f7-5d64-4a03-ad0d-32e54e210929 (ami-0e578d0294bddd205)

Concerns Qlik Sense Enterprise on Windows May 2021

Installed QWS AMI: QSE_May_2021_BYOL_1627384765
Windows 2019 server, single server node

Issue: 2021-08-31 16:24:44.829 UTC [924] FATAL: password authentication failed for user "qliksenserepository" 2021-08-31 16:24:44.829 UTC [924] DETAIL: Password does not match for user "qliksenserepository". Connection matched pg_hba.conf line 84: "host all all 127.0.0.1/32 md5"

I have two other installations both running monitors(lic and ops), both had the same issue until I changed the qliksenserepository password. They were both older April 2020 versions.

After a brand new install for this issue the event log was filled with the usual postgres error above.
QMC is working and preparing to finish setup's.

I performed the following steps to try and correct the qliksenserepository authentication error:

(note this is a summary of multiple self-help topics on this same issue, however many of those suggestions I used again predate by years this new version.)

Services turned on/off as needed. logging service always off now.

Verified I could logon as both postgres and qliksenserepository(both QLogs and QSMQ) worked with same pw.
Verified pw correct in qlik utils
Verified all services pinging in qlik utils, only 3 ports not but I assume those were not used
such as 80.

Ran the PS utilities to correct cached PW's as line by line and the loop method.
2019 Server eg:
Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
cd "C:\Program Files\Qlik\Sense\QseEventProcessor"
.\Configure-Service.ps1 localhost 4432 qliksenserepository thepw
etc.

FIPS security is disabled by default.
Checked data connections pw's to reset but no userid's on install, only the qlogs_reader userid. Didn't change monitors to use service account at this time.
At one point had the failure to decrypt password in the system log but that went away.

Finally when all of that failed I changed the qliksenserepository password and the errors in event logs still persist.

Main errors below, repeat ~1000 times.

Someone should be able to tell the exact code causing this issue from below.

What is the resolution?

Thanks

log errors:

Main source:


C:\ProgramData\Qlik\Sense\Log\HybridDeploymentService\Trace\IP-0A0009C4_HybridDeploymentService_20210830.log (983 hits)
3 20210830T132612.771-04:00 INFO IP-0A0009C4 9 IP-0A0009C4\QSServiceAccount "Request starting HTTP/1.1 GET https://ip-10-0-9-196.ec2.internal:5927/v1/deployments " 7620
4 20210830T132616.059-04:00 ERROR IP-0A0009C4 9 IP-0A0009C4\QSServiceAccount Connection id ""0HMBBRD1L2FQ8"", Request id ""0HMBBRD1L2FQ8:00000001"": An unhandled exception was thrown by the application. Npgsql.PostgresException (0x80004005): 28P01: password authentication failed for user "qliksenserepository"↵↓ at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext()↵↓--- End of stack trace from previous location where exception was thrown ---↵↓ at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext()↵↓--- End of stack trace from previous location where exception was thrown ---↵↓ at Npgsql.NpgsqlConnector.AuthenticateMD5(String username, Byte[] salt, Boolean async)↵↓ at Npgsql.NpgsqlConnector.Authenticate(String username, NpgsqlTimeout timeout, Boolean async)↵↓ at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)↵↓ at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)↵↓ at Npgsql.NpgsqlConnection.<>c__DisplayClass32_0.<<Open>g__OpenLong|0>d.MoveNext()↵↓--- End of stack trace from previous location where exception was thrown ---↵↓ at Npgsql.NpgsqlConnection.Open()↵↓ at Marten.Services.ManagedConnection.Execute[T](Func`2 func)↵↓ at Marten.Schema.DatabaseSchemaGenerator.Generate(StoreOptions options, String[] schemaNames)↵↓ at Marten.DocumentStore..ctor(StoreOptions options)↵↓ at Marten.DocumentStore.For(Action`1 configure)↵↓ at HybridDeploymentService.Startup.<>c__DisplayClass9_0.<ConfigureServices>b__4(IServiceProvider p)↵↓ at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProviderEngineScope scope)↵↓ at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, ServiceProviderEngineScope scope)↵↓ at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitTransient(TransientCallSite transientCallSite, ServiceProviderEngineScope scope)↵↓ at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService(IServiceProvider sp, Type type, Type requiredBy, Boolean isDefaultParameterRequired)↵↓ at lambda_method(Closure , IServiceProvider , Object[] )↵↓ at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)↵↓ at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)↵↓ at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()↵↓ at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()↵↓ at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)↵↓ at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)↵↓ at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()↵↓ at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()↵↓ at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)↵↓ at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)↵↓ at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIIndexMiddleware.Invoke(HttpContext httpContext)↵↓ at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext)↵↓ at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)↵↓ at HybridDeploymentService.Startup.<>c.<<Configure>b__10_0>d.MoveNext()↵↓--- End of stack trace from previous location where exception was thrown ---↵↓ at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application) 7620

Another source:

C:\ProgramData\Qlik\Sense\Log\QlikMobilityRegistrar\202108271520188990400.log (1 hit)
Line 11: 10 20210827T152048.457-0400 ip-10-0-9-196.ec2.internal 1.1.0 Global INFO password authentication failed for user "qliksenserepository"

[Sebastian_Linser · Accepted solution]

Hello @twales 

i am glad we can provide you with a workaround.

Search for all Configure-Service.ps1 files in C:\program files\qlik\sense 

replace the line

[string]$postgresHome = "$senseInstallPath\Repository\PostgreSQL\9.6",

with 

[string]$postgresHome = "$senseInstallPath\Repository\PostgreSQL\12.5",

after that reset the passwords once more for all the Micro Services and restart the Qlik Sense Services under services.msc

Since we use PostgreSQL 12.5 in a fresh install beginning form May2021 this one was not updated at all.

[twales · Accepted solution]

Thanks very much for your persistence in getting this resolved.   It was even affecting the CPU utilization, now that's no longer an issue nor are there error events for qsr.

[twales]

FYI,

Just installed the Nov 2021, 11/15 version QSE AMI and still the issue persists in ps1's and all need updating.

Thanks

[twales · Accepted solution]

Now installing the 02/2022 (Feb) AMI and the issue still persists.  (Major event failures)

[string]$postgresHome = "$senseInstallPath\Repository\PostgreSQL\9.6",

In case anyone needs the same fix:

Essentially you need to find all the files named Configure-Service.ps1 in the C:\ProgramData\Qlik\Sense directory tree.  Just search for Configure-Service.ps1 and they will all appear.

Open each one and change the line below(it will be near the top) to:

[string]$postgresHome = "$senseInstallPath\Repository\PostgreSQL\12.5",

You're just changing the 9.6 to 12.5, as that's the current Qlik Postgres version using.

Once corrected all ps1's do:

Open powershell window as administrator and run each of the scripts below:

The first one is commented b/c it doesn't exist on this instance.

Set-ExecutionPolicy RemoteSigned -Scope CurrentUser

-----cd "C:\Program Files\Qlik\Sense\QseEventProcessor\install"
------.\Configure-Service.ps1 localhost 4432 qliksenserepository YourPassword

cd "C:\Program Files\Qlik\Sense\AppDistributionService"
.\Configure-Service.ps1 localhost 4432 qliksenserepository YourPassword

cd "C:\Program Files\Qlik\Sense\HybridDeploymentService"
.\Configure-Service.ps1 localhost 4432 qliksenserepository YourPassword

cd "C:\Program Files\Qlik\Sense\Licenses"
.\Configure-Service.ps1 localhost 4432 qliksenserepository YourPassword

cd "C:\Program Files\Qlik\Sense\NotifierService\install"
.\Configure-Service.ps1 localhost 4432 qliksenserepository YourPassword

cd "C:\Program Files\Qlik\Sense\MobilityRegistrarService\install"
.\Configure-Service.ps1 localhost 4432 qliksenserepository YourPassword

cd "C:\Program Files\Qlik\Sense\PrecedentsService\install"
.\Configure-Service.ps1 localhost 4432 qliksenserepository YourPassword

cd "C:\Program Files\Qlik\Sense\NLAppSearch\install"
.\Configure-Service.ps1 localhost 4432 qliksenserepository YourPassword

cd "C:\Program Files\Qlik\Sense\NLBroker\install"
.\Configure-Service.ps1 localhost 4432 qliksenserepository YourPassword

cd "C:\Program Files\Qlik\Sense\DataPrepService\install"
.\Configure-Service.ps1 localhost 4432 qliksenserepository YourPassword

After this is completed, stop and restart all Qlik services.  Note I keep my running when making all the changes and found it makes no difference.

Once restarted, check the event viewer app logs and there should be no more massive errors logged, in fact none will appear, just info and warnings.

[Sebastian_Linser]

Hello @twales,

you are correct, the issue can only be fixed in an initial release and the one where this will make it in the installer is May 2022.

best regards

Sebastian

[twales]

Hi Sebastian,

Well if you're there, nothing changed.  I just installed the 8/2023 BYOL AMI on a new server and both issues, the incorrect Postgres directory(12.5) and having to battle the authentication qsr issues filling up events.   Changed the folder to 14 and the qsr finally went away after running the ps1's a few times using the same PW I started with.

And this one had another issue never before faced on any other server, the web sockets TO. Note it's the same AWS network used for all servers with minor variations but basically identical.  I did reply to another thread on this topic  and they stated the culprit is network appliances.  Well AWS has a default 350s TO for all NAT's and LB's etc.  So it's not that. 

Had to set it to 30s on ping to stop the disconnects on client accessing applications, myself in this instance.  Tried 40 but it will still occur.  Tested both Chrome and Firefox.

Thanks,