Failed Qlik Sense Enterprise on Windows August 2021 and May 2021 - Installing AWS AMI: QSE_August/May_2021_BYOL

[twales]

Concerns failures on both August and May 2021 AMI's

August 2021 Update 2:

QSE_August_2021_BYOL_Upd2_1634038484-590d28f7-5d64-4a03-ad0d-32e54e210929 (ami-0e578d0294bddd205)

Concerns Qlik Sense Enterprise on Windows May 2021

Installed QWS AMI: QSE_May_2021_BYOL_1627384765
Windows 2019 server, single server node

Issue: 2021-08-31 16:24:44.829 UTC [924] FATAL: password authentication failed for user "qliksenserepository" 2021-08-31 16:24:44.829 UTC [924] DETAIL: Password does not match for user "qliksenserepository". Connection matched pg_hba.conf line 84: "host all all 127.0.0.1/32 md5"

I have two other installations both running monitors(lic and ops), both had the same issue until I changed the qliksenserepository password. They were both older April 2020 versions.

After a brand new install for this issue the event log was filled with the usual postgres error above.
QMC is working and preparing to finish setup's.

I performed the following steps to try and correct the qliksenserepository authentication error:

(note this is a summary of multiple self-help topics on this same issue, however many of those suggestions I used again predate by years this new version.)

Services turned on/off as needed. logging service always off now.

Verified I could logon as both postgres and qliksenserepository(both QLogs and QSMQ) worked with same pw.
Verified pw correct in qlik utils
Verified all services pinging in qlik utils, only 3 ports not but I assume those were not used
such as 80.

Ran the PS utilities to correct cached PW's as line by line and the loop method.
2019 Server eg:
Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
cd "C:\Program Files\Qlik\Sense\QseEventProcessor"
.\Configure-Service.ps1 localhost 4432 qliksenserepository thepw
etc.

FIPS security is disabled by default.
Checked data connections pw's to reset but no userid's on install, only the qlogs_reader userid. Didn't change monitors to use service account at this time.
At one point had the failure to decrypt password in the system log but that went away.

Finally when all of that failed I changed the qliksenserepository password and the errors in event logs still persist.

Main errors below, repeat ~1000 times.

Someone should be able to tell the exact code causing this issue from below.

What is the resolution?

Thanks

log errors:

Main source:


C:\ProgramData\Qlik\Sense\Log\HybridDeploymentService\Trace\IP-0A0009C4_HybridDeploymentService_20210830.log (983 hits)
3 20210830T132612.771-04:00 INFO IP-0A0009C4 9 IP-0A0009C4\QSServiceAccount "Request starting HTTP/1.1 GET https://ip-10-0-9-196.ec2.internal:5927/v1/deployments " 7620
4 20210830T132616.059-04:00 ERROR IP-0A0009C4 9 IP-0A0009C4\QSServiceAccount Connection id ""0HMBBRD1L2FQ8"", Request id ""0HMBBRD1L2FQ8:00000001"": An unhandled exception was thrown by the application. Npgsql.PostgresException (0x80004005): 28P01: password authentication failed for user "qliksenserepository"↵↓ at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext()↵↓--- End of stack trace from previous location where exception was thrown ---↵↓ at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext()↵↓--- End of stack trace from previous location where exception was thrown ---↵↓ at Npgsql.NpgsqlConnector.AuthenticateMD5(String username, Byte[] salt, Boolean async)↵↓ at Npgsql.NpgsqlConnector.Authenticate(String username, NpgsqlTimeout timeout, Boolean async)↵↓ at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)↵↓ at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)↵↓ at Npgsql.NpgsqlConnection.<>c__DisplayClass32_0.<<Open>g__OpenLong|0>d.MoveNext()↵↓--- End of stack trace from previous location where exception was thrown ---↵↓ at Npgsql.NpgsqlConnection.Open()↵↓ at Marten.Services.ManagedConnection.Execute[T](Func`2 func)↵↓ at Marten.Schema.DatabaseSchemaGenerator.Generate(StoreOptions options, String[] schemaNames)↵↓ at Marten.DocumentStore..ctor(StoreOptions options)↵↓ at Marten.DocumentStore.For(Action`1 configure)↵↓ at HybridDeploymentService.Startup.<>c__DisplayClass9_0.<ConfigureServices>b__4(IServiceProvider p)↵↓ at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProviderEngineScope scope)↵↓ at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, ServiceProviderEngineScope scope)↵↓ at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitTransient(TransientCallSite transientCallSite, ServiceProviderEngineScope scope)↵↓ at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService(IServiceProvider sp, Type type, Type requiredBy, Boolean isDefaultParameterRequired)↵↓ at lambda_method(Closure , IServiceProvider , Object[] )↵↓ at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)↵↓ at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)↵↓ at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()↵↓ at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()↵↓ at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)↵↓ at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)↵↓ at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()↵↓ at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()↵↓ at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)↵↓ at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)↵↓ at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIIndexMiddleware.Invoke(HttpContext httpContext)↵↓ at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext)↵↓ at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)↵↓ at HybridDeploymentService.Startup.<>c.<<Configure>b__10_0>d.MoveNext()↵↓--- End of stack trace from previous location where exception was thrown ---↵↓ at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application) 7620

Another source:

C:\ProgramData\Qlik\Sense\Log\QlikMobilityRegistrar\202108271520188990400.log (1 hit)
Line 11: 10 20210827T152048.457-0400 ip-10-0-9-196.ec2.internal 1.1.0 Global INFO password authentication failed for user "qliksenserepository"

[Sebastian_Linser]

Hello @twales 

That one is easy you need to update all micro services from the service dispatcher as well since they have the wrong initial password.

Do to that search for "configure-service.ps1" in  c:\program files\qlik\sense

Run all of them and adjust the password for the qliksenserepository to the new password you set.

Then restart the service dispatcher and the error should be gone.

Then on Monday please log a ticket so we can have the backend team adjust the password in the AMI instance.

Best regards

Sebastian

[twales]

Yes, most of those ps1's were run before I found 3 more directories and eliminated 1 or 2 not existing anymore.  Although all except one had errors on execution.  Wherever there was an access denied (2 or 3) I gave myself perms to the path, not any one file, just the directory tree.

All but one had some sort of exception in accessing some file, not found, etc. In fact not one of the config.params existed nor .conf's.  There was one that did exist but there were errors on executing the file in the script. So all of the ones taht state access denied on a file, they don't exist in the path specified.

For example, one states:

>> Write-PgPassFile : Exception calling "WriteAllLines" with "3" argument(s): "Access to the path 'C:\Program Files\Qlik\Sense\PrecedentsService\install\pgpass.conf' is denied."

Well pgpass.conf isn't a file that exists.

In any event none of this is working and it's more of a mess than when I corrected our two 2016 servers for this same issue.

The installs are a hot mess, I wish they would just fix them, period.

So can this be corrected, is there a more updated version w/install fixes?

Thanks

& "C:\Program Files\Qlik\Sense\AppDistributionService\Configure-Service.ps1" localhost 4432 qliksenserepository thepw@

& "C:\Program Files\Qlik\Sense\HybridDeploymentService\Configure-Service.ps1" localhost 4432 qliksenserepository thepw@

& "C:\Program Files\Qlik\Sense\Licenses\Configure-Service.ps1" localhost 4432 qliksenserepository thepw@

&"C:\Program Files\Qlik\Sense\NotifierService\install\Configure-Service.ps1" localhost 4432 qliksenserepository thepw@

& "C:\Program Files\Qlik\Sense\MobilityRegistrarService\install\Configure-Service.ps1" localhost 4432 qliksenserepository thepw@

& "C:\Program Files\Qlik\Sense\PrecedentsService\install\Configure-Service.ps1" localhost 4432 qliksenserepository thepw@

& "C:\Program Files\Qlik\Sense\NLAppSearch\install\Configure-Service.ps1" localhost 4432 qliksenserepository thepw@

& "C:\Program Files\Qlik\Sense\NLBroker\install\Configure-Service.ps1" localhost 4432 qliksenserepository thepw@

& "C:\Program Files\Qlik\Sense\DataPrepService\install\Configure-Service.ps1" localhost 4432 qliksenserepository thepw@

PS C:\Users\twales> & "C:\Program Files\Qlik\Sense\PrecedentsService\install\Configure-Service.ps1" localhost 4432 qliksenserepository thepw@
PrecedentService configuration started.
WARNING: Skiping the database initialization. No superuser or password specified.
Write-PgPassFile : Exception calling "WriteAllLines" with "3" argument(s): "Access to the path 'C:\Program Files\Qlik\Sense\PrecedentsService\install\pgpass.conf' is denied."
At C:\Program Files\Qlik\Sense\PrecedentsService\install\Configure-Service.ps1:62 char:5
+ Write-PgPassFile $databaseUserPassword
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-PgPassFile], MethodInvocationException
+ FullyQualifiedErrorId : UnauthorizedAccessException,Write-PgPassFile

PS C:\Users\twales> PS C:\Users\twales> & "C:\Program Files\Qlik\Sense\PrecedentsService\install\Configure-Service.ps1" localhost 4432 qliksenserepository thepw@
>> PrecedentService configuration started.
>> WARNING: Skiping the database initialization. No superuser or password specified.
>> Write-PgPassFile : Exception calling "WriteAllLines" with "3" argument(s): "Access to the path 'C:\Program Files\Qlik\Sense\PrecedentsService\install\pgpass.conf' is denied."
>> At C:\Program Files\Qlik\Sense\PrecedentsService\install\Configure-Service.ps1:62 char:5
>> + Write-PgPassFile $databaseUserPassword
>> + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> + CategoryInfo : NotSpecified: (:) [Write-PgPassFile], MethodInvocationException
>> + FullyQualifiedErrorId : UnauthorizedAccessException,Write-PgPassFile
>>
PS C:\Users\twales> & "C:\Program Files\Qlik\Sense\NLAppSearch\install\Configure-Service.ps1" localhost 4432 qliksenserepository thepw@
NLAppSearch configuration started.
WARNING: Skiping the database initialization. No superuser or password specified.
Saving the modified settings.
Exporting the copy of the invocation parameters.
Out-File : Access to the path 'C:\ProgramData\Qlik\Sense\NLAppSearch\configure.params' is denied.
At C:\Program Files\Qlik\Sense\NLAppSearch\install\Configure-Service.ps1:85 char:5
+ Out-File -InputObject $params -FilePath "$paramsLocation\configur ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OpenError: (:) [Out-File], UnauthorizedAccessException
+ FullyQualifiedErrorId : FileOpenFailure,Microsoft.PowerShell.Commands.OutFileCommand

PS C:\Users\twales> & "C:\Program Files\Qlik\Sense\NLBroker\install\Configure-Service.ps1" localhost 4432 qliksenserepository thepw@
NLBroker configuration started.
WARNING: Skiping the database initialization. No superuser or password specified.
Creating schema 'nl_broker'.
Invoke-Query : Exception calling "Start" with "0" argument(s): "The system cannot find the file specified"
At C:\Program Files\Qlik\Sense\NLBroker\install\install-utils\Postgres.ps1:56 char:12
+ $ret = Invoke-Query $command
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Invoke-Query], MethodInvocationException
+ FullyQualifiedErrorId : Win32Exception,Invoke-Query

PS C:\Users\twales> & "C:\Program Files\Qlik\Sense\DataPrepService\install\Configure-Service.ps1" localhost 4432 qliksenserepository thepw@
DataPrepService configuration started.
WARNING: Skiping the database initialization. No superuser or password specified.
Creating schema 'dataprep_service'.
Invoke-Query : Exception calling "Start" with "0" argument(s): "The system cannot find the file specified"
At C:\Program Files\Qlik\Sense\DataPrepService\install\install-utils\Postgres.ps1:56 char:12
+ $ret = Invoke-Query $command
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Invoke-Query], MethodInvocationException
+ FullyQualifiedErrorId : Win32Exception,Invoke-Query

[Sebastian_Linser]

Could you try one more time but changing into the root folder of every configure-service.ps1 before execution then just call it and add the values by hand. If that works for one try the next the same way but with the parameters you wrote up.

[twales]

Running PS as admin.  I checked for perms again I have full control on all these folders and sub-folders.  App, hybrid, licenses worked, the rest failed.  

I also decided for one test to give myself full control on not just the install folder but the install-utils folder where the postgres.ps1 resides, still same result.  This is really not necessary but I wanted to eliminate any issues from having access.

Note on one of the services I also tried the parameter entry you asked me to do with same results.

PS C:\Program Files\Qlik\Sense\NotifierService\install>

cmdlet Configure-Service.ps1 at command pipeline position 1
Supply values for the following parameters:
databaseHost: localhost
databasePort: 4432
databaseUser: qliksenserepository
databaseUserPassword: passwordhere
NotifierService configuration started.
WARNING: Skiping the database initialization. No superuser or password specified.
Creating schema 'qlik_notifier_service'.
Invoke-Query : Exception calling "Start" with "0" argument(s): "The system cannot find the file specified"
At C:\Program Files\Qlik\Sense\NotifierService\install\install-utils\Postgres.ps1:56 char:12
+ $ret = Invoke-Query $command
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Invoke-Query], MethodInvocationException
+ FullyQualifiedErrorId : Win32Exception,Invoke-Query

Thanks

This is consistent:

Invoke-Query : Exception calling "Start" with "0" argument(s): "The system cannot find the file specified"
At C:\Program Files\Qlik\Sense\NotifierService\install\install-utils\Postgres.ps1:56 char:12

PS C:\Program Files\Qlik\Sense\NotifierService\install> .\Configure-Service.ps1 localhost 4432 qliksenserepository thepw@
NotifierService configuration started.
WARNING: Skiping the database initialization. No superuser or password specified.
Creating schema 'qlik_notifier_service'.
Invoke-Query : Exception calling "Start" with "0" argument(s): "The system cannot find the file specified"
At C:\Program Files\Qlik\Sense\NotifierService\install\install-utils\Postgres.ps1:56 char:12
+ $ret = Invoke-Query $command
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Invoke-Query], MethodInvocationException
+ FullyQualifiedErrorId : Win32Exception,Invoke-Query

PS C:\Program Files\Qlik\Sense\MobilityRegistrarService> cd "C:\Program Files\Qlik\Sense\MobilityRegistrarService\install"
>> .\Configure-Service.ps1 localhost 4432 qliksenserepository thepw@
MobilityRegistrarService configuration started.
WARNING: Skiping the database initialization. No superuser or password specified.
Creating schema 'qlik_mobility_registrar_service'.
Invoke-Query : Exception calling "Start" with "0" argument(s): "The system cannot find the file specified"
At C:\Program Files\Qlik\Sense\MobilityRegistrarService\install\install-utils\Postgres.ps1:56 char:10
+ $ret = Invoke-Query $command
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Invoke-Query], MethodInvocationException
+ FullyQualifiedErrorId : Win32Exception,Invoke-Query

PS C:\Windows\system32> cd "C:\Program Files\Qlik\Sense\PrecedentsService\install"
PS C:\Program Files\Qlik\Sense\PrecedentsService\install> .\Configure-Service.ps1 localhost 4432 qliksenserepository thepw@
PrecedentService configuration started.
WARNING: Skiping the database initialization. No superuser or password specified.
Creating schema 'precedents_service'.
Invoke-Query : Exception calling "Start" with "0" argument(s): "The system cannot find the file specified"
At C:\Program Files\Qlik\Sense\PrecedentsService\install\install-utils\Postgres.ps1:56 char:12
+ $ret = Invoke-Query $command
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Invoke-Query], MethodInvocationException
+ FullyQualifiedErrorId : Win32Exception,Invoke-Query

PS C:\Program Files\Qlik\Sense\NLBroker\install> .\Configure-Service.ps1 localhost 4432 qliksenserepository thepw@
NLBroker configuration started.
WARNING: Skiping the database initialization. No superuser or password specified.
Creating schema 'nl_broker'.
Invoke-Query : Exception calling "Start" with "0" argument(s): "The system cannot find the file specified"
At C:\Program Files\Qlik\Sense\NLBroker\install\install-utils\Postgres.ps1:56 char:12
+ $ret = Invoke-Query $command
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Invoke-Query], MethodInvocationException
+ FullyQualifiedErrorId : Win32Exception,Invoke-Query

PS C:\Program Files\Qlik\Sense\NLBroker\install>

PS C:\Program Files\Qlik\Sense\DataPrepService\install> .\Configure-Service.ps1 localhost 4432 qliksenserepository thepw@
DataPrepService configuration started.
WARNING: Skiping the database initialization. No superuser or password specified.
Creating schema 'dataprep_service'.
Invoke-Query : Exception calling "Start" with "0" argument(s): "The system cannot find the file specified"
At C:\Program Files\Qlik\Sense\DataPrepService\install\install-utils\Postgres.ps1:56 char:12
+ $ret = Invoke-Query $command
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Invoke-Query], MethodInvocationException
+ FullyQualifiedErrorId : Win32Exception,Invoke-Query

PS C:\Program Files\Qlik\Sense> cd "C:\Program Files\Qlik\Sense\DataPrepService\install"
PS C:\Program Files\Qlik\Sense\DataPrepService\install> .\Configure-Service.ps1 localhost 4432 qliksenserepository thepw@
DataPrepService configuration started.
WARNING: Skiping the database initialization. No superuser or password specified.
Creating schema 'dataprep_service'.
Invoke-Query : Exception calling "Start" with "0" argument(s): "The system cannot find the file specified"
At C:\Program Files\Qlik\Sense\DataPrepService\install\install-utils\Postgres.ps1:56 char:12
+ $ret = Invoke-Query $command
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Invoke-Query], MethodInvocationException
+ FullyQualifiedErrorId : Win32Exception,Invoke-Query

[Sebastian_Linser]

Hello @twales ,

Could  you try with August 2021 which has been released recently as a new AMI image and see if the problem persists?

best regards

Sebastian

[twales]

The kind of time I spent to setup both our dev and prod servers for 2021 it's not possible.  I have dev's working on the dev server now.

They need to patch this thing ASAP so I can apply the update.

What if it doesn't work on Aug 2021 either, then what.

for whatever reason it's running and all my setups worked but this issue/error persists in the logs thousands of times.

This is a huge waste of resources, I hope they're actually addressing this.

Thanks for your help.

[twales]

I have applied the AWS BYOL August 2021 Update 2 AMI and still the same failures occur as described above.

Brand new installs twice on 2 new servers just to verify.  Even used the simplest passwords I could.  5 out of the 9 .ps1's failed.  No changes in event errors, still thousands. Related to the dispatcher as that's when they all  appear.

[Sebastian_Linser]

Hello @twales 

you are correct, i saw the same errors in a test today. I have identified the problematic services and will file another product defect towards R&D to get this fixed asap. I will update you here once its done.

best regards

Sebastian

[twales]

Hello Sebastian, 

Sounds good! 

A patch would be nice, not another AMI.   I can't keep installing servers and setting up, too much time is lost.

Thanks