How to configure Qlik Sense to use a dedicated PostgreSQL database

[Sebastian_Linser]

This article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

The Qlik Sense Repository Database (QSR) can be moved to a dedicated standalone PostgreSQL instance not hosted on the same machine as other Qlik Sense Services. It is also possible to move an already existing QSR from a local Sense install, to a dedicated PostgresSQL database.

Content:

• Set up a postgresSQL database
• Simplified checklist of steps:
• Moving the Qlik Sense Share Folders as well?
• Backup
• Restore
• Changing the Connection String to QSR and QSMQ Database
• Changing the Connection Stringto SenseServices Databases.
• Changing the Connection String to Licenses Database ( April 2019 and up )
• Remove Database Service dependency and Postgres Server
• Finalize

Set up a postgresSQL database

Instructions for this might vary slightly. Please refer to the Qlik Online Help for details.  We recommend watching this video.

Qlik Sense Enterprise server has a set of five databases. For more details visit Installing and configuring PostgreSQL on help.qlik.com.

1. The Qlik Sense repository database (QSR):

The QSR is the primary database in your Qlik Sense deployment.

2. The Qlik Sense services database (SenseServices):

The SenseServices database contains schemas for each of the Qlik Sense services and allows growth independently of the Qlik Sense Repository Database, while still sharing the same PostgreSQL instance and login role.

3. The Qlik Sense message queue database (QSMQ):

The QSMQ database provides a light-weight method of passing messages internally between services in Qlik Sense Enterprise. The NOTIFY and LISTEN functionality in PostgreSQL allows services to be notified about new messages that have been written to the messaging table.

4. The Qlik Sense logging database (QLogs): No longer supported after May 2021

The QLogs database centralizes logging by collecting log messages from all Qlik Sense nodes in your deployment and stores them in a PostgreSQL database.

5. The Qlik Sense license service database (Licenses):

The Licenses database is used to sync the assignments between the Qlik license backend and the local installation.

Simplified checklist of steps:

1. Download a valid version of postgresSQL.  For Sense June 2017 and up, 9.6.x is required, from November 2019 version 11.5.x as well. From May 2021 version 12.5 can be used. Review the system requirements of your version before proceeding.
2. Install based on instructions found on the Qlik Online Help for Qlik Sense; select the version you are running on then select from Menu , Deploy - Qlik Sense installation - Installing and configuring PostgreSQL.

See Installing and configuring PostgreSQL 

Moving the Qlik Sense Share Folders as well?

Follow How to change the shared path in Qlik Sense 

Backup 

Depending on the version, you need to backup different databases, on the current Qlik Sense central node hosting the Qlik Sense Repository Database service, follow the backup instructions to obtain a database copy of each Qlik Sense Database Present . 
   Example for February 2021  you need to backup the databases reported in section "Installing and configuring PostgreSQL"; referring to the Help site these are : 
    QSR , QSMQ , SenseServices , Licenses , if configured QLogs    

Restore

On the dedicated remote postgresSQL host, perform a restore   of the backed up Databases of previous point. 

Changing the Connection String to QSR and QSMQ Database 

1. On the Qlik Sense central node, all services but the Qlik Sense Repository Service are shut down.
2. Open QlikSenseUtil. By default, this will be installed in: C:\Program Files\Qlik\Sense\Repository\Util\QlikSenseUtil\QlikSenseUtil.exe
3. Open the Connection String Editor and Read the current configuration, decrypting the current data.
   
   
4. Replace all mentions of the current database hostname.  In our example, the new database resides on 172.16.16.103. Instead of an IP you can also use the FQDN of the db server.
5. You need to repeat this step for the   sections <add name="QSMQ" and <add name="postgres"
6. Hit Save Value in config file encrypted, committing any changes. 

Changing the Connection String to SenseServices Databases. 

       Execute steps in Moved Postgress database to new host, but some qliksense is still accessing the SenseServices old da...; check if different version of article are present for different Qlik Sense version .

Changing the Connection String to Licenses Database ( April 2019 and up ) 

Starting from April 2019 new License Service implementation is using this database.

1.   Open an elevated Powershell and change directory to C:\Program Files\Qlik\Sense\Licenses
2.    Run the script .\Configure-Service.ps1 and provide:
   • new Hostname or IP  
   • database port default is 4432
   • database user qliksenserepository
   • databasepassword is the qliksenserepository password  used during configuration of  the new database.
3. example: 
4. cd  'C:\Program Files\Qlik\Sense\Licenses'
5. .\Configure-Service.ps1 172.16.16.102 4432 qliksenserepository Password123!

It will look like this:

Windows PowerShell
Copyright (C) 2014 Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> cd \
PS C:\> cd  'C:\Program Files\Qlik\Sense\Licenses'
PS C:\Program Files\Qlik\Sense\Licenses> .\Configure-Service.ps1 172.16.16.102 4432 qliksenserepository Password123!
Licenses configuration started.
WARNING: Skipping the database initialization. No superuser or password specified.
Reading the settings file.
Saving the modified settings.
Licenses configuration successful.
PS C:\Program Files\Qlik\Sense\Licenses>

repeat this for:

C:\Program Files\Qlik\Sense\NotifierService\install\Configure-Service.ps1
C:\Program Files\Qlik\Sense\AppDistributionService\Configure-Service.ps1
C:\Program Files\Qlik\Sense\HybridDeploymentService\Configure-Service.ps1
C:\Program Files\Qlik\Sense\MobilityRegistrarService\install\Configure-Service.ps1
C:\Program Files\Qlik\Sense\PrecedentsService\install\Configure-Service.ps1
C:\Program Files\Qlik\Sense\NLAppSearch\install\Configure-Service.ps1

there might come new services or disappear with new releases, please check in the C:\Program Files\Qlik\Sense\ folder for "Configure-Service.ps1" to find additional one in your version of Qlik Sense.

Note: If you have installed Qlik Sense in a different directory the Configure-Service.ps1 would have to be updated as in this example:

File: Configure-Service.ps1 Folders: Various (9 microservices)
Before:
11 | [string] $senseInstallPath
"$env:ProgramFiles\Qlik Sense",
After:
11
[string] $senseInstallPath = "D:\Qlik",

Remove Database Service dependency and Postgres Server

After successfully verifying your new setup, remove the Qlik Sense Repository service dependencies and the old Postgres database. This is done to avoid issues with future Upgrade of Environment. 

1. In an elevated command prompt, execute the following on prompt:
   
   c:\sc config "QlikSenseRepositoryService" depend= ""
   c:\sc config "QlikLoggingService" depend= ""
   
   This will return: 
   
   [SC] ChangeServiceConfig SUCCESS
   
   
2. Restart the Qlik Sense Repository Service.
   
   Check that Qlik Sense Repository Database doesn't start (confirms [SC] ChangeServiceConfig SUCCESS ).
   
   
3. Remove the (local) Qlik Sense Repository Database
   
   Find the most recent PostgreSQL.msi in C:\ProgramData\Package Cache\  and remove it. This removes the service but no old local database data. This is needed to prevent issues on future upgrades of Environment.
   
   You can find the precise folder with this command depending on the Postgresql Version you used installing the inital release.
   
   9.6:
   
   c:\cd ProgramData\Package Cache
   c:\ProgramData\Package Cache\dir /s PostgreSQL.msi
   
   The folder containing PostgreSQL.msi will be revealed. Right click on the PostgreSQL.msi file and select uninstall from the menu.
   
   12.5:
   
   c:\cd ProgramData\Package Cache
   c:\ProgramData\Package Cache\dir /s PostgreSQL125.msi
   
   The folder containing PostgreSQL.msi will be revealed. Right click on the PostgreSQL.msi file and select uninstall from the menu.
   

Finalize

1. Restart the Qlik Sense Repository Service and verify that Qlik Sense Repository Database Service doesn't exist when followed the previous steps.
2. Start all remaining Sense Services 
3. Progress on the startup sequence (including the connection to the new database) can be tracked in the _System_Repository log files located in C:\ProgramData\Qlik\Sense\Log\Repository\Trace
4. Verify functionality.

5. In Multi-node scenario you need to repeat steps "Changing the Connection Strings to Databases" on all Rim nodes (QlikSenseUtil for Repository.exe.config and Configure-Service.ps1 for the micro services), and if required configure "Update the Qlik Logging Service Connection String" (note that is EOL with Qlik Sense Enterprise on Windows February 2022) .   

Environment

Qlik Sense Enterprise on Windows 

[aadil_madarveet]

Hi,

I am running Qlik Sense Feb 2021 P8.

I am trying to migrate the PostgreSQL db to a separate standalone node to implement HA.

I have followed your instructions step by step. 

All the services are running now, but somehow the proxy is not kicking in. Still shows this error in the system proxy log. 

The proxy configurations are still pointing to localhost for QlikSenseDatabaseService.

24	20211028T142549.340+0000	ERROR	server	System.Proxy.Proxy.Communication.Rest.ProxyRestClient	9	59602e12-d054-4368-bea7-0f0f4062e0a8	DEVSERVER\senseadmin	When contacting https://localhost:4242/	No connection could be made because the target machine actively refused it [::1]:4242↵↓Unable to connect to the remote server	   at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)↵↓   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)↵↓   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)↵↓   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)↵↓   at Qlik.Sense.Common.Communication.REST.Client.WebClientEx.GetWebResponse(WebRequest request, IAsyncResult result)↵↓   at System.Net.WebClient.DownloadBitsResponseCallback(IAsyncResult result)		59602e12-d054-4368-bea7-0f0f4062e0a8

After following this step, 

"Right click on the PostgreSQL.msi file and select uninstall from the menu."

The QlikSenseRespositoryDatabase service has got removed from the box. It does not appear on the server anymore.

[Sebastian_Linser]

Hello @aadil_madarveet 

since the response in the logs is that you can't reach the repository service, it seems that this one is not up. So please check in the repository logs first why the DB is not connecting towards your new instance.

best regards

Sebastian

[Lokeshb31]

Hi,

We are running Qliksense Feb 2021 Patch3 on Windows Server 2019. We want to move qliksense repository to new DB host for High Availability.  AS per backup instructions stated in https://help.qlik.com/en-US/sense-admin/August2022/Subsystems/DeployAdministerQSE/Content/Sense_Depl...  I am taking a backup but shall we backup and restore all DBs like SenseServices, QSMQ, Licenses? We are not using centralized logging and QSMQ/SenseServices are asked to be backuped up in case of linked cloud environment(What does it mean? We have on-prem Qliksense installation) 

Will it be ok if we only restore QSR to another DB, update only that configuration in Qliksenseutil.exe and it will work? Do I need to delete default Qliksense Repository database service in that case or I can keep it up? 

Please help to address this confusion. 

[Sebastian_Linser]

@Lokeshb31  

The basic principle is that what you marked above

- if you have a could deployment, so connection to tennantid.(eu|us|ap).qlikcloud.com and you are pushing apps from Qlik Sense into this instance you need the SenseServices and QSMQ table,

- if you use a signed license key (SLK) then you need the Licenses database if you use LEF and Controlnumber (which is highly likley in Feb2021) then you dont need this database.

Since you migrate you have at least to create empty DB's, the script here works just fine.

https://help.qlik.com/en-US/sense-admin/August2022/Subsystems/DeployAdministerQSE/Content/Sense_Depl...

If you just move the only QSR the worst what can happen is that you reconfigure the cloud tennant one more time in the QMC, the rest will work just fine.

Beside the Qliksenseutil.exe you also want to update the configure-service.ps1 files to get the new database connection for the microservices (appsettings.json files) as described above.

[Lokeshb31]

@Sebastian_Linser : I am not migrating DB to cloud, we just want to shift Qliksense repository from Default central node to another standalone separate database node managed by DBA team, both environments are on-premise, no cloud is involved here. Do we need to backup/restore all DBs in this case?  Our Feb 2021 repository is 9.6(default) and the one we are shifting QSR is Postgres 12.  What I did is, just modified Qliksenseutil connection string for QSR  as per below screenshot and restarted all services. Its all working fine. 

Could you please suggest what other steps I need to do? 

[Sebastian_Linser]

The create script is identical for on prem and cloud. You still need to create the other DB's as a emtpy one. Migrating only if you fullfill the conditions in the last post. Otherwise they will be seeded by first Qlik service dispatcher start.

To see which additional connection strings you have to update check with that command

C:\Program Files\Qlik\Sense\dir /s Configure-Service.ps1

all you find need to be updated.

[Lokeshb31]

@Sebastian_Linser : In this link its mentioned to update configure.ps1 with below command i.e. localhost DB credentials. 

.\Configure-Service.ps1 localhost 4432 qliksenserepository Password123!

It should be .\Configure-Service.ps1 <newDBhost> <newDBport> qliksenserepository <newpassword>

Right? 

I tried with .\Configure-Service.ps1 <newDBhost> <newDBport> qliksenserepository <newpassword> 

But  out of all Configure.ps1 locations ( found using dir /s Configure-Service.ps1), it threw error for below ones

D:\Program Files\Qlik\Sense\DataPrepService\install

D:\Program Files\Qlik\Sense\PrecedentsService\install

D:\Program Files\Qlik\Sense\MobilityRegistrarService\install

D:\Program Files\Qlik\Sense\NotifierService\install

 Error message:

WARNING: Skiping the database initialization. No superuser or password specified.
Creating schema 'dataprep_service'.
Invoke-Query : Exception calling "Start" with "0" argument(s): "The system cannot find the file specified"
At D:\Program Files\Qlik\Sense\DataPrepService\install\install-utils\Postgres.ps1:56 char:12
+ $ret = Invoke-Query $command
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Invoke-Query], MethodInvocationException
+ FullyQualifiedErrorId : Win32Exception,Invoke-Query

Please help to address the issue.

[Sebastian_Linser]

are youy within the folder when you tun the Query? 

so cd "D:\Program Files\Qlik\Sense\DataPrepService\install" and then running the command?

You can alternativly also run the command without the arguments and you will be promted to enter them.

This one here "Invoke-Query : Exception calling "Start" with "0" argument(s): "The system cannot find the file specified"" could be related to the postgres binary it searches. Which is not because you installed it on the D drive but because you have the folder 9.6 hardcoded in the original file.

Check for this line in Configure-Services.Ps1 and adjust it to your system

[string]$postgresHome = "$senseInstallPath\Repository\PostgreSQL\9.6",

e.g.

[string]$postgresHome = "$senseInstallPath\Repository\PostgreSQL\12.5",

[Lokeshb31]

@Sebastian_Linser I edited path to [string]$postgresHome = "$senseInstallPath\Repository\PostgreSQL\12.5", but its not working, the obvious reason is, I dont have 12.5 installed on same host, its on another host. What changes I need to do in file? 

[Lokeshb31]

[CmdletBinding()]
Param(
    [Parameter(Mandatory = $True)] [string]$databaseHost,
    [Parameter(Mandatory = $True)] [int]$databasePort,
    [Parameter(Mandatory = $True)] [string]$databaseUser,
    [Parameter(Mandatory = $True)] [string]$databaseUserPassword,

    [string]$databaseSuperUser,
    [string]$databaseSuperUserPassword,

    [string]$senseInstallPath = "$env:ProgramFiles\Qlik\Sense",
    [string]$senseDataPath = "$env:ProgramData\Qlik\Sense",
    [string]$postgresHome = "$senseInstallPath\Repository\PostgreSQL\9.6",
    [string]$hostname = "localhost",
    [string]$serviceName = "DataPrepService",
    [string]$databaseName = "SenseServices",
    [string]$databaseSchema = "dataprep_service",
    [string]$paramsLocation = "$senseDataPath\$serviceName"
)

$ErrorActionPreference = "Stop";
$contentLocation = $PSScriptRoot

. $contentLocation/install-utils/Json.ps1
. $contentLocation/install-utils/Postgres.ps1
. $contentLocation/install-utils/Encoding.ps1
. $contentLocation/install-utils/Parameters.ps1
. $contentLocation/install-utils/Security.ps1

# Set postgres execution environment
$global:psqlPath = "$postgresHome/bin/psql.exe"
$global:pgPassPath = "$contentLocation/pgpass.conf"
$global:postgresPort = $databasePort
$global:postgresHost = $databaseHost
$global:postgresDB = $databaseName

Write-Output "DataPrepService configuration started."

try {
    if ($databaseSuperUser -and ($databaseSuperUserPassword -or $databaseHost -eq "localhost")) {
        Write-Output "Database superuser specified, initializing the database."
        
        # Set postgres user context
        $global:postgresUser = $databaseSuperUser
        Write-PgPassFile $databaseSuperUserPassword

        # Create database in external postgres instance
        Set-Database $databaseName $databaseSuperUser

        # Create service user accounts with CREATE privilege in database
        Set-ServiceUser $databaseUser $databaseUserPassword $databaseName
        Write-Output "Database initialization successful."
    } else {
        Write-Warning "Skiping the database initialization. No superuser or password specified."
    }

    # Reconfigure postgres user context
    $global:postgresUser = $databaseUser
    Write-PgPassFile $databaseUserPassword

    # Create database schema
    Set-Schema $databaseSchema $databaseUser

    # Read default appsettings
    $appsettings = Get-Content "$contentLocation\..\appsettings.json" -raw | ConvertFrom-Json

    # Set config values
    $appsettings.Postgres.Host = $databaseHost
    $appsettings.Postgres.Port = $databasePort
    $appsettings.Postgres.Database = $databaseName
    $appsettings.Postgres.Schema = $databaseSchema
    $appsettings.Postgres.Password = $databaseUserPassword | Encrypt-WithMachineKey
    $appsettings.Postgres.Username = $databaseUser

    # Save modified appsettings
    Write-Output "Saving the modified settings."
    $json = $appsettings | ConvertTo-Json -Depth 10 | Format-Json
    Out-File -InputObject $json -FilePath "$contentLocation\..\appsettings.json"

    # Create a copy of the invocation parameters for the upgrade
    Write-Output "Exporting the copy of the invocation parameters."
    $params = $PSBoundParameters | Export-JsonParameters | Encrypt-WithMachineKey
    New-Item -ItemType Directory -Force -Path $paramsLocation | Out-Null
    Out-File -InputObject $params -FilePath "$paramsLocation\configure.params" -Force
    
    Write-Output "DataPrepService configuration successful."
} finally {
    Remove-PgPassFile
}

Above is Configure.pS1