Qlik Community

Qlik Associative Big Data Index Discussions

Ask a question or participate in a discussion about Qlik Associative Big Data Index here.

Partner
Partner

Found 0 persistencemanager service(s)

Hi,

I'm in the process of setting up my first QABDI instance and have gotten as far as to running the start_indexing_env.sh. In the output of this script I get an endless stream of  "[console-info]-[118] Found 0 persistencemanager service(s)" messages accompanied with a single 

[19-07-30T22:04:12:455]-[pers_mgr_srv-error]-[42] Failed to setup Persistence database. Error: Failed to get current model version. database is locked Ref:1817
[19-07-30T22:04:12:497]-[pers_mgr_srv-critical]-[42]

[19-07-30T22:04:12:497]-[pers_mgr_srv-critical]-[42] ********************** CRITICAL! *********************
[19-07-30T22:04:12:497]-[pers_mgr_srv-critical]-[42] PersistenceManager: Failed to intialize persistence service. Abort Starting of the service. Ref:287
[19-07-30T22:04:12:497]-[pers_mgr_srv-critical]-[42] ********************** CRITICAL! *********************

I have found no information on what the Persistence database is or  how to configure the persistencemanager services. Any clues on where to proceed?

 

Thanks in advance!

BR,

-Christian

Labels (1)
1 Solution

Accepted Solutions
Employee
Employee

Re: Found 0 persistencemanager service(s)

Hi

This is due to a locking issue with the PersistenceDB and azurefile system, adding a mount option to the storage class yaml will fix it:

A sample yaml is included - azurefile-sc.yaml:

kind: StorageClass

apiVersion: storage.k8s.io/v1

metadata:

  name: azurefile

provisioner: kubernetes.io/azure-file

parameters:

  skuName: Standard_LRS

mountOptions:

  - nobrl


also please find attached a sample deployment guide with all the yaml files (now updated to include AKS)

 

11 Replies
Partner
Partner

Re: Found 0 persistencemanager service(s)

I found this in the log file "/home/output/logs/pers_mgr_srv_qabdi-indexingmanager_55010.qlog". How do I determine why the database is locked?

[19-07-31T09:51:23:431]-[pers_mgr_srv-info]-[43] RegistryServiceClient connects to localhost:55057

[19-07-31T09:51:23:467]-[pers_mgr_srv-info]-[43] Successfully got ABDI license.

[19-07-31T09:52:23:671]-[pers_mgr_srv-error]-[43] Failed to setup Persistence database. Error: Failed to get current model version. database is locked Ref:1817

[19-07-31T09:52:23:690]-[pers_mgr_srv-critical]-[43] 

 

[19-07-31T09:52:23:690]-[pers_mgr_srv-critical]-[43] ********************** CRITICAL! *********************

[19-07-31T09:52:23:690]-[pers_mgr_srv-critical]-[43] PersistenceManager: Failed to intialize persistence service. Abort Starting of the service. Ref:287

[19-07-31T09:52:23:690]-[pers_mgr_srv-critical]-[43] ********************** CRITICAL! *********************

Partner
Partner

Re: Found 0 persistencemanager service(s)

Did you mount your PersistentVolume? If so, did you enable the privileged setting?

helm upgrade <release-name> <repo/chart> -f <yaml> --set image.priviledged=true

 

Blog: WhereClause   Twitter: @treysmithdev
Partner
Partner

Re: Found 0 persistencemanager service(s)

Thanks for the reply! I appreciate all ideas that help me solve this issue 🙂

Which persistent volume are you referring to? The one created with the command

./exec_in_all_pods.sh <cluster name> 'mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport
<FileSystemID>.efs.<AWS_REGION>.amazonaws.com:/ /home/efs'

 As I am using Azure (AKS), I have not run this command, nor the preceding ones that create the /home/efs directory nor creates the efs-provisioner pod.

I have found no reference of this shared mount in the documentation. Should I just mount any (otherwise unused) common fileshare from Azure Files into the directory /home/efs to get things working?

Where is it configured that it is specifically the path "/home/efs" that should be expected?

Partner
Partner

Re: Found 0 persistencemanager service(s)

Within your kubernetes cluster you'll need a persistent storage location set up. Once that is available to the K8s cluster, you will need to mount it in your pods. 

The cmd you listed should be usable by switching the EFS location with your Azure location and probably the size parameters as well. 

Here are some yaml settings provided for Azure Files:

Static:

## BDI values
## Default values: <a href="https://github.com/qlik-trial/elastic-charts/blob/master/charts/bdi/values.yaml" target="_blank">https://github.com/qlik-trial/elastic-charts/blob/master/charts/bdi/values.yaml</a>

## Static volumes are not available for config.
config:
  enabled: true

  persistence:
    ## The name of the StorageClass created with the cluster
    ## ref: <a href="https://github.com/qlik-trial/oxpecker-k8s/blob/master/azure/azurefile-sc.yaml" target="_blank">https://github.com/qlik-trial/oxpecker-k8s/blob/master/azure/azurefile-sc.yaml</a>
    storageClass: "azurefile"

data:
  persistence:
    ## Enables persistence for `data`
    enabled: true

    volume:
      ## Enables a static volume.
      enabled: true

      plugin:
        ## The Azure Files plugin is built in directly to Kubernetes
        azureFile:
          ## The name of the secret that contains Azure credentials
          ## ref: <a href="https://docs.microsoft.com/en-us/azure/aks/azure-files-volume##create-a-kubernetes-secret" target="_blank">https://docs.microsoft.com/en-us/azure/aks/azure-files-volume##create-a-kubernetes-secret</a>
          secretName: azure-secret
          ## The name of the Azure File share
          shareName: aksshare
          ## Permissions of the share
          readOnly: false

output:
  persistence:
    ## Enables persistence for `output`
    enabled: true

    volume:
      ## Enables a static volume.
      enabled: true

      plugin:
        ## The Azure Files plugin is built in directly to Kubernetes
        azureFile:
          ## The name of the secret that contains Azure credentials
          ## ref: <a href="https://docs.microsoft.com/en-us/azure/aks/azure-files-volume##create-a-kubernetes-secret" target="_blank">https://docs.microsoft.com/en-us/azure/aks/azure-files-volume##create-a-kubernetes-secret</a>
          secretName: azure-secret
          ## The name of the Azure File share
          shareName: aksshare
          ## Permissions of the share
          readOnly: false

Dynamic:

## BDI values
## Default values: <a href="https://github.com/qlik-trial/elastic-charts/blob/master/charts/bdi/values.yaml" target="_blank">https://github.com/qlik-trial/elastic-charts/blob/master/charts/bdi/values.yaml</a>

config:
  enabled: true

  persistence:
    ## The name of the StorageClass created with the cluster
    ## ref: <a href="https://github.com/qlik-trial/oxpecker-k8s/blob/master/azure/azurefile-sc.yaml" target="_blank">https://github.com/qlik-trial/oxpecker-k8s/blob/master/azure/azurefile-sc.yaml</a>
    storageClass: "azurefile"

data:
  enabled: true

  persistence:
    ## The name of the StorageClass created with the cluster
    ## ref: <a href="https://github.com/qlik-trial/oxpecker-k8s/blob/master/azure/azurefile-sc.yaml" target="_blank">https://github.com/qlik-trial/oxpecker-k8s/blob/master/azure/azurefile-sc.yaml</a>
    storageClass: "azurefile"

output:
  enabled: true

  persistence:
    ## The name of the StorageClass created with the cluster
    ## ref: <a href="https://github.com/qlik-trial/oxpecker-k8s/blob/master/azure/azurefile-sc.yaml" target="_blank">https://github.com/qlik-trial/oxpecker-k8s/blob/master/azure/azurefile-sc.yaml</a>
    storageClass: "azurefile"

 

Blog: WhereClause   Twitter: @treysmithdev
Partner
Partner

Re: Found 0 persistencemanager service(s)

I have already earlier set up my persistent volumes (config, data, output) using the static file example that you also provided, of course modifying it to match my environment. These three volumes get mapped properly and I have tested them by logging in to e.g. the bastion and creating empty files in /home/data, /home/output and /home/ubuntu/dist/runtime/config, all of this works and the empty created files are properly visible in the Azure storage. The other way around also applies so that when I add files to the data storage in Azure, they show up properly in the cluster.

Did I understand correctly, that apart from these three documented volumes (that currently work as expected), I also need to mount one more volume (that is not mentioned in the documentation at help.qlik.com) at the path /home/efs in order for the BDI services to work? I can of course test this and see what happens.

Partner
Partner

Re: Found 0 persistencemanager service(s)

Did you only mount this directory to the bastion pod? In other words, if you open up a bash session in the indexingmanager pod, do you still see the folder? If so, you need to mount on the rest of your pods. 

Also, did you set image.privileged = true? 

 

There is no other storage/volume necessary aside from the one persistent volume with the three directories.

Blog: WhereClause   Twitter: @treysmithdev
Partner
Partner

Re: Found 0 persistencemanager service(s)

I have not mounted any directory manually. The only mount that I have are the three ones as defined by the helm chart (config, data and output), all of these appear correctly mounted on those pods they should be mounted on (e.g. the nginx pod does not have the data and output mounts) - the tests for this are further down my post.

I just retried the creation of the entire cluster (after first deleting it) using the command 

helm install --name qabdi bt_qlik/bdi -f my_values.yaml --set image.privileged=true

with the my_values.yaml looking like this:

##License settings

acceptLicense: true

license:
  key: [redacted]

## Enable management console

managementconsole:
  ingress:
    enabled: true

nginx-ingress:
  enabled: true

## Enable qsl manager

qslmanager:
  service:
    type: LoadBalancer



## BDI values
## Default values: <a href="https://github.com/qlik-trial/elastic-charts/blob/master/charts/bdi/values.yaml" target="_blank">https://github.com/qlik-trial/elastic-charts/blob/master/charts/bdi/values.yaml</a>

## Static volumes are not available for config.
config:
  enabled: true

  persistence:
    ## The name of the StorageClass created with the cluster
    ## ref: <a href="https://github.com/qlik-trial/oxpecker-k8s/blob/master/azure/azurefile-sc.yaml" target="_blank">https://github.com/qlik-trial/oxpecker-k8s/blob/master/azure/azurefile-sc.yaml</a>
    storageClass: "azurefile"

data:
  persistence:
    ## Enables persistence for `data`
    enabled: true

    volume:
      ## Enables a static volume.
      enabled: true

      plugin:
        ## The Azure Files plugin is built in directly to Kubernetes
        azureFile:
          ## The name of the secret that contains Azure credentials
          ## ref: <a href="https://docs.microsoft.com/en-us/azure/aks/azure-files-volume##create-a-kubernetes-secret" target="_blank">https://docs.microsoft.com/en-us/azure/aks/azure-files-volume##create-a-kubernetes-secret</a>
          secretName: azure-secret
          ## The name of the Azure File share
          shareName: data
          ## Permissions of the share
          readOnly: false

output:
  persistence:
    ## Enables persistence for `output`
    enabled: true

    volume:
      ## Enables a static volume.
      enabled: true

      plugin:
        ## The Azure Files plugin is built in directly to Kubernetes
        azureFile:
          ## The name of the secret that contains Azure credentials
          ## ref: <a href="https://docs.microsoft.com/en-us/azure/aks/azure-files-volume##create-a-kubernetes-secret" target="_blank">https://docs.microsoft.com/en-us/azure/aks/azure-files-volume##create-a-kubernetes-secret</a>
          secretName: azure-secret
          ## The name of the Azure File share
          shareName: output
          ## Permissions of the share
          readOnly: false

Then I log into the bastion and run the command

./dist/runtime/scripts/indexer/start_indexing_env.sh

Which should start the indexing services properly but the following error ends up in the logs.

[19-07-31T16:42:55:965]-[pers_mgr_srv-info]-[62] 

**************************************************************************************
****************************************-=-=-=****************************************
New logging session started
Binary:      
Time:        2019-07-31T16:42:55
Console:     Yes
File:        /home/output/logs/pers_mgr_srv_qabdi-indexingmanager_55010.qlog
Debug:       0 ""
Trace:       0 ""
File Size:   5242880 Bytes
File Count:  1
Prefix:      pers_mgr_srv
Log Pattern: [%C-%m-%dT%H:%M:%S:%e]-[%^%n-%l%$]-[%t] %v
**************************************************************************************
**************************************************************************************

[19-07-31T16:42:56:000]-[pers_mgr_srv-info]-[62] RegistryServiceClient connects to localhost:55057
[19-07-31T16:42:56:032]-[pers_mgr_srv-info]-[62] Successfully got ABDI license.
[19-07-31T16:43:56:204]-[pers_mgr_srv-error]-[62] Failed to setup Persistence database. Error: Failed to get current model version. database is locked Ref:1817
[19-07-31T16:43:56:217]-[pers_mgr_srv-critical]-[62] 

[19-07-31T16:43:56:217]-[pers_mgr_srv-critical]-[62] ********************** CRITICAL! *********************
[19-07-31T16:43:56:217]-[pers_mgr_srv-critical]-[62] PersistenceManager: Failed to intialize persistence service. Abort Starting of the service. Ref:287
[19-07-31T16:43:56:217]-[pers_mgr_srv-critical]-[62] ********************** CRITICAL! *********************

The persistence db file gets created in the expected location but it has a zero size.

ls -l /home/output/PersistenceManagerOutput/persistence.db 
-rwxrwxrwx 1 root root 0 Jul 31 16:42 /home/output/PersistenceManagerOutput/persistence.db

Tests of the mounts on each pod:

The data mount works correctly:

$ ./exec_in_all_pods.sh qabdi 'touch /home/data/touchfile'
--- Launching command touch /home/data/touchfile on qabdi-bastion-5878f667c-rrnkr
---
--- Launching command touch /home/data/touchfile on qabdi-indexer-0
---
--- Launching command touch /home/data/touchfile on qabdi-indexingmanager-84f7df86c4-fwsx7
---
--- Launching command touch /home/data/touchfile on qabdi-license-6475f54f8f-9wpz2
touch: /home/data/touchfile: No such file or directory
command terminated with exit code 1
---
--- Launching command touch /home/data/touchfile on qabdi-managementconsole-6b566864f4-vdf7m
touch: cannot touch '/home/data/touchfile': No such file or directory
command terminated with exit code 1
---
--- Launching command touch /home/data/touchfile on qabdi-nginx-ingress-controller-8679fbbfd5-f6stn
touch: cannot touch '/home/data/touchfile': No such file or directory
command terminated with exit code 1
---
--- Launching command touch /home/data/touchfile on qabdi-nginx-ingress-default-backend-6fdfdb594-kkf2j
OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "exec: \"touch\": executable file not found in $PATH": unknown
command terminated with exit code 126
---
--- Launching command touch /home/data/touchfile on qabdi-qslexecutor-785c5ddbd9-5h5rr
---
--- Launching command touch /home/data/touchfile on qabdi-qslmanager-fc7b8bb49-sl6f6
---
--- Launching command touch /home/data/touchfile on qabdi-qslworker-0
---
--- Launching command touch /home/data/touchfile on qabdi-restapi-7ddfb5f9cc-2dkv2
---
--- Launching command touch /home/data/touchfile on qabdi-symbolserver-0

 The output mount works correctly:

$ ./exec_in_all_pods.sh qabdi 'touch /home/output/touchfile'
--- Launching command touch /home/output/touchfile on qabdi-bastion-5878f667c-rrnkr
---
--- Launching command touch /home/output/touchfile on qabdi-indexer-0
---
--- Launching command touch /home/output/touchfile on qabdi-indexingmanager-84f7df86c4-fwsx7
---
--- Launching command touch /home/output/touchfile on qabdi-license-6475f54f8f-9wpz2
touch: /home/output/touchfile: No such file or directory
command terminated with exit code 1
---
--- Launching command touch /home/output/touchfile on qabdi-managementconsole-6b566864f4-vdf7m
touch: cannot touch '/home/output/touchfile': No such file or directory
command terminated with exit code 1
---
--- Launching command touch /home/output/touchfile on qabdi-nginx-ingress-controller-8679fbbfd5-f6stn
touch: cannot touch '/home/output/touchfile': No such file or directory
command terminated with exit code 1
---
--- Launching command touch /home/output/touchfile on qabdi-nginx-ingress-default-backend-6fdfdb594-kkf2j
OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "exec: \"touch\": executable file not found in $PATH": unknown
command terminated with exit code 126
---
--- Launching command touch /home/output/touchfile on qabdi-qslexecutor-785c5ddbd9-5h5rr
---
--- Launching command touch /home/output/touchfile on qabdi-qslmanager-fc7b8bb49-sl6f6
---
--- Launching command touch /home/output/touchfile on qabdi-qslworker-0
---
--- Launching command touch /home/output/touchfile on qabdi-restapi-7ddfb5f9cc-2dkv2
---
--- Launching command touch /home/output/touchfile on qabdi-symbolserver-0
---

 The config mount works correctly:

$ ./exec_in_all_pods.sh qabdi 'touch /home/ubuntu/dist/runtime/config/touchfile'
--- Launching command touch /home/ubuntu/dist/runtime/config/touchfile on qabdi-bastion-5878f667c-rrnkr
---
--- Launching command touch /home/ubuntu/dist/runtime/config/touchfile on qabdi-indexer-0
---
--- Launching command touch /home/ubuntu/dist/runtime/config/touchfile on qabdi-indexingmanager-84f7df86c4-fwsx7
---
--- Launching command touch /home/ubuntu/dist/runtime/config/touchfile on qabdi-license-6475f54f8f-9wpz2
touch: /home/ubuntu/dist/runtime/config/touchfile: No such file or directory
command terminated with exit code 1
---
--- Launching command touch /home/ubuntu/dist/runtime/config/touchfile on qabdi-managementconsole-6b566864f4-vdf7m
---
--- Launching command touch /home/ubuntu/dist/runtime/config/touchfile on qabdi-nginx-ingress-controller-8679fbbfd5-f6stn
touch: cannot touch '/home/ubuntu/dist/runtime/config/touchfile': No such file or directory
command terminated with exit code 1
---
--- Launching command touch /home/ubuntu/dist/runtime/config/touchfile on qabdi-nginx-ingress-default-backend-6fdfdb594-kkf2j
OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "exec: \"touch\": executable file not found in $PATH": unknown
command terminated with exit code 126
---
--- Launching command touch /home/ubuntu/dist/runtime/config/touchfile on qabdi-qslexecutor-785c5ddbd9-5h5rr
---
--- Launching command touch /home/ubuntu/dist/runtime/config/touchfile on qabdi-qslmanager-fc7b8bb49-sl6f6
---
--- Launching command touch /home/ubuntu/dist/runtime/config/touchfile on qabdi-qslworker-0
---
--- Launching command touch /home/ubuntu/dist/runtime/config/touchfile on qabdi-restapi-7ddfb5f9cc-2dkv2
---
--- Launching command touch /home/ubuntu/dist/runtime/config/touchfile on qabdi-symbolserver-0

 

Partner
Partner

Re: Found 0 persistencemanager service(s)

When you deleted the cluster did you purge the data? In other words the locked db is a new file?

What are your settings in the output/config/indexer/persistence_manager.json file?

I think there is a secondary helm repo, qlik/bdi. You can try using the chart from there, which might be a newer version. 

Blog: WhereClause   Twitter: @treysmithdev
Partner
Partner

Re: Found 0 persistencemanager service(s)

I have now recreated the cluster once again (making sure to delete all files in the volumes also). No changes there, still in the same situation.

The contents of my output/config/indexer/persistence_manager_service.json file (there is no persistence_manager.json file):

{
	"license_config" : 
	{
		"address" : "qabdi-license",
		"async_server_threads" : 1,
		"port" : 9200,
		"threads_per_cq" : 1
	},
	"registry" : 
	{
		"address" : "qabdi-indexingmanager",
		"async_server_threads" : 1,
		"port" : 55057,
		"threads_per_cq" : 1
	},
	"server_config" : 
	{
		"address" : "qabdi-indexingmanager",
		"async_server_threads" : 1,
		"persistence_target" : "database",
		"port" : 55010,
		"threads_per_cq" : 1
	},
	"service_outputs" : 
	{
		"backup_paths" : [],
		"name" : "service_outputs",
		"paths" : 
		[
			{
				"key" : "",
				"path" : "/home/output/symbol",
				"type" : "SymbolOutput"
			},
			{
				"key" : "",
				"path" : "/home/output/index",
				"type" : "IndexOutput"
			},
			{
				"key" : "",
				"path" : "/home/output/symbol_position",
				"type" : "SymbolPositions"
			}
		],
		"root_folder" : "/home/output"
	}
}

 

I checked the helm repos, bt_qlik is just the local alias (name) I have used for the repository at the url https://qlik.bintray.com/qabdicharts. In some documentation the local name bt_qlik has been given and in others documentations the local name has been qlik, so I don't think there are two repos.