I can build the image (base image is redhat ubi10), but when I run start_replicate.sh, it fails because this script calls /opt/attunity/replicate/bin/areplicate stop but this file does not exist.
I also found articles that a non-systemd installations of replicate can be started, stopped etc through a command that is named as <instancename> and that the default instancename is areplicate.
So, why is my installation missing the <instancename> command/file?
It seems that you’re missing the areplicate instance command because it is installed only when the RPM installer sets up systemd — in non-systemd environments (like your Docker container), setting systemd=no during install skips creating that wrapper.
You used systemd=no to disable systemd support in Docker, which blocks installation of the default areplicate instance command—hence it's absent.
Thanks for your response. But if the instance command is skipped on systemd installation, I don't understand the following things:
* Why does the start_replicate.sh script expect this command? * Why says this page that non-service instances are managed with this instance command? Is a non-service instance not the same as a non-systemd-installation?
Hi @sis-qlik, I have coming up to 6 years experience running Qlik in a containerized environment, both in Fargate and Docker - the latter for local and CI pipeline builds.
If you'd like to share your Dockerfile I can maybe help shed some light. My personal experience is the docker images need a little massaging to get running.
This isn't some marketing ploy, but I've also made the decision to try and share some of the things I've learnt working in this way. It's very early days right now but I've created a github repo (I'm still refining the CI pipeline right now) which may also help - although its currently geared to working in docker with the hope to provide examples for running in AWS ECS too.
My personal experience is the docker images need a little massaging to get running.
Yeah, definitely. I wrote my own startup script that fills the gap between the installer and the docke image. The main challenge was to prepare the environment for the attunity user who runs replicate.
Thanks for sharing your repo. I will have a look, perhaps I will find some optimizations.
The script start_replicate.sh that is contained in the RPM file of replicate under opt/attunity/replicate/addons/samples/docker, had several problems in my docker image.
Therefore I wrote my own start script for replicate.
The original script calls the command <replicate>/bin/areplicate which is not installed (which is by design for non-systemd installations according to @Rahul_Kale, see his answer in this thread)
So instead of areplicate, I had to use <replicate>/bin/repctl.sh
But calling repctl.sh throwed error while loading shared libraries because the lib folder of replicate was not in the environment variable LD_LIBRARY_PATH
I fixed this by sourcing <replicate>/bin/arep_login.sh (taken from this thread)
But since replicate is started as user attunity, the sourcing had no effect because it was done by root user who runs the start script
Therefore I wrote the command export LD_LIBRARY_PATH=$LD_LIBRARY_PATH into the file <replicate>/bin/site_arep_login.sh which seems to be loaded by repctl.sh
The file site_arep_login.sh is for site specific stuff
Because of this export command, LD_LIBRARY_PATH is also set for user attunity and replicate starts
Hope this helps other users who want to build a docker image
Full custom start script
#!/bin/bash
# !!! MODIFIED COPY OF STARTSCRIPT EMBEDDED IN REPLICATE-RPM !!!
# Expect four parameters:
# 1. Data folder
# 2. Admin password
# 3. Rest port
# 4. license file, or empty to indicate that no license needs to be imported.
# Create data folder and grant user attunity ownership of it
if [ -z $1 ] || [ -z $2 ] || [ -z $3 ]; then
echo "Usage: start-replicate_sis.sh <Data folder> <Admin password> <Rest port> [<license file>]"
exit 1
fi
mkdir -p "$1"
chown attunity:attunity "$1"
source /opt/attunity/replicate/bin/arep_login.sh
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH" >> /opt/attunity/replicate/bin/site_arep_login.sh
# Change admin password
su attunity -c "/opt/attunity/replicate/bin/repctl.sh -d $1 setserverpassword $2"
if [ ! -z "$4" ]; then
su attunity -c "/opt/attunity/replicate/bin/repctl.sh -d $1 importlicense license_file=$4"
fi
# Run Attunity Replicate
su attunity -c "/opt/attunity/replicate/bin/repctl.sh -d $1 service start rest_port=$3"
Great you got it running. A couple of things to consider:
You have to persist the ReplicateDataFolder between restarts of your container (default is /replicate/data). This is where the logs and the sqllite backend of Qlik live (this has the imported tasks, endpoints, source transaction log read/write progress, target write progress, etc). If you don't persist this directory then Qlik will start from new every start up (i.e. zero logs and zero tasks).
You will want to set a MasterKey externally and import it upon every fresh start of replicate. This is the private key that all sensitive data in the previously mentioned sqllite database is encrypted against. If you don't set it your endpoint passwords will not be able to be decrypted and you wont resume Qlik. Here's an example of you will likely want in the start up cript, before the service start:
u attunity -c "/opt/attunity/replicate/bin/repctl.sh -d ${ReplicateDataFolder} setmasterkey $5 master_key_scope=1" | sed 's/setmasterkey .*/setmasterkey [REDACTED]/'
assuming you passed your private key as the 5 parameter
Consider capturing SIGTERM and handling it by stopping the qlik replicate tasks via the command line. There is some loose documentation around this. This is one of the things I am going to look to build in the future, i.e. a qlikcli that you can call from a terminal to stop/start tasks that end up being API calls.
Considering gracefully stopping qlik replicate service before closing the container. This isn't super important but if you dont then pid files get left in a temp folder that means Qlik struggles to start the next time you start the container. Alternatively you can have this in your startup script before starting the service.
rm -f ${ReplicateDataFolder}/tmp/*pid
Passwords and masterkey information is sent to stdout when you import them. You can see in my example above I remove it from output but consider doing the same for your password import - if thats something you/your company consider a risk.
Similarly you need to think about how sensitive data gets in to the container in the first place. In docker you might pass them via a docker-compose file or the docker command but both of these mean having sensitive data in plain text. If you use some sort of password vault consider how you might get this information at runtime and in memory before running commands with sensitive information like password, masterkey or even task endpoint passwords! If you search on the forums I have provided details of using a Qlik Addon that obtains passwords from AWS Secrets Manager at runtime. This way qlik task endpoint passwords would not contain anything sensitive and qlik handles getting the information from that vault at runtime.
Finally, exposing logs. If you want to be able to see your logs without bashing in to your container consider something like I've done on my repo. In the start up script, after service start up I do this:
_ReplicateLogs="${ReplicateDataFolder}/logs"
declare -A tailed
start_tail() {
local logfile="$1"
# Skip archived logs
if [[ "$logfile" == *__*.log ]]; then
return
fi
# Skip if already tailed
if [[ -n "${tailed["$logfile"]+x}" ]]; then
return
fi
tailed["$logfile"]=1
echo "Tailing log file: $logfile"
tail -F "$logfile" 2>/dev/null | sed -u "s|^|$(basename "$logfile"): |" &
}
# Tail all existing active logs
for logfile in "$_ReplicateLogs"/*.log; do
start_tail "$logfile"
done
# Watch for new log files
inotifywait -m -e create "$_ReplicateLogs" |
while read -r path action file; do
if [[ "$file" == *.log ]]; then
start_tail "$_ReplicateLogs/$file"
fi
done &
# Keep container alive forever
wait -n
This means all logs in the log folder, excluding archived logs (which qlik does automatically every time you stop a task), are output to stdout. It prefixes every log entry with the file name as there is a log file for replicate commands (the repctl.sh commands) - repcmd.log , replicate service logs - repsrv.log, and then a log for every task you have, these always start reptask_<taskname with _ replacing spaces>.log.
That code also requires the package inotify-tools to be installed in the container. Stick it in your dockerfile!
