Export/Import with Git for Qlik Replicate Tasks

[fruityhost]

Every few months I go looking for things in my company that could run smoother, and this time I landed on Qlik Replicate. So I asked the engineers and consultants how they handle changes to tasks.

In short: tasks reuse is export, import, and hand-editing the JSON in between. Version control is a folder of exported files, if someone remembered to export. Transformations stay stuck in the task they were written in. And none of it ever settles, because new sources and targets keep arriving.

So we built a first version of Git versioning for Qlik Replicate Tasks. Here is how it works:

1. A browser extension (Gitoqlok, iykyk) grabs the list of Replicate tasks from the console links a Git repository to it (GitHub, GitLab, Bitbucket, Azure DevOps). The task definition goes in as the first commit.

1. Repository then contains the JSON that Replicate's Export button gives you: endpoints, table selection, transformations, task settings. Plus a README generated from it: source, target, endpoints, the table list.

1. Change and commit. Edit the task in the console as usual. Next time you open the popup it shows what changed, line by line. Here I added one transformation on a table, and it shows exactly the 28 lines it touched. Type a message, Commit & Push.

1. Go back. History dropdown, Restore button. Any version goes back into Replicate as a full replacement, deleted table selections included.

So the versioning and rollback part is done. My next idea was import: a task committed from DEV gets imported into TEST or PROD straight from Git, any saved version.

Then the sheer number of transformations inside a single task got me thinking: maybe transformations deserve versioning of their own... Imagine an import that pulls just one piece, a transformation or the logging levels, from another task you've already configured.

Some additional details: our install doesn't have EM, so this uses the console's API.

Would appreciate your thoughts, any feedback. If anyone's interested, I'd be happy to share the solution!

#QlikReplicate #Qlik #DataEngineering #CDC #DevOps 

[AlexPolorotov]

That's what our team looked for! 

[Vegy]

We built an in house CLI for Qlik that utilises the same Web API that the qlik page uses. If you open up developer mode you can see the end points, json payloads, and responses. This will allow you to create a task import that way.

Why do this instead of using the repctl cli? Because that only supports importing from flat files, which means your end point secrets need to be in plain text OR you build your own C addon to allow Qlik to get passwords from your secrets vault. I have also done this (I have a post on here somewhere with the code I added).

Currently we create our task files in JSON on Git, some of it is static but the table list is pulled in by terraform at apply stage to allow us to easily add tables without people having to write the full JSON. Terraform then uploads it to S3 and our qlik container pulls from S3 buckets at start up.

We put the AWS Secrets Manager ARN in the password fields, and then as part of our qlikcli task import we look for secret arns in the task json that has been downloaded at start up, then replace the ARNS with the plain text secret value but in memory, before importing to Qlik.

This way we never have any secrets stored at rest, not even momentarily

[Vegy]

Just to add to this, I will be, in the near future, building a python CLI for qlik utilising this method and sharing on my public github. Some of my current work - which is around containerising Qlik, can be found here https://github.com/VegyBS/qlik-replicate-docker and no this isnt a sales ploy its all open, etc

[Vegy]

Finally, the C addon I wrote can be found here on these forums https://community.qlik.com/t5/Qlik-Replicate/AWS-Secrets-Manager-lookup/m-p/2543080, I wouldnt call it prod ready and I don't know C so it was a case of utilising claude. It worked as intended though. I personally think this is the best world really

[john_wang]

Hello  ,

Thank you so much for your outstanding support! We’re really glad to have someone with such strong expertise in Qlik/QEM. We truly appreciate your help and support!

Best Regards,

John.