topic Re: Kafka SMT for Kafka Sinks in Qlik Replicate

Kafka SMT for Kafka Sinks

chris_s — Mon, 13 Mar 2023 09:30:28 GMT

Hi Qlik Community

We use Qlik Replicate to repliacate a database as CDC Events into Kafka. This works quite performant and we are happy with the producing side.

But we are missing something on the consumer side of things.
We would like to have a mechanism to flatten the CDC Events back to the original structure. This should be possible by taking the headers.operation into account:

REFRESH => perform upsert on DB
UPDATE => perform update on DB
INSERT => perform insert on DB
DELETE => perform delete on DB (or soft delete)

Our current solution is to implement that as views in the DB. But this is cumbersome to manage.

We saw that there is a Kafka single message transformation (SMT) to do this task: https://github.com/michelin/kafka-connect-transforms-qlik-replicate & https://blogit.michelin.io/open-sourcing-kafka-connect-transforms-for-qlik-replicate/

But this project has not has any release in the last 2 years.

Is there a similar Artifact from Qlik?

Re: Kafka SMT for Kafka Sinks

john_wang — Mon, 13 Mar 2023 15:05:41 GMT

Hello @chris_s ,

Thanks for you opening the article, really an interesting topic. In Qlik Replicate, the message format can be tailor-made include removing the beforeImage, flatten the hierarchical structure of the section etc.

Hope it helps.

Regards,

John.

Re: Kafka SMT for Kafka Sinks

chris_s — Tue, 14 Mar 2023 07:09:23 GMT

Hi John
Thanks for the answer. Unfortunately it is not enough for our use case. We already produce the messages and have several consumers (which are happy with the current message format).

Hence we are looking for some kind of process-logic which interpret the headers fields and do the propper actions on a receiving db.

And probably is "flatten" a missleading word for that. Flatten would also be possible on a kafka sink connector using an existing SMT: https://docs.confluent.io/platform/current/connect/transforms/flatten.html