Kafka SMT for Kafka Sinks

[chris_s]

Hi Qlik Community

We use Qlik Replicate to repliacate a database as CDC Events into Kafka. This works quite performant and we are happy with the producing side. 

But we are missing something on the consumer side of things. 
We would like to have a mechanism to flatten the CDC Events back to the original structure. This should be possible by taking the headers.operation into account:

• REFRESH => perform upsert on DB
• UPDATE => perform update on DB
• INSERT => perform insert on DB
• DELETE  => perform delete on DB (or soft delete)

Our current solution is to implement that as views in the DB. But this is cumbersome to manage.

We saw that there is a Kafka single message transformation (SMT) to do this task: https://github.com/michelin/kafka-connect-transforms-qlik-replicate & https://blogit.michelin.io/open-sourcing-kafka-connect-transforms-for-qlik-replicate/ 

But this project has not has any release in the last 2 years.

Is there a similar Artifact from Qlik? 

[john_wang]

Hello @chris_s ,

Thanks for you opening the article, really an interesting topic. In Qlik Replicate, the message format can be tailor-made include removing the beforeImage, flatten the hierarchical structure of the section etc. 

Hope it helps.

Regards,

John.

[chris_s]

Hi John
Thanks for the answer. Unfortunately it is not enough for our use case. We already produce the messages and have several consumers (which are happy with the current message format).

Hence we are looking for some kind of process-logic which interpret the headers fields and do the propper actions on a receiving db.

And probably is "flatten" a missleading word for that. Flatten would also be possible on a kafka sink connector using an existing SMT: https://docs.confluent.io/platform/current/connect/transforms/flatten.html