Skip to main content
Announcements
UPGRADE ADVISORY for Qlik Replicate 2024.5: Read More
cancel
Showing results for 
Search instead for 
Did you mean: 
chris_s
Contributor II
Contributor II

Kafka SMT for Kafka Sinks

Hi Qlik Community

 

We use Qlik Replicate to repliacate a database as CDC Events into Kafka. This works quite performant and we are happy with the producing side. 

But we are missing something on the consumer side of things. 
We would like to have a mechanism to flatten the CDC Events back to the original structure. This should be possible by taking the headers.operation into account:

  • REFRESH => perform upsert on DB
  • UPDATE => perform update on DB
  • INSERT => perform insert on DB
  • DELETE  => perform delete on DB (or soft delete)

Our current solution is to implement that as views in the DB. But this is cumbersome to manage.

We saw that there is a Kafka single message transformation (SMT) to do this task: https://github.com/michelin/kafka-connect-transforms-qlik-replicate & https://blogit.michelin.io/open-sourcing-kafka-connect-transforms-for-qlik-replicate/ 

But this project has not has any release in the last 2 years.

 

Is there a similar Artifact from Qlik? 

Labels (1)
2 Replies
john_wang
Support
Support

Hello @chris_s ,

Thanks for you opening the article, really an interesting topic. In Qlik Replicate, the message format can be tailor-made include removing the beforeImage, flatten the hierarchical structure of the section etc. 

Hope it helps.

Regards,

John.

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
chris_s
Contributor II
Contributor II
Author

Hi John
Thanks for the answer. Unfortunately it is not enough for our use case. We already produce the messages and have several consumers (which are happy with the current message format).

Hence we are looking for some kind of process-logic which interpret the headers fields and do the propper actions on a receiving db.

And probably is "flatten" a missleading word for that. Flatten would also be possible on a kafka sink connector using an existing SMT: https://docs.confluent.io/platform/current/connect/transforms/flatten.html