topic Kafka target Change Table message key configurations in Qlik Replicate

Kafka target Change Table message key configurations

mouse_keyboard — Thu, 25 Jul 2024 03:39:45 GMT

Hi,

My kafka destination is connected to the SQL Server as source, connection is set as below:

- separate topic for each table

- transformation rules to add prefix to schema

On Full Load:

- Message key is using primary key, and thus the partitioning also is using primary key

On Change Data Capture:

- Message key becomes the timestamp like "202407080525482000000000000000040249"+"B"

Just wondering it is possible to make Full Load and CDC both share the same message key? (There are around 200 tables with mixtures of primary key and composite key, thus I cannot go into each table and define a transformation rule on $partition column separately)

Re: Kafka target Change Table message key configurations

john_wang — Thu, 25 Jul 2024 04:00:29 GMT

Hello @mouse_keyboard ,

Welcome to Qlik Community forum and thanks for reaching out here!

I'm not sure what's the purpose of the transform to "resolve kafka topics". If you intended to specify the topic name , or topic name rule, then please use the $topic, see User Guide Overriding the default settings.

Hope this helps.

John.

Re: Kafka target Change Table message key configurations

mouse_keyboard — Thu, 25 Jul 2024 04:20:51 GMT

Hi John,

please ignore the topic name related transformation. My main question is how to make it such that both the full load and the CDC table share the same message key (primary key)

Re: Kafka target Change Table message key configurations

john_wang — Thu, 25 Jul 2024 04:48:35 GMT

Hello @mouse_keyboard ,

Thanks for the update however the behavior cannot be reproduced in my local labs.

IN my env both Full Load and CDC Change Processing produce the correct message key, see my screenshot, the first row is produced in Full Load stage, the 2nd rows is CDC message. The message key is the table's PK value "2":

Regards,

John.