topic Re: Change KEY values going to Kafka in Qlik Replicate

Change KEY values going to Kafka

fj40wdh — Wed, 10 Jul 2024 13:40:41 GMT

We have a Qlik task with an Oracle source and a Kafka target (key columns defined). The app that updates Oracle changes data in the unique index, in the table. The problem for us, is we are using the unique key in Oracle as a primary key downstream of Kafka. The index data change is causing a Primary Key change down stream which is causing update problems (can't find the current record because the key changed). The question is, can we change just the key values in Qlik? I know there is before and after info, but we need to change the key values going to Kafka.

"key":
{"FACILITY":{"string":"BNP"},
"TICKET_NUMBER":{"string":"94054100"},
"WHSE":{"string":"1"},
"CATALOG_ID":{"string":"0073530040"},
"Q_LEVEL":{"string":"0"},
"UTC_NUMBER":{"string":" },
"QTY_TYPE":{"string":"PR"},
"LOC_ZONE":{"string":"3"},
"LOC_ROW":{"string":"D"},
"LOC_SECTION":{"string":"3A"},
"LOC_TIER":{"string":"3ZZ"},
"LOC_BIN":{"string":"ZZZ"}},
"value"
_data": { ...
_beforeData": { ...

Re: Change KEY values going to Kafka

john_wang — Wed, 10 Jul 2024 13:54:56 GMT

Hello @fj40wdh ,

Thanks for reaching out to Qlik Community!

I'm uncertain about the impact of changing the primary key or unique index on the downstream components of Kafka. Are you concerned about how this change might affect which partition the messages are sent to?

If yes, we have a similar discussion, Kafka Partition strategy by primary key.

Hope this helps.
John.

Re: Change KEY values going to Kafka

fj40wdh — Wed, 10 Jul 2024 14:47:38 GMT

Thanks!

I am not concerned about the key in Kafka. We will only be changing the value of one of the columns in the key. The partition will not be affected. Just wondering if there is a key variable like the before variable $BI__UTC_NUMBER or current value $UTC_NUMBER that be accessed to change the key value while leaving the current value as is?

Re: Change KEY values going to Kafka

john_wang — Wed, 10 Jul 2024 15:33:43 GMT

Hello @fj40wdh ,

Thanks for the update.

By default the Kafka message contains the record before image in "beforeData" section (see ②), however you still can use the $BI__<columnName> to get additional value of the before image (see ①), a sample of the column "name" before image and after image:

Hope this helps.

John.

Re: Change KEY values going to Kafka

fj40wdh — Wed, 10 Jul 2024 17:05:12 GMT

Thanks

I'm aware of the before image and after image. I'm need to know about the KEY "image" see the example in my original post. when you look at the topics in Kafka there is a KEY section, a CURRENT DATA section, and a BEFORE DATA section. All of this comes from Qlik Replicate. I know I can change the Before and After image data, I need to know if there is a way to change just the KEY data,

Note

You can change the after image and it will change the key, but I only need to change the key.

Thanks.

Re: Change KEY values going to Kafka

john_wang — Thu, 11 Jul 2024 01:11:55 GMT

Hello @fj40wdh ,

I guess I did not get the question well. Would you please export the task and attach the json - obfuscating sensitive information as needed.

Thank you,

John.

Re: Change KEY values going to Kafka

fj40wdh — Thu, 11 Jul 2024 13:12:17 GMT

Here's the file

Re: Change KEY values going to Kafka

fj40wdh — Thu, 11 Jul 2024 15:13:43 GMT

Here's a file with the record from Kafka, you can see the KEY section, then VALUE section, under the value section is the AFTER and BEFORE data.

Re: Change KEY values going to Kafka

john_wang — Fri, 12 Jul 2024 07:10:15 GMT

Hello @fj40wdh ,

Thanks for the information. Would you please open a support ticket with source table creation DDL and we'd like to reproduce the behavior in our internal labs to understand the issue. I'd love to work with out support team to move the case forward.

thanks,

John.