topic Re: Confluent Schema Registry | Subject naming strategy in Qlik Replicate

Confluent Schema Registry | Subject naming strategy

ernesto_costa — Tue, 20 Feb 2024 15:15:52 GMT

Hi,

when sending data to Confluent Kafka using Schema Registry, and using in the target configuration the following configuration:

What is being created on the schema registry is the following:

I don't understand why the highlighted part is being added to the subject name of the schema since the strategy is only to use the topic and record name, which is present in the part which is NOT highlighted.

Besides, in the task definition, we have the following configuration:

Is there a way to, when using the topic and record name subject naming strategy, to create it only with the topic name and record name?

Taking the above example, to create a schema in Schema Registry named "raw.EG3DTRGN.ORDHVDT1.DataRecord-value"?

Re: Confluent Schema Registry | Subject naming strategy

SushilKumar — Thu, 22 Feb 2024 11:54:03 GMT

Hello Team,

Request you to follow below section in Qlik help link.

Data message publishing

Schema Registry subject properties

Setting general connection properties #Setting general connection properties | Qlik Replicate Help

Regards,

Sushil Kumar

Re: Confluent Schema Registry | Subject naming strategy

john_wang — Sat, 24 Feb 2024 10:59:55 GMT

Hello @ernesto_costa ,

You may custom the namespace expression even with the "Include Namespace" enabled. For example change the expression from ~~com.attunity.queue.msg.{{TASK_NAME}}.~~{{TARGET_TABLE_OWNER}}.{{TARGET_TABLE_NAME}} to {{TARGET_TABLE_OWNER}}.{{TARGET_TABLE_NAME}}:

Then the result in Kafka schams topic:

{
"keytype": "SCHEMA",
"subject": "dbo.kit-dbo.kit.DataRecord-value",
"version": 1,
"magic": 1
}

where "dbo" is the schema name and "kit" is table name in SQL Server source DB.

Hope this helps.

John.

Re: Confluent Schema Registry | Subject naming strategy

SushilKumar — Mon, 26 Feb 2024 06:22:41 GMT

Hello Team,

If our response has been helpful, please consider clicking "Accept as Solution". This will assist other users in easily finding the answer.

Regards,

Sushil Kumar

Re: Confluent Schema Registry | Subject naming strategy

ernesto_costa — Mon, 26 Feb 2024 16:31:49 GMT

I'm going to validate and then provide feedback.

Thank you.

Re: Confluent Schema Registry | Subject naming strategy

john_wang — Tue, 27 Feb 2024 01:12:10 GMT

Thank you for the update @ernesto_costa , Good luck.

Re: Confluent Schema Registry | Subject naming strategy

ernesto_costa — Fri, 01 Mar 2024 16:37:02 GMT

Hi John,

the suggestion you made does not fix the issue.

By choosing to include the namespace the minimum name possible to achieve is by removing the entries in the namespace text box, as seen below

and what is generated in schema registry is what can be seen below

As it can be seen, the expected result would be

cdc.aspect4.EG3DT.XLPLKDP1-value
cdc.aspect4.EG3DT.XLPLKDP1-key

But what is created has the following subjects

cdc.aspect4.EG3DT.XLPLKDP1-.DataRecord-value
cdc.aspect4.EG3DT.XLPLKDP1-.KeyRecord-key

Btw, the DataRecord and KeyRecord entries cannot be removed or else the task fails to start as seen below:

So, our question remains unresolved.

Re: Confluent Schema Registry | Subject naming strategy

john_wang — Sun, 03 Mar 2024 11:02:56 GMT

Hello @ernesto_costa ,

Thanks for the update.

I got below result in my labs, where "dbo" is source table owner name, "kit" is the table name:

{
"keytype": "SCHEMA",
"subject": "dbo.kit-value",
"version": 1,
"magic": 1
}

We need 2 steps:

1. In Kafka endpoint setting, set "Subject name strategy" to "Topic Name":

2. In Task setting, set Namespace to "{{TARGET_TABLE_OWNER}}.{{TARGET_TABLE_NAME}}":

Hope this helps.

John.

Re: Confluent Schema Registry | Subject naming strategy

ernesto_costa — Mon, 04 Mar 2024 11:08:16 GMT

Hi John,

thanks, going to validate and will provide feedback ASAP.

Re: Confluent Schema Registry | Subject naming strategy

john_wang — Mon, 04 Mar 2024 13:33:09 GMT

Hello @ernesto_costa ,

Thanks for the feedback!

Good luck,

John.