QR: Confluent Cloud vs Kafka endpoint

[Jon_Donker]

Hi,

We're embarking on an upgrade of our Qlik Replicate platform to 2024-11

We have the endpoint "Confluent Cloud" now available to us.

My question is - what does the "Confluent Cloud" endpoint has to offer over our existing Kafka endpoints that are going to Confluent cloud?

Is there a reason to switch our existing connections over?

[john_wang · Accepted solution]

Hello @Jon_Donker ,

Qlik Replicate’s Confluent endpoint is optimized for seamless integration with Confluent Cloud's Kafka environment.
It provides:

• Built-in support for API Key/Secret authentication

• Confluent Schema Registry integration

• Automatic topic creation and management via Confluent Cloud APIs (CC APIs)

• Secure connectivity using TLS encryption and SASL mechanisms tailored specifically for Confluent Cloud

You can see that the fields are not exact the same:

Kafka EP (Endpoint):

Confluent Cloud EP:

We recommend upgrading the endpoint connection to the new version for improved compatibility and support.

Hope this helps.

John.

[Jon_Donker]

Is there any expected message structure difference between the two endpoints?

[john_wang]

Hello @Jon_Donker ,

No, No difference is expected.

Regares,

John.

[Jon_Donker]

Hi,

We're doing some unit testing between the two connectors and discovered a difference:

Can you please confirm that this is a difference the two connectors?

[john_wang]

Hello @Jon_Donker ,

Thank you so much for the update.

You are right, there is a bit difference:

In the regular Kafka endpoint, the message key looks like this:

"key": "3",

However, in the new Confluent Cloud endpoint, the key is structured as:

"key": {
    "ID": {
        "int": 3
    }
},

This appears to be an enhancement in the new endpoint. It includes not only the primary key (PK) column(s) values but also their data types. This structure is especially useful when the PK is composed of multiple columns, as it ensures greater accuracy and clarity in message serialization.

Feel free to let me know if you need any additional information.

Regards,

John.