Kafka - Does Replicate guarantee that a message is delivered. Kafka endpoint internal parameters and default settings.

[sanketmistry]

Hi Team,

QR version: 7.0.0.652

We use Kafka one of the endpoint.

We would like to know the what is the setting for below internal parameters.

enable.idempotence 

request.required.acks

If possible , could you please share the internal values set for Kafka (rdfkakfaproperties)

Thanks,

Sanket.

[john_wang]

Hello @sanketmistry ,

The recommended internal parameters are:

rdkafkaProperties = "enable.idempotence=true;acks=all;max.in.flight.requests.per.connection=1"
rdkafkaTopicProperties = "acks=all"

With above settings that messages are always delivered once and only once.

Hope this helps.

Regards,

John.

[sanketmistry]

Hi John,

Does it mean, no duplicates. 

Since producer replicates data in batches, is there any chance the duplicates in case of failure.

Regards,

Sanket.

[Steve_Nguyen]

@sanketmistry

1. Replicate can not avoid duplicates on Kafka target.
Replicate can avoid it on targets that has a unique index, and a duplicate constraint error is generated by the target.
2. Please also note, that with Kafka target, duplicates rows can occur even at later stages, please see the explanation in this link: https://community.qlik.com/t5/Knowledge/Kafka-Does-Replicate-guarantee-that-a-message-is-delivered-o...