article Kafka - Does Replicate guarantee that a message is delivered only once and exactly the right order? in Official Support Articles

Kafka - Does Replicate guarantee that a message is delivered only once and exactly the right order?

Melissa_Potvin — Thu, 20 Jan 2022 15:52:02 GMT

[publishing on behalf of Global Support]

Replicate guarantees that messages are delivered to Kafka at least once.

Each message contains a “change sequence” field (same as in CT tables), which is monotonically increasing. In case a message was produced to Kafka more than once, the customer is able to detect it and ignore that message.

Replicate produces messages in batches. At the same time, different batches are sent to different broker machines (depends which broker is the leader of which partitions at a given time).

It is possible that record X is produced to broker B1 and record X+1 is produced to broker B2.

Broker B2 might respond fast and return an acknowledgment for X+1, while broker B1 might be slower (or down) and record X will get into recovery or fail.
In that case, Replicate task will start sending the stream of records as follows:

Replicate v6.2 and lower -- from the earliest record that failed (record X).
Replicate v6.3 and higher -- from the beginning of the failed transaction.

NOTES

It is possible that some of the following records (X+1) will be duplicated on Kafka.
As mentioned above, the customer can easily filter out these duplicates, if exist.

Re: Kafka - Does Replicate guarantee that a message is delivered only once and exactly the right ord

Prabodh — Fri, 14 Aug 2020 20:57:26 GMT

Does Replicate support idempotent producer?

Re: Kafka - Does Replicate guarantee that a message is delivered only once and exactly the right ord

Anonymous — Fri, 14 Aug 2020 12:59:18 GMT

@Melissa_Potvin I have a question or two concerning this and how perhaps the Replicate product has evolved.

We are still running some replication to Kafka on Replicate version 5.5, and we identified cases where change records would be dropped if a leader change occurred in the cluster. Replicate would not see this as a failure and would not re-send the batch.

We had to employ an internal parameter to force acknowledgment (rdKafkaTopicProperties set to acks =all) to guarantee delivery. We still use it to this day.

Has this scenario changed in newer versions of Replicate?

Re: Kafka - Does Replicate guarantee that a message is delivered only once and exactly the right ord

Steve_Nguyen — Fri, 14 Aug 2020 13:35:59 GMT

In 6.4xx and newer,

you can use the following :

You can change the value by doing these steps:

1. Open the Kafka endpoint, click on the Advanced tab.

2. Click on Internal Parameters.

3. Add an internal parameter named "rdkafkaTopicProperties" give it a value of "request.required.acks=all"

Re: Kafka - Does Replicate guarantee that a message is delivered only once and exactly the right order?

JitenderR — Mon, 23 May 2022 21:45:11 GMT

Is the above good with latest 2021.11 version as well?