idea Re: Support tomb-stoning methodology for Kafka target endpoints in Suggest an Idea

Support tomb-stoning methodology for Kafka target endpoints

Jose_Pena — Thu, 18 Jun 2020 19:29:22 GMT

When there is a change to PK fields on the source system, Replicate sends to Kafka the previous PK data as part of the envelope for the update message. However, the update message has the new PK data as key. The result is that it can go to a different Kafka partition than the old message. This can impact to downstream consumers since there are no ordering guarantees for messages on different partitions.

The idea is to send to Kafka a tombstone record for keys that no longer exist on the source system, not just to add the previous PK as metadata in the message envelope on a completely different key (the “beforeData” for each message).

Here is a use case: suppose there is an Email table in the source database where primary key fields are PersonId and EmailType. Suppose there is a record where PersonId=1 and EmailType=Office. Last, suppose someone were to UPDATE that record to have PersonId=1 and EmailType=Personal.

Current behavior:

Partition A:
Message1: Key - { “personId”: 1, “emailType”: “office” } / Value – { “address”: something@domain.com , “beforeData”: null}

Partition B:
Message1: Key - { “personId”: 1, “emailType”: “personal” } / Value – { “address”: something@domain.com , “beforeData”: { “personId”: 1, “emailType”: “office” }

In current behavior, the consumer is required to look at the “beforeData” for each message and create a tombstone record, with a potential processing for the original key from the new partition (B). This adds complexity on the consuming side.

Tomb-stoning methodology:

Partition A:
Message1: Key - { “personId”: 1, “emailType”: “office” } / Value – { “address”: something@domain.com }
Message2: Key - { “personId”: 1, “emailType”: “office” } / Value – null

Partition B:
Message1: Key - { “personId”: 1, “emailType”: “personal” } / Value – { “address”: something@domain.com }

With the tomb-stoning methodology, Message2 would be the tombstone message for the key { “personId”: 1, “emailType”: “office” }, handled by the producer and avoiding extra complexity on the producing side.

Re: Support tomb-stoning methodology for Kafka target endpoints - Status changed to: Open - Collecting Feedback

Ola_Mayer — Mon, 06 Jul 2020 13:46:58 GMT

Re: Support tomb-stoning methodology for Kafka target endpoints

pgalluzzo — Tue, 06 Apr 2021 13:17:50 GMT

I vote for this also!

Re: Support tomb-stoning methodology for Kafka target endpoints

BradA — Tue, 06 Apr 2021 13:31:33 GMT

I'd at least like to see source endpoint deletes replicated to Kafka configurable to use either the Replicate classic method for generating "delete" messages or to generate a true Kafka tombstone message where the key is provided, but the message body is NULL. This configuration option should be applied regardless of whether the source system PK was changed or just deleted.

Re: Support tomb-stoning methodology for Kafka target endpoints

MichaelMockus — Wed, 28 Jul 2021 21:24:10 GMT

Any movement on this? This is a common feature in most other change data capture products including several of your competitors.

Without this feature, data written into kafka that is maintained and compacted violates the data rules when it is partitioned at all, as a record will show as active in 2 partitions, thought it has truly just had a primary key update.

Re: Support tomb-stoning methodology for Kafka target endpoints

tkdrahn — Fri, 24 Sep 2021 17:23:21 GMT

This is causing data integrity issues throughout our system? When will it be fixed?

Re: Support tomb-stoning methodology for Kafka target endpoints

Tzachi_Nissim — Tue, 16 Nov 2021 10:25:36 GMT

Hi,

Thanks to all that commented and voted on this ideation. We hear you and are looking to address this as one of the higher priority items in our backlog. I can't provide an ETA yet, but I will when I have one.

Regards,

Tzachi

Re: Support tomb-stoning methodology for Kafka target endpoints

bobvecchione — Wed, 04 May 2022 21:08:46 GMT

Hi folks,

This has made it up the priority list and we are planning to enable the Replicate Engine to support updates ok PKs in the Nov 22 release - which will enable support of Tombstoning.

--bobv--

From now on, please track this idea from the Ideation por...

Meghann_MacDonald — Wed, 02 Aug 2023 16:35:35 GMT

From now on, please track this idea from the Ideation portal.

Link to new idea

Meghann

NOTE: Upon clicking this link 2 tabs may open - please feel free to close the one with a login page. If you only see 1 tab with the login page, please try clicking this link first: Authenticate me! then try the link above again. Ensure pop-up blocker is off.

Re: Support tomb-stoning methodology for Kafka target endpoints - Status changed to: Closed - Archived

Ideation — Wed, 02 Aug 2023 16:35:36 GMT