Skip to main content

Suggest an Idea

Vote for your favorite Qlik product ideas and add your own suggestions.

Announcements
This page is no longer in use. To suggest an idea, please visit Browse and Suggest.

Support tomb-stoning methodology for Kafka target endpoints

Jose_Pena
Employee
Employee

Support tomb-stoning methodology for Kafka target endpoints

When there is a change to PK fields on the source system, Replicate sends to Kafka the previous PK data as part of the envelope for the update message. However, the update message has the new PK data as key. The result is that it can go to a different Kafka partition than the old message. This can impact to downstream consumers since there are no ordering guarantees for messages on different partitions.

The idea is to send to Kafka a tombstone record for keys that no longer exist on the source system, not just to add the previous PK as metadata in the message envelope on a completely different key (the “beforeData” for each message).

Here is a use case: suppose there is an Email table in the source database where primary key fields are PersonId and EmailType. Suppose there is a record where PersonId=1 and EmailType=Office. Last, suppose someone were to UPDATE that record to have PersonId=1 and EmailType=Personal.

Current behavior:

Partition A:
Message1: Key - { “personId”: 1, “emailType”: “office” } / Value – { “address”: something@domain.com , “beforeData”: null}

Partition B:
Message1: Key - { “personId”: 1, “emailType”: “personal” } / Value – { “address”: something@domain.com , “beforeData”: { “personId”: 1, “emailType”: “office” }

In current behavior, the consumer is required to look at the “beforeData” for each message and create a tombstone record, with a potential processing for the original key from the new partition (B). This adds complexity on the consuming side.

Tomb-stoning methodology:

Partition A:
Message1: Key - { “personId”: 1, “emailType”: “office” } / Value – { “address”: something@domain.com }
Message2: Key - { “personId”: 1, “emailType”: “office” } / Value – null

Partition B:
Message1: Key - { “personId”: 1, “emailType”: “personal” } / Value – { “address”: something@domain.com }

With the tomb-stoning methodology, Message2 would be the tombstone message for the key { “personId”: 1, “emailType”: “office” }, handled by the producer and avoiding extra complexity on the producing side.

9 Comments
Ola_Mayer
Employee
Employee
 
Status changed to: Open - Collecting Feedback
pgalluzzo
Contributor
Contributor

I vote for this also!

BradA
Contributor III
Contributor III

I'd at least like to see source endpoint deletes replicated to Kafka configurable to use either the Replicate classic method for generating "delete" messages or to generate a true Kafka tombstone message where the key is provided, but the message body is NULL.     This configuration option should be applied regardless of whether the source system PK was changed or just deleted.

 

 

MichaelMockus
Contributor III
Contributor III

Any movement on this? This is a common feature in most other change data capture products including several of your competitors. 

Without this feature, data written into kafka that is maintained and compacted violates the data rules when it is partitioned at all, as a record will show as active in 2 partitions, thought it has truly just had a primary key update.

tkdrahn
Contributor
Contributor

This is causing data integrity issues throughout our system?  When will it be fixed?

Tzachi_Nissim
Employee
Employee

Hi,

Thanks to all that commented and voted on this ideation. We hear you and are looking to address this as one of the higher priority items in our backlog. I can't provide an ETA yet, but I will when I have one.

Regards,

Tzachi

bobvecchione
Employee
Employee

Hi folks,

This has made it up the priority list and we are planning to enable the Replicate Engine to support updates ok PKs in the Nov 22 release - which will enable support of Tombstoning.

 

--bobv--

Meghann_MacDonald

From now on, please track this idea from the Ideation portal. 

Link to new idea

Meghann

NOTE: Upon clicking this link 2 tabs may open - please feel free to close the one with a login page. If you only see 1 tab with the login page, please try clicking this link first: Authenticate me! then try the link above again. Ensure pop-up blocker is off.

Ideation
Explorer II
Explorer II
 
Status changed to: Closed - Archived