How can we verify FL and CDC when target is Confluent Kafka on GCP?

[judian_lin]

Hi All,

We want to replicate from RDBMS to  Confluent Kafka on GCP, but don't know how to verify FL and CDC.

Does anyone have an idea about it?

Qlik Replicate 

[john_wang · Accepted solution]

Hello @judian_lin ,

Thanks for reaching out to Qlik Community!

I'm assuming you want to find a way to compare the data for example rows number etc between the source RDBMS and Kafka target after task run. looks to me we may bread the process to 2 stage:

1- Full Load stage

2- CDC stage

You may set the task (if the task set to Full Load and CDC) to Stop after the Full Load done:

then compare the 2 sides (rows operation = REFRESH). after the comparison done, RESUME the task and compare the CDC stage rows (rows with operation is equal to INSERT/UPDATE/DELETE). Please check the operation values in User Guide:

REFRESH – Insertion of a record to the target during Full Load

INSERT – Insertion of new target record

UPDATE – Update of an existing target record

DELETE – Deletion of a target record

Hope this helps.

John.