topic Re: How to manage multiple changes with CDC technology in Talend Studio

How to manage multiple changes with CDC technology

castiellll — Sat, 16 Nov 2024 04:50:17 GMT

Hello,

I have been thinking about the way to manage multiple changes when using CDC.

Explanation :

Let's say we have a really bad model with 2 tables users and users_status (1-1). We want to merge them into one table dim.users in a DWH.

CDC is activated for both tables so we can track changes in both and our Talend flow has the user table with an inner join (lookup) on the user_status table (very simple).

The problem is when there is a change in only one of the tables ? how do you manage the fact that if a changes happens in only one of these tables and not in the other ??

Example:

table user : (id, name) --> (1,'toto'), (2,'titi')

table user_status (id_user, status) --> (1,true), (2,false)

if we change the status of 'titi' to true then the CDC will get (2,true) for the user_status change table but nothing for the user change table. So our Talend flow won't update anything.

Maybe am I missing something in the CDC configuration ?

Thank you in advance for your help.

Regards,

Sofiane.

Re: How to manage multiple changes with CDC technology

Anonymous — Wed, 04 Sep 2019 10:23:17 GMT

Hello,

We are supposing that you have referred this online user guide about:TalendHelpCenter:Change Data Capture scenarios

The cdc tables only store the change type and primary key. This is a limitation of our current solution, how ever the triggers and components could be customized to save every change record.

Best regards

Sabrina

Re: How to manage multiple changes with CDC technology

castiellll — Wed, 04 Sep 2019 14:11:40 GMT

Hello,

Thank you for you answer but this is not answering my question.

My CDC is managed in the DB not in Talend. The question is more about a best architecture practice.

Good day.