topic Re: Real time change data capture in Talend Studio

Real time change data capture

Anonymous — Sat, 16 Nov 2024 08:49:51 GMT

I want to create a talend big data batch job where we need to compare 2 tables..tables have a primary key..while comparing if any changes are detected we need to update or insert in the second table..in short I want to do a real cdc .. please help me on this as I am a newbie to talend...

Re: Real time change data capture

Anonymous — Sun, 21 Jan 2018 17:49:19 GMT

Any help on real cdc with talend big data batch job...

Re: Real time change data capture

Anonymous — Thu, 25 Jan 2018 10:12:04 GMT

Hello,

So far, talend CDC components are not available in bigdata batch job.

If you want to capture the changed data and only load these changed data into target table to achieve table sync, you can compare tables by using tMap.

Best regards

Sabrina

Re: Real time change data capture

Anonymous — Sat, 27 Jan 2018 17:00:50 GMT

Hello,

Thanks for the reply.. my table contains billion records , so which would be the best option of table compare like hbase or hive components in big data Hadoop environment .. and what would be the logic.. I just have a pk column in table and no date column..

Thanks in advance

Re: Real time change data capture

Anonymous — Tue, 30 Jan 2018 03:51:48 GMT

Hello,

Please refer to this online document about:TalendHelpCenter: Best Practice: Change Data Capture with Spark in Big Data.

Hope it will help.

Best regards

Sabrina