topic Re: How to run the ETL job automatically when trigger Fires (CDC) in Talend Studio

How to run the ETL job automatically when trigger Fires (CDC)

hami1 — Thu, 15 Jun 2017 08:57:46 GMT

Hi Team ,

I would like to design a Talend Job which will trigger automatically whenever there is a change in the tables(Inserts/Updates) and process those changes to corresponding databases. I am using Talend Open Studio , Mysql - DB's. Would it be possible to trigger the talend Job only there is a change occured in the tables ?

Thanks,

hmd.

Re: How to run the ETL job automatically when trigger Fires (CDC)

Anonymous — Thu, 15 Jun 2017 09:37:54 GMT

Hello
The CDC feature is only available in Talend Enterprise Subscription product, for more information about CDC, please read this page:
https://help.talend.com/reader/sSAk2HQITuEIWJdkl3e1Rw/xMY3zAjqxlBYJS7sin671w

Regards
Shong

Re: How to run the ETL job automatically when trigger Fires (CDC)

vapukov — Thu, 15 Jun 2017 11:58:49 GMT

Shong!

For proper CDC - use 3rd party tools for capture CDC from MySQL to Kafka and then Talend Listen Kafka and parse changes

Trigger Based CDC can be created with OpenSource version as well - add triggers, read the table + realise logic for multi subscribers, it as well allow handle situations when same records was - INSERTED/UPDATED and than DELETED from table.

All problem of any trigger based solution - it not fire Job by trigger, it will be or permanent loop with SELECT from table, or scheduled Job.

For scheduled Job - You do not need trigger in many cases, just need have 2 timestamp columns - CreatedAt, UpdatedAt ...

Tools and products for real CDC (I leave in list only not Talend competitors):
- MariaDB MaxScale - https://mariadb.com/products/technology/maxscale , it part of Enterprise packet, but subscription required if You use it for more than 3 server in production environment.

- Maxwell - https://github.com/zendesk/maxwell

- http://debezium.io/docs/connectors/mysql/

- Apache NiFi 1.2+ https://nifi.apache.org