How to run the ETL job automatically when trigger Fires (CDC)
Hi Team ,
I would like to design a Talend Job which will trigger automatically whenever there is a change in the tables(Inserts/Updates) and process those changes to corresponding databases. I am using Talend Open Studio , Mysql - DB's. Would it be possible to trigger the talend Job only there is a change occured in the tables ?
For proper CDC - use 3rd party tools for capture CDC from MySQL to Kafka and then Talend Listen Kafka and parse changes
Trigger Based CDC can be created with OpenSource version as well - add triggers, read the table + realise logic for multi subscribers, it as well allow handle situations when same records was - INSERTED/UPDATED and than DELETED from table.
All problem of any trigger based solution - it not fire Job by trigger, it will be or permanent loop with SELECT from table, or scheduled Job.
For scheduled Job - You do not need trigger in many cases, just need have 2 timestamp columns - CreatedAt, UpdatedAt ...
Tools and products for real CDC (I leave in list only not Talend competitors): - MariaDB MaxScale - https://mariadb.com/products/technology/maxscale , it part of Enterprise packet, but subscription required if You use it for more than 3 server in production environment.
