topic Re: Streaming table changes to Kafka in Talend Studio

Streaming table changes to Kafka

Anonymous — Sat, 16 Nov 2024 06:53:17 GMT

Hi,

I'm trying to capture SQL Server table changes in real-time and send them to a Kafka topic.

I know that I can use a CDC table and connect it to Kafka directly (CDC table -> Kafka), but is there another alternative using Talend (something like CDC table -> Talend -> Kafka, etc)?

I've tried using Talend's CDC feature, but it seems that it only works for batch jobs and not real-time... Do correct me if I'm wrong though.

Regards,

Veronica

Re: Streaming table changes to Kafka

vapukov — Fri, 11 Jan 2019 07:41:28 GMT

Hi,

any solution would be not a real-time (unless you not design sending a message direct from triggers), all will be - loop by schedule or infinite loop

so, you could run Talend job in loop - 1 sec delay or less

or you could check in internet SQL CDC to Kafka and find few other solution for read SQL Server CDC tables, but it still will be not a real real-time

Re: Streaming table changes to Kafka

Anonymous — Mon, 14 Jan 2019 02:21:48 GMT

Hi,

Yes, I am aware that I can create real-time streams from SQL Server directly to Kafka, but I was wondering whether it's possible to do it in a more user-friendly way with Talend.

Thanks for the input!

Re: Streaming table changes to Kafka

vapukov — Mon, 14 Jan 2019 02:50:36 GMT

as I know - not, all what you can do it schedule job in a loop (schedule or design Job as infinite loop)

with some other tools it same - you must define query interval