topic Re: Kafka component deployment issue in Talend Studio

Kafka component deployment issue

Anonymous — Sat, 16 Nov 2024 06:15:27 GMT

Hi All,

Greetings,

We have developed Kafka integration ( consume message from Apache Kafka topic and load into back end Database ) in "open studio for big data version 7.1 " and deployed into "open studio for ESB run time version 7.1".

1. Kafka consumer integration is working fine in local "open studio for big data"

2. Same integration is not working in "open studio for ESB runt time" , we have added Kafka jars in Run time server as well

3. We have not seen any errors in "open studio for ESB run time" logs

My question is -

Can we deploy Kafka component integration ( which we developed in open studio for big data ) into open studio for ESB run time ?

Note - we have developed kafka integration in open studio for big data as we don't see kafka component in "open studio for ESB"

Appreciate for your help.

Regards,

Satya

Re: Kafka component deployment issue

vapukov — Tue, 26 Mar 2019 01:38:11 GMT

you can not import unsupported components from TOS BD into TOS ESB

generally and not too many reasons for this

if you develop a standalone job - you could run it from OS (as normal batch or powershell script)
if you develop route - TOS for ESB contain separate component cKafka for work with Kafka brockers. You could receive message in the route with cKafka component and transfer context to DI job for ETL transformations

cKafka

cTalendJob

regards, Vlad

Re: Kafka component deployment issue

Anonymous — Tue, 26 Mar 2019 03:23:08 GMT

Hi Vlad,

Appreciate your response.

As per my understanding,

"tKafka" components not part of Talend open studio for ESB, its part of open studio for Big Data.

Can you please suggest how can we achieve below requirement with Talend open studio products ?

our requirement is -

1. Apache Kafka will publish message to Topic

2. Talend job should subscribe message from Topic and load into back end database.

3. Talend job should not be cron job or scheduler. it should listen Apache Kafka and subscribe if found any latest messages.

And, what is cKafka component ? where can i find cKafka component in open studio for ESB ?

Can you please provide more details on "If you develop route" in your response.

Thank you much for all your help.

Regards,

Sagar

Re: Kafka component deployment issue

vapukov — Tue, 26 Mar 2019 03:48:36 GMT

@Satya17 wrote:

Can you please provide more details on "If you develop route" in your response.

Thank you much for all your help.

Regards,

Sagar

Hi,

check first (just from a google search 😉 )

https://cleverbuilder.com/articles/mediation-vs-integration/

and of course, install and check all available demos:

https://help.talend.com/reader/EDncCk~GQGYJDo13J1a77g/zOczPWm3KuUqQw78e5doMw

+ Talend Help

it helps you to have an idea - what is it and how it works

regarding to:

3. Talend job should not be cron job or scheduler. it should listen Apache Kafka and subscribe if found any latest messages.

it exactly about routes, but and DI job could be started and do an infinite loop

regards, Vlad