Kafka component deployment issue

[Anonymous]

Hi All,

Greetings,

We have developed Kafka integration ( consume message from Apache Kafka topic and load into back end Database ) in "open studio for big data version 7.1 " and deployed into "open studio for ESB run time version 7.1".

1. Kafka consumer integration is working fine in local "open studio for big data"

2. Same integration is not working in "open studio for ESB runt time" , we have added Kafka jars in Run time server as well

3. We have not seen any errors in "open studio for ESB run time" logs 

My question is -

Can we deploy Kafka component integration ( which we developed in open studio for big data ) into open studio for ESB run time ?

Note - we have developed kafka integration in open studio for big data as we don't see kafka component in "open studio for ESB"

Appreciate for your help.

Regards,

Satya

[vapukov]

Hi 

you can not import unsupported components from TOS BD into TOS ESB

generally and not too many reasons for this

• if you develop a standalone job - you could run it from OS (as normal batch or powershell script)
• if you develop route - TOS for ESB contain separate component cKafka for work with Kafka brockers. You could receive message in the route with cKafka component and transfer context to DI job for ETL transformations

cKafka

cTalendJob

regards, Vlad

[Anonymous]

Hi Vlad,

Appreciate your response. 

As per my understanding, 

"tKafka" components not part of Talend open studio for ESB, its part of open studio for Big Data.

Can you please suggest how can we achieve  below requirement with Talend open studio products ?

our requirement is -

1. Apache Kafka will publish message to Topic

2. Talend job should subscribe message from Topic and load into back end database.

3. Talend job should not be cron job or scheduler. it should listen Apache Kafka and subscribe if found any latest messages.

And, what is cKafka component ? where can i find cKafka component in open studio for ESB ?

Can you please provide more details on "If you develop route" in your response.

Thank you much for all your help.

Regards,

Sagar

[vapukov]

---

@Satya17 wrote:

Can you please provide more details on "If you develop route" in your response.

Thank you much for all your help.

Regards,

Sagar

---

Hi,

check first (just from a google search 😉  )

https://cleverbuilder.com/articles/mediation-vs-integration/

and of course, install and check all available demos:

https://help.talend.com/reader/EDncCk~GQGYJDo13J1a77g/zOczPWm3KuUqQw78e5doMw

+ Talend Help 

it helps you to have an idea - what is it and how it works 

regarding to:

3. Talend job should not be cron job or scheduler. it should listen Apache Kafka and subscribe if found any latest messages.

it exactly about routes, but and DI job could be started and do an infinite loop

regards, Vlad