tKafkaInput and tKafkaOutput manual commit and flush

SCCC1643798450 — Fri, 15 Nov 2024 23:13:41 GMT

Hi Team,

In one of our job we are using tKafkaInput and tKafkaOutput where we will receive the message, transform it and send to other kafka topic.

tKafkaInput -> Transformation -> tKafkaOutput.

When I see the Talend Java code I could see tKafkaInput create infinite while look and inside that poll the messages from Kafka topic. After polling the message we have forloop to process each message. I have a requirement to manually flush and commit the messages once all the messages in the for loop processing done.

I dont see any way I can include Java code to flush and commit after the for loop ends and before while loop ends. Is there anyway we can manually flush the messages in bulk?

Thanks

Sachin

Re: tKafkaInput and tKafkaOutput manual commit and flush

Anonymous — Thu, 10 Feb 2022 08:03:16 GMT

Hello

The source files of component are located in <studio installation directory>\plugins\org.talend.designer.components.bigdata_7.3.1.20200214_1052\components\tKafkaInput. If you are interested, back up the component and have a play with the source code.

In addition, I would suggest you to open an issue on Talend Bugtracker if you have feature request.

Regards

Shong

topic tKafkaInput and tKafkaOutput manual commit and flush in Talend Studio

tKafkaInput and tKafkaOutput manual commit and flush

Re: tKafkaInput and tKafkaOutput manual commit and flush