How to read data by using kafka input component

Venkata_Sivaji1 — Sat, 16 Nov 2024 03:11:18 GMT

Team,

How to setup kafka input component to read data from a topic which is already created(Able to see data in that topic).Used the properties as attached in screen shot. Kafka connection got succeful when job triggered but kafka input is not returning anything.Zookeeper and kafka server is in running status.Can some one please what could be the issue.

What is the consumer group id we can give here.

What is the consumer group id need to give . Given topic names as group id as well .

Re: How to read data by using kafka input component

DBeltritti1 — Fri, 26 Feb 2021 17:17:32 GMT

Does it ever start? I experienced a lot of delays on starting pulling messages on a consumer group id that already exists but it eventually starts.

However if you use a new consumer group id (like "INDIA_PRODUCTS1" for example) it pulls right away.

I was trying to get batches in output file of 500 records at the time and committing them after saving to file and ran into that same issue that you have, this was my setup:

infinite loop --> KafkaInput (stop after 500 msgs) --> outputFile --> KafkaCommit

The first iteration ran fast, but for the second to get the KafkaInput to start it took around 3 min of wait time.

Switched to KafkaInput (no batches setup) --> tflowtoiterate --> tjava to count msgs --> if message less than 500 then to hashoutput else if = 500 then thashinput from buffer to file --> KafkaCommit --> reset count

That way you can commit only what's already saved in your file and keep pulling new messages

topic How to read data by using kafka input component in Talend Studio

How to read data by using kafka input component

Re: How to read data by using kafka input component