How to read data by using kafka input component

[Venkata_Sivaji1]

Team,

How to setup kafka input component to read data from a topic which is already created(Able to see data in that topic).Used the properties as attached in screen shot. Kafka connection got succeful when job triggered but kafka input is not returning anything.Zookeeper and kafka server is in running status.Can some one please what could be the issue.

What is the consumer group id we can give here.

What is the consumer group id need to give . Given topic names as group id as well .

[DBeltritti1]

Does it ever start? I experienced a lot of delays on starting pulling messages on a consumer group id that already exists but it eventually starts.

However if you use a new consumer group id (like "INDIA_PRODUCTS1" for example) it pulls right away.

I was trying to get batches in output file of 500 records at the time and committing them after saving to file and ran into that same issue that you have, this was my setup:

infinite loop --> KafkaInput (stop after 500 msgs) --> outputFile --> KafkaCommit

The first iteration ran fast, but for the second to get the KafkaInput to start it took around 3 min of wait time.

Switched to KafkaInput (no batches setup) --> tflowtoiterate --> tjava to count msgs --> if message less than 500 then to hashoutput else if = 500 then thashinput from buffer to file --> KafkaCommit --> reset count

That way you can commit only what's already saved in your file and keep pulling new messages