How to maintain offsets of a consumer group in tKafkaInput

[KarthikGs]

Hi all,

I'm using tKafkaInput to read kafka topic data on daily basis.

But topic will receive data on a hourly basis, so i need to read only the latest data which was not consumed by my consumer group specified in the tkafkaInput component.

e.g: 1M messages where published to my kafka topic, my consumer group has read 1M  and was stopped consuming.While my consumer is at rest, 1 more Million records were published.

  so when i use the latest option and auto-commit offsets.The consumer is reading only the data which is being published.

Seems like the above settings doesn't allow consumer to maintain offset of the topic.

How to maintain to maintain the exact offsets and which settings shall i use?

 I'm using kafka version 0.9.0.1

Thanks in advance

[KarthikGs · Accepted solution]

Just enter in to advanced settings tab and check the "Load the offset with message". And the remaining props in the Basic settings should be checked as shown.

This fixed the problem, and the consumer group is reading the messages from the latest offset

[Anonymous]

Can you please explain how the option "Load the offset with the message" commits the offset?

As per documentation, It only adds a readonly column of offset value in the output.