topic Re: Problem with running tKafkaInput in a loop in Talend Studio

Problem with running tKafkaInput in a loop

Anonymous — Sat, 16 Nov 2024 10:29:22 GMT

Hi,
I am using Open Studio for Big Data v 6.1.1 and have a pipeline where I am reading data from Kafka and joining it with a small DB and writing the output back to Kafka. The lookup data in DB can change periodically, so my pipeline is:
tLoop -> tJava --(On Component Ok)--> tKafkaInput -> tMap (Lookup using tHiveInput) -> tKafkaOutput
I am using tHiveInput for lookup flow and loading of data in tMap ('Load once' configuration). In tKafkaInput, I have configured option 'Stop after a maximum total duration' setting to 600000 ms i.e. 10 mins. So after every 10 mins., tKafkaInput flow is stopped and started again, with tMap again loading lookup data from Hive.
The problem here is that after one iteration of tLoop is over, in the next iteration tKafkaInput doesn't read any input data. From the logs, I see that consumer id from previous iteration is already present and new iteration consumer is not getting any partition assigned to it.
Can someone please help me here?
Is there any other way I can design my pipeline and achieve the same use case?
Thanks,
Ajay

Re: Problem with running tKafkaInput in a loop

Anonymous — Tue, 26 Jul 2016 09:07:23 GMT

Hi,
Could you please post your job setting screenshots into forum? What's the java code in tJava component? How did you set tKafkaInput component?
More information will be helpful for us to address your issue.
Best regards
Sabrina

Re: Problem with running tKafkaInput in a loop

Anonymous — Tue, 26 Jul 2016 12:11:15 GMT

Thanks for the reply Sabrina. Job and component configuration files are attached.
Regards,
Ajay

Re: Problem with running tKafkaInput in a loop

Anonymous — Tue, 26 Jul 2016 21:25:00 GMT

For the same consumer group Id I think you need to start reading the offset storage from "the latest".
Also i would suggest you separate all components after the tjava in a subjob