topic Re: Use multiple tkafkainputs in one job in Talend Studio

Use multiple tkafkainputs in one job

MyrtoD — Fri, 11 Jun 2021 14:05:26 GMT

Hi ,

I want to create a job which read from kafka. I am trying to implement 4 tkafkainput components but at the end only one consumer raise.

I have connected my tkafkinputs with tkafkaconnection via on component ok.

Re: Use multiple tkafkainputs in one job

Anonymous — Tue, 15 Jun 2021 01:27:09 GMT

If all tKafkainput components use the same connection, tKafkainput keep listening to server if you don't set the message batches or the condition is not met, the next subjob will not be triggered. Have a try to create different kafka connections for each tKafkainput component?

Regards

Shong

Re: Use multiple tkafkainputs in one job

MyrtoD — Tue, 15 Jun 2021 06:11:20 GMT

@Shicong Hong

Hi,

I tried to have 4 tkafkaconnection connected with the tkafkainputs but only one consumer raised at the end.

Re: Use multiple tkafkainputs in one job

Anonymous — Wed, 16 Jun 2021 03:14:22 GMT

Only the first tKafkaInput_1 is running? can you share a screenshot of basic settings of this component?

Re: Use multiple tkafkainputs in one job

MyrtoD — Wed, 16 Jun 2021 06:11:05 GMT

@Shicong Hong

Re: Use multiple tkafkainputs in one job

MyrtoD — Wed, 16 Jun 2021 06:11:37 GMT

Re: Use multiple tkafkainputs in one job

Anonymous — Thu, 17 Jun 2021 06:58:42 GMT

@Myrto Damianou , please change your job design as below and run again.

tPrejob--oncomponentok--tKafkaConnection1--oncomponentok--tKafkaConnection2

tKafkaInput1(use tKafkaConnection1)..babababa...

tKafKaInput2(use tKafKaConnection2)...babababa....

No trigger connector between tKafkaInput1 and tKafkaInput2 and ..., go to Job settings panel-->Extra and check the 'Enable Multi thread execution' to make all the subjobs run parallel.

Regards

Shong

Re: Use multiple tkafkainputs in one job

MyrtoD — Thu, 17 Jun 2021 14:15:08 GMT

@Shicong Hong

I tried it but only consumer raised again.

It is vey weird because when I have 4 separate jobs it raises 4 consumers but it is not efficient