Use multiple tkafkainputs in one job

[MyrtoD]

Hi ,

I want to create a job which read from kafka. I am trying to implement 4 tkafkainput components but at the end only one consumer raise.

I have connected my tkafkinputs with tkafkaconnection via on component ok.

[Anonymous]

Hi

If all tKafkainput components use the same connection, tKafkainput keep listening to server if you don't set the message batches or the condition is not met, the next subjob will not be triggered. Have a try to create different kafka connections for each tKafkainput component?

Regards

Shong

[MyrtoD]

@Shicong Hong​ 

Hi,

I tried to have 4 tkafkaconnection connected with the tkafkainputs but only one consumer raised at the end.

[Anonymous]

Only the first tKafkaInput_1 is running? can you share a screenshot of basic settings of this component?

[MyrtoD]

@Shicong Hong​ 

[MyrtoD]

[Anonymous]

@Myrto Damianou​ , please change your job design as below and run again.

tPrejob--oncomponentok--tKafkaConnection1--oncomponentok--tKafkaConnection2

tKafkaInput1(use tKafkaConnection1)..babababa...

tKafKaInput2(use tKafKaConnection2)...babababa....

No trigger connector between tKafkaInput1 and tKafkaInput2 and ..., go to Job settings panel-->Extra and check the 'Enable Multi thread execution' to make all the subjobs run parallel.

Regards

Shong

[MyrtoD]

@Shicong Hong​ 

I tried it but only consumer raised again.

It is vey weird because when I have 4 separate jobs it raises 4 consumers but it is not efficient