Talend job fails when 1 or more Kafka brokers become unavailable

[RR1622110713]

Hi,

We are using

tKafkaInput

in our talend job to fetch messages from Kafka.

There are 5 kafka brokers in Production which are listed in the component’s settings.

Usually – there is no issue with it and the job is working as expected.

Yet we noticed that when one of the kafka brokers is not available the job fails with this message:

WARN NetworkClient: Connection to node 4 could not be established. Broker may not be available.

Can you please suggest what might be the problem, and if the job should be able to work even when one or more kafka brokers are down?

thanks

[Anonymous]

Hi

This might be a new feature, don't stop the job if one of brokers is unavailable. As a workaround, try to iterate each broker and fetch messages in a child job, call the child job using tRunJob component and unchecked 'Die on error' box on tRunJob, eg:

main job:

tFixedFlowInput--main--tFlowToiterate--iterate-->tRunJob

pass the current broker to child job.

child job:

tKafkaInput--main--tLogRow

in child job, define a context variable and use context varaible in Broker list field.

Regards

Shong

[RR1622110713]

thanks, will try that