Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi,
We are using
tKafkaInputin our talend job to fetch messages from Kafka.
There are 5 kafka brokers in Production which are listed in the component’s settings.
Usually – there is no issue with it and the job is working as expected.
Yet we noticed that when one of the kafka brokers is not available the job fails with this message:
WARN NetworkClient: Connection to node 4 could not be established. Broker may not be available.
Can you please suggest what might be the problem, and if the job should be able to work even when one or more kafka brokers are down?
thanks
Hi
This might be a new feature, don't stop the job if one of brokers is unavailable. As a workaround, try to iterate each broker and fetch messages in a child job, call the child job using tRunJob component and unchecked 'Die on error' box on tRunJob, eg:
main job:
tFixedFlowInput--main--tFlowToiterate--iterate-->tRunJob
pass the current broker to child job.
child job:
tKafkaInput--main--tLogRow
in child job, define a context variable and use context varaible in Broker list field.
Regards
Shong
thanks, will try that