Skip to main content
Announcements
Introducing Qlik Answers: A plug-and-play, Generative AI powered RAG solution. READ ALL ABOUT IT!
cancel
Showing results for 
Search instead for 
Did you mean: 
SCCC1643798450
Contributor
Contributor

tKafkaInput Exception Handling

Hi All,

I am new to Talend and trying with Talend 7.1 Open Studio. We have a requirement where we will receive the message on Kafka Topic and then we will persist it into Database. For this I have used tKafkaInput Component that consuming the messages. Now the issue is after tKafkaInput component we have tJavaRow that do the transformation of the message that we received on Kafka Topic and if any error occurred in tJavaRow component I could see entire process terminate.

My requirement is if any error occurred in tJavaRow or while persisting record in DB I'll have to retry the message 3 times and then send the message to other Kafka error topic instead of terminating the process. My process should be running.

What is the way to handle this requirement?

Thanks

Labels (3)
4 Replies
Anonymous
Not applicable

Why would you need to retry 3 times? If something goes wrong with the transformation the first time, it will happen every other time. If the record does not make it to the DB, similar is true (although it could just be because the DB is down I guess). Also, what logic do you have the in tJavaRow? The tJavaRow is very powerful (especially if you have some Java knowledge), but you ca end up "reinventing the wheel" if you are not careful. Try bringing the logic out into other components. This way you can capture errors far easier.

 

If the message errors at all during this process, you should pass it to your error topic immediately. This will be far more efficient and prevent some incredibly convoluted process that is not necessary (given the example you have shared.....I may have missed something, if so please feel free to let me know).

SCCC1643798450
Contributor
Contributor
Author

We want to have retry because like you said DB down or some network glitch happened and it will get up immediately so may be in retry the message will get process. Using tLogCatcher we could able to send message to error topic but this also terminating the process. In tJavaRow we are enriching the JSON message that we are getting from Kafka.

What is a right way for not to terminate the Kafka Process if any error occurred in any of the component.

Anonymous
Not applicable

OK, first you want to untick the "Die on Error" tick box in all components that have this option. You will also need to implement some try/catch Java logic in your tJavaRow to ensure a failure in your transformation does not fail the job. Maybe have an "error" column which will output the the error in the "catch" section. After the tJavaRow you will probably need a tMap to send the output to either the DB (on success) or to send it round again if it fails.

 

The looping (or retry) will be the hardest part of this. This is not impossible at all, but will require some thought on how it will loop. This is one of the reasons that I questioned whether it was necessary. You will need to enable a process where the message is tried, fails up to 3 times, for each of the tries under 3 for each message you will need to send it round again. If it fails 3 times, then the message needs to be sent to your error topic. A simple try--> success vs try--> fail process which is rerun later after an alert as to what the error was, would be a lot easier to implement. But as I said, it can be done. But you may need to build the basic process first, test it, find where the errors occur, what the errors reveal themselves as, then start to implement the looping functionality once you fully understand the parameters of the problem. Then come back and show us what you have, what can happen and we can help you through that process.

SCCC1643798450
Contributor
Contributor
Author

sure will try this,