topic Re: Error handling in Bigdata job in Talend Studio

Error handling in Bigdata job

Anonymous — Sat, 16 Nov 2024 08:54:51 GMT

Hi Team,

What are the ways to handle errors in Talend BigData Batch job?

Some of the bigdata components do not have "OnSubJobError" for eg. tHiveInput. How should we enable error handling in spark?

Re: Error handling in Bigdata job

Anonymous — Thu, 28 Dec 2017 07:04:27 GMT

Hello,

The onSubjobError connector is not supported by most Spark components.

Best regards

Sabrina

Re: Error handling in Bigdata job

Anonymous — Thu, 28 Dec 2017 07:07:40 GMT

Hi Sabrina,

Then how we should catch errors? Any best practice around that?

Re: Error handling in Bigdata job

Anonymous — Thu, 28 Dec 2017 10:02:59 GMT

Hello,

What's kind of error do you want to handle? To capture an exception in Spark jobs?

Best regards

Sabrina

Re: Error handling in Bigdata job

Anonymous — Thu, 28 Dec 2017 10:14:39 GMT

Hi Sabrina,

Lets say, I am performing below operation.

tHiveInput --> tMap (Some Transformation) --> tHiveOutput

Even though there is some issue in transformation, YARN application gets completed successfully. I am not able to record/track any issue, unless I go and check the logs.

Re: Error handling in Bigdata job

Anonymous — Thu, 28 Dec 2017 10:15:14 GMT

Hi Sabrina,

Lets say, I am performing below operation.

tHiveInput --> tMap (Some Transformation) --> tHiveOutput

Even though there is some issue in transformation, YARN application gets completed successfully. I am not able to record/track any issue, unless I go and check the logs.

Re: Error handling in Bigdata job

prawanth — Tue, 01 Oct 2019 15:19:58 GMT

Hi Guys,

We are facing the similar situation building big data batch, did you know the resolution / best practices around handling errors in big data batch ?

Thanks,

Re: Error handling in Bigdata job

prawanth — Tue, 01 Oct 2019 18:57:09 GMT

Below is the response from Talend on the same question.

"You wouldn't find components such as tlogcatcher, tstatcatcher in BD jobs. It is always recommended to have a DI orchestrator job for your spark jobs and then trigger spark jobs from DI jobs. You can pass context of statistics, error and other information using the orchestrator DI job to BD job. Orchestration should always happen through DI, the purpose of BD jobs is to do the processing of huge data. https://www.talend.com/blog/2016/10/05/talend-job-design-patterns-best-practices-part-3/ Error handling is explained in detail in the above blog."

cheers !!