topic Handle exception and store the invalid record in mongoDB. in Talend Studio

Handle exception and store the invalid record in mongoDB.

raowaqasakram — Fri, 15 Nov 2024 22:14:16 GMT

I'm extracting the data from MongoDB and then doing some processing on that in tJavaRow. After that I'm storing it into MySQL DB.

My requirement is that if any record is invalid/corrupt & exception occurs (while processing data or while storing in DB) then this record should not load to MySQL db.

For example ,

When the job is started , it load 50 records from MongoDB. Let's say the 37th record have some missing field.

Current Behaviour :

So currently NullPointerException is being thrown when I'm trying to process data in tJavaRow & the job is being terminated immediately.

Expected Behaviour :

My job should skip the 37th record (that have problem) and store other 49 records in MySQL DB.

The 37th record should be printed (using tLogRow) or stored in MongoDB.

My findings :

I have tried tLogCatcher but there is two problems in this.When it throws exception

It is showing the error message (Which is fine). But it isn't printing the complete (37th) record which loaded from mongoDB.
Then the job is terminated and NO record is being saved to MySQL.

Re: Handle exception and store the invalid record in mongoDB.

Anonymous — Wed, 28 Dec 2022 03:33:23 GMT

Go to the tip of the day page, and read the tip of Step 27, 2022. This tip shows how to catch the exception and continue to process next record, output the invalid data to a file.

Regards

Shong

Re: Handle exception and store the invalid record in mongoDB.

raowaqasakram — Fri, 30 Dec 2022 16:24:52 GMT

Hi @Shicong Hong ,

Thanks for your reply.

I have read this. But It didn't worked for my scenario.

My job design looks like

Design

tMongoDbInput --> tJavaRow --> tMap --> tDbOutput.

As per your tip , you have used tJavaFlex & I'm using tJavaRow.

I have tried to use all the instructions/code of tJavaRow in try-catch block in tJavaRow.

It is catching the exception and also printing for the buggy/invalid record but it is not "rejecting" that record .

For example , the 37th record have some NullPointerExcep. Thus this is showing NullPointer and then using the value from previous line and sending the 37th record to tMap.

Kindly suggest a way to do this which is compatible to my design.

Re: Handle exception and store the invalid record in mongoDB.

Anonymous — Tue, 03 Jan 2023 03:08:26 GMT

You should use tJavaFlex instead of tJavaRow, the job looks like:

tMongoDBInput--main-tFlowToIterate--iterate--tJavaFlex--iterate--tFixedFlowInput1--tMap--tDBoutput

what's the problem with this design?