topic [resolved] Best Way to Catch errors on Bulk Jobs in Talend Studio

[resolved] Best Way to Catch errors on Bulk Jobs

Anonymous — Thu, 23 Feb 2017 15:51:42 GMT

Talend Open Studio for Data Integration
Version: 6.3.1
Hello Community
Where on the Run to change our Jobs into Bulk ones and we would like to catch the errors for it
with normal Jobs we had some TDies and an Tlog Catcher which sends some States to an mail adress.
But i have read that the Bulk Jobs doesn't give so much Feedback when the Job Fails.
So, whats the best way ?
regards john

Re: [resolved] Best Way to Catch errors on Bulk Jobs

Anonymous — Thu, 02 Mar 2017 09:56:21 GMT

Okay may it's too common ... here is my Job.
It's an Multi Salesforce Bulk Job with 2 Database Components:
1. Data from Oracle to Salesforce
2. Query from Salesforce to Salesforce
Multi Job Sub Job 1 Sub Job 2 Sub Job 3

It's just in progress .. so any improvements are welcome.
best regards john

Re: [resolved] Best Way to Catch errors on Bulk Jobs

TRF — Thu, 02 Mar 2017 12:56:06 GMT

Hi,
Rejected records from tSalesforceOutputBulk are not catched by tLogCatcher.
Just connect a tFileOutputDelimited (or other component) after tSalesforceOutputBulk using the "Reject" flow (right click).
At the end, depending on the number of rejected records you can decide to send an email or not.
Regards,
TRF

Re: [resolved] Best Way to Catch errors on Bulk Jobs

Anonymous — Thu, 02 Mar 2017 15:12:57 GMT

Hello TRF,
How about the T-Assert component which is mentioned here: http://bekwam.blogspot.de/2011/04/three-error-handling-strategies-in.html
For my understanding T-Assert Catches Everything ... or is this not correct at all
Could i merge error and Status warnings on different mails components with T-Assert ?
best regards john

Re: [resolved] Best Way to Catch errors on Bulk Jobs

TRF — Thu, 02 Mar 2017 16:01:35 GMT

Maybe it may helps but from my experience with tSalesforceOutputBulkExec, the best is to push rejected records to a tHashOutput or a tFileXxxx depending on what you want to do with.

Re: [resolved] Best Way to Catch errors on Bulk Jobs

Anonymous — Fri, 03 Mar 2017 08:30:14 GMT

Hello TRF
What about the Error Handling ? Should i use some T'Dies or does the T-Assert Catch the job also.
ATM. I've made a test on Sub Job 1:
I set up an another Data Type at an Row on the Tmap component which provoked an conversion error.
But either the tlog/tstat - nor the t-assert catcher ( added on job for testing but not on the picture) catches the expetion in the Bulk Job ... is that normal ?
best regards john

Re: [resolved] Best Way to Catch errors on Bulk Jobs

TRF — Fri, 03 Mar 2017 09:04:47 GMT

Hi,
Yes it's normal as rejected records are not considered as exceptions (a chance!)
You'll catch an exception as soon as the component fails, for example because of a timeout or something like that.
So, if you want to throw an exception on rejected records, it's your responsability to put a tDie at the end of the rejected flow.
Regards,
TRF

Re: [resolved] Best Way to Catch errors on Bulk Jobs

Anonymous — Fri, 03 Mar 2017 12:22:13 GMT

Is an conversion Error an rejection ?
Confused ATM
May i don't see the whole thing ATM.
best regards john

Re: [resolved] Best Way to Catch errors on Bulk Jobs

TRF — Fri, 03 Mar 2017 13:18:36 GMT

Each record which cannot be injected due to an error at runtime give a rejection (invalid date, string too long, unknown parent) that you can catch using the reject flow after the tSalesforce component.
If you have an error due to a SOQL syntax error (for example, unknown field name) or any technical error, the full Salesforce job is rejected and an exception is thrown.
If you want to keep it easy, remember you can use string datatype for all fields you push into tSalesforce component (which works in the background with a csv format).
Regards,
TRF

Re: [resolved] Best Way to Catch errors on Bulk Jobs

Anonymous — Mon, 06 Mar 2017 14:39:46 GMT

Hello TRF,
So as i can see, the exception catching is different to an non bulk job.

If you want to keep it easy, remember you can use string datatype for all fields you push into tSalesforce component (which works in the background with a csv format).

Yes but if you do so there are some drawbacks on the sf side.
For example: A string Field cannot really evaluated for Reports or sth. else. if you have set all Fields in Salesforce as a String or is this only a problem with non bulk jobs.
best regards john

Re: [resolved] Best Way to Catch errors on Bulk Jobs

TRF — Mon, 06 Mar 2017 14:44:14 GMT

I didn't say to design fields as strings on SF side (of course!!!).
I was just speaking about the datatype for flows.

Re: [resolved] Best Way to Catch errors on Bulk Jobs

Anonymous — Tue, 07 Mar 2017 09:53:55 GMT

Hello TRF
Oh sorry i don't get this at first (confused from my further jobs) ... yes makes sense thnx for info
best regards john