[resolved] Best Way to Catch errors on Bulk Jobs

[Anonymous]

Talend Open Studio for Data Integration
Version: 6.3.1
Hello Community 
Where on the Run to change our Jobs into Bulk ones and we would like to catch the errors for it
with normal Jobs we had some TDies and an Tlog Catcher which sends some States to an mail adress.
But i have read that the Bulk Jobs doesn't give so much Feedback when the Job Fails. 
So, whats the best way ? 
regards john 

[Anonymous]

Okay may it's too common ... here is my Job.
It's an  Multi Salesforce Bulk Job with 2 Database Components:
1. Data from Oracle to Salesforce
2. Query from Salesforce to Salesforce
    Multi Job                               Sub Job 1                         Sub Job 2                              Sub Job 3
         
It's just in progress .. so any improvements are welcome. 
best regards john 

[TRF]

Hi,
Rejected records from tSalesforceOutputBulk are not catched by tLogCatcher.
Just connect a tFileOutputDelimited (or other component) after tSalesforceOutputBulk using the "Reject" flow (right click).
At the end, depending on the number of rejected records you can decide to send an email or not.
Regards,
TRF

[Anonymous]

Hello TRF,
How about the T-Assert component which is mentioned here:   http://bekwam.blogspot.de/2011/04/three-error-handling-strategies-in.html 
For my understanding T-Assert Catches Everything ... or is this not correct at all 
Could i merge error and  Status warnings on different mails components  with T-Assert ?
best regards john 

[TRF]

Maybe it may helps but from my experience with tSalesforceOutputBulkExec, the best is to push rejected records to a tHashOutput or a tFileXxxx depending on what you want to do with.

[Anonymous]

Hello TRF 
What about the Error Handling ?  Should i use some T'Dies or does the T-Assert Catch the job also.
ATM. I've made a test on Sub Job 1:
I set up an another  Data  Type at an Row on  the Tmap component which provoked an conversion error. 
But either the tlog/tstat - nor the t-assert catcher ( added on job for testing but not on the picture) catches the expetion in the Bulk Job ... is that normal ?     
best regards john 

[TRF]

Hi,
Yes it's normal as rejected records are not considered as exceptions (a chance!)
You'll catch an exception as soon as the component fails, for example because of a timeout or something like that.
So, if you want to throw an exception on rejected records, it's your responsability to put a tDie at the end of the rejected flow.
Regards,
TRF

[Anonymous]

Is an conversion Error an rejection ? 
Confused ATM
May i don't see the whole thing  ATM.
best regards john 

[TRF · Accepted solution]

Each record which cannot be injected due to an error at runtime give a rejection (invalid date, string too long, unknown parent) that you can catch using the reject flow after the tSalesforce component.
If you have an error due to a SOQL syntax error (for example, unknown field name) or any technical error, the full Salesforce job is rejected and an exception is thrown.
If you want to keep it easy, remember you can use string datatype for all fields you push into tSalesforce component (which works in the background with a csv format).
Regards,
TRF

[Anonymous]

Hello TRF,
So as i can see, the exception catching is different to an non bulk job.  

If you want to keep it easy, remember you can use string datatype for all fields you push into tSalesforce component (which works in the background with a csv format).

Yes but if you do so there are some drawbacks on the sf side. 
For example: A string Field cannot really evaluated for Reports or sth. else. if you have set all Fields in Salesforce as a String or is this only a problem with non bulk jobs.
best regards john