topic Re: tSalesforceBulkExecute, 65000 row limit? in Talend Studio

tSalesforceBulkExecute, 65000 row limit?

Anonymous — Sat, 16 Nov 2024 11:36:30 GMT

Hi, I have a job that uses tSalesforceOutputBulk and tSalesforceBulkExec to write data to Salesforce using the bulk API. I'm using TOS-DI 5.5 and for my Salesforce connection I'm using version 25. I'm reading from a MySQL table with 200k rows, mapping to a tSalesforceOutputBulk components which writes the data to a CSV files, then triggering a tSalesforceBulkExec component which reads the CSV file and performs the Salesforce inserts. The inserts are working correctly except that only 65000 of the 200000 rows are processed. Is this a limitation of Talend? CSV? Java? or am I missing a setting that would allow for all the rows to be processed?

Re: tSalesforceBulkExecute, 65000 row limit?

Anonymous — Mon, 23 Jun 2014 04:06:37 GMT

Hi,
Is there any truncation error printed on console? Have you tried to use "Rejects" row from tSalesforceBulkExec to see if there is any rejected row?
Best regards
Sabrina

Re: tSalesforceBulkExecute, 65000 row limit?

Anonymous — Mon, 23 Jun 2014 06:36:46 GMT

Hi,
Is there any truncation error printed on console? Have you tried to use "Rejects" row from tSalesforceBulkExec to see if there is any rejected row?
Best regards
Sabrina

There is also a Salesforce console that you or your Salesforce Admin can log on to, to see what's happened with Bulk Loads. Sometimes this gives more information than Talend has. As said, Rejects are first port of call.

Re: tSalesforceBulkExecute, 65000 row limit?

Anonymous — Mon, 23 Jun 2014 19:36:59 GMT

Hi and thanks so much for the replies. There are no truncation errors present in the Talend console. I am outputting the results of the tSalesforceBulkExec component to success and error files. The issue is not really with some rows failing and some completing successfully, it's just that only 65k rows make it Salesforce. In the Salesforce bulk data load monitoring tool it shows 65k rows being processed, some failing and some successful and the results are consistent with the tSalesforceBulkExec output that I can see in Talend. The strange thing to me is that when I process a file with >200k rows with tSalesforceBulkExec, exactly 65k rows make it to the bulk load job in Salesforce. I'm beginning to think that maybe Salesforce has limited the data on input, but really wanted to see if any others have also seen this same behavior. Thanks, cheers!

Re: tSalesforceBulkExecute, 65000 row limit?

Anonymous — Mon, 23 Jun 2014 19:58:25 GMT

Hi and thanks so much for the replies. There are no truncation errors present in the Talend console. I am outputting the results of the tSalesforceBulkExec component to success and error files. The issue is not really with some rows failing and some completing successfully, it's just that only 65k rows make it Salesforce. In the Salesforce bulk data load monitoring tool it shows 65k rows being processed, some failing and some successful and the results are consistent with the tSalesforceBulkExec output that I can see in Talend. The strange thing to me is that when I process a file with >200k rows with tSalesforceBulkExec, exactly 65k rows make it to the bulk load job in Salesforce. I'm beginning to think that maybe Salesforce has limited the data on input, but really wanted to see if any others have also seen this same behavior. Thanks, cheers!

I've loaded several millions of rows to Salesforce with the Bulk Loader, so if there is a fault, it may be an issue with later versions of Talend.
I believe that Talend Enterprise 5.1.1 was, at least, ok.

Re: tSalesforceBulkExecute, 65000 row limit?

Anonymous — Tue, 24 Jun 2014 04:20:06 GMT

Hi,
Could you please show us your job design screenshots? Did you use tMap in your work flow?
Best regards
Sabrina

Re: tSalesforceBulkExecute, 65000 row limit?

Anonymous — Tue, 24 Jun 2014 16:48:31 GMT

I'm going to run some more tests and get screenshots together but short answer is yes my job goes like this:
tSalesforceConnection -> tFileInputDelim -> tMap -> tSalesforceOutputBulk -> (on component ok) -> tSalesforceOutputBulkExec -> tFileOutputDelim (success) & tfileOutputDelim (error)

Re: tSalesforceBulkExecute, 65000 row limit?

Anonymous — Tue, 24 Jun 2014 21:41:10 GMT

Very possible that you are exceeding the limits and being throttled.
http://www.salesforce.com/us/developer/docs/api_asynch/Content/asynch_api_concepts_limits.htm

Re: tSalesforceBulkExecute, 65000 row limit?

Anonymous — Wed, 25 Jun 2014 09:53:30 GMT

Hi plsteffens,
Is there any update for your issue?
Best regards
Sabrina

Re: tSalesforceBulkExecute, 65000 row limit?

_AnonymousUser — Wed, 25 Mar 2015 14:37:32 GMT

It would be nice to get an update on this issue.
.
Regards,
Derek