topic Re: [resolved] tSalesforceInput PK or Autonumber chunking in Talend Studio

[resolved] tSalesforceInput PK or Autonumber chunking

JR1 — Wed, 01 Feb 2017 14:39:46 GMT

Hi all

I would like to know if the tSalesforceInput component supports PK or autonumber chunking in Bulk Query mode. Please find the details here: PK Chunking Header. This can be crucial for the performance when exporting large numbers of records.

Thanks & Regards, Joachim

Re: [resolved] tSalesforceInput PK or Autonumber chunking

TRF — Wed, 01 Feb 2017 15:19:15 GMT

Hi,
You may associate tSalesforceInput with a tSalesforceConnection in which "For salesforce bulk component" option is checked.
In this case, in tSalesforceInput choose "Query Mode" as "Bulk Query" (or "Mass Query" depending on TOS version) and that's all (if I understood correctly your question).
Salesforce will process in parallel mode to retrieve the selected records (as for a bulk Insert/Update/Delete/Upsert).
Hope this help,
TRF

Re: [resolved] tSalesforceInput PK or Autonumber chunking

JR1 — Wed, 01 Feb 2017 17:31:59 GMT

Hi

No, unfortunately this does not answer my question and it is a bit more technical than that. I know how to tell Talend to use the Salesforce Bulk API. I cannot find the option to set the PK Chunking Header in the requests and as far as I can see this is not possible. Please see the article mentioned above to fully understand it.

Regards, Joachim

Re: [resolved] tSalesforceInput PK or Autonumber chunking

JR1 — Thu, 02 Feb 2017 09:47:43 GMT

If anyone is interested: there is a recent Talendforge case regarding this: https://jira.talendforge.org/browse/TDI-38096

Apparently this is not supported at the moment. It definitely should be as this would dramatically increase the performance of the Bulk API in case millions of records are to be extracted.

Re: [resolved] tSalesforceInput PK or Autonumber chunking

TRF — Thu, 02 Feb 2017 10:38:19 GMT

Hi,
What about using directly the Salesforce Bulk API from Talend (without using tSalesforce components)?

Re: [resolved] tSalesforceInput PK or Autonumber chunking

JR1 — Thu, 02 Feb 2017 10:54:17 GMT

This would mean rebuilding all the functionality the tSalesforceInput component already provides for Bulk Mode in a custom Java component including building the WSC, creating the job, collecting results, error handling, etc. It would undermine the purpose of using Talend in the first place if I just use it to program my own scripts with it (this, one could do without Talend). I have no immediate need for this but tSalesforceInput must definitely support this in the near future.