I am trying to streamline the loading of ContentVersion records (attachments) to salesforce by using a tFlowtoIterate with enable parallel execution to parallelize a call to a child job. I have also tried this without a child job and got the same result.
ContentVersions have to be loaded serially into salesforce, they can't be grouped in to batches greater than 1 record owing to various restrictions in salesforce.
So I am trying to work around this by setting up many threads all running a single record into salesforce.
To start with the job runs very fast with multiple threads executing to load into salesforce. However after n records the tSalesforceOutput component fails with the following string index error.
This error does not occur if I set the tRunJob to use separate processes for each execution however the overhead of setting up a new process for each iteration slows the process down so much it is no faster than the serial method.
So I suspect that something in tSalesforceOutput is not thread safe.
Has anyone seen this behaviour before or know if my hunch is correct?
----
Exception in component tRunJob_1 (PutAttachmentParent) java.lang.RuntimeException: Child job running failed. java.lang.StringIndexOutOfBoundsException: String index out of range: -8 at ihs.putattachmentparent_0_3.PutAttachmentParent$1tRunJob_1Thread.run(PutAttachmentParent.java:4970) at routines.system.ThreadPoolWorker.runIt(TalendThreadPool.java:159) at routines.system.ThreadPoolWorker.runWork(TalendThreadPool.java:150) at routines.system.ThreadPoolWorker.access$0(TalendThreadPool.java:145) at routines.system.ThreadPoolWorker$1.run(TalendThreadPool.java:122) at java.lang.Thread.run(Unknown Source)
----Here is the exception log if I don't use a child job
18:30:37,049 INFO [SalesforceWriter] Done. 18:31:26,508 INFO [SalesforceWriter] Prepare to insert record 1. 18:31:27,250 INFO [SalesforceWriter] Has inserted record 1 successfully. 18:31:27,250 INFO [SalesforceWriter] Has inserted 1 records successfully. 18:31:27,250 INFO [SalesforceWriter] Has rejected 0 records. 18:31:27,250 INFO [SalesforceWriter] Done. 18:31:27,828 INFO [SalesforceWriter] Failed to insert record 1. 18:31:27,828 INFO [SalesforceWriter] Has inserted 1 records successfully. 18:31:27,828 INFO [SalesforceWriter] Has rejected 1 records. 18:31:27,828 INFO [SalesforceWriter] Done. Exception in component tSalesforceOutput_1 (PutAttachmentParallel) java.lang.StringIndexOutOfBoundsException: String index out of range: 13 at java.lang.String.substring(Unknown Source) at java.text.MessageFormat.subformat(Unknown Source) at java.text.MessageFormat.format(Unknown Source) at java.text.Format.format(Unknown Source) at org.talend.daikon.i18n.I18nMessages.getFormattedMessage(I18nMessages.java:63) at org.talend.daikon.i18n.ClassBasedI18nMessages.getMessage(ClassBasedI18nMessages.java:76) at org.talend.components.salesforce.runtime.SalesforceWriter.getPastForm(SalesforceWriter.java:738) at org.talend.components.salesforce.runtime.SalesforceWriter.handleSuccess(SalesforceWriter.java:511) at org.talend.components.salesforce.runtime.SalesforceWriter.doInsert(SalesforceWriter.java:351) at org.talend.components.salesforce.runtime.SalesforceWriter.insert(SalesforceWriter.java:328) at org.talend.components.salesforce.runtime.SalesforceWriter.write(SalesforceWriter.java:185) at ihs.putattachmentparallel_0_3.PutAttachmentParallel$1tDBInput_2Thread.run(PutAttachmentParallel.java:8802) at routines.system.ThreadPoolWorker.runIt(TalendThreadPool.java:159) at routines.system.ThreadPoolWorker.runWork(TalendThreadPool.java:150) at routines.system.ThreadPoolWorker.access$0(TalendThreadPool.java:145) at routines.system.ThreadPoolWorker$1.run(TalendThreadPool.java:122) at java.lang.Thread.run(Unknown Source)
The Salesforce output component actually works better with a tFlowToIterate component placed in front. Could you please show us your current job design screenshots? Did you use global variables(Put, Get) in your work flow?
Yes I tried the tFlowToIterate pattern with the same result.
In this example I've tried running 4 in parallel but by splitting the flow using a randomly assigned number. when I set the enable parallel execution on the iterate links and set the number to >1 I get the error mentioned in my post.
The tMaps doing the lookup to salesforce to get the attachment data are operating in the reload each row mode to query for a specific salesforce record each time.
