tSalesforceInput bulk mode problem

[Alinea]

Hi,

We're using talend data integration 6.5.1.
We try to export our clients from Salesforce (Account object) and use this query:
"SELECT Id, PersonEmail
FROM Account"

There's about 4M records.

To make this request, we use tSalesforceInput, with the following options:
- connexion type: basic
- query mode: bulk
- timeout: 1800000

Our Talend job fails 2 minutes after beginning with the following error:
"Exception in component tSalesforceInput_2 (Alim_RCU_Oneshot)
org.talend.components.api.exception.ComponentException: org.talend.daikon.exception.error.DefaultErrorCode@7ef82753:{failedBatch=[BatchInfo id='7510Q000001gX65QAE'
jobId='7500Q000001zNZUQA2'
state='InProgress'
stateMessage='null'
createdDate='java.util.GregorianCalendar[time=1553695116000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2019,MONTH=2,WEEK_OF_YEAR=13,WEEK_OF_MONTH=4,DAY_OF_MONTH=27,DAY_OF_YEAR=86,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=1,HOUR_OF_DAY=13,MINUTE=58,SECOND=36,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
systemModstamp='java.util.GregorianCalendar[time=1553695121000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2019,MONTH=2,WEEK_OF_YEAR=13,WEEK_OF_MONTH=4,DAY_OF_MONTH=27,DAY_OF_YEAR=86,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=1,HOUR_OF_DAY=13,MINUTE=58,SECOND=41,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
numberRecordsProcessed='0'
numberRecordsFailed='0'
totalProcessingTime='0'
apiActiveProcessingTime='0'
apexProcessingTime='0'
]
}
at org.talend.components.salesforce.runtime.SalesforceBulkRuntime.doBulkQuery(SalesforceBulkRuntime.java:525)
at org.talend.components.salesforce.runtime.SalesforceBulkQueryInputReader.executeSalesforceBulkQuery(SalesforceBulkQueryInputReader.java:114)
at org.talend.components.salesforce.runtime.SalesforceBulkQueryInputReader.start(SalesforceBulkQueryInputReader.java:56)
at org.talend.codegen.flowvariables.runtime.FlowVariablesReader.start(FlowVariablesReader.java:73)
at XXX_flux_talend.alim_rcu_oneshot_0_1.Alim_RCU_Oneshot.tSalesforceInput_2Process(Alim_RCU_Oneshot.java:3751)
at XXX_flux_talend.alim_rcu_oneshot_0_1.Alim_RCU_Oneshot.tFileInputDelimited_3Process(Alim_RCU_Oneshot.java:1935)
at XXX_flux_talend.alim_rcu_oneshot_0_1.Alim_RCU_Oneshot.runJobInTOS(Alim_RCU_Oneshot.java:7694)
at XXX_flux_talend.alim_rcu_oneshot_0_1.Alim_RCU_Oneshot.main(Alim_RCU_Oneshot.java:7301)"

On salesforce side, there's no error, we can see that our bulk job lasted 4 minutes et ended well. We can even see the results.

We already made a case in salesforce but they told us that the problem was in Talend.

How can I get my component to work?

Thanks

[TRF]

salesforce's support always answers the problem is on Talend's side as soon as you tell you're using this tool.

Don't know why 

Anyway, it sounds like a network communication error but it's difficult to be sure.

I suggest you to retry for 1 or 2 times (change network connection if possible and avoid Wifi connection).

If the problem persists, try the same using DataLoader and if you got the same error, renew your case for slaesforce support (they can't say "we don't this f$$$$g tool).

[Alinea]

Hi, thanks for your answer.

Works like a charm on DataLoader.
I runned the job on my computer but also on a server (so no wifi) and it always fails after 2 minutes.

[TRF]

Did you miss a 0 for the timeout value? (just to be sure)

[Alinea]

Unfortunatly, no. But whenever I try to change timeout (for a lower or a greater value) it doesn't change anything.

[TRF]

What if you don't use a bulk query?

[Alinea]

It works if I don't use it in a tMap lookup.
And I must select a batch size of 1000 (where 2000 is the maximum), otherwise it doesn't work.

That's not very efficient.

[TRF]

You should have a problem somewhere, I never saw this before.

Can you share your job design including your tMap?

[Alinea]

First try: mapping between Account and Addresses using tMap and bulk mode.

My job:

Parameters:

Advanced parameters:

Result:

---

Démarrage du job POC_SFSC_Export a 12:40 15/04/2019.
[statistics] connecting to socket on port 3602
[statistics] connected
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
Exception in component tSalesforceInput_2 (POC_SFSC_Export)
org.talend.components.api.exception.ComponentException: org.talend.daikon.exception.error.DefaultErrorCode@5ab956d7:{failedBatch=[BatchInfo id='7511t000004ecjjAAA'
jobId='7501t000004crI0AAI'
state='InProgress'
stateMessage='null'
createdDate='java.util.GregorianCalendar[time=1555324853000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2019,MONTH=3,WEEK_OF_YEAR=16,WEEK_OF_MONTH=3,DAY_OF_MONTH=15,DAY_OF_YEAR=105,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=3,AM_PM=0,HOUR=10,HOUR_OF_DAY=10,MINUTE=40,SECOND=53,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
systemModstamp='java.util.GregorianCalendar[time=1555324856000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2019,MONTH=3,WEEK_OF_YEAR=16,WEEK_OF_MONTH=3,DAY_OF_MONTH=15,DAY_OF_YEAR=105,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=3,AM_PM=0,HOUR=10,HOUR_OF_DAY=10,MINUTE=40,SECOND=56,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
numberRecordsProcessed='0'
numberRecordsFailed='0'
totalProcessingTime='0'
apiActiveProcessingTime='0'
apexProcessingTime='0'
]
}
at org.talend.components.salesforce.runtime.SalesforceBulkRuntime.doBulkQuery(SalesforceBulkRuntime.java:525)
at org.talend.components.salesforce.runtime.SalesforceBulkQueryInputReader.executeSalesforceBulkQuery(SalesforceBulkQueryInputReader.java:114)
at org.talend.components.salesforce.runtime.SalesforceBulkQueryInputReader.start(SalesforceBulkQueryInputReader.java:56)
at org.talend.codegen.flowvariables.runtime.FlowVariablesReader.start(FlowVariablesReader.java:73)
at alinea_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.tSalesforceInput_2Process(POC_SFSC_Export.java:3080)
at alinea_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.tSalesforceInput_1Process(POC_SFSC_Export.java:1496)
at alinea_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.runJobInTOS(POC_SFSC_Export.java:6998)
at alinea_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.main(POC_SFSC_Export.java:6453)
[statistics] disconnected
Job POC_SFSC_Export terminé à 12:43 15/04/2019. [Code sortie=1]

---

From salesforce point of view - lots (it's in french but it says finished and no error):

Salesforce job is "running" (but in fact all lots are finished):

tmap_bulk1.PNG

[Alinea]

2nd try: Same thing but with query mode.

Result : java heap space in lookup (nearly one hour of duration).

---

Exception in component tSalesforceInput_2 (POC_SFSC_Export)
org.talend.components.api.exception.ComponentException: org.talend.daikon.exception.error.DefaultErrorCode@5ab956d7:{failedBatch=[BatchInfo id='7511t000004ecjjAAA'
jobId='7501t000004crI0AAI'
state='InProgress'
stateMessage='null'
createdDate='java.util.GregorianCalendar[time=1555324853000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2019,MONTH=3,WEEK_OF_YEAR=16,WEEK_OF_MONTH=3,DAY_OF_MONTH=15,DAY_OF_YEAR=105,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=3,AM_PM=0,HOUR=10,HOUR_OF_DAY=10,MINUTE=40,SECOND=53,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
systemModstamp='java.util.GregorianCalendar[time=1555324856000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2019,MONTH=3,WEEK_OF_YEAR=16,WEEK_OF_MONTH=3,DAY_OF_MONTH=15,DAY_OF_YEAR=105,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=3,AM_PM=0,HOUR=10,HOUR_OF_DAY=10,MINUTE=40,SECOND=56,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
numberRecordsProcessed='0'
numberRecordsFailed='0'
totalProcessingTime='0'
apiActiveProcessingTime='0'
apexProcessingTime='0'
]
}
at org.talend.components.salesforce.runtime.SalesforceBulkRuntime.doBulkQuery(SalesforceBulkRuntime.java:525)
at org.talend.components.salesforce.runtime.SalesforceBulkQueryInputReader.executeSalesforceBulkQuery(SalesforceBulkQueryInputReader.java:114)
at org.talend.components.salesforce.runtime.SalesforceBulkQueryInputReader.start(SalesforceBulkQueryInputReader.java:56)
at org.talend.codegen.flowvariables.runtime.FlowVariablesReader.start(FlowVariablesReader.java:73)
at XXX_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.tSalesforceInput_2Process(POC_SFSC_Export.java:3080)
at XXX_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.tSalesforceInput_1Process(POC_SFSC_Export.java:1496)
at XXX_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.runJobInTOS(POC_SFSC_Export.java:6998)
at XXX_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.main(POC_SFSC_Export.java:6453)Démarrage du job POC_SFSC_Export a 12:48 15/04/2019.
[statistics] connecting to socket on port 3452
[statistics] connected
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:984)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:491)
at javax.crypto.Cipher.doFinal(Cipher.java:2376)
at sun.security.ssl.CipherBox.decrypt(Unknown Source)
at sun.security.ssl.InputRecord.decrypt(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at sun.security.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net. www.http.ChunkedInputStream.fastRead(Unknown Source)
at sun.net. www.http.ChunkedInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at sun.net. www.protocol.http.HttpURLConnection$HttpInputStream.read(Unknown Source)
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
[statistics] disconnected
[statistics] disconnected
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at com.sforce.ws.parser.MXParser.fillBuf(MXParser.java:2962)
at com.sforce.ws.parser.MXParser.more(MXParser.java:3014)
at com.sforce.ws.parser.MXParser.parseAttribute(MXParser.java:2036)
at com.sforce.ws.parser.MXParser.parseStartTag(MXParser.java:1809)
at com.sforce.ws.parser.MXParser.nextImpl(MXParser.java:1276)
at com.sforce.ws.parser.MXParser.next(MXParser.java:1111)
at com.sforce.ws.parser.XmlInputStream.next(XmlInputStream.java:136)
at com.sforce.ws.parser.XmlInputStream.peek(XmlInputStream.java:187)
at com.sforce.ws.bind.XmlObject.loadAfterStartTag(XmlObject.java:308)
at com.sforce.ws.bind.XmlObject.load(XmlObject.java:294)
at com.sforce.ws.bind.TypeMapper.readSingle(TypeMapper.java:674)
at com.sforce.ws.bind.TypeMapper.readArray(TypeMapper.java:580)
at com.sforce.ws.bind.TypeMapper.readObject(TypeMapper.java:558)
Job POC_SFSC_Export terminé à 13:41 15/04/2019. [Code sortie=1]