topic Re: tSalesforceInput bulk mode problem in Talend Studio

tSalesforceInput bulk mode problem

Alinea — Sat, 16 Nov 2024 06:04:54 GMT

Hi,

We're using talend data integration 6.5.1.
We try to export our clients from Salesforce (Account object) and use this query:
"SELECT Id, PersonEmail
FROM Account"

There's about 4M records.

To make this request, we use tSalesforceInput, with the following options:
- connexion type: basic
- query mode: bulk
- timeout: 1800000

Our Talend job fails 2 minutes after beginning with the following error:
"Exception in component tSalesforceInput_2 (Alim_RCU_Oneshot)
org.talend.components.api.exception.ComponentException: org.talend.daikon.exception.error.DefaultErrorCode@7ef82753:{failedBatch=[BatchInfo id='7510Q000001gX65QAE'
jobId='7500Q000001zNZUQA2'
state='InProgress'
stateMessage='null'
createdDate='java.util.GregorianCalendar[time=1553695116000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2019,MONTH=2,WEEK_OF_YEAR=13,WEEK_OF_MONTH=4,DAY_OF_MONTH=27,DAY_OF_YEAR=86,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=1,HOUR_OF_DAY=13,MINUTE=58,SECOND=36,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
systemModstamp='java.util.GregorianCalendar[time=1553695121000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2019,MONTH=2,WEEK_OF_YEAR=13,WEEK_OF_MONTH=4,DAY_OF_MONTH=27,DAY_OF_YEAR=86,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=1,HOUR_OF_DAY=13,MINUTE=58,SECOND=41,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
numberRecordsProcessed='0'
numberRecordsFailed='0'
totalProcessingTime='0'
apiActiveProcessingTime='0'
apexProcessingTime='0'
]
}
at org.talend.components.salesforce.runtime.SalesforceBulkRuntime.doBulkQuery(SalesforceBulkRuntime.java:525)
at org.talend.components.salesforce.runtime.SalesforceBulkQueryInputReader.executeSalesforceBulkQuery(SalesforceBulkQueryInputReader.java:114)
at org.talend.components.salesforce.runtime.SalesforceBulkQueryInputReader.start(SalesforceBulkQueryInputReader.java:56)
at org.talend.codegen.flowvariables.runtime.FlowVariablesReader.start(FlowVariablesReader.java:73)
at XXX_flux_talend.alim_rcu_oneshot_0_1.Alim_RCU_Oneshot.tSalesforceInput_2Process(Alim_RCU_Oneshot.java:3751)
at XXX_flux_talend.alim_rcu_oneshot_0_1.Alim_RCU_Oneshot.tFileInputDelimited_3Process(Alim_RCU_Oneshot.java:1935)
at XXX_flux_talend.alim_rcu_oneshot_0_1.Alim_RCU_Oneshot.runJobInTOS(Alim_RCU_Oneshot.java:7694)
at XXX_flux_talend.alim_rcu_oneshot_0_1.Alim_RCU_Oneshot.main(Alim_RCU_Oneshot.java:7301)"

On salesforce side, there's no error, we can see that our bulk job lasted 4 minutes et ended well. We can even see the results.

We already made a case in salesforce but they told us that the problem was in Talend.

How can I get my component to work?

Thanks

Re: tSalesforceInput bulk mode problem

TRF — Fri, 12 Apr 2019 15:38:28 GMT

salesforce's support always answers the problem is on Talend's side as soon as you tell you're using this tool.

Don't know why

Anyway, it sounds like a network communication error but it's difficult to be sure.

I suggest you to retry for 1 or 2 times (change network connection if possible and avoid Wifi connection).

If the problem persists, try the same using DataLoader and if you got the same error, renew your case for slaesforce support (they can't say "we don't this f$$$$g tool).

Re: tSalesforceInput bulk mode problem

Alinea — Fri, 12 Apr 2019 15:50:22 GMT

Hi, thanks for your answer.

Works like a charm on DataLoader.
I runned the job on my computer but also on a server (so no wifi) and it always fails after 2 minutes.

Re: tSalesforceInput bulk mode problem

TRF — Fri, 12 Apr 2019 16:07:29 GMT

Did you miss a 0 for the timeout value? (just to be sure)

Re: tSalesforceInput bulk mode problem

Alinea — Mon, 15 Apr 2019 08:16:31 GMT

Unfortunatly, no. But whenever I try to change timeout (for a lower or a greater value) it doesn't change anything.

Re: tSalesforceInput bulk mode problem

TRF — Mon, 15 Apr 2019 08:46:19 GMT

What if you don't use a bulk query?

Re: tSalesforceInput bulk mode problem

Alinea — Mon, 15 Apr 2019 09:21:49 GMT

It works if I don't use it in a tMap lookup.
And I must select a batch size of 1000 (where 2000 is the maximum), otherwise it doesn't work.

That's not very efficient.

Re: tSalesforceInput bulk mode problem

TRF — Mon, 15 Apr 2019 10:06:05 GMT

You should have a problem somewhere, I never saw this before.

Can you share your job design including your tMap?

Re: tSalesforceInput bulk mode problem

Alinea — Mon, 15 Apr 2019 11:46:48 GMT

First try: mapping between Account and Addresses using tMap and bulk mode.

My job:

Parameters:

Advanced parameters:

Result:

Démarrage du job POC_SFSC_Export a 12:40 15/04/2019.
[statistics] connecting to socket on port 3602
[statistics] connected
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
Exception in component tSalesforceInput_2 (POC_SFSC_Export)
org.talend.components.api.exception.ComponentException: org.talend.daikon.exception.error.DefaultErrorCode@5ab956d7:{failedBatch=[BatchInfo id='7511t000004ecjjAAA'
jobId='7501t000004crI0AAI'
state='InProgress'
stateMessage='null'
createdDate='java.util.GregorianCalendar[time=1555324853000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2019,MONTH=3,WEEK_OF_YEAR=16,WEEK_OF_MONTH=3,DAY_OF_MONTH=15,DAY_OF_YEAR=105,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=3,AM_PM=0,HOUR=10,HOUR_OF_DAY=10,MINUTE=40,SECOND=53,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
systemModstamp='java.util.GregorianCalendar[time=1555324856000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2019,MONTH=3,WEEK_OF_YEAR=16,WEEK_OF_MONTH=3,DAY_OF_MONTH=15,DAY_OF_YEAR=105,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=3,AM_PM=0,HOUR=10,HOUR_OF_DAY=10,MINUTE=40,SECOND=56,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
numberRecordsProcessed='0'
numberRecordsFailed='0'
totalProcessingTime='0'
apiActiveProcessingTime='0'
apexProcessingTime='0'
]
}
at org.talend.components.salesforce.runtime.SalesforceBulkRuntime.doBulkQuery(SalesforceBulkRuntime.java:525)
at org.talend.components.salesforce.runtime.SalesforceBulkQueryInputReader.executeSalesforceBulkQuery(SalesforceBulkQueryInputReader.java:114)
at org.talend.components.salesforce.runtime.SalesforceBulkQueryInputReader.start(SalesforceBulkQueryInputReader.java:56)
at org.talend.codegen.flowvariables.runtime.FlowVariablesReader.start(FlowVariablesReader.java:73)
at alinea_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.tSalesforceInput_2Process(POC_SFSC_Export.java:3080)
at alinea_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.tSalesforceInput_1Process(POC_SFSC_Export.java:1496)
at alinea_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.runJobInTOS(POC_SFSC_Export.java:6998)
at alinea_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.main(POC_SFSC_Export.java:6453)
[statistics] disconnected
Job POC_SFSC_Export terminé à 12:43 15/04/2019. [Code sortie=1]

From salesforce point of view - lots (it's in french but it says finished and no error):

Salesforce job is "running" (but in fact all lots are finished):

tmap_bulk1.PNG

Re: tSalesforceInput bulk mode problem

Alinea — Mon, 15 Apr 2019 13:05:10 GMT

2nd try: Same thing but with query mode.

Result : java heap space in lookup (nearly one hour of duration).

Exception in component tSalesforceInput_2 (POC_SFSC_Export)
org.talend.components.api.exception.ComponentException: org.talend.daikon.exception.error.DefaultErrorCode@5ab956d7:{failedBatch=[BatchInfo id='7511t000004ecjjAAA'
jobId='7501t000004crI0AAI'
state='InProgress'
stateMessage='null'
createdDate='java.util.GregorianCalendar[time=1555324853000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2019,MONTH=3,WEEK_OF_YEAR=16,WEEK_OF_MONTH=3,DAY_OF_MONTH=15,DAY_OF_YEAR=105,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=3,AM_PM=0,HOUR=10,HOUR_OF_DAY=10,MINUTE=40,SECOND=53,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
systemModstamp='java.util.GregorianCalendar[time=1555324856000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2019,MONTH=3,WEEK_OF_YEAR=16,WEEK_OF_MONTH=3,DAY_OF_MONTH=15,DAY_OF_YEAR=105,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=3,AM_PM=0,HOUR=10,HOUR_OF_DAY=10,MINUTE=40,SECOND=56,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
numberRecordsProcessed='0'
numberRecordsFailed='0'
totalProcessingTime='0'
apiActiveProcessingTime='0'
apexProcessingTime='0'
]
}
at org.talend.components.salesforce.runtime.SalesforceBulkRuntime.doBulkQuery(SalesforceBulkRuntime.java:525)
at org.talend.components.salesforce.runtime.SalesforceBulkQueryInputReader.executeSalesforceBulkQuery(SalesforceBulkQueryInputReader.java:114)
at org.talend.components.salesforce.runtime.SalesforceBulkQueryInputReader.start(SalesforceBulkQueryInputReader.java:56)
at org.talend.codegen.flowvariables.runtime.FlowVariablesReader.start(FlowVariablesReader.java:73)
at XXX_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.tSalesforceInput_2Process(POC_SFSC_Export.java:3080)
at XXX_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.tSalesforceInput_1Process(POC_SFSC_Export.java:1496)
at XXX_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.runJobInTOS(POC_SFSC_Export.java:6998)
at XXX_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.main(POC_SFSC_Export.java:6453)Démarrage du job POC_SFSC_Export a 12:48 15/04/2019.
[statistics] connecting to socket on port 3452
[statistics] connected
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:984)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:491)
at javax.crypto.Cipher.doFinal(Cipher.java:2376)
at sun.security.ssl.CipherBox.decrypt(Unknown Source)
at sun.security.ssl.InputRecord.decrypt(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at sun.security.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net. www.http.ChunkedInputStream.fastRead(Unknown Source)
at sun.net. www.http.ChunkedInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at sun.net. www.protocol.http.HttpURLConnection$HttpInputStream.read(Unknown Source)
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
[statistics] disconnected
[statistics] disconnected
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at com.sforce.ws.parser.MXParser.fillBuf(MXParser.java:2962)
at com.sforce.ws.parser.MXParser.more(MXParser.java:3014)
at com.sforce.ws.parser.MXParser.parseAttribute(MXParser.java:2036)
at com.sforce.ws.parser.MXParser.parseStartTag(MXParser.java:1809)
at com.sforce.ws.parser.MXParser.nextImpl(MXParser.java:1276)
at com.sforce.ws.parser.MXParser.next(MXParser.java:1111)
at com.sforce.ws.parser.XmlInputStream.next(XmlInputStream.java:136)
at com.sforce.ws.parser.XmlInputStream.peek(XmlInputStream.java:187)
at com.sforce.ws.bind.XmlObject.loadAfterStartTag(XmlObject.java:308)
at com.sforce.ws.bind.XmlObject.load(XmlObject.java:294)
at com.sforce.ws.bind.TypeMapper.readSingle(TypeMapper.java:674)
at com.sforce.ws.bind.TypeMapper.readArray(TypeMapper.java:580)
at com.sforce.ws.bind.TypeMapper.readObject(TypeMapper.java:558)
Job POC_SFSC_Export terminé à 13:41 15/04/2019. [Code sortie=1]

Re: tSalesforceInput bulk mode problem

Alinea — Mon, 15 Apr 2019 13:16:40 GMT

2nd try: same thing but with query mode (and 2000 batch size).

Result : java heap space in lookup.

Re: tSalesforceInput bulk mode problem

Alinea — Mon, 15 Apr 2019 15:20:20 GMT

Third try: juste a simple salesforce input that goes to a csv file.

Results:

- bulk mode: error

- query mode (2000 batch size): ok (but last nealry 2 hours)

Detailled errors:

- Bulk mode:

Démarrage du job POC_SFSC_Export a 14:05 15/04/2019.
[statistics] connecting to socket on port 3621
[statistics] connected
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
Exception in component tSalesforceInput_3 (POC_SFSC_Export)
org.talend.components.api.exception.ComponentException: org.talend.daikon.exception.error.DefaultErrorCode@52719fb6:{failedBatch=[BatchInfo id='7511t000004eeMSAAY'
jobId='7501t000004csSGAAY'
state='InProgress'
stateMessage='null'
createdDate='java.util.GregorianCalendar[time=1555329956000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2019,MONTH=3,WEEK_OF_YEAR=16,WEEK_OF_MONTH=3,DAY_OF_MONTH=15,DAY_OF_YEAR=105,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=3,AM_PM=1,HOUR=0,HOUR_OF_DAY=12,MINUTE=5,SECOND=56,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
systemModstamp='java.util.GregorianCalendar[time=1555329956000,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2019,MONTH=3,WEEK_OF_YEAR=16,WEEK_OF_MONTH=3,DAY_OF_MONTH=15,DAY_OF_YEAR=105,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=3,AM_PM=1,HOUR=0,HOUR_OF_DAY=12,MINUTE=5,SECOND=56,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0]'
numberRecordsProcessed='0'
numberRecordsFailed='0'
totalProcessingTime='0'
apiActiveProcessingTime='0'
apexProcessingTime='0'
]
}
at org.talend.components.salesforce.runtime.SalesforceBulkRuntime.doBulkQuery(SalesforceBulkRuntime.java:525)
at org.talend.components.salesforce.runtime.SalesforceBulkQueryInputReader.executeSalesforceBulkQuery(SalesforceBulkQueryInputReader.java:114)
at org.talend.components.salesforce.runtime.SalesforceBulkQueryInputReader.start(SalesforceBulkQueryInputReader.java:56)
at org.talend.codegen.flowvariables.runtime.FlowVariablesReader.start(FlowVariablesReader.java:73)
at XXX_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.tSalesforceInput_3Process(POC_SFSC_Export.java:1423)
at XXX_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.runJobInTOS(POC_SFSC_Export.java:5368)
at XXX_flux_talend.poc_sfsc_export_0_1.POC_SFSC_Export.main(POC_SFSC_Export.java:4823)
[statistics] disconnected
Job POC_SFSC_Export terminé à 14:08 15/04/2019. [Code sortie=1]

Re: tSalesforceInput bulk mode problem

KrupalGharat — Thu, 14 Nov 2019 07:29:14 GMT

Have you found any solution for this?

I too am facing the same issue!!

Re: tSalesforceInput bulk mode problem

TRF — Thu, 14 Nov 2019 08:08:48 GMT

Did you try to do the same thing using Data Loader?

Re: tSalesforceInput bulk mode problem

Alinea — Mon, 18 Nov 2019 09:36:46 GMT

Yes we upgraded our talend version from 6.5 to 7.1
It seems to resolve our problems.

Re: tSalesforceInput bulk mode problem

Alinea — Mon, 18 Nov 2019 09:36:50 GMT

Yes we upgraded our talend version from 6.5 to 7.1

It seems to resolve our problems.

Re: tSalesforceInput bulk mode problem

Alinea — Mon, 18 Nov 2019 09:37:12 GMT

Yes and it works like a charm