topic Re: Memory issue for Linkedin/salesforce sync integration job in Talend Studio

Memory issue for Linkedin/salesforce sync integration job

gt0731 — Sat, 16 Nov 2024 07:40:09 GMT

Attached zip job is crashing due to too much processing at once and not enough RAM to do it and therefore needs simplifying and generally updating to make it better overall.

Any help on it would be much appreciated ?

Also, Advise on what measure need to be taken here at memory level.

Re: Memory issue for Linkedin/salesforce sync integration job

Jesperrekuh — Wed, 12 Sep 2018 18:10:30 GMT

which version are you on.
Please show a screenshot... we will point out what to adjust and how.
Memory and processing power is system related... so it works here and fails at your side ...

What did you already try to improve performance?!?

Re: Memory issue for Linkedin/salesforce sync integration job

gt0731 — Wed, 12 Sep 2018 18:23:57 GMT

I already attached complete zipped folder about this job. Are you able to
extract the folder ?

I tried to implement
Apex soap api and at memory still struggling to figure out the process

Re: Memory issue for Linkedin/salesforce sync integration job

Jesperrekuh — Wed, 12 Sep 2018 19:27:57 GMT

Again, what version are you running? I need to be able to import your project and your Talend version could differ from mine. And what is the struggling part?

Love to help... but it's like asking... please solve world peace for me... and hand me over all of your shizzle ... and expect people here within the community just know what to do... and we have our own talend apocalypse too )

Re: Memory issue for Linkedin/salesforce sync integration job

gt0731 — Thu, 13 Sep 2018 17:40:39 GMT

Sorry as i posted at my night time. I missed out prompt reply

I am using Talend Open Studio for Data Integration v6.3.1.2016.

I attached the screenshot of where the job crashes due to lack of memory. I am assuming your job will run all the way through as it isn't running the whole process where all the files on our servers and compared to pitchbook to produce the complete spreadsheet as at job tFileList_2.
I tried
I have already tried to increase the Ram argument under the Advanced setting to add the -Xmx2048M but this hasnt worked. I am not a Talend expert at all and have only just started using it and therefore my knowledge of it is very basic.

Re: Memory issue for Linkedin/salesforce sync integration job

gt0731 — Thu, 13 Sep 2018 17:45:30 GMT

I also attached screenshot of job as well. files called linkedin_sync.jpg where in i highlighted the painful area

Re: Memory issue for Linkedin/salesforce sync integration job

gt0731 — Thu, 13 Sep 2018 17:46:24 GMT

any help on it would be much appreciated ?

Re: Memory issue for Linkedin/salesforce sync integration job

Anonymous — Fri, 14 Sep 2018 06:50:58 GMT

Hello,

Could you please let us know if this article helps?

https://community.talend.com/t5/Migration-Configuration-and/OutOfMemory-Exception/ta-p/21669

Best regards

Sabrina

Re: Memory issue for Linkedin/salesforce sync integration job

gt0731 — Fri, 14 Sep 2018 18:22:36 GMT

@xdshi @Dijke

I will implement the your mentioned link as well.

Here i am posting the exact error log for your refrence.

This is the error log:

Starting job RMS_LinkedIn_Sync at 17:26 14/09/2018.

[statistics] connecting to socket on port 3825
[statistics] connected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
[statistics] disconnected
at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.attr(Cur.java:3044)
at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.attr(Cur.java:3065)
at org.apache.xmlbeans.impl.store.Locale$SaxHandler.startElement(Locale.java:3263)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.reportStartTag(Piccolo.java:1082)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseAttributesNS(PiccoloLexer.java:1802)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseOpenTagNS(PiccoloLexer.java:1521)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseTagNS(PiccoloLexer.java:1362)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseXMLNS(PiccoloLexer.java:1293)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseXML(PiccoloLexer.java:1261)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.yylex(PiccoloLexer.java:4812)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.yylex(Piccolo.java:1290)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.yyparse(Piccolo.java:1400)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.parse(Piccolo.java:714)
at org.apache.xmlbeans.impl.store.Locale$SaxLoader.load(Locale.java:3479)
at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1277)
at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1264)
at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:345)
at org.openxmlformats.schemas.spreadsheetml.x2006.main.WorksheetDocument$Factory.parse(Unknown Source)
at org.apache.poi.xssf.usermodel.XSSFSheet.read(XSSFSheet.java:194)
at org.apache.poi.xssf.usermodel.XSSFSheet.onDocumentRead(XSSFSheet.java:186)
at org.apache.poi.xssf.usermodel.XSSFWorkbook.onDocumentRead(XSSFWorkbook.java:354)
at org.apache.poi.POIXMLDocument.load(POIXMLDocument.java:166)
at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:240)
at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:296)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tFileInputExcel_8Process(RMS_LinkedIn_Sync.java:20957)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tFileInputDelimited_2Process(RMS_LinkedIn_Sync.java:29394)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tFileList_2Process(RMS_LinkedIn_Sync.java:19152)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tFileDelete_8Process(RMS_LinkedIn_Sync.java:18439)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tFileDelete_7Process(RMS_LinkedIn_Sync.java:18263)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tFileDelete_4Process(RMS_LinkedIn_Sync.java:18087)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tRunJob_7Process(RMS_LinkedIn_Sync.java:61891)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tRunJob_6Process(RMS_LinkedIn_Sync.java:61661)
Job RMS_LinkedIn_Sync ended at 17:27 14/09/2018. [exit code=1]

Re: Memory issue for Linkedin/salesforce sync integration job

Anonymous — Mon, 17 Sep 2018 03:04:08 GMT

Hello,

Have you already tried the solutions mentioned in article? Is it working for you? Don't hesitate to post your issue here.

Best regards

Sabrina

Re: Memory issue for Linkedin/salesforce sync integration job

gt0731 — Mon, 17 Sep 2018 18:26:25 GMT

@Dijke @xdshi

please help on it
I implemented the solution now i am out of java heap space error but also getting few errors.

I am reattaching the job. please try to reproduce the issue.
I am sending you the back up of the job as it stands now?
you can try then test/make amendments as necessary.

More errors level
at tfileexcelinput
Its that error which is causing the output excel files which now write in 2007 to have date formatting issues so the input file that it relates to when reading is not working properly
Salesforce object issue
Attached is the output file thats created - as you can see it is suggesting that none of the Companies or the Contacts are on Salesforce, so it has worked to a certain extent but its not reading the output download files from Salesforce correctly (Contacts, Accounts, Relationships etc etc), which is why its suggesting that all the contacts need to be added to Salesforce along with all the Companies
Attaching this file for your reference " "

Any help on it would be much appreciated ?

LinkedIn sync 2018 09 17.zip

Re: Memory issue for Linkedin/salesforce sync integration job

Jesperrekuh — Mon, 17 Sep 2018 20:19:24 GMT

@gt0731

Ive imported your jobflows but there's a version mismatch (noworries)... but I still was able to look at your jobs ... and I think your job design makes sense but not from a memory perspective.

* DONT read unrelated/not used columns in your lookups. That will drain your memory. so the "unmatched" stuff is inefficient. String based columns/variables will drain memory.

* DONT let columns (which you dont use, previous point) 'travel' to the next tMap. Prevent using multiple tMap if you could do it in one, due to excessive memory usage.

DO:

* Solution - split your big main job into multiple subjobs... and in your case.... are able to run independent and parallel.

* Solution : read/import into a temp file or a buffer/tHashInput and only columns you need for the (un)matching part. Limit memory load.

* Suggestion : first preprocess/read xls files to a tmp csv file because xls is very memory consuming. Run it as an "independent job" into csv and in your matching process/read this csv into tHashOutput. Reuse this.

* Suggestion : if everything of the above doesnt help, import into temporary tables, join/query in database and simply process output with Talend... so you prevent matching in Talend.

Hope this helps... rebuilding your job is up to you