Memory issue for Linkedin/salesforce sync integration job

[gt0731]

 Attached  zip job is crashing due to too much processing at once and not enough RAM to do it and therefore needs simplifying and generally updating to make it better overall.

Any help on it would be much appreciated ?

Also, Advise on  what measure need to be taken here at memory level.

[Jesperrekuh]

which version are you on.
Please show a screenshot... we will point out what to adjust and how.
Memory and processing power is system related... so it works here and fails at your side ...

What did you already try to improve performance?!?

[gt0731]

I already attached complete zipped folder about this job. Are you able to
extract the folder ?

I tried to implement
Apex soap api and at memory still struggling to figure out the process

[Jesperrekuh]

Again, what version are you running? I need to be able to import your project and your Talend version could differ from mine. And what is the struggling part?

Love to help... but it's like asking... please solve world peace for me... and hand me over all of your shizzle ... and expect people here within the community just know what to do... and we have our own talend apocalypse too )

[gt0731]

Sorry as i posted at my night time. I missed out prompt reply


I am using Talend Open Studio for Data Integration v6.3.1.2016.

I attached the screenshot of where the job crashes due to lack of memory. I am assuming your job will run all the way through as it isn't running the whole process where all the files on our servers and compared to pitchbook to produce the complete spreadsheet as at job tFileList_2.
I tried
I have already tried to increase the Ram argument under the Advanced setting to add the -Xmx2048M but this hasnt worked. I am not a Talend expert at all and have only just started using it and therefore my knowledge of it is very basic.

[gt0731]

I also attached screenshot of job as well. files called linkedin_sync.jpg where in i highlighted the painful area

[gt0731]

any help on it would be much appreciated ?

[Anonymous]

Hello,

Could you please let us know if this article helps?

https://community.talend.com/t5/Migration-Configuration-and/OutOfMemory-Exception/ta-p/21669

Best regards

Sabrina

[gt0731]

@xdshi @Dijke

I will implement the your mentioned link as well. 

Here i am posting the exact error log for your refrence. 

This is the error log:

Starting job RMS_LinkedIn_Sync at 17:26 14/09/2018.

[statistics] connecting to socket on port 3825
[statistics] connected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
[statistics] disconnected
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
[statistics] disconnected
at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.attr(Cur.java:3044)
at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.attr(Cur.java:3065)
at org.apache.xmlbeans.impl.store.Locale$SaxHandler.startElement(Locale.java:3263)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.reportStartTag(Piccolo.java:1082)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseAttributesNS(PiccoloLexer.java:1802)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseOpenTagNS(PiccoloLexer.java:1521)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseTagNS(PiccoloLexer.java:1362)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseXMLNS(PiccoloLexer.java:1293)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseXML(PiccoloLexer.java:1261)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.yylex(PiccoloLexer.java:4812)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.yylex(Piccolo.java:1290)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.yyparse(Piccolo.java:1400)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.parse(Piccolo.java:714)
at org.apache.xmlbeans.impl.store.Locale$SaxLoader.load(Locale.java:3479)
at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1277)
at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1264)
at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:345)
at org.openxmlformats.schemas.spreadsheetml.x2006.main.WorksheetDocument$Factory.parse(Unknown Source)
at org.apache.poi.xssf.usermodel.XSSFSheet.read(XSSFSheet.java:194)
at org.apache.poi.xssf.usermodel.XSSFSheet.onDocumentRead(XSSFSheet.java:186)
at org.apache.poi.xssf.usermodel.XSSFWorkbook.onDocumentRead(XSSFWorkbook.java:354)
at org.apache.poi.POIXMLDocument.load(POIXMLDocument.java:166)
at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:240)
at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:296)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tFileInputExcel_8Process(RMS_LinkedIn_Sync.java:20957)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tFileInputDelimited_2Process(RMS_LinkedIn_Sync.java:29394)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tFileList_2Process(RMS_LinkedIn_Sync.java:19152)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tFileDelete_8Process(RMS_LinkedIn_Sync.java:18439)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tFileDelete_7Process(RMS_LinkedIn_Sync.java:18263)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tFileDelete_4Process(RMS_LinkedIn_Sync.java:18087)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tRunJob_7Process(RMS_LinkedIn_Sync.java:61891)
at linkedinsync.rms_linkedin_sync_0_10.RMS_LinkedIn_Sync.tRunJob_6Process(RMS_LinkedIn_Sync.java:61661)
Job RMS_LinkedIn_Sync ended at 17:27 14/09/2018. [exit code=1]

[Anonymous]

Hello,

Have you already tried the solutions mentioned in article? Is it working for you? Don't hesitate to post your issue here.

Best regards

Sabrina