Memory issue for Linkedin/salesforce sync integration job
Attached zip job is crashing due to too much processing at once and not enough RAM to do it and therefore needs simplifying and generally updating to make it better overall.
Any help on it would be much appreciated ?
Also, Advise on what measure need to be taken here at memory level.
please help on it I implemented the solution now i am out of java heap space error but also getting few errors.
I am reattaching the job. please try to reproduce the issue. I am sending you the back up of the job as it stands now? you can try then test/make amendments as necessary.
More errors level at tfileexcelinput Its that error which is causing the output excel files which now write in 2007 to have date formatting issues so the input file that it relates to when reading is not working properly Salesforce object issue Attached is the output file thats created - as you can see it is suggesting that none of the Companies or the Contacts are on Salesforce, so it has worked to a certain extent but its not reading the output download files from Salesforce correctly (Contacts, Accounts, Relationships etc etc), which is why its suggesting that all the contacts need to be added to Salesforce along with all the Companies Attaching this file for your reference " "
Ive imported your jobflows but there's a version mismatch (noworries)... but I still was able to look at your jobs ... and I think your job design makes sense but not from a memory perspective.
* DONT read unrelated/not used columns in your lookups. That will drain your memory. so the "unmatched" stuff is inefficient. String based columns/variables will drain memory.
* DONT let columns (which you dont use, previous point) 'travel' to the next tMap. Prevent using multiple tMap if you could do it in one, due to excessive memory usage.
DO:
* Solution - split your big main job into multiple subjobs... and in your case.... are able to run independent and parallel.
* Solution : read/import into a temp file or a buffer/tHashInput and only columns you need for the (un)matching part. Limit memory load.
* Suggestion : first preprocess/read xls files to a tmp csv file because xls is very memory consuming. Run it as an "independent job" into csv and in your matching process/read this csv into tHashOutput. Reuse this.
* Suggestion : if everything of the above doesnt help, import into temporary tables, join/query in database and simply process output with Talend... so you prevent matching in Talend.
Hope this helps... rebuilding your job is up to you
