topic Re: JAVA Heap Space Error while import an Excel file in Talend Studio

JAVA Heap Space Error while import an Excel file

Anonymous — Sat, 16 Nov 2024 13:32:22 GMT

Hello Experts,
I am a new bie in Talend platform.i have JDK 1.6 and Talend 4.0. Now i am trying to load the data from an Excel file which contain more than 1 lacs records. While importing the excel file its erroring out with lang.OutOfMemoryError: Java heap space

at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:559)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:514)
at org.eclipse.equinox.launcher.Main.run(Main.java:1311)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.util.Arrays.copyOf(Unknown Source)
at java.util.ArrayList.toArray(Unknown Source)
at org.talend.repository.ui.wizards.metadata.connection.files.excel.ExcelReader.readSheet(Unknown Source)
at org.talend.repository.ui.wizards.metadata.connection.files.excel.ExcelFileStep1Form$9$1.run(Unknown Source)
at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:134)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3855)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3476)
at org.eclipse.jface.operation.ModalContext$ModalContextThread.block(ModalContext.java:173)
at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:388)
at org.eclipse.jface.dialogs.ProgressMonitorDialog.run(ProgressMonitorDialog.java:507)
at org.talend.repository.ui.wizards.metadata.connection.files.excel.ExcelFileStep1Form.viewSheet(Unknown Source)
at org.talend.repository.ui.wizards.metadata.connection.files.excel.ExcelFileStep1Form.viewExcelFile(Unknown Source)
at org.talend.repository.ui.wizards.metadata.connection.files.excel.ExcelFileStep1Form.access$4(Unknown Source)
at org.talend.repository.ui.wizards.metadata.connection.files.excel.ExcelFileStep1Form$7.modifyText(Unknown Source)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:167)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1003)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1027)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1008)
at org.eclipse.swt.widgets.Text.wmCommandChild(Text.java:2526)
at org.eclipse.swt.widgets.Control.WM_COMMAND(Control.java:4082)
at org.eclipse.swt.widgets.Control.windowProc(Control.java:3949)
at org.eclipse.swt.widgets.Display.windowProc(Display.java:4589)
at org.eclipse.swt.internal.win32.OS.CallWindowProcW(Native Method)
at org.eclipse.swt.internal.win32.OS.CallWindowProc(OS.java:2313)
at org.eclipse.swt.widgets.Text.callWindowProc(Text.java:255)
at org.eclipse.swt.widgets.Control.windowProc(Control.java:4036)
at org.eclipse.swt.widgets.Text.windowProc(Text.java:2170)
at org.eclipse.swt.widgets.Display.windowProc(Display.java:4589)
at org.eclipse.swt.internal.win32.OS.SetWindowTextW(Native Method)

Plz shed some light on it
Thanks in advance..

Re: JAVA Heap Space Error while import an Excel file

Anonymous — Sat, 20 Feb 2010 09:55:10 GMT

Hello
Go to windows-->preference-->talend-->Run/Debug-->Job run vm arguments, and locates more memory.
Best regards
shong

Re: JAVA Heap Space Error while import an Excel file

Anonymous — Sat, 20 Feb 2010 12:10:07 GMT

Hi,
Thanks for ur quick reply...i changed to -Xms128M and -Xmx512M..but still no luck.My memory congifuration is 2GB.
Plz Let me know if i need to verify anything.....Plz do reply..
Thanks...

Re: JAVA Heap Space Error while import an Excel file

Anonymous — Sun, 21 Feb 2010 03:31:39 GMT

Hello
See 3281. As jandry said, try to
decrease the memory setting in the Preferences and run it again. If that doesn't work, export the job as a script and try running it outside of the Studio via the Command Prompt.
Best regards

shong

Re: JAVA Heap Space Error while import an Excel file

Anonymous — Mon, 22 Feb 2010 17:20:36 GMT

Hi,
I am running TIS version 3.2.1 and have tried many combinations of the memory setting but still get :-
Could not create the Java virtual machine.
Error occurred during initialization of VM
Could not reserve enough space for object heap
I have 2.5Gb of memory.
Current settings are :-
-vmargs
-Xms256M
-Xmx1536M
-XX:MaxPermSize=128m
I want to be able to run the debugger, so exporting the job and running from the command line is not feasible.
One of the posts says :-
'Moving the JAVA_HOME/bin entry in the Path variable to the first position should solve the problem.'
but I do not know how to do this??
Also, running from the command line is great until you want to run debugger - I need to run debugger.
Can anyone help?

Re: JAVA Heap Space Error while import an Excel file

Anonymous — Tue, 23 Feb 2010 06:29:08 GMT

Thanks Shong for all ur help....my java heap space error got resolved....
But i have one doubt.My Excel input has 1 lacs records with some null values.While loading the data it is inserting only 55523 records into the database and skipped the null rows.
My question 1> is it(55523 records) the maximum limit ?
2>Is there any way that we can load the records having NULL values ?

Thanks for ur time......

Re: JAVA Heap Space Error while import an Excel file

Anonymous — Tue, 23 Feb 2010 07:13:42 GMT

Hello

My question 1> is it(55523 records) the maximum limit ?
2>Is there any way that we can load the records having NULL values ?

There is no a limitation and don't skip the null rows, please check if you have checked the option 'Stop to read on empty rows' on advanced settings.
Best regards
shong

Re: JAVA Heap Space Error while import an Excel file

_AnonymousUser — Tue, 23 Feb 2010 17:16:30 GMT

adding JAVA_HOME/bin to the PATH environment on windows:

Right click on my computer->properties->advanced->environment variables

Go down to the system variables and edit the PATH variable.

Add the full path to Java's bin directory to the front of the string, followed by a semi-colon:

c:\Program Files\java\jdk1.6.0_07\bin\; <existing path string>

I am not sure if this is your problem, but it's worth a shot.

Thanks,

Thomas

Re: JAVA Heap Space Error while import an Excel file

Anonymous — Tue, 23 Feb 2010 17:42:55 GMT

Hi Thomas,
This did not resolve my issue.
Added bugtracker - http://www.talendforge.org/bugs/view.php?id=11752 to track issue.

Re: JAVA Heap Space Error while import an Excel file

Anonymous — Tue, 15 May 2018 06:48:18 GMT

Hello,

Instead of importing/Scanning large excel file take 10-20 records for schema of that file after imported the file change the location to your large excel file and change -Generation mode to "Less memory consumed for large excel(Event mode) in "tFileInputExcel" component Advanced settings and give column count also(I have 60 columns in my file) see this count by "Metadat->FileExcel->Scanned file->metadata->Columns there you can find the column count.

Hope it will work.

Thanks.