JAVA Heap Space Error while import an Excel file

[Anonymous]

Hello Experts,
I am a new bie in Talend platform.i have JDK 1.6 and Talend 4.0. Now i am trying to load the data from an Excel file which contain more than 1 lacs records. While importing the excel file its erroring out with lang.OutOfMemoryError: Java heap space

at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:559)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:514)
at org.eclipse.equinox.launcher.Main.run(Main.java:1311)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.util.Arrays.copyOf(Unknown Source)
at java.util.ArrayList.toArray(Unknown Source)
at org.talend.repository.ui.wizards.metadata.connection.files.excel.ExcelReader.readSheet(Unknown Source)
at org.talend.repository.ui.wizards.metadata.connection.files.excel.ExcelFileStep1Form$9$1.run(Unknown Source)
at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:134)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3855)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3476)
at org.eclipse.jface.operation.ModalContext$ModalContextThread.block(ModalContext.java:173)
at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:388)
at org.eclipse.jface.dialogs.ProgressMonitorDialog.run(ProgressMonitorDialog.java:507)
at org.talend.repository.ui.wizards.metadata.connection.files.excel.ExcelFileStep1Form.viewSheet(Unknown Source)
at org.talend.repository.ui.wizards.metadata.connection.files.excel.ExcelFileStep1Form.viewExcelFile(Unknown Source)
at org.talend.repository.ui.wizards.metadata.connection.files.excel.ExcelFileStep1Form.access$4(Unknown Source)
at org.talend.repository.ui.wizards.metadata.connection.files.excel.ExcelFileStep1Form$7.modifyText(Unknown Source)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:167)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1003)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1027)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1008)
at org.eclipse.swt.widgets.Text.wmCommandChild(Text.java:2526)
at org.eclipse.swt.widgets.Control.WM_COMMAND(Control.java:4082)
at org.eclipse.swt.widgets.Control.windowProc(Control.java:3949)
at org.eclipse.swt.widgets.Display.windowProc(Display.java:4589)
at org.eclipse.swt.internal.win32.OS.CallWindowProcW(Native Method)
at org.eclipse.swt.internal.win32.OS.CallWindowProc(OS.java:2313)
at org.eclipse.swt.widgets.Text.callWindowProc(Text.java:255)
at org.eclipse.swt.widgets.Control.windowProc(Control.java:4036)
at org.eclipse.swt.widgets.Text.windowProc(Text.java:2170)
at org.eclipse.swt.widgets.Display.windowProc(Display.java:4589)
at org.eclipse.swt.internal.win32.OS.SetWindowTextW(Native Method)

Plz shed some light on it
Thanks in advance..

[Anonymous]

Hello
Go to windows-->preference-->talend-->Run/Debug-->Job run vm arguments, and locates more memory.
Best regards
shong

[Anonymous]

Hi,
Thanks for ur quick reply...i changed to -Xms128M and -Xmx512M..but still no luck.My memory congifuration is 2GB.
Plz Let me know if i need to verify anything.....Plz do reply..
Thanks...

[Anonymous]

Hello
See 3281. As jandry said, try to
decrease the memory setting in the Preferences and run it again. If that doesn't work, export the job as a script and try running it outside of the Studio via the Command Prompt.
Best regards

shong

[Anonymous]

Hi,
I am running TIS version 3.2.1 and have tried many combinations of the memory setting but still get :-
Could not create the Java virtual machine.
Error occurred during initialization of VM
Could not reserve enough space for object heap
I have 2.5Gb of memory.
Current settings are :-
-vmargs
-Xms256M
-Xmx1536M
-XX:MaxPermSize=128m
I want to be able to run the debugger, so exporting the job and running from the command line is not feasible.
One of the posts says :-
'Moving the JAVA_HOME/bin entry in the Path variable to the first position should solve the problem.'
but I do not know how to do this??
Also, running from the command line is great until you want to run debugger - I need to run debugger.
Can anyone help?

[Anonymous]

Thanks Shong for all ur help....my java heap space error got resolved....
But i have one doubt.My Excel input has 1 lacs records with some null values.While loading the data it is inserting only 55523 records into the database and skipped the null rows.
My question 1> is it(55523 records) the maximum limit ?
2>Is there any way that we can load the records having NULL values ?

Thanks for ur time......

[Anonymous]

Hello

My question 1> is it(55523 records) the maximum limit ?
2>Is there any way that we can load the records having NULL values ?

There is no a limitation and don't skip the null rows, please check if you have checked the option 'Stop to read on empty rows' on advanced settings.
Best regards
shong

[_AnonymousUser]

adding JAVA_HOME/bin to the PATH environment on windows:

Right click on my computer->properties->advanced->environment variables

Go down to the system variables and edit the PATH variable.

Add the full path to Java's bin directory to the front of the string, followed by a semi-colon:

c:\Program Files\java\jdk1.6.0_07\bin\; <existing path string>

I am not sure if this is your problem, but it's worth a shot.

Thanks,

Thomas

[Anonymous]

Hi Thomas,
This did not resolve my issue.
Added bugtracker - http://www.talendforge.org/bugs/view.php?id=11752 to track issue.

[Anonymous]

Hello,

Instead of importing/Scanning large excel file take 10-20 records for schema of that file after imported the file change the location to your large excel file and change -Generation mode to "Less memory consumed for large excel(Event mode) in  "tFileInputExcel" component Advanced settings and give column count also(I have 60 columns in my file) see this count  by "Metadat->FileExcel->Scanned file->metadata->Columns  there you can find the column count.

Hope it will work.

Thanks.