[resolved] java.lang.OutOfMemoryError: Java heap space talend

[_AnonymousUser]

Hi All,
I have a simple job with the following components
tMySqlInput ---->tMap---->tMySqlOutput
This is for inserting/updating the records from 1 MySql Db to another.This is working fine for a table with 125000 records .I am getting an error when tried with a table of 11587500 records.The error is:
Exception in thread "Thread-0" java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedList.listIterator(Unknown Source)
at java.util.AbstractList.listIterator(Unknown Source)
at java.util.AbstractSequentialList.iterator(Unknown Source)
at routines.system.RunStat.sendMessages(RunStat.java:261)
at routines.system.RunStat.run(RunStat.java:225)
at java.lang.Thread.run(Unknown Source)
Exception in thread "main" java.lang.Error: java.lang.OutOfMemoryError: Java heap space.
Please help.
Thanks,
scsdev

[Anonymous]

Hi,
I think the reference outOfMemory on Talend Help Center will give you a little help.
Best regards
Sabrina

[_AnonymousUser]

As per the help center ,I have increased my JVM memory upto 1500M.Still its showing the same memory error.I cann't allocate more than 1500M for JVM.So I am thinking of redesigning my job.Is there any option to execute the same task in batches.That is,first fetch 10000 records and insert/update to the target db.Then fetch next 10000 and so on.Please help me in redesigning the job.My task is to insert/update 11587508 records from DB1 to DB2.
thanks,
scsdev

[Anonymous]

Hi,
In Talend, all the rows are collected into the memory and after finishing the collection, the component will delivery each data one by one, this often causes OutOfHeapSpace problems for a large of data set.
Try to check the option enable stream' in the advanced settings of tMysqlInput.
Best regards
Sabrina

[_AnonymousUser]

Hi,
I tried by selecting the option 'Enable Stream' and it worked.Thanks for your help.But I am not able to sync all the records.Showing error like
1. Incorrect string value: '\xEF\xBF\xBD Sh...' for column
2. Data truncation: Out of range value for column
What is the exact issue?How can i correct this?
Thanks,
scsdev

[Anonymous]

Hi,

Incorrect string value: '\xEF\xBF\xBD Sh...' for column

For your issue one
What is your source file? Would you mind giving us a example for that, what's more, the screenshot of your job schema will be very appreciated.

Data truncation: Out of range value for column

For your issue two
Set a length for your data in schema, see the following screenshot for details.
Best regards
Sabrina

[_AnonymousUser]

Hi,
My first issue 'Incorrect string value: '\xEF\xBF\xBD Sh...' for column' was due to the database encoding.It is corrected when i changed the charset to utf8 for the table.
thanks,
scsdev

[Anonymous]

Hi,
Thanks for your feedback, plus how about your issue2? Is it resolved?
Best regards
Sabrina

[_AnonymousUser]

Hi Sabrina,
Second issue still exists.I couldn't find a solution for that.The issue is for a column with datatype float.I have set the length as 9 and precision as 3.It is the same setting available for source db also.But the job is showing the 'Out of range' error for the value 1000000.000.
Thanks,
scsdev

[Anonymous · Accepted solution]

Hi,

I have set the length as 9 and precision as 3.It is the same setting available for source db also

In your MySql DB, you should change your table and set the data type as float(11,3), due to your input source " 1000000.000"
Float(m,d), m represents the total length in bytes and d is the precise.
Best regards
Sabrina