topic Re: [resolved] java.lang.OutOfMemoryError: Java heap space talend in Talend Studio

[resolved] java.lang.OutOfMemoryError: Java heap space talend

_AnonymousUser — Fri, 01 Feb 2013 03:25:32 GMT

Hi All,
I have a simple job with the following components
tMySqlInput ---->tMap---->tMySqlOutput
This is for inserting/updating the records from 1 MySql Db to another.This is working fine for a table with 125000 records .I am getting an error when tried with a table of 11587500 records.The error is:
Exception in thread "Thread-0" java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedList.listIterator(Unknown Source)
at java.util.AbstractList.listIterator(Unknown Source)
at java.util.AbstractSequentialList.iterator(Unknown Source)
at routines.system.RunStat.sendMessages(RunStat.java:261)
at routines.system.RunStat.run(RunStat.java:225)
at java.lang.Thread.run(Unknown Source)
Exception in thread "main" java.lang.Error: java.lang.OutOfMemoryError: Java heap space.
Please help.
Thanks,
scsdev

Re: [resolved] java.lang.OutOfMemoryError: Java heap space talend

Anonymous — Fri, 01 Feb 2013 04:02:38 GMT

Hi,
I think the reference outOfMemory on Talend Help Center will give you a little help.
Best regards
Sabrina

Re: [resolved] java.lang.OutOfMemoryError: Java heap space talend

_AnonymousUser — Fri, 01 Feb 2013 05:26:52 GMT

As per the help center ,I have increased my JVM memory upto 1500M.Still its showing the same memory error.I cann't allocate more than 1500M for JVM.So I am thinking of redesigning my job.Is there any option to execute the same task in batches.That is,first fetch 10000 records and insert/update to the target db.Then fetch next 10000 and so on.Please help me in redesigning the job.My task is to insert/update 11587508 records from DB1 to DB2.
thanks,
scsdev

Re: [resolved] java.lang.OutOfMemoryError: Java heap space talend

Anonymous — Fri, 01 Feb 2013 06:48:16 GMT

Hi,
In Talend, all the rows are collected into the memory and after finishing the collection, the component will delivery each data one by one, this often causes OutOfHeapSpace problems for a large of data set.
Try to check the option enable stream' in the advanced settings of tMysqlInput.
Best regards
Sabrina

Re: [resolved] java.lang.OutOfMemoryError: Java heap space talend

_AnonymousUser — Fri, 01 Feb 2013 09:59:40 GMT

Hi,
I tried by selecting the option 'Enable Stream' and it worked.Thanks for your help.But I am not able to sync all the records.Showing error like
1. Incorrect string value: '\xEF\xBF\xBD Sh...' for column
2. Data truncation: Out of range value for column
What is the exact issue?How can i correct this?
Thanks,
scsdev

Re: [resolved] java.lang.OutOfMemoryError: Java heap space talend

Anonymous — Fri, 01 Feb 2013 10:43:02 GMT

Hi,

Incorrect string value: '\xEF\xBF\xBD Sh...' for column

For your issue one
What is your source file? Would you mind giving us a example for that, what's more, the screenshot of your job schema will be very appreciated.

Data truncation: Out of range value for column

For your issue two
Set a length for your data in schema, see the following screenshot for details.
Best regards
Sabrina

Re: [resolved] java.lang.OutOfMemoryError: Java heap space talend

_AnonymousUser — Mon, 04 Feb 2013 08:19:56 GMT

Hi,
My first issue 'Incorrect string value: '\xEF\xBF\xBD Sh...' for column' was due to the database encoding.It is corrected when i changed the charset to utf8 for the table.
thanks,
scsdev

Re: [resolved] java.lang.OutOfMemoryError: Java heap space talend

Anonymous — Mon, 04 Feb 2013 08:28:40 GMT

Hi,
Thanks for your feedback, plus how about your issue2? Is it resolved?
Best regards
Sabrina

Re: [resolved] java.lang.OutOfMemoryError: Java heap space talend

_AnonymousUser — Mon, 04 Feb 2013 08:47:46 GMT

Hi Sabrina,
Second issue still exists.I couldn't find a solution for that.The issue is for a column with datatype float.I have set the length as 9 and precision as 3.It is the same setting available for source db also.But the job is showing the 'Out of range' error for the value 1000000.000.
Thanks,
scsdev

Re: [resolved] java.lang.OutOfMemoryError: Java heap space talend

Anonymous — Mon, 04 Feb 2013 09:55:40 GMT

Hi,

I have set the length as 9 and precision as 3.It is the same setting available for source db also

In your MySql DB, you should change your table and set the data type as float(11,3), due to your input source " 1000000.000"
Float(m,d), m represents the total length in bytes and d is the precise.
Best regards
Sabrina

Re: [resolved] java.lang.OutOfMemoryError: Java heap space talend

_AnonymousUser — Mon, 04 Feb 2013 10:07:51 GMT

Hi,
Thank you very much.It helped me.
scsdev

Re: [resolved] java.lang.OutOfMemoryError: Java heap space talend

Anonymous — Mon, 04 Feb 2013 10:16:06 GMT

Hi,
Welcome, Please feel free to post your issue on talend forum.
Best regards
Sabrina

Re: [resolved] java.lang.OutOfMemoryError: Java heap space talend

Jackson0123 — Wed, 08 Oct 2025 12:22:49 GMT

This error java.lang.OutOfMemoryError: Java heap space indicates that NiFi is running out of memory while processing your flow. Since you are using NiFi 1.0 with around 123 processors and also seems to handle about 200 flow files of 50 MB each, the data load is quite higher than usual. Even with 4 GB of heap memory configured in bootstrap.conf, NiFi may not be able to handle this workload efficiently due to memory constraints and known inefficiencies in that version.

Upgrading to a newer version of NiFi can be considered as the best way recommended. This is suggested because in most cases, later releases will include several memory and performance optimizations that will help you to handle such circumstances. Along with this change, consider reducing the number of concurrent tasks, processing smaller batches, setting appropriate back pressure limits, and ensuring repositories use disk storage instead of memory. Monitoring heap usage through GC logs can also help identify memory-heavy processors and prevent similar issues in the future. If you are interested in learning about the types of OutOfMemoryError, you can check out this blog: Types of OutOfMemoryError, Causes, and Solutions.