Java heap space issue while reading large of data from source
Hi,
I am trying to read large amount of data approx 3 to 5 millions records from SQL server like source(Heidisql). I keep getting the Java heap space error:
error is: Exception in thread "Thread-0" java.lang.OutOfMemoryError: Java heap space at java.util.LinkedList.listIterator(Unknown Source) at java.util.AbstractList.listIterator(Unknown Source)
I read some of the community posts about it also documentation around it. Things I tried are :
1. Enable Stream option is checked
2. Increasing JVM parameters ie xms2048M and Xmx4096M
Anything else i can do to handle this?
My job design is tmysqlinput--->tLogrow--->tMap--->tDBoutput.
In your job, there are some cache component consuming two much memory. such as tMap, tLogRow. For a large set of data, please try to store the data on disk instead of memory on tMap to allocate more memory to execute the job.
In your job, there are some cache component consuming two much memory. such as tMap, tLogRow. For a large set of data, please try to store the data on disk instead of memory on tMap to allocate more memory to execute the job.
The tLogRow component is used to display data or results in the Run console. You can use tfileoutputxxx to replace it. Could you please post your tMap component setting screenshot here which will be helpful for us to address your issue?
The tLogRow component is used to display data or results in the Run console. You can use tfileoutputxxx to replace it. Could you please post your tMap component setting screenshot here which will be helpful for us to address your issue?
@xdshi :Thanks for your Suggestion! I have removed tLogRow component from my job and now re-testing my job with this code. Please find attached the screenshot of the tMap config settings.
Also, i noticed no temp files are being generated at the defined path. What could be reason for the same?
