Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

Announcements
Save $650 on Qlik Connect, Dec 1 - 7, our lowest price of the year. Register with code CYBERWEEK: Register
cancel
Showing results for 
Search instead for 
Did you mean: 
_AnonymousUser
Specialist III
Specialist III

Getting error while processing large amount of data

Talend is installed in a 32 bits system(4GB RAM). I was migrating 20GB data with some transformation. It was showing error like that

connecting to socket on port 3968
connected
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
disconnected
disconnected
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Unknown Source)
at java.lang.Class.getDeclaredField(Unknown Source)
at java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl.<init>(Unknown Source)
I have made  the following changes 
1) In tmap  set "Store temp data" option as true and also set a directory for temp data
2)Also set JVM arguments as per my system configuration
Still I am getting that error.
Please help me on that issue.
Thanks in advance.
Labels (3)
14 Replies
_AnonymousUser
Specialist III
Specialist III
Author

Hi,
I already set the batch size. In spite of that, Job is taking 20 hours to migrate 25 GB of data.
Is that issue related to Talend or Database ?
Thanks 
Anonymous
Not applicable

Hi,
What's the row rate? Would you mind sharing your job design screenshot with us?
Best regards
Sabrina
_AnonymousUser
Specialist III
Specialist III
Author

Hi,
I am not able to upload any screenshot.
(tinput1,tinput2) --> tMap -->tfilter --> toutput
tOutput is being loaded with 1200 rows/s. Total data volume is 25 GB.
Thanks
Babul
_AnonymousUser
Specialist III
Specialist III
Author

Hi Sabrina,
Please help me if there is any way to enhance the processing speed.
Thanks 
Babul
Anonymous
Not applicable

Hi Babul,
Performance issue is usually caused by the DB connection or the job design. You can register and log in as a Community member first after that  "UPLOAD"  button should be available on your end or could you please export the job and email it to us?
Best regards
Sabrina