Getting error while processing large amount of data
Talend is installed in a 32 bits system(4GB RAM). I was migrating 20GB data with some transformation. It was showing error like that
connecting to socket on port 3968
connected
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
disconnected
disconnected
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Unknown Source)
at java.lang.Class.getDeclaredField(Unknown Source)
at java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl.<init>(Unknown Source)
I have made the following changes
1) In tmap set "Store temp data" option as true and also set a directory for temp data
2)Also set JVM arguments as per my system configuration
Still I am getting that error.
Please help me on that issue.
Thanks in advance.
Hi, I already set the batch size. In spite of that, Job is taking 20 hours to migrate 25 GB of data. Is that issue related to Talend or Database ? Thanks
Hi, I am not able to upload any screenshot. (tinput1,tinput2) --> tMap -->tfilter --> toutput tOutput is being loaded with 1200 rows/s. Total data volume is 25 GB. Thanks Babul
Hi Babul,
Performance issue is usually caused by the DB connection or the job design. You can register and log in as a Community member first after that "UPLOAD" button should be available on your end or could you please export the job and email it to us?
Best regards
Sabrina