Getting error while processing large amount of data

[_AnonymousUser]

Talend is installed in a 32 bits system(4GB RAM). I was migrating 20GB data with some transformation. It was showing error like that

connecting to socket on port 3968
connected
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
disconnected
disconnected
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Unknown Source)
at java.lang.Class.getDeclaredField(Unknown Source)
at java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl.<init>(Unknown Source)
I have made  the following changes 
1) In tmap  set "Store temp data" option as true and also set a directory for temp data
2)Also set JVM arguments as per my system configuration
Still I am getting that error.
Please help me on that issue.
Thanks in advance.

[Anonymous]

Hi,
Does this issue only repro on your specific job?
Did you edit the JVM parameters in .ini file for studio itself or open the Run view, click Advanced settings tab and then edit the JVM parameters for your current job?
Best regards
Sabrina

[_AnonymousUser]

Hi Sabrina,
I have set it in run view > Advanced settings tab.
Still getting error :rolleyes:
Please help me

[Anonymous]

Hi,
What's your current JVM parameters? Could you also show us your job design? Which components are you using in it?
Best regards
Sabrina

[_AnonymousUser]

Hi Sabrina,
Could you please share your email address so that I can send you all the details about that job.
Thanks
Babul

[Anonymous]

Hi Babul,
What component (tMysqlInput?, tPostgresInput?, ???) are you using to read the data?  If you are using the tMysqlInput component, there is an 'Enable stream' option on the component's Advanced settings tab that may be of help to you.
Hope this helps.
 

[_AnonymousUser]

Hi,
I am using postgres database  as source database as well as target database.
Is there any such option ?
Thank
Babul

[Anonymous]

Let's see...
the tPostgressqlInput component has a 'use cursor' option on the 'Advanced settings' tab which allows you to set the cursor size and may help you better manage the amount of data you are bringing in on your select. 
the PostgressqlBulkExec component may also be of use to you though I do not see any options that let you determine different ways to handle the incoming data. 
Not having a Postgres database to test with, I can only guess on what I see.  That said, experiment a little with the above two and see how that works. 

[_AnonymousUser]

Hi,
Now I am facing a new issue. In my 25 GB data migration, execution speed is very low (260 rows/s).
I am using postgres database as source database as well as target database.
Is there any way to improve the execution speed ??
Thanks 
Babul

[Anonymous]

Hi,
Here is an option "Use Batch Size" in tPostgresqloutput which is used to activate the batch mode for data processing.
Best regards
Sabrina