topic Getting error while processing large amount of data in Talend Studio

Getting error while processing large amount of data

_AnonymousUser — Wed, 06 Jan 2016 13:59:52 GMT

Talend is installed in a 32 bits system(4GB RAM). I was migrating 20GB data with some transformation. It was showing error like that

connecting to socket on port 3968
connected
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
disconnected
disconnected
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Unknown Source)
at java.lang.Class.getDeclaredField(Unknown Source)
at java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl.<init>(Unknown Source)
I have made the following changes
1) In tmap set "Store temp data" option as true and also set a directory for temp data
2)Also set JVM arguments as per my system configuration
Still I am getting that error.
Please help me on that issue.
Thanks in advance.

Re: Getting error while processing large amount of data

Anonymous — Thu, 07 Jan 2016 06:55:33 GMT

Hi,
Does this issue only repro on your specific job?
Did you edit the JVM parameters in .ini file for studio itself or open the Run view, click Advanced settings tab and then edit the JVM parameters for your current job?
Best regards
Sabrina

Re: Getting error while processing large amount of data

_AnonymousUser — Thu, 07 Jan 2016 09:45:14 GMT

Hi Sabrina,
I have set it in run view > Advanced settings tab.
Still getting error :rolleyes:
Please help me

Re: Getting error while processing large amount of data

Anonymous — Thu, 07 Jan 2016 10:40:21 GMT

Hi,
What's your current JVM parameters? Could you also show us your job design? Which components are you using in it?
Best regards
Sabrina

Re: Getting error while processing large amount of data

_AnonymousUser — Thu, 07 Jan 2016 10:45:16 GMT

Hi Sabrina,
Could you please share your email address so that I can send you all the details about that job.
Thanks
Babul

Re: Getting error while processing large amount of data

Anonymous — Thu, 07 Jan 2016 12:55:34 GMT

Hi Babul,
What component (tMysqlInput?, tPostgresInput?, ???) are you using to read the data? If you are using the tMysqlInput component, there is an 'Enable stream' option on the component's Advanced settings tab that may be of help to you.
Hope this helps.

Re: Getting error while processing large amount of data

_AnonymousUser — Thu, 07 Jan 2016 13:15:32 GMT

Hi,
I am using postgres database as source database as well as target database.
Is there any such option ?
Thank
Babul

Re: Getting error while processing large amount of data

Anonymous — Thu, 07 Jan 2016 14:30:50 GMT

Let's see...
the tPostgressqlInput component has a 'use cursor' option on the 'Advanced settings' tab which allows you to set the cursor size and may help you better manage the amount of data you are bringing in on your select.
the PostgressqlBulkExec component may also be of use to you though I do not see any options that let you determine different ways to handle the incoming data.
Not having a Postgres database to test with, I can only guess on what I see. That said, experiment a little with the above two and see how that works.

Re: Getting error while processing large amount of data

_AnonymousUser — Tue, 12 Jan 2016 12:01:50 GMT

Hi,
Now I am facing a new issue. In my 25 GB data migration, execution speed is very low (260 rows/s).
I am using postgres database as source database as well as target database.
Is there any way to improve the execution speed ??
Thanks
Babul

Re: Getting error while processing large amount of data

Anonymous — Wed, 13 Jan 2016 05:49:15 GMT

Hi,
Here is an option "Use Batch Size" in tPostgresqloutput which is used to activate the batch mode for data processing.
Best regards
Sabrina

Re: Getting error while processing large amount of data

_AnonymousUser — Thu, 21 Jan 2016 05:34:54 GMT

Hi,
I already set the batch size. In spite of that, Job is taking 20 hours to migrate 25 GB of data.
Is that issue related to Talend or Database ?
Thanks

Re: Getting error while processing large amount of data

Anonymous — Thu, 21 Jan 2016 06:28:01 GMT

Hi,
What's the row rate? Would you mind sharing your job design screenshot with us?
Best regards
Sabrina

Re: Getting error while processing large amount of data

_AnonymousUser — Thu, 21 Jan 2016 08:04:21 GMT

Hi,
I am not able to upload any screenshot.
(tinput1,tinput2) --> tMap -->tfilter --> toutput
tOutput is being loaded with 1200 rows/s. Total data volume is 25 GB.
Thanks
Babul

Re: Getting error while processing large amount of data

_AnonymousUser — Fri, 22 Jan 2016 07:54:21 GMT

Hi Sabrina,
Please help me if there is any way to enhance the processing speed.
Thanks
Babul

Re: Getting error while processing large amount of data

Anonymous — Fri, 22 Jan 2016 08:36:46 GMT

Hi Babul,
Performance issue is usually caused by the DB connection or the job design. You can register and log in as a Community member first after that "UPLOAD" button should be available on your end or could you please export the job and email it to us?
Best regards
Sabrina