topic Re: Java heap space talend in Talend Studio

Java heap space talend

Anonymous — Sat, 16 Nov 2024 06:10:58 GMT

Hi All,

I have a simple job with the following components
tMySqlInput ---->tMap---->tMySqlOutput
This is for inserting/updating the records from 1 MySql Db to another.This is working fine for a table with 100 000 records .I am getting an error when tried with a table of 3 995 586 records

I check the option enable stream and I modify the Use specific JVM arguments In the Run view, open the Advanced Settings tab and in the file TOS_ESB-win-x86_64

but still dont work any solutions plzzzzzzzz

Re: Java heap space talend

manodwhb — Tue, 02 Apr 2019 12:09:33 GMT

@AnisBensalah ,can you please let me know what JVM configuration are you using? And can you also do the operation of tMap on store on disk.

Re: Java heap space talend

Anonymous — Tue, 02 Apr 2019 12:40:03 GMT

Hi,

You can increase the heap space for specific jobs or for all the jobs by changing the Xms and Xmx parameters.

https://community.talend.com/t5/Design-and-Development/Pass-JVM-arguments-in-quot-Use-specific-JVM-arguments-quot-as/td-p/80414

But it is not a single solution that fits all sizes. You need to still think about the possibility of parking the data in tMap and other components using temporary disk space. Please refer the detail below from below link.

https://help.talend.com/reader/EJfmjmfWqXUp5sadUwoGBA/J4xg5kxhK1afr7i7rFA65w

It means that you are not holding the entire data in memory which usually creates the issue like in your use. It will hold the overflowing data in disk temporarily and this way you will not run out of memory and you dont have to keep on expanding the memory.

I would also suggest you to do a memory run of the job to optimize the memory allocation of your job before moving to production. In this way, you will know what is the optimum value by changing the Xms and Xmx parameters.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: Java heap space talend

Anonymous — Tue, 02 Apr 2019 15:49:52 GMT

hi @nthampi ,@manodwhb thnx for repling me but it s the same probleme

in fact there is no error but the job it stays in loop and does not end since 2 hours of the execution of the job what am I to do

Re: Java heap space talend

Anonymous — Tue, 02 Apr 2019 16:01:16 GMT

Hi,

Did you check at which component, it is taking more time to process? Is there any DB load as part of your job? Most probably it could be due to low commit size.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: Java heap space talend

Anonymous — Tue, 02 Apr 2019 16:12:57 GMT

there is no loading from the base and there is no transformation to the new base

I think that it stops at the levels of the loading . in the tdbinput

Re: Java heap space talend

Anonymous — Tue, 02 Apr 2019 18:55:27 GMT

Hi,

I feel the tDBOutput is taking more time. Could you please change the highlighted parameters and see the performance difference. Please reach an optimum level for these parameters based on your system resource availability.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: Java heap space talend

Anonymous — Wed, 03 Apr 2019 11:41:44 GMT

the problem is not resolved but she gave me another error

Exception in component tDBOutput_1 (g)
java.sql.SQLException: Could not retrieve transation read-only status server

Re: Java heap space talend

Anonymous — Wed, 03 Apr 2019 12:58:12 GMT

Hi,

It seems multiple write requests without commit is happening at same time.

https://stackoverflow.com/questions/21651671/odd-sqlexception-could-not-retrieve-transation-read-only-status-server

Could you please check whether commit is correct?

I also saw another post in stackoverflow as below. Please think about this option too.

https://stackoverflow.com/questions/51935611/connecting-talnd-to-mysql-transation-read-only-status-server

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: Java heap space talend

Anonymous — Wed, 03 Apr 2019 14:52:19 GMT

hi @nthampi ,

I resolved the 2nd error its just it an error in the connection parameter

but for the load data no

I changed the number of commit every and the number of rows but still the same 0 ligne effected in the DB

Re: Java heap space talend

Anonymous — Thu, 04 Apr 2019 06:42:40 GMT

@AnisBensalah

I am sorry. I did not get what you are trying to convey. Could you please explain the current issue with some screenshots?

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: Java heap space talend

Anonymous — Thu, 04 Apr 2019 10:55:50 GMT

hi @nthampi,

well i want to insert 4 million records from DB to another DB for that i use a simple job with the following components
tMySqlInput ---->tMap---->tMySqlOutput but when i start the job it remains running for an hour or two and it gives me only 200 record and finally no result ,and in the end therse no error, I do not know why

Re: Java heap space talend

Anonymous — Thu, 04 Apr 2019 11:37:13 GMT

Hi,

Could you please use Bulk components to load the data to target table? So you can download the data from source table and if you are using tparallelize, you can download multiple partitions of data from source table at same time.

Once the data is ready, you can use Bulk components to load them to target table. Once you do this method, your overall processing time will be reduced from hours to minutes.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: Java heap space talend

Anonymous — Fri, 05 Apr 2019 09:28:58 GMT

thnx @ nikhilthampi it s work

Re: Java heap space talend

sushantk19 — Tue, 30 Jun 2020 10:18:37 GMT

@AnisBensalah @nthampi _: I am also facing the same issue. I have a simple job design where i am extracting the data from source and loading to my target table( around 3 to 5 million). what do you mean by bulk components? how to parallelize it? my job design is as follows:

tMysqlinput--->tLogRow--->tMap-->tDBoutput

Re: Java heap space talend

sushantk19 — Tue, 30 Jun 2020 10:18:37 GMT

tMysqlinput--->tLogRow--->tMap-->tDBoutput