topic Re: tMap Job is hanging at 10000 rows in Talend Studio

tMap Job is hanging at 10000 rows

Anonymous — Wed, 20 Nov 2019 01:28:26 GMT

I am transferring data from a SQL Server table to an Oracle table. It first worked then I deleted the data because I think the primary keys were getting duplicated so I tried it again and it hangs every time now. I'm not sure if I changed something or what. Please tell me what is wrong with this.

Here it is hanging:

Here is the tMap info:

Re: tMap Job is hanging at 10000 rows

Anonymous — Wed, 20 Nov 2019 02:20:46 GMT

Hi,

I would recommend to attach the diagrams as attachments or in line to the post. Many of our contributors may have restrictions to use unknown links.

Coming to the point, please check whether you are having enough memory in the Job server where you are running the job. If you are using lookup, I would recommend to use disk usage for tMap (refer Talend help for details and sample use cases) to reduce the impact of data volume getting stored in memory. It would be also a good idea to increase the Xms and Xmx parameters of the Talend job.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: tMap Job is hanging at 10000 rows

Anonymous — Wed, 20 Nov 2019 03:44:53 GMT

In terms of job server do you mean the computer I am on running on?

After asking this I decided to try the same job on my other computer and it worked but I got a unique constraint violated error and I think it is because the Numeric.sequence is creating duplicates, how can I fix this error?

Re: tMap Job is hanging at 10000 rows

Raghuram_Puram — Wed, 20 Nov 2019 13:32:25 GMT

For unique key constrain error:

See if there is data in the table already and you are trying to insert the same again.

Also check if you are using "insert or update" option in target load component.

Re: tMap Job is hanging at 10000 rows

Anonymous — Wed, 20 Nov 2019 14:35:10 GMT

Hi,

First of all, thanks for updating the images 🙂

Coming to your query, there could be multiple reasons for the job hanging. I would recommend to try below items to see if the process speed is increasing.

a) The first step I will take is to make sure that Die on Error check box is selected. This will help you to identify if there is any error.

b) As Raghu told in previous post, make sure you are using Insert/Update mechanism and select a key in the output schema of tOracleOutput to tell the DB that insert and update need to happen based on the key selected.

c) If you are fine to do commit in between (please note that you will not be able to do rollback if you are using this option), select "Commit every" value in Advanced properties. If you are using a Oracle connection component to pass the DB connectivity, select Auto Commit option in Advanced properties.

d) You can also play with Batch size to adjust the number of records in each batch (There is no magic number here and you may have to adjust the value to see the performance of underlying DB).

e) In the Run Tab of Talend job, go to Advanced settings and change the Xms and XMx parameters to higher values. Please make sure that you are not giving a huge value which will consume entire system memory.

Please let us know how things are going after changing these options and feel free to come back if the issue persists. If the issue is resolved, please don't forget to mark the topic as resolved 🙂

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: tMap Job is hanging at 10000 rows

Anonymous — Wed, 20 Nov 2019 14:53:03 GMT

hey,
Looking at your output schema, "addrerss_id" is your key which has a sequence generator. so you won't be getting duplicate key issue since it has increment factor as 1. The only reason could be your the table must be having that ID already. so try truncate and load. (first truncate the table and then load it back)

Regards,

Ajinkya

Re: tMap Job is hanging at 10000 rows

Anonymous — Thu, 21 Nov 2019 00:27:17 GMT

I switched computers and ran the same job and it worked again for one time then stopped working everytime after it is stilling hanging at 10000 rows - I have tried making the xms and xmx by a couple hundred.

Re: tMap Job is hanging at 10000 rows

Anonymous — Thu, 21 Nov 2019 00:58:05 GMT

Could it also be a db server issue because I share the server with a bunch of other people?

Re: tMap Job is hanging at 10000 rows

Anonymous — Thu, 21 Nov 2019 02:23:18 GMT

Hi,

I guess the issue could be at Oracle Output component. Could you please share the screenshot of the component (both basic and advanced)? Also please share the target table structure to check the PK of the table.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved