topic Re: Performance Issue with parallel jobs in Talend Studio

Performance Issue with parallel jobs

rrigoni — Sat, 16 Nov 2024 03:05:44 GMT

Hi all,

I need to process a big file. The process is complex. I decided to split this file in several files and created a several instances of this job. This way this instances running parallel. It works. But I realize that performance is decreased over time. The first job was executed in 4 minutes. The job number 400, for example,was executed in 50 minutes. Do you know if Talend has some cache to clean or other parameter that helps to maintain the initial performance?

Thanks a lot

Re: Performance Issue with parallel jobs

tnewbie — Thu, 05 Mar 2020 02:53:53 GMT

I guess the issue is not with file processing but with data getting committed to the target database and probably your jobs are waiting for the table to be available for committing the data before moving over to the next job/thread. I strongly suspect that your partitions are causing a kind of mutating locks on the table where each of them is waiting for the other to finish. I would recommend to focus on these aspects. On the other hand, in my view parallelism should confine to the number of cores available on the CPU if you increase your parallel threads more than what your server can handle then it can affect your perform adversely. Lastly check for the memory and CPU utilization when these jobs are running, that might give you the exact problem as to what is slowing down.

Re: Performance Issue with parallel jobs

manodwhb — Thu, 05 Mar 2020 03:26:41 GMT

@rrigoni ,when your jobs are running any other jobs are running?

Re: Performance Issue with parallel jobs

rrigoni — Fri, 06 Mar 2020 09:45:06 GMT

No, only my jobs

Re: Performance Issue with parallel jobs

rrigoni — Fri, 06 Mar 2020 09:46:05 GMT

Hi, the target is a file. We are processing all these files and when all files finish, we're appending in only one file. We don't write at tables.
With resources that i have, i have monitored CPU and memory and always has free resources. But i will ask help to someone with more privileges than me to monitoring machine.

Thanks for help!!

Re: Performance Issue with parallel jobs

manodwhb — Fri, 06 Mar 2020 09:48:04 GMT

Ok.