Performance Issue with parallel jobs

[rrigoni]

Hi all,

I need to process a big file. The process is complex. I decided to split this file in several files and created a several instances of this job. This way this instances running parallel. It works. But I realize that performance is decreased over time. The first job was executed in 4 minutes. The job number 400, for example,was executed in 50 minutes. Do you know if Talend has some cache to clean or other parameter that helps to maintain the initial performance?

Thanks a lot

[tnewbie]

I guess the issue is not with file processing but with data getting committed to the target database and probably your jobs are waiting for the table to be available for committing the data before moving over to the next job/thread. I strongly suspect that  your partitions are causing a kind of mutating locks on the table where each of them is waiting for the other to finish. I would recommend to focus on these aspects. On the other hand, in my view parallelism should confine to the number of cores available on the CPU if you increase your parallel threads more than what your server can handle then it can affect your perform adversely. Lastly check for the memory and CPU utilization when these jobs are running, that might give you the exact problem as to what is slowing down.

[manodwhb]

@rrigoni ,when your jobs are running any other jobs are running?

[rrigoni]

No, only my jobs

[rrigoni]

Hi, the target is a file. We are processing all these files and when all files finish, we're appending in only one file. We don't write at tables.
With resources that i have, i have monitored CPU and memory and always has free resources. But i will ask help to someone with more privileges than me to monitoring machine.

Thanks for help!!

[manodwhb]

Ok.