I have developed DB migration jobs using Talend Studio.
The jobs are working OK, but I need to reduce their execution time.
I have job “A” which takes 50 mins when it’s executed alone, and this job takes a day when it’s run with 9 other jobs (These 10 jobs are for the same schema)
One of the jobs uses dynamic schema to handle most of the tables, and rest of the jobs are to run heavy tables individually.
I have enabled parallel execution, and job profile setting is “-Xmx10240m”.
The server Remote Engine runs on has 96GB RAM.
I have tried changing setting on Remote Engine, set Xms to 1024MB and set Xmx to 10240MB and 32,768MB, but it did not make the execution any faster.
Could anyone suggest what I can do for faster execution?
In addition to increasing the JVM parameters value, have a try to use tParallelize to control the execution order of jobs, let some jobs are executed first, and some jobs are executed in parallel to avoid too much concurrent access to the same database/table.
What all these jobs do? Do they writing in the same database or even table? In such cases I suggest you check in your database for blockades caused by SQL statements.
It should also not a surprise if you cause heavy load on the database the hole system is getting slower.
You could also check if the database provides enough connections available to run in parallel. Often in the default settings the number of parallel connections is very limited.
