Performance problem

[Anonymous]

Hello,
I made talend job which calls other exported talend jobs with their sh files. For this purpose I use tSystem. So far so good. My problem is when I put Oracle connection (tOracleConnection) in the exported talend jobs the performance goes down very very badly.
In my main Talend job I calls 10 other exported talend jobs by sh files. If I don 't have tOracleConnection in the exported jobs everything is fine - I have immediately execution of all 10 jobs. If I put tOracleConnection then the processing time for every next called jobs is increased progressive - 1th job for 2 sec, 2th job for 5sec, 3th job for 50sec...10th job for 5-10 min. Of course I put tOracleClose in every jobs and I release the connection. I cannot use tRunJob because all jobs will live as standalone applications and they will be exported with sh files. Maybe there is Talend pool of DB connections which size must be increased ?!
Any idea how I can sove this performance problem?
Regards

[Anonymous]

Sorry, all jobs contain different tOracleConnection and tOracleClose. When I start the main job, it calls 10 others (just tOracleConnection and tOracleClose, no other components) and they are executed fine but for 5 min. archenroot I would like to say thank you again for your ideas. I'm continuing with the investigation how this problem can be resolved.

[Anonymous]

Dont thank me, the issue still exists...hehe...
You schould know how the 10 subprocesses are executed. If you do not use any log file, just edit the .sh files of the jobs and put to the end of statement something like this:

>>/home/user/talend/job1.log 2>&1

This will create log file with both error and standard messages and you will see how the processes behave.

[Anonymous]

Hi Roman,
maybe it is unrelated - maybe not. I've discussed with an Oracle DBA about another topic - and one of result was, that connecting to Oracle DB itself is time expensive. It simply takes time to connect. So if you have no shared connection or pool, connecting itself may be delaying a lot. (even it doesn't explain increasing response time in your jobs). I even used subjobs working with the same connection, was ok. Just I didn't need many parallel subjobs so far Just tring to point out an idea where the delay could originate, still I have no solution for you.
Carpe diem
Gabriel

[Anonymous]

Hi
Currently, tDBConnection doesn't support JDBC pool.
'Shared DB Connection Name' is a workaround, but you said "all jobs contain different tOracleConnection and tOracleClose".
I have only opinion at TOS side, but i'm not sure. Can you separate your ten jobs into multi thread in your main job by Checking 'Multi thread excution'?
Or these jobs must be excuted in logic order?
Best regards!
Pedro

[Anonymous]

Hi Pedro,
Thank you for your answer.
Unfortunately all jobs must follow logic order and multi threads are not possible.
Regards
Roman

[Anonymous]

Try to run them one by one.

[Anonymous]

I think that I found the problem, unfortunately I cannot be 100% sure but:
I start same job - main and subjobs in Windows 32bit JDK and on Linux 64bit JDK. The result is : perfect performance on 32bit JDK and very very bad on 64bit JDK (the usual environment for my jobs). So my conclusion is that all performance problem is coming from the 64bit JDK. I followed many documentations and I discovered that on 64bit Java it's possible big big delays because platform and garbage collectors reasons. I will try to do some tuning of my 64bit JDK, but I'm sure 80% that this is the main reason for my problems with Talend performance. Did Anybody meet 64bit JDK problems with Talend jobs ?
Regards
Roman

[Anonymous]

I didn't face that issue and run jobs on RHEL64bit with JDK 1.6 update 2x 64bit. Did you google for this issue. Send me the jdbc drivers directly from your Talend installation I could try to simulate parallel connection in simple java project in eclipse or any other ide.
Ladislav