topic Re: Slow Job-Generation in Talend Studio

Slow Job-Generation

DaKaLKa — Wed, 09 Dec 2015 09:39:11 GMT

Hallo,
I have migrated some project from Talend Open Studio 5.6.1 to 6.1.0 - mostly with no problems. But the job generation seems to be very slow.
So I created some very simple jobs, a MainJob and 10 empty subjobs. The MainJob contains 10 tRunJob-Components that call the empty subjobs. Here a comparison of the job-generation:
Talend 5.6.1: < 1s
Talend 6.0.1: ~ 2.5s
Talend 6.1.0: ~ 5s
Then I migrated some "production" jobs to version 6.1.0. This are mostly normal etl-processes to build up dimension-tables for a DWH-database.
Job-Generation:
Talend 5.6.1: ~10s
Talend 6.1.0: ~51s
And this were quite "small" jobs. I have a bigger Job that needs about 4-5 Minutes to export on 5.6.1. After 15 Minutes I killed the export-process on 6.1.0.
Is there anything setting to speedup the job-generation? Any maven settings? Any parallelization for job generation?

Best regards,
Paul Mandalka

Re: Slow Job-Generation

Anonymous — Mon, 14 Dec 2015 02:32:32 GMT

Hi,
Is there any error message in log which is located in <Talend Studio installation path>/workspace/.metadata/.log when you kill your processing?
Does this issue only repro studio 6.1.0?
Have you tried to generate one job from one studio to another to see if the speed is normal?

Best regards
Sabrina

Re: Slow Job-Generation

DaKaLKa — Tue, 15 Dec 2015 09:22:45 GMT

Hi,
no there is no error message. I think, the process would be generated but it takes very long.
If I run one simple job from Talend, there is almost not difference:
5.6.2: < 0.5s
6.0.1: ~1.2s
6.1.0: ~1.3s
This where very simple Jobs in a very small demo project.
My normal project to build up a DWH has about 700 Jobs.
A comparison in Job-Generation if I run the job in studio:
5.6.1: < 0.5s (second run)
6.1.0: ~ 5.5s (second run)
Building the Job as Standalone Job
5.6.1: ~ 6s
6.1.0: ~ 25s
This is for a "simple" job with a MSSQL-Server as source and 5 Lookup tables, two tMaps and a MSSQL-Output.
For me it seems, that there is a high overhead for every job and at the end of the job generation in the new version.
There are no error messages and no missing Jar-Files.

Re: Slow Job-Generation

Anonymous — Mon, 29 Feb 2016 19:26:21 GMT

Hi Sabrina,
I'm having the same issue. We have a big job that calls almost 30 Subjobs. It takes me about 30 mins to build that big job. However, it was only about 5 mins when I was using Talend v5.5.

Re: Slow Job-Generation

DaKaLKa — Tue, 08 Mar 2016 10:48:30 GMT

Hi,
are there any possibilities to gain more performance from the Job build?
I noticed that all job are generated over and over again e.g. I have a tRunJob with a dynamic job and I have selected about 200 Job als possible Jobs. It I generate the start-Job, all 200 Jobs are generated every time, even if nothing has changed in the jobs.
This is not very efficient. Are there any hidden settings for that?
Kind regards,
Paul Mandalka

Re: Slow Job-Generation

Anonymous — Tue, 08 Mar 2016 11:22:56 GMT

I am having the exact same issue. This needs resolving. I have raised a support call.
Talend Studio should be able to identify which jobs have been modified and ONLY compile those (obviously jobs containing jobs which have been modified will need to be recompiled as well). We have a situation where a job that took 5 to 10 mins to compile in 5.6.1 takes almost an hour in 6.1.1.

Re: Slow Job-Generation

DaKaLKa — Tue, 15 Mar 2016 18:48:39 GMT

Hi,
any news on that?

Re: Slow Job-Generation

Anonymous — Fri, 15 Apr 2016 16:52:29 GMT

We had this same issue, which was resolved in the most recent patch Patch_20160316_TPS-1251_v1

Re: Slow Job-Generation

ananyasingh — Fri, 29 Apr 2016 13:11:52 GMT

Hi,
We have recently migrated to Talend 6.1.1(Enterprise version) and we are facing the same issue.
Please let me know if there is any solution for this.
Thanks.

Re: Slow Job-Generation

Anonymous — Fri, 29 Apr 2016 14:38:44 GMT

Hi,
We are using Talend 6.1.1 and we have the same issue.

Re: Slow Job-Generation

Anonymous — Wed, 04 May 2016 15:05:13 GMT

Hi,
am getting same issue jobs are taking too much time to generate and run.

can anybody has solution.
Talend 6.1
Coludera 5.5

Re: Slow Job-Generation

Anonymous — Sat, 14 May 2016 08:14:07 GMT

Hi,
We too are facing same issue with jobs taking too much time to generate & run the jobs.
Does any one have got solution for this issue.
Talend 6.1.1 (Enterprise Edition)
Thanks & Regards
Ashok Karnam

Re: Slow Job-Generation

Anonymous — Sat, 14 May 2016 15:28:55 GMT

We had the same issue when we upgraded from Talend 5.6 to 6.1. The generate/build step started taking double the amount of 5.6.
Here are my observations and steps followed to bring down to normal.
1. Talend 5.6 requires Java 7 jdk 1.7 (Must)
2. Though Talend 6.1 still support jdk 1.7 but it is highly recommended to have Java 8 jdk 1.8. This is big! give this a try.
3. Talend 6.1 generate/build process includes maven, this is another reason the build process takes more time. This was optional in 5.6 but it is part of build in 6.1.
4. Talend 6.1 requies patch 1237 to address slowness in setComponentPath (a step in generate). If you are generating via TAC, goto commandline section and notice the time it took for initRemote, logonProject, setComponentPath and buildJob.
4.1 If you see delay in initRemote - this is probably connection to TAC is very slow
4.2 if it's logonProject - you may require svn cleanup and delete unwanted jobs in repository
4.3 if it's setComponentPath - the patch 1237 will fix it.
4.4 If it's buildJob step - java upgrade will hopefully fix it.
5. It is always advised to run svn cleanup and also delete unwanted jobs from repository.
Hope it helps
Thanks
Rohan Suresh

Re: Slow Job-Generation

Anonymous — Tue, 17 May 2016 10:20:43 GMT

Hi ashok.karnam,
Does this issue repro on all jobs on v 6.1.1 or just a specific job? Are you using JDK 1.8?
Best regards
Sabrina

Re: Slow Job-Generation

Anonymous — Tue, 17 May 2016 10:34:28 GMT

Sabrina,
Generation time taken is high for all the jobs & not specific to any particular job.
Even for the jobs with minimal tComponents (~2-3) it takes longer time say around 3-5 mins.
Java version - java version "1.8.0_92"
Can you advice on, what could be the typical time taken for Generating & Deploying a job having minimal tComponents with no complexity/transformations.
Thanks & Regards
Ashok Karnam

Re: Slow Job-Generation

Anonymous — Wed, 18 May 2016 10:11:28 GMT

Hi Ashok Karnam,
The job-generation consuming time is determined by many elements, job design, data volumn, your prod environment, the component you are using and so on.
It's a little hard for us to repro your slow job-generation issue on our side.
Would you mind report a ticket for your issue on Talend Support Portal? In this way, our collegues from support team can give you a remote assistance on this performance issue.
Best regards
Sabrina

Re: Slow Job-Generation

Anonymous — Wed, 18 May 2016 10:57:44 GMT

Thanks Sabrina,
Job design is very simple job, populating data from source table to target table without any transformations & having bare minimal components - tMySQLInput, tMap, tMySQLOutput with a volume of 10,000 rows. Input & Output databases are within LAN.
As suggested, I will create a ticket on Talend support portal for the said issue.
Thanks & Regards
Ashok Karnam

Re: Slow Job-Generation

Anonymous — Thu, 19 May 2016 17:58:09 GMT

My experiences is, the speed of the SVN has the most influence. If you have less network latency you can increase the build speed significant. We also see under Windows it runs much slower than under Linux.

Re: Slow Job-Generation

Anonymous — Mon, 23 May 2016 11:57:38 GMT

Even we face the same problem while migrating from talend 5.6.1 to 6.1.1.
6.1.1 is mush slower compared to 5.6.1 ?
kindly recommend some possible solution for this problem.
Thanks and regards,
Amritha

Re: Slow Job-Generation

Anonymous — Fri, 27 May 2016 03:59:23 GMT

We are using Talend 6.1.1 and we have the same issue.