topic Re: Studio Performance in Talend Studio

Studio Performance

Anonymous — Sat, 16 Nov 2024 11:21:17 GMT

I am using 5.3.1 Studio/TAC on a RedHat install.   32 CPU cores, 256gb ram.
Our team is running into considerable performance concerns with our configuration. Everything is installed on a single server(described above), TAC, CMDLINE, SVN, JobServer. The server is more than capable of handling the traffic and typically never exceeds 50% cpu load or RAM usage.
Performance of the jobs is not the concern. It is the check in, check out, save and basic operation of the studio while plugged into the SVN(mod_dav_svn). I have been in touch with Talend support about this problem repeatedly as well as read and read and read some more about possible problems/solutions. None of the recommended configurations below produced any valuable performance gains. Increasing memory settings for TAC, CMDLINE, SVN, JobServer also yielded no noticeable improvement. SymLinks, SVNPathAuthz Off, SVNINMemoryCacheSize, SVNCompressionLevel also all yeilded no noteable improvement.
The time's recorded in the attached chart were produced from a test repository and job set, and are showing better than average performance time. If our development team only ever had to open one or two jobs during the course of a days work, this would not be a problem. However, when these performance times are added across 10-15 jobs needing small modification the performance is a very big problem. If it is the consensus of this group that these recorded times are "as good as it gets" then our solution will be to push all development to a sandbox environment, uploading completed work into a SVN. Essentially rendering the Enterprise portion of the tool useless.
I am currently in the process of installing more tracking tools to understand exactly where the performance is bad. Any recommendations on how to further troubleshoot and understand this problem are greatly appreciated.

Problem :
1.)    Opening a job from a connected repository takes a considerable amount of time.
2.)    Closing/Saving a job from a connected repository takes a considerable amount of time.
3.)    Creating a job from a connected repository takes a considerable amount of time.
4.)    Deleting a job from a connected repository takes a considerable amount of time.

Solutions :
                According to my reading/research the following remedy exists for the above problems.
1.)    Co-located SVN/TAC can be problematic, separating the repository and web server increases performance
2.)    Number of Jobs in a Repository – Reduce the number of jobs in a repository to gain performance.
3.)    Favor separate SVN repository over several projects in a single repository.

All testing was done with identical set of jobs imported into different repository configurations. Approximately 65 jobs total, with 2 context objects, and 3 metadata connection items. The run times shown below are recorded in seconds.
                Three jobs were opened. The time it took to edit was recorded and graphed in the chart below

OPEN_TEMP_CARD_PORT

OPEN_TEMP_CARD

OPEN_TEMP_CIRC_CIRC

Three different jobs were closed/saved. Time it took to edit was recorded and graphed in the chart below

CLOSE_SAVE_TEMP_CARD_PORT

CLOSE_SAVE_TEMP_CARD

CLOSE_SAVE_TEMP_CIRC_CIRC

                OPEN ALL … represents the time it takes to display 100% of all the jobs in a repository from a trunjob object
                CREATE
                DELETE
                EMPTY RECYCLE BIN

Sandbox - This is the control test for the environment. This is a disconnected project, IE: NO SVN. This represent the fastest possible execution time of each task.
191 -- 68u - TAC running on our 191 machine, and connecting to SVN on 68u, No repository separation
191 – 68u -REPTEST – TAC running on 191, SVN on 68u, Separate Repository
191 – 191 -- TAC running 191, SVN running 191, Separate repository
68u-68u-65 jobs – TAC running 68, SVN running 68, No repository separation
68u – 68u REPTEST – TAC running 68, SVN running 68, Separate Repository
68u—68u –Add 100jobs – TAC running 68, SVN running 68, Separate repository
68u—68u ---REPTEST(30 Jobs)— TAC running 68, SVN running 68, Separate repository

Re: Studio Performance

Anonymous — Tue, 13 Jan 2015 21:01:26 GMT

Thanks for the detailed post. I am experiencing a similar issue and plan to open a support ticket.
Did you see this knowledge base article? Did not apply to me.
https://help.talend.com/search/all?query=Performance+issues+during+Tomcat+startup+and+SVN+checkout&content-lang=en
I also notice that talend is leaving behind many files in /tmp directory like:
/tmp/talend936181116480188931TemporaryTalendSvnkitConfigDir
I suspect this is a related problem.
I can check out a whole project with svn client in less than a minute, but it take 1 - 3 minutes to open a job.
I have less than 1ms latency between the client and server - This is something you could check on. network latency can be a big problem.

Re: Studio Performance

Anonymous — Tue, 13 Jan 2015 21:31:59 GMT

We have disabled the Symlinks in a setenv.sh file called on startup.-->
JAVA_OPTS="-Dsvnkit.symlinks=false"
Offsite client -60-80ms
Onsite Client - 17-20ms
Jobserver - 0.3 - 0.3ms
TACServer - 0.3 - 0.3ms
Also:
Update the "catalina.sh" file with (TAC) :
JAVA_OPTS = "$JAVA_OPTS -Xms4096m -Xmx16384m -XX:MaxPermSize=2048m" --Considerable TAC(web page performance gain)
Update "start_cmdline.sh" (CMDLINE)with:
java -Xms5120m -Xmx8192m -XX:MaxPermSize=2048m --Compile/Generate time improved. Jobs no longer seem to get "stuck" in deploying status
Update "start_rs.sh" (JOBSERVER) with:
# set the JVM arguments here
MY_JMV_ARGS="-Xms8192m -Xmx16384m" -- Fewer JVM related errors. Does not seem to change performance characteristics.
update http.conf (SVN -mod_dav_svn) with :
KeepAlive On
MaxKeepAliveRequests 1000
SVNPathAuthz off
SVNInMemoryCacheSize 4194304
SVNCacheTextDeltas On
SVNCacheFullTexts On
SVNCompressionLevel 9

Re: Studio Performance

Anonymous — Wed, 14 Jan 2015 06:13:56 GMT

Hi,

Would you mind opening a ticket on our Talend Support Portal so that the Support team can feed your issue with additional information coming from your case?
Many thanks.

Best regards
Sabrina

Re: Studio Performance

Anonymous — Thu, 15 Jan 2015 18:46:46 GMT

Resolution was either upgrade to 5.3.2 ++, Or apply a patch.
I applied the patch and ran new benchmarks.
Solution tested: Patch--
bugs_integrate_patch_v5.3.1.r104014_20140304
****************2014-03-04*******************
TPS-599 : Problems of SVN performances using the studio of 5.3.1 (TUP-974,TDI-27085) (zwzhao)

Re: Studio Performance

Anonymous — Thu, 15 Jan 2015 19:00:59 GMT

I am using Talend 5.4.1 and experiencing Studio slowness issues. Not sure if you and I are having the same issue, but seems similar. Every piece of infrastructure is fast but Talend Studio is slow.

Re: Studio Performance

Anonymous — Tue, 14 Apr 2015 23:46:51 GMT

Update:
The patch had a significant improvement on the smaller test repository used to generate these results. No noticeable improvement in the larger production environment. (about 200 jobs and 15 development staff. )
The performance is pretty terrible. We have no choice but to shed all the old versions and revisions of our jobs and create a new repository.
In the long term:
We are exploring other options to circumvent the Talend SVN implementation using sandbox mode. The entire SVN/Talend experience has just been too unreliable and problematic for an enterprise implementation.

Re: Studio Performance

Anonymous — Tue, 14 Apr 2015 23:56:05 GMT

We are upgrading to 5.6.1 and are not experiencing this slowness. The only unpredictable part is log on which takes 1 - 10 minutes. Opening jobs is a few seconds now and other parts of the start up process are faster as well.

Re: Studio Performance

Anonymous — Tue, 28 Jul 2015 14:21:01 GMT

Well, the code repository management is really poor in Talend, on top of this mixed with internal versioning system built in studio.
There is missing enterprise grade guide on this.
We are also experiencing issues with 5.4.1 EE in Studio while opening the job, I am already suspicious about the SVN implementation as soon as sometimes the commit reports error, but after restart everything is committed.... 😞

Re: Studio Performance

Anonymous — Mon, 12 Sep 2016 10:40:29 GMT

Is there a solution to this issue. We're using 5.4.1 and are facing the same issue.