topic Re: TIS + SVN slowly ... in Talend Studio

TIS + SVN slowly ...

Anonymous — Sat, 16 Nov 2024 14:04:24 GMT

Hello !
We have installed TIS 3.0.2 r20205 with SVN (version 1.5.4). Since mid-january, we are 7 developpers (only 3 before), and time to open project and to save jobs take more time...
Before, to open a project, we put 1 minute, now 2 minutes...
I see that current version available for SVN is 1.5.8, we have 1.5.4, do we have to upgrade SVN ? If yes, is there any risks ?
What can we do / test to wait less ? Is there a configuration better than an other ?
Could you help us ?
Thanks.
Anaïs

Re: TIS + SVN slowly ...

Anonymous — Wed, 18 Feb 2009 16:14:53 GMT

I've received your repository (private channel).
First I would like to give an advice related to backup: it's a good thing (even mandatory) to backup your SVN repository, but it's not done the right way. You should not archive the repository directory itself. You have to dump it with the command "svnadmin dump". In a second step, I advice you to create incremental backup instead of full backup (I may explain this procedure in another topic).
You have 18k revisions. Here is the dispatch between months:

$ egrep '^r+ \|' repo.svnlog | awk '{print $5}' | awk -F"-" '{print $1$2}' | uniq -c
   8655 200902
   5291 200901
   3722 200812
    588 200811

So obviously you have much more revisions recently: at the end of february, you should have 15k only for february.
I've installed the repository on our test machine (Core2Quad 2.4GHz, memory 4GB, Debian Linux, SVN 1.4.2) and I must admit that it works nice. I mean that it takes 30s to open your biggest project (D_EDI), including 18s for template initialization. Opening a job takes less than 5 seconds (5 seconds when it contains tRunJobs).
I don't think that you open your projects very often, do you? So what's more interesting it the time to wait when opening a job. How long does it take?
I have a doubt that updating SVN would have any effect.

Re: TIS + SVN slowly ...

Anonymous — Wed, 18 Feb 2009 16:20:15 GMT

If you have to wait for initializing templates, you should increase the used memory, I suggest using 256m as perm size

Re: TIS + SVN slowly ...

Anonymous — Wed, 25 Feb 2009 16:09:26 GMT

I've made new test, with the same repository but with a Windows server as Subversion repository. The machine has very low performances compared to our 4 cores 2.4GHz 4GB memory.
The performances are the same as on the Linux server. The first job needs 10 seconds to open and next jobs open in less than 3 seconds.

Re: TIS + SVN slowly ...

Anonymous — Wed, 25 Feb 2009 16:12:32 GMT

What may also slow the process: a poor connection to the server (what is the result of a ping) or an antivirus running.

Re: TIS + SVN slowly ...

Anonymous — Thu, 26 Feb 2009 13:14:03 GMT

Hello,
Sorry, I didn't see before your answers...
For the SVN backup, we use command "svnadmin hotcopy". It's not right ?
In fact, at the beginning, we were only 3 developpers... and now we are 7 !
When we were 3, there was no problem ! But since we are 7 ...
I don't think problem come from a poor connection to the server (because server is just near us)...
For antivirus, I will go to see our administrator to desactive it... but problem seems to appear since more developpers use the repository.
Anaïs

Re: TIS + SVN slowly ...

Anonymous — Thu, 26 Feb 2009 14:53:40 GMT

So, we desactived antivirus on the server and on our machine. And it's the same.
To open project D_EDI, It takes 3 minutes 15 (and not 30 seconds...).
To open project D_DWH_EvtProduct 2 minutes. Next, to open job, the first : 12 seconds, the second 5 seconds.
We open projects very often, because we don't work all the day on the same project, so we lose many time !
I also modify initializing templates (used memory to 256m as perm size)... but no result...
Any other suggestions ?
Thanks !
Anaïs

Re: TIS + SVN slowly ...

Anonymous — Thu, 26 Feb 2009 15:54:49 GMT

Sorry, I didn't see before your answers...

Did you subscribe to the topic? If yes, didn't you receive the emails?

For the SVN backup, we use command "svnadmin hotcopy". It's not right ?

It's perfectly right. I never use this method so I thought you had made a simple copy of the repository directory.

In fact, at the beginning, we were only 3 developpers... and now we are 7 !
When we were 3, there was no problem ! But since we are 7 ...

Theoretically the number of developers should not have an impact on the repository efficiency! On Talend source code the repository is used by much more than 7 developers and it runs very well.

We open projects very often, because we don't work all the day on the same project, so we lose many time !

I saw you had many project, and I imagine that you "switch project" very often! If it takes 3 minutes to switch, it's really a pain I admit.

Any other suggestions ?

When the project is launching, there are several steps:
* Initialize Project Connection
* Logon In Project
* Synchronize Libraries
* Loading Workbench
* Generation Engine Initialization in Progress
Can you tell us which steps take time (and how much?)
During project launch, how is the CPU usage?

Re: TIS + SVN slowly ...

Anonymous — Thu, 26 Feb 2009 16:30:28 GMT

Sorry, I didn't see before your answers...

Did you subscribe to the topic? If yes, didn't you receive the emails?
In fact, I didn't subscribe to the topic, so I didn't receive emails... I subscribed now...

When the project is launching, there are several steps:
* Initialize Project Connection
* Logon In Project
* Synchronize Libraries
* Loading Workbench
* Generation Engine Initialization in Progress
Can you tell us which steps take time (and how much?)
During project launch, how is the CPU usage?

In fact, the last step takes lot of time. 2 minutes 30 to open the project, 2 minutes 15 for the last step...
So, for the 4 other, it takes only 15 seconds.
Concerning CPU usage, here is a screenshot.
Thanks.
Anaïs

Re: TIS + SVN slowly ...

Anonymous — Thu, 26 Feb 2009 17:09:41 GMT

This is a very interesting information you're giving :-)
There is a bug in 3.0.2 concerning cache management on generation engine templates. Can you please:
1. close Talend Integration Suite studio
2. create an empty file workspace/.JETEmitters/FirstCompilationMarker (with no filename extension)
3. relaunch the studio, it should last as long as usual
4. close the studio
5. relaunch the studio, it should be much faster
Does it behave as expected?

Re: TIS + SVN slowly ...

Anonymous — Fri, 27 Feb 2009 07:53:31 GMT

It works !
Open project takes now less than 1 minute ! That's good.
One more question : is there somewhere a list of bugs known that we can see ? Because, for this bug, if we were informed before, we didn't lose time ! In fact, I think that we have to be informed of all bugs resolved for our version ... No ?
Thanks.
Regards.
Anaïs

Re: TIS + SVN slowly ...

Anonymous — Fri, 27 Feb 2009 09:17:35 GMT

It works !
Open project takes now less than 1 minute ! That's good.

Nice to read this!

One more question : is there somewhere a list of bugs known that we can see ? Because, for this bug, if we were informed before, we didn't lose time !

Yes, you can read the the Bugtracker Changelog Page on Talend Open Studio project (this is a TOS bug, not a TIS bug).
Unfortunately, this bug has been solved but not declared in the bugtracker, so you wouldn't have been able to see it. We have just added it :6466

Re: TIS + SVN slowly ...

Anonymous — Tue, 03 Mar 2009 16:18:38 GMT

Hello,

Yes, you can read the the Bugtracker Changelog Page on Talend Open Studio project (this is a TOS bug, not a TIS bug).

I didn't know this page and it is very usefull ! However, I can't change project with my login. When I connect, project is TIS Studio, and if I try to change to 'Talend open Studio' for example, I'm automatically redirected to project TIS Studio...
Is there a problem with my account ?
Could you help me ?
Thanks !
Anaïs