I am using 5.3.1 Studio/TAC on a RedHat install. 32 CPU cores, 256gb ram. Our team is running into considerable performance concerns with our configuration. Everything is installed on a single server(described above), TAC, CMDLINE, SVN, JobServer. The server is more than capable of handling the traffic and typically never exceeds 50% cpu load or RAM usage. Performance of the jobs is not the concern. It is the check in, check out, save and basic operation of the studio while plugged into the SVN(mod_dav_svn). I have been in touch with Talend support about this problem repeatedly as well as read and read and read some more about possible problems/solutions. None of the recommended configurations below produced any valuable performance gains. Increasing memory settings for TAC, CMDLINE, SVN, JobServer also yielded no noticeable improvement. SymLinks, SVNPathAuthz Off, SVNINMemoryCacheSize, SVNCompressionLevel also all yeilded no noteable improvement. The time's recorded in the attached chart were produced from a test repository and job set, and are showing better than average performance time. If our development team only ever had to open one or two jobs during the course of a days work, this would not be a problem. However, when these performance times are added across 10-15 jobs needing small modification the performance is a very big problem. If it is the consensus of this group that these recorded times are "as good as it gets" then our solution will be to push all development to a sandbox environment, uploading completed work into a SVN. Essentially rendering the Enterprise portion of the tool useless. I am currently in the process of installing more tracking tools to understand exactly where the performance is bad. Any recommendations on how to further troubleshoot and understand this problem are greatly appreciated.
Problem: 1.) Opening a job from a connected repository takes a considerable amount of time. 2.) Closing/Saving a job from a connected repository takes a considerable amount of time. 3.) Creating a job from a connected repository takes a considerable amount of time. 4.) Deleting a job from a connected repository takes a considerable amount of time.
Solutions: According to my reading/research the following remedy exists for the above problems. 1.) Co-located SVN/TAC can be problematic, separating the repository and web server increases performance 2.) Number of Jobs in a Repository – Reduce the number of jobs in a repository to gain performance. 3.) Favor separate SVN repository over several projects in a single repository.
All testing was done with identical set of jobs imported into different repository configurations. Approximately 65 jobs total, with 2 context objects, and 3 metadata connection items. The run times shown below are recorded in seconds. Three jobs were opened. The time it took to edit was recorded and graphed in the chart below
Three different jobs were closed/saved. Time it took to edit was recorded and graphed in the chart below
OPEN ALL … represents the time it takes to display 100% of all the jobs in a repository from a trunjob object CREATE DELETE EMPTY RECYCLE BIN
Sandbox - This is the control test for the environment. This is a disconnected project, IE: NO SVN. This represent the fastest possible execution time of each task. 191 -- 68u - TAC running on our 191 machine, and connecting to SVN on 68u, No repository separation 191 – 68u-REPTEST – TAC running on 191, SVN on 68u, Separate Repository 191 – 191 -- TAC running 191, SVN running 191, Separate repository 68u-68u-65 jobs – TAC running 68, SVN running 68, No repository separation 68u – 68u REPTEST – TAC running 68, SVN running 68, Separate Repository 68u—68u –Add 100jobs – TAC running 68, SVN running 68, Separate repository 68u—68u ---REPTEST(30 Jobs)—TAC running 68, SVN running 68, Separate repository
Thanks for the detailed post. I am experiencing a similar issue and plan to open a support ticket. Did you see this knowledge base article? Did not apply to me. https://help.talend.com/search/all?query=Performance+issues+during+Tomcat+startup+and+SVN+checkout&c... I also notice that talend is leaving behind many files in /tmp directory like: /tmp/talend936181116480188931TemporaryTalendSvnkitConfigDir I suspect this is a related problem. I can check out a whole project with svn client in less than a minute, but it take 1 - 3 minutes to open a job. I have less than 1ms latency between the client and server - This is something you could check on. network latency can be a big problem.
We have disabled the Symlinks in a setenv.sh file called on startup.--> JAVA_OPTS="-Dsvnkit.symlinks=false" Offsite client -60-80ms Onsite Client - 17-20ms Jobserver - 0.3 - 0.3ms TACServer - 0.3 - 0.3ms Also: Update the "catalina.sh" file with (TAC) : JAVA_OPTS = "$JAVA_OPTS -Xms4096m -Xmx16384m -XX:MaxPermSize=2048m" --Considerable TAC(web page performance gain) Update "start_cmdline.sh" (CMDLINE)with: java -Xms5120m -Xmx8192m -XX:MaxPermSize=2048m --Compile/Generate time improved. Jobs no longer seem to get "stuck" in deploying status Update "start_rs.sh" (JOBSERVER) with: # set the JVM arguments here MY_JMV_ARGS="-Xms8192m -Xmx16384m" -- Fewer JVM related errors. Does not seem to change performance characteristics. update http.conf (SVN -mod_dav_svn) with : KeepAlive On MaxKeepAliveRequests 1000 SVNPathAuthz off SVNInMemoryCacheSize 4194304 SVNCacheTextDeltas On SVNCacheFullTexts On SVNCompressionLevel 9
Resolution was either upgrade to 5.3.2 ++, Or apply a patch. I applied the patch and ran new benchmarks. Solution tested: Patch-- bugs_integrate_patch_v5.3.1.r104014_20140304 ****************2014-03-04******************* TPS-599 : Problems of SVN performances using the studio of 5.3.1 (TUP-974,TDI-27085) (zwzhao)
Update: The patch had a significant improvement on the smaller test repository used to generate these results. No noticeable improvement in the larger production environment. (about 200 jobs and 15 development staff. ) The performance is pretty terrible. We have no choice but to shed all the old versions and revisions of our jobs and create a new repository. In the long term: We are exploring other options to circumvent the Talend SVN implementation using sandbox mode. The entire SVN/Talend experience has just been too unreliable and problematic for an enterprise implementation.
We are upgrading to 5.6.1 and are not experiencing this slowness. The only unpredictable part is log on which takes 1 - 10 minutes. Opening jobs is a few seconds now and other parts of the start up process are faster as well.
Well, the code repository management is really poor in Talend, on top of this mixed with internal versioning system built in studio. There is missing enterprise grade guide on this. We are also experiencing issues with 5.4.1 EE in Studio while opening the job, I am already suspicious about the SVN implementation as soon as sometimes the commit reports error, but after restart everything is committed.... :-(