Slow Job-Generation

One Star

Slow Job-Generation

Hallo,
I have migrated some project from Talend Open Studio 5.6.1 to 6.1.0 - mostly with no problems. But the job generation seems to be very slow.
So I created some very simple jobs, a MainJob and 10 empty subjobs. The MainJob contains 10 tRunJob-Components that call the empty subjobs. Here a comparison of the job-generation:
Talend 5.6.1: < 1s
Talend 6.0.1: ~ 2.5s
Talend 6.1.0: ~ 5s
Then I migrated some "production" jobs to version 6.1.0. This are mostly normal etl-processes to build up dimension-tables for a DWH-database.
Job-Generation:
Talend 5.6.1: ~10s
Talend 6.1.0: ~51s
And this were quite "small" jobs. I have a bigger Job that needs about 4-5 Minutes to export on 5.6.1. After 15 Minutes I killed the export-process on 6.1.0.
Is there anything setting to speedup the job-generation? Any maven settings? Any parallelization for job generation?

Best regards,
Paul Mandalka
Moderator

Re: Slow Job-Generation

Hi,
Is there any error message in log which is located in <Talend Studio installation path>/workspace/.metadata/.log when you kill your processing?
Does this issue only repro studio 6.1.0?
Have you tried to generate one job from one studio to another to see if the speed is normal?


Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Slow Job-Generation

Hi,
no there is no error message. I think, the process would be generated but it takes very long.
If I run one simple job from Talend, there is almost not difference:
5.6.2: < 0.5s
6.0.1: ~1.2s
6.1.0: ~1.3s
This where very simple Jobs in a very small demo project.
My normal project to build up a DWH has about 700 Jobs.
A comparison in Job-Generation if I run the job in studio:
5.6.1: < 0.5s (second run)
6.1.0: ~ 5.5s (second run)
Building the Job as Standalone Job
5.6.1: ~ 6s
6.1.0: ~ 25s
This is for a "simple" job with a MSSQL-Server as source and 5 Lookup tables, two tMaps and a MSSQL-Output.
For me it seems, that there is a high overhead for every job and at the end of the job generation in the new version.
There are no error messages and no missing Jar-Files.
One Star

Re: Slow Job-Generation

Hi Sabrina,
I'm having the same issue. We have a big job that calls almost 30 Subjobs. It takes me about 30 mins to build that big job. However, it was only about 5 mins when I was using Talend v5.5.
One Star

Re: Slow Job-Generation

Hi,
are there any possibilities to gain more performance from the Job build?
I noticed that all job are generated over and over again e.g. I have  a tRunJob with a dynamic job and I have selected about 200 Job als possible Jobs. It I generate the start-Job, all 200 Jobs are generated every time, even if nothing has changed in the jobs.
This is not very efficient. Are there any hidden settings for that?
Kind regards,
Paul Mandalka
Sixteen Stars

Re: Slow Job-Generation

I am having the exact same issue. This needs resolving. I have raised a support call. 
Talend Studio should be able to identify which jobs have been modified and ONLY compile those (obviously jobs containing jobs which have been modified will need to be recompiled as well). We have a situation where a job that took 5 to 10 mins to compile in 5.6.1 takes almost an hour in 6.1.1.
One Star

Re: Slow Job-Generation

Hi,
any news on that?
One Star

Re: Slow Job-Generation

We had this same issue, which was resolved in the most recent patch Patch_20160316_TPS-1251_v1
Four Stars

Re: Slow Job-Generation

Hi,
We have recently migrated to Talend 6.1.1(Enterprise version) and we are facing the same issue.
Please let me know if there is any solution for this.
Thanks.
Six Stars

Re: Slow Job-Generation

Hi,
We are using Talend 6.1.1 and we have the same issue.
Four Stars

Re: Slow Job-Generation

Hi,
am getting same issue jobs are taking too much time to generate and run.

can anybody has solution.
Talend 6.1
Coludera 5.5
Four Stars

Re: Slow Job-Generation

Hi,
We too are facing same issue with jobs taking too much time to generate & run the jobs.
Does any one have got solution for this issue.
Talend 6.1.1 (Enterprise Edition)
Thanks & Regards
Ashok Karnam
Thanks & Regards
Ashok Karnam
One Star

Re: Slow Job-Generation

We had the same issue when we upgraded from Talend 5.6 to 6.1. The generate/build step started taking double the amount of 5.6. 
Here are my observations and steps followed to bring down to normal.
1. Talend 5.6 requires Java 7 jdk 1.7 (Must)
2. Though Talend 6.1 still support jdk 1.7 but it is highly recommended to have Java 8 jdk 1.8. This is big! give this a try.
3. Talend 6.1 generate/build process includes maven, this is another reason the build process takes more time. This was optional in 5.6 but it is part of build in 6.1.
4. Talend 6.1 requies patch 1237 to address slowness in setComponentPath (a step in generate). If you are generating via TAC, goto commandline section and notice the time it took for initRemote, logonProject, setComponentPath and buildJob.
4.1 If you see delay in initRemote - this is probably connection to TAC is very slow
4.2 if it's logonProject - you may require svn cleanup and delete unwanted jobs in repository
4.3 if it's setComponentPath - the patch 1237 will fix it.
4.4 If it's buildJob step - java upgrade will hopefully fix it.
5. It is always advised to run svn cleanup and also delete unwanted jobs from repository.
Hope it helps
Thanks
Rohan Suresh
Moderator

Re: Slow Job-Generation

Hi ashok.karnam,
Does this issue repro on all jobs on v 6.1.1 or just a specific job? Are you using JDK 1.8?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: Slow Job-Generation

Sabrina,
Generation time taken is high for all the jobs & not specific to any particular job.
Even for the jobs with minimal tComponents (~2-3) it takes longer time say around 3-5 mins.
Java version - java version "1.8.0_92"
Can you advice on, what could be the typical time taken for Generating & Deploying a job having minimal tComponents with no complexity/transformations.
Thanks & Regards
Ashok Karnam
Thanks & Regards
Ashok Karnam
Moderator

Re: Slow Job-Generation

Hi Ashok Karnam,
The job-generation consuming time is determined by many elements, job design, data volumn, your prod environment, the component you are using and so on.
It's a little hard for us to repro your slow job-generation issue on our side.
Would you mind report a ticket for your issue on Talend Support Portal? In this way, our collegues from support team can give you a remote assistance on this performance issue.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: Slow Job-Generation

Thanks Sabrina,
Job design is very simple job, populating data from source table to target table without any transformations & having bare minimal components - tMySQLInput, tMap, tMySQLOutput with a volume of 10,000 rows. Input & Output databases are within LAN. 
As suggested, I will create a ticket on Talend support portal for the said issue.
Thanks & Regards
Ashok Karnam
Thanks & Regards
Ashok Karnam
Seventeen Stars

Re: Slow Job-Generation

My experiences is, the speed of the SVN has the most influence. If you have less network latency you can increase the build speed significant. We also see under Windows it runs much slower than under Linux.
One Star

Re: Slow Job-Generation

Even we face the same problem while migrating from talend 5.6.1 to 6.1.1.
6.1.1 is mush slower compared to 5.6.1 ? 
kindly recommend some possible solution for this problem.
Thanks and regards,
Amritha
One Star

Re: Slow Job-Generation

We are using Talend 6.1.1 and we have the same issue.
Seventeen Stars

Re: Slow Job-Generation

Hey guys, only to state your system is slow does not help. Please post a short description of the systems.
Following information would be fine:
1. Which Code repository you use (Git/SVN) and which product you use here
2. If the Code repository does not run on the same server as the TAC please post the network latency (measured with ping starting from the TAC server)
3. Which operation system runt the TAC and the commandline
4. Does the command line runs on the same server as the TAC?
5. On Windows, are there any virus scanner running?
6. Which java version you use.
7. Please check in the TAC the view Commandline and post how long the single steps takes.
8. Do you use user components?
Four Stars

Re: Slow Job-Generation

As suggested, description of the systems we are using -
1. Which Code repository you use (Git/SVN) and which product you use here? -- SubVersion svn, version 1.9.4 (r1740329)
2. If the Code repository does not run on the same server as the TAC please post the network latency (measured with ping starting from the TAC server)? -- Code repository and TAC both on the same server.
3. Which operation system runt the TAC and the commandline? -- CentOS release 6.7 (Final)
4. Does the command line runs on the same server as the TAC? -- CommandLine & TAC both on the same server.
5. On Windows, are there any virus scanner running? -- Not Applicable
6. Which java version you use. -- Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
7. Please check in the TAC the view Commandline and post how long the single steps takes. Generating a job takes nearly 4-5 mins.
8. Do you use user components? -- No 
Thanks & Regards
Ashok Karnam
Thanks & Regards
Ashok Karnam
Seventeen Stars

Re: Slow Job-Generation

Hi, I guess I have found the problem:
https://help.talend.com/search/all?query=Compatible+version+control+systems&content-lang=en
According to this page SVN version 1.9.4 is not supported. I know similar problems in customer installation and we solved that only by switching back to SVN version 1.8.11
Do you have installed the SVN with the package management or do you use Collabnet Edge?
I usually use the Collabnet Edge server because of the nice web administration interface but also here, we cannot use the latest release because it contains SVN 1.9.x
We have saved an older version of this server for our self:
http://jan-lolling.de/talend/CollabNetSubversionEdge-4.0.13_linux-x86_64.tar.gz
The installation guid from the recent release can be applied to this older version. Please do not do the suggested update within this server - you would run in the same issues.
 
One Star

Re: Slow Job-Generation

This is not a SVN issue --- I am using TOS 6.2.1 with local projects and have just starting noticing the much slower build run times recently.  I am running Java 8 on both my laptop and target (HP Itanium) server, with no virus scan or other significant processes in the background.  Nothing has changed in my configuration recently except upgrading to latest TOS.  I would say there has been an order of magnitude increase in both compile and run times.
One Star

Re: Slow Job-Generation

I have similar problem with my Talend Studio. I don't connect to SVN due to slowness, and local projects take atleast 20 to 25 minutes to build and reach the execution path. This is very frustrating, I have 2 simple jobs with very few components.
Please help me how to improve my talend studio performance.
One Star

Re: Slow Job-Generation

According to this page SVN version 1.9.4 is not supported. I know similar problems in customer installation and we solved that only by switching back to SVN version 1.8.11 

Hello!
 I use 

Talend Data Management Platform 6.2.1
svnserve, version 1.8.16
java 1.8
RHEL 7.2
When I start to generate job on Conductor >> Job Conductor tab I see "Generating" status and on monitoring >> commandline tab I see last command:
CommandGroupCommand startGroup -o 'Generate job for task \'test_new\'';initRemote '<server ip>/org.talend.administrator' -ul '<my admin user>' -up 

and it lasts like forever...
The job is very simple, it consists of Input >> tConvertType >> tMap >> Output  and it is one of the first jobs created on this server.
Is there any ideas how can we fix it?
Thanks!
Client-Server connection is good.
One Star

Re: Slow Job-Generation

Same for me, after update Talend 6.2.1 became very slow on several environments. 
Seven Stars

Re: Slow Job-Generation

Hi,
I have been facing the same problem with Talend Ver6.2.1.  We are doing versioning on the GitHub And executing a simple job is taking as much as 15 minutes. It's totally unusable this way. We are using Talend Data Services and connecting remotely to TAC.
Any help would be greatly appreciated. 
One Star

Re: Slow Job-Generation

Same for me on several environments. Maybe its fixed in 6.3 version? someone tried?
Moderator

Re: Slow Job-Generation

Hi Vaidas,
Could you please indicate what's the build version you are using? Have you already reported a ticket about your issue to talend support portal?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.