One Star

Job generation from Talend Administration Console = SLOW

Hi,
I did several Talend projects, running on both Linux and Windows systems and everytime I notice that job generation from the Talend Administration Console takes way more time and cpu power then building or exporting the job from the Studio. This is not 2x, but more like 500x times slower.
Anyone also experienced this behaviour? and how to solve?
Thanks!
19 REPLIES
Community Manager

Re: Job generation from Talend Administration Console = SLOW

Hi
Which version are you using? Have you done a testing for the performance if you open Commandline and export directly the job script from Commandline?
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Job generation from Talend Administration Console = SLOW

Hi Shong,

I have been also facing same issue while running the job through TAC the generation is taking too much memory and too much CPU usage and still it is taking more than 72 hours also.

More Information:
1. TAC installed & Running in my Local Machine
2. commandLine ,SVN Server and Job Remote Server also running in same machine
3. Talend Integration GUI/Studio also opened in same machine.
4. I tested the performance without starting Talend GUI but all the remaining services mentioned above as well as with Talend GUI also started.
But in both the cases I am not able to generate the job for about 72 hours and still it is generating mode only.
I observed one more thing , when i am trying to run job which has designed with less number of components(small jobs) are generating very much fast.
and if the job is complex(more number of sub jobs are there calling more jobs in same project) am not able to generate job in TAC, where as when am trying to run the job through Talend Studio i can able to run any kind of job (either complex or simple).

Temporary solution:
the temporary solution which i am using now is that am exporting the complex job into my local machine and then designing one more job call the batch file of the exported complex job , Then calling that batch file using TAC which is generating the job very fast and running fine.

But I cant able to monitor my each and every jobs (sub jobs designed in complex job) job flow until it gets failed any where. even that time also i have to go to Log file and search for error in which job it failed.

So this is what my problem.
Please suggest me any permanent solution for this.
The Talend Version am using Talend Professional Edition 4.1.2 in windows7Machine.

Thanks In Advance,
Srinivas. L
One Star

Re: Job generation from Talend Administration Console = SLOW

Hi Shong,
The Talend Version am using is Talend Professional Edition 4.1.2
Thanks ,
Srinivas. L
One Star

Re: Job generation from Talend Administration Console = SLOW

Hello, We are having the issue using Talend Enterprise Data Integration 5.4.1 on CentOS were you able to find a workaround? Thanks in advance for your time?
One Star

Re: Job generation from Talend Administration Console = SLOW

Hi,
I am running the talend esb job from the windows service in the windows 2008 sever where other jobs are also being run at a time. When running the same job for each file it takes not more than 4 second to process using talend studio but from windows service at a minute only 17 files get processed. And it does not end up here, the speed gradually decreases and reach one file per minute and I have not used database either to log.
I see in the memory in the task manager and my wrapper occupies only about 1.4 MB so this could not be the memory problem either.
Can anyone help me out??
Moderator

Re: Job generation from Talend Administration Console = SLOW

Hi dhirajdhakal,
What's the talend product build version are you using? Is there any error info for your issue?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Job generation from Talend Administration Console = SLOW

HI Sabrina,
I am using TOS 5.3.1 version. No there is no info regarding this, only what happens is the speed gradually falls down and each component takes very long time to process , sometimes even 4-5 seconds.
From few days some wierd thing is happening. The processing slows down upto 23:59 processing upto only one file per minute and when the clock makes 00:00 the speed increases rapidly and reaches up to 15-16 files per minute. And again the speed gradually falls down and reaches upto 4 files/minute after 2:00.
I don't know what is the problem, is it due to the server or in the talend itself.
Thanks!!!
Moderator

Re: Job generation from Talend Administration Console = SLOW

Hi dhirajdhakal,
Would you mind sharing your job design screenshots with us so that we can see if there is any optimistic solution for your issue.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Job generation from Talend Administration Console = SLOW

I am also facing the same issue while generating it is taking more time. Please give any suggestion.
Moderator

Re: Job generation from Talend Administration Console = SLOW

Hi,
I am also facing the same issue while generating it is taking more time. Please give any suggestion.

Please give us more description about your current situation.(job design, talend product version...).
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Job generation from Talend Administration Console = SLOW

Yep same here using 5.4.1 much, much faster to generate and run in Studio (on an ordinary desktop machine) than via TAC or CommandLine (on a powerfull server) which take approximately them same, never more then 15 minutes or so but still very odd.
One Star

Re: Job generation from Talend Administration Console = SLOW

Hi Sabrina,
I have attached the images herewith with main job and the one subjob. There are similar 3 more subjobs in it. It is giving issue when running from Windows Service.
One Star

Re: Job generation from Talend Administration Console = SLOW

oh the above one is only the sub job
due to pixel problem the main job can't be posted.
One Star

Re: Job generation from Talend Administration Console = SLOW

I have again attached the mainjob
One Star

Re: Job generation from Talend Administration Console = SLOW

Hi Sabrina
Sorry for uploading lots of comments. But i could not post the main job, anyway the main job only transfers some file here-and-to. And gets the response from the sub jobs which I have already posted,
Can u plz give the solution??
Moderator

Re: Job generation from Talend Administration Console = SLOW

Hi,
I have seen your pic and found there is tRunjob component in your workflow. Have you checked out option "Use an independent process to run subjob" which use an independent process to run the subjob. This helps in solving issues related to memory limits.
TalendHelpCenter:tRunJob.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Job generation from Talend Administration Console = SLOW

Hi Sabrina,

I tried to make the sub-job independent but it gives some other issue. As we have used tBufferOutput Component in each subjob on making the job independent the value is not returned and we are given exception.
Thanks!!
One Star

Re: Job generation from Talend Administration Console = SLOW

I've tried using "Use an independent process to run subjob" it can help with memory issues whilst the job is running but does not seem to affect either the time to "generate" the job or the memory usage when doing so (i.e. the commandLine process that is generating the job can run out of java heap space).
Not applicable

Re: Job generation from Talend Administration Console = SLOW

Hi,
I migrate from 5.3 to 5.4.1 r111943-20131212-1133, and i'am having trouble with an exported job.
That job works fine with 5.3, (TOS GUI and autonomous job) and with 5.4.1 it takes too much time to execute as an autonomous job.
If running in TOS GUI 5.4.1, it takes about 20mn to complete, but when running in autonomous job it is 1 or 2 hours !...
It is not a problem with TOS 5.3.
Have an idea ?
Thx