One Star

Virtual server job deployment

Hello,
during testing job server high availability, I have come across following issue:
1. Create a virtual server VS consisting of two job servers A,B
2. Create a job (from an exported zip, not from svn) and schedule it for repetitive triggering on the virtual server VS. Set it to restart in case of unavailable job server.
3. Let's say the task is being executed on job server A. In case of failure of this server (shutdown,...), the job will be executed on B for the time A is down.
4. If server B is on again, the job may be scheduled back to B on one of its following triggerings.
However, here comes the problem: if, for example, the server with job server B was rebooted, at step 4, I get following error: "Generated job not found", and need to redeploy the job.
If I tried to schedule the job just on job server A (instead of VS) and rebooted the job server A between two triggerings, the job got deployed on the server A and there was no error.
My guess is that if there a job server for which a job is scheduled is offline for some time and then is available again, the TAC automatically redeploys the jobs to it. However, if the job is scheduled for a virtual server and one of its job servers is unavailable for some time, the virtual server itself is not detected down, so the job doesn't get redeployed.
The last paragraph is under assumption that after system rebooting or job server restarting, jobs need to be redeployed - I'm not sure about that, however, it looks to me like that based on the behaviour.
Could you please explain this behaviour to me and tell me how to solve it to avoid manual redeployment after server restart?
Thanks
Jan
3 REPLIES
Community Manager

Re: Virtual server job deployment

Hi Jan
It seems like a bug. I agree with you that the job should be redeployed automatically once the job server is available again.
Which version are you using? I have consulted to our developer and was told that a similar issue was reported before and have been fixed.
I would suggest you to report a jira issue and open a ticket in our support portal, our colleagues from support team will follow up your issue and assist you.
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Virtual server job deployment

Hello shong,
thank you for your answer. I'm using version 5.4.1. After some investigation, I think the root cause is more likely in the job conductor, specifically in generated jobs storing (I considered it to be a job server issue because I have TACs and job servers on same machines and so also switched off both the TAC and job server).
So, I guess the problem really is in following:
As I want high availability, I'm using two TAC instances on two servers (TAC-A, TAC-B). When I upload a job through TAC-A, it is stored in the generatedJobs folder just on the machine with TAC-A.
Now, when the machine with TAC-A is down and TAC-B makes an attempt to redeploy the job, it fails, as the generated job cannot be found.
The failure in the scenario described in my first post may be caused because TAC-B notices the job server A running before TAC-A is up (tomcat takes more time to start) and, of course, fails to redeploy the job. *
(Manually) synchronizing the content of the generatedJobs folders on both machines seems to solve this problem. However, I must admit that I'd like if there was another way to get TACs really synchronized when working in a cluster mode - other than file copying outside talend.
Maybe I would suggest following. When TAC notices that a job server just started to be available, it may check for each job if it is already deployed on the job server with version corresponding to the one in the TAC database. If yes, there is no need to redeploy the job. This would probably avoid my issue at least for most of time.
However, maybe it would really be more convenient to move to jira - I'll think about this and maybe post an issue there.
Regards
Jan
---
* N.B:
However, what if both TAC-A and TAC-B are up and TAC-B is trying to deploy the job? In case of error (because, afaik, TAC-B looks only into its local generatedJobs folder), will it switch to TAC-A which has the job?
Community Manager

Re: Virtual server job deployment

Hi Jan
As you said, for development issue, it is better to move to Jira and discuss with our developers, please report a jira issue and post the link here, it is convenient for us to follow up your issue.
Thanks!
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business