SVN in Talend TAC

[kate]

Hi, 

I have small confusion regarding SVN configuration working with Tac.

My question is, Is that possible we have svn server and deployment server is same server ?

We configure our svn in Tac and use tac url in Talend's web app url field in manage connections.

If we do settings like this, what's the difference between when we deploy our jobs from Tac on deployment server and committed talend code via svn in same deployment server ? If yes, Do they share same location or different in server ?

And we deploy the build of talend job in Tac, where it keeps on deployment server ? Do we define this location in Tac configuration ?

I'm little confuse here how these two things work all together, SVN with TAC with Deployment. 

Any help will be appreciated !!

[jlolling]

To be honest, your question is a bit confusing.

Anyway. You can run your SVN server where ever you want! It does not matter. In most cases the SVN server is also used by other projects than Talend and therefore the server often already exists in the company when we install Talend.

The only thing you should check is the SVN server version - must be supported by Talend.

The svn server url in the TAC will directly be delivered to the studios and the studios talks directly to the svn server (same applies to git).

To separate development and production you should use different branches - just the normal case for svn or git.

[kate]

Hi Jan, thanks for your response.

So you mean to say, My deployment server can be my svn server as well. Correct ?

So please correct me if am wrong, we take a build from studio connected to svn and deploy it from tac. And tac keeps that build in nexus artifact repository. Is it correct ?

Second, I have never used publish of jobs in Talend. How its different from normal deployment explained above, like take a build of job from studio and then deploy that zip file in TAC ?

My assumption, correct me If am wrong, When we publish the job from studio with snapshot option, it pushes the latest build of job from studio directly to nexus artifact repository which we have already configured in tac.

If that's a case, then even after doing publishing why we go to tac and find the job from Nexus option to deploy in Execution task (which appears when we click on ADD option in job conductor) ? What it does extra when we select the job from Nexus repository to deploy ? Dont you think so once its published, job should be automatically appears in job conductor ?

Please correct me if am going in wrong direction..!

[jlolling]

What do you mean with development server? A TAC or the Jobserver or what?

[kate]

Jan I have attached the same image with my understanding, please have a look and share your words.

I have couple of more doubst Jan on same topic that where the Nexus artifact repository actually sits physically ? Like SVN folder structure is in Machine 2 for example, Is it the same way Nexus sits in Machine 3 ?

Here is my complete understanding Jan, What I'm thinking is,

1) When we build a job from Talend studio and import a zip file into Tac's execution task, Tac place that zip file to Nexus repository. And according to me Nexus artifact repository is always present in job server, the server where we want to deploy our Talend jobs and run as well.

2) When we publish our job from Talend studio, the zip file of job goes to Nexus artifact repository in job server. And then after that we point that job from Tac as a execution Task. Is my understanding correct Jan in point 1 and 2 mentioned above ?

These are two ways of deployment, that's how I'm think. Please put my understanding in right direction if am wrong.

Secondly, If am thinking right in above mentioned points why we have different ways of deployment, like first is import zip file in Tac and second is publishing a job ?

Third, What is the difference between Add normal task and Artifact task when we add a job to deploy in job conductor ?

[rhall_2_0]

Hi Kate,

I think you are confusing the old system with the new system. In the old days (and it is still used by some customers now) the CommandLine would generate and build the job from the SVN when you published a job using the TAC. This meant recompiling every time (...a very poor way of working). What the Nexus does is allow us to compile once (from the Studio when you publish) and store the compiled artefact. What the TAC does with this is simply pull the compiled artefact from the Nexus when it is needed. 

You can still pull the code from SVN and compile it using the Commandline (orchestrated by the TAC interface) if you want, but I would not. In fact, I would advocate completely removing the CommandLine (which is just a headless Studio) and publish your jobs to the artefact repository (Nexus) directly from the Studio. 

[kate]

Thanks rhall for joining conversation.

Can you please share your answers to below points ?

Like you said when we publish the job from studio, it goes to Nexus repository. I agree. 

Here are small doubts:

1) Is Nexus repository is also a compiled code managing directory like structure like SVN has ?

2) When we build Talend job from studio and import .zip file into Tac, Does it goes to same Nexus repository ?

3) When we publish our job from Talend studio, the compiled code zip file of job goes to Nexus artifact repository in job server. And then after that we point that job from Tac as a execution Task ? Is it correct understanding ?

4) What is the difference between Add normal task and Artifact task when we add a job to deploy in job conductor ?

5) If am thinking right in above mentioned points then why we have different ways of deployment in TAC, like first is to import zip file in Tac and second is publishing a job ?

[rhall_2_0]

---

@kate wrote:

Thanks rhall for joining conversation.

Can you please share your answers to below points ?

Like you said when we publish the job from studio, it goes to Nexus repository. I agree. 

Here are small doubts:

1) Is Nexus repository is also a compiled code managing directory like structure like SVN has ?

 The Nexus repository is just compiled code. Its structure is related to the Group ID, Artifact ID and Version values you specify when you publish. The structure is not related to the SVN.

2) When we build Talend job from studio and import .zip file into Tac, Does it goes to same Nexus repository ?

 This sounds like you are by-passing the Nexus. When you "build" in the Studio, it does not publish to the Nexus repository. I don't believe jobs published on Tac like this go to the Nexus repository. There is not enough information supplied to put them in a the Nexus (I've just tested and cannot see them in the Nexus). Once you deploy the job you will find the zip file's folder structure underneath the /jobserver/agent/TalendJobSeversFiles/repository folder.

3) When we publish our job from Talend studio, the compiled code zip file of job goes to Nexus artifact repository in job server. And then after that we point that job from Tac as a execution Task ? Is it correct understanding ?

This seems correct. Although the Nexus does not have to be on the JobServer. The Nexus repository can be anywhere in your architecture.

4) What is the difference between Add normal task and Artifact task when we add a job to deploy in job conductor ?

 "Add normal" allows you to select where to add the job from (zip, SVN...using commandline, and Nexus). "Artifact task" just makes it easier if you are always adding from a Nexus repository. This way is the recommended way now.

5) If am thinking right in above mentioned points then why we have different ways of deployment in TAC, like first is to import zip file in Tac and second is publishing a job ?

This is to support older implementations. Talend used to make you compile your code from SVN using the CommandLine. Now it works in a better way. However, many users will have built CommandLine scripts to automate this process. If Talend stopped supporting this mechanism over night, then it would alienate a lot of customers and lead to a lot of rework. 

Unfortunately it has not been made very clear to newer customers that the CommandLine is pretty much deprecated now. Most things you could do with the commandline, you can now do with the metaservlet (not quite everything, but most things).  

---

[kate]

Thanks for this wonderful sharing rhall.

I have just one last doubt about the deployment we do using build jobs from Talend studio.

When we build the job from talend studio, it generates the zip file. And after that when we import this zip file in TAC's execution task where it goes in the back-end ? I mean where that job is kept in job server after importing zip file into TAC in job conductor ? Like in case of publish it goes to artifact repository, but what happen in the case of importing zip file type of deployment in Tac ? I hope it makes sense

[jlolling]

Hi all,

The Publisher in the TAC creates so called artefacts and put them in the Nexus. 

The publisher must be used for services and can be used for batch jobs.

I strongly recommend using the Publisher for all kind of jobs.

The advantage compared to the build of the job in the Job Conductor (with its generate-function) is, the artefact in the Nexus is stable and (in case of using a release repository) it cannot be overwritten.

If you build a job within the Job Conductor you cannot rely on it because the Job Conductor can decide to build a job again. The Job Conductor will never build a job if you have created the Job Conductor entry based on a Nexus artefact.

We have also 2 different TACs (at least). One TAC is dedicated to the development and has a command-line connected to it. Here we only build artefacts and organize the projects and the developer rights.

The other TAC is only to run jobs. We setup here the same projects like in the Dev-TAC with exactly(!) the same name but typed as "no storage" - means there is no source repository. This way we cannot build a job in production and always build the artefacts in the development - mostly using different branches for the different stages (feature-branches, release branches etc).

[kate]

Jan thanks for new comments.

I'm sorry but your new comments are still not cleared to me.

Can we make it more simpler ? 

Lets say, following machines:

Machine-1 (SVN server)

Machine-2(Nexus Rpo server)

Machine-3(Job server or Test Server)

Machine-4( Where my actual talend is installed, where talend services are running- ADMIN area)

Machine-5( I'm sitting here and developing Talend jobs and my studio is connected to SVN which is machine-1)

Case-1, I create one job on Machine-5 and named as TalendJob.  Once job is committed to SVN, the job would be available in SVN server which is Machine-1. Now My job is ready to test on Machine-3 which is my test server, so when I published that job it goes to Machine-2 which is Nexus repository. Then I will open my Tac and will select a job from nexus from execution task and deploy that. So when I will run the job from Tac which is actually sitting on Machine-4 will point to  the TalendJob job in Nexus Repo which is in Machine-2 and will execute into Machine-3 environment which is test server.

Case-2,  I create one job on Machine-5 and named as TalendJob.  Once job is committed to SVN, the job would be available in SVN server which is Machine-1. Now My job is ready to test on Machine-3 which is my test server, so when I build a job I will get a zip file of that job. I will take that zip and will create a new task in TAC's job conductor and then will import that zip file over there. So when I import that zip file where it goes ? Does it goes to Nexus repo in Machine-2 ? Is it same like case-1 deployment and running procedure in backend ? 

Can you please share your comment on both cases Jan ? And correct me where my wrong..!!

[rhall_2_0]

Jan,

I neglected to mention the Publish option within TAC simply because I seldom use it (if ever). From my perspective. I prefer having my team publish from the Studio as Snapshots into the Nexus. I have written a Job which uses Maven to move and re-version artefacts into different repositories (we have DEV, TEST and PROD in Nexus). This allows us to guarantee that an artefact that meets developer unit testing requirements, is then tested without potential change and then progressed to prod with nothing but a version amendment to indicate stage of the workflow (DEV, TEST, PROD). This job also tags our SVN so that we can see the point in time when particular jobs/routes/services were code complete, which means we can branch from that point if we need to make amendments in the future. This is all done using the metaservlet (which is getting more awesome between versions) and we have completely done away with the horrible CommandLine.

[rhall_2_0]

Kate, 

Forget about Case 2. Case 2 leaves you open to issues creeping in. If Talend were starting today they would not have the option of following the Case 2 route. What you want to do is forget about Jobs being built as Zip files and just publish them to your Nexus repository. Once there, everyone can have access to them and your TAC is configured to easily make use of them for TEST or PROD purposes. As soon as you start manually moving Zip files around and compiling new builds left, right and centre, you will end up losing track of where your compiled code is in relation to your artefacts.

[kate]

Thanks rhall for your valuable comments, cleared so much dobuts Just needs your last closing comments to following small question, I really would like to know as am curious. I'm agree with your comments but sill want to know 1) When we import zip file into Job conductor's execution task, Does it goes to same Nexus repo ? 2) Jan said, Nexus repo's artefacts cannot be overwritten, they are stable when we publish jobs. So it means when we publish a job twice, it creates two Artefact IDS ?

[jlolling]

I totally agree with you. The command line is crap and not really reliable.

Talend has a plan to replace it and provide a maven plugin to build the job source code. I hope we will see first results in Talend 7.

[jlolling]

Case 1 is the correct way. I would not use Case 2.

[jlolling]

We have a separate Gitlab server (take it as your SVN) because a lot of other developers use it also.

We have a separate Nexus server - same reason because other developers also creates artefacts.

We have a development-TAC server which has a TAC with command line -> only to develop and build jobs

We have a Test-TAC (without command line) which is supposed to run the jobs for tests 

We have a Test-Job server (steered fron the Test-TAC)

We have a productive TAC (without command line) and this TAC is dedicated to steer the jobs in production.

We have a productive job server which is steered from the productive TAC.

[kate]

Sure Jan, you will not use case-2

I will not use it unless I know the what happens exactly in case-2 ? I want to know what wrong happens in case 2 so that I could also think that case 1 is always right.  Just for knowledge purpose I want to know where the zip file goes when we import it from Tac's job conductor ?

If it goes somewhere then why we should not use it, I'm looking for that perception.

[jlolling]

The problem is, you will never know the actual source on which the job is based because the developer can change something, create the zip and change it again!

You should always build testable artefacts automatically and not by hand in a studio.

[kate]

Ah Okay..!! So you mean Jan, Zip build also goes to Nexus. But the problem is if we take a zip and import that in job conductor, in-between changes can happen. So to protect and avoid that, we should use a architecture where everything is interconnected from studio--publish--TAC---- nexus. If we do that we would be having all trace who did what and when. Is it correct Jan...?

[pmambekar]

Hello,

I'm also trying to achieve similar workflow where we wanted to write a Jenkins job that uses Maven and re-version the artifact into different repo. Are you also deploying the artifacts to the target runtime using meta-servlet APIs? Or that deployment is still done via TAC? Is there a documentation on community that helped you achieve your workflow?