From Thursday, July, 9, 3:00 PM Pacific,
our Community site will be in
read-only mode
through Sunday, July 12th.
Thank you for your patience.

In TAC Architecture , Nexus , Git , databases

Highlighted
Five Stars

In TAC Architecture , Nexus , Git , databases

Hi All,

 

I am kind of new to TAC . 

While going through the TAC architecture I had some doubts.

 

In TAC ,

we have 2 repositories  & 3 databases: 

1. NEXUS

2. GIT

 

1. Audit

2. Admin

3 . Monitoring

 

So My Questions are  :

1. What  are exact roles of these 2 repositories in TAC .

2. Whenever we schedule Job in TAC we point TAC to deploy job from NEXUS Artifact repo, We have all versions of job there so why do we need to maintain source code ?
3. How they are inter connected to each other inside TAC ?

4. If we remove old versions of job from NEXUS repo (Only maintain New version),  does it affect my TAC server ? If Yes then How ?
(For ex. We have publish two version of Job (0.1 & 0.2) and if we remove 01. and only maintain 0.2 in Nexus.)
5. What are the role of databases in TAC?

6. how TAC server communicate with all repo & Database.

 

 

TAC-Arch.pngThanks in advance. 


Accepted Solutions
Highlighted
Five Stars

Re: In TAC Architecture , Nexus , Git , databases

Hi,

 

We have now moved to Talend Cloud - but from memory of using TAC the best answer I can think of to 1) and 2) would be

 

- GIT has the 'source code' ( xml, code snippets, contexts etc) and represents the editable version of the jobs in Studio

- Nexus has the built/compiled version of a job - akin to a deployable version ( as well as stores other dependencies/third party libraries etc) 

 

3) TAC has configuration to both the nexus server ( from our experience usually 1:1 with TAC instances ) and Via Workspaces/Projects to GIT - in our case we also have GIT configured on each user account - so we can keep audit trail of users updating jobs.

4) Removing Jobs from Nexus isn't ideal - suspect if you removed all references (running tasks/scheduled jobs etc it might be ok) but if you have jobs calling other jobs this could get messy. One option might be to maintain multiple nexus repositories ( i.e. dev / staging / preprod / prod ) probably a way of doing this - likely easier with paid version of nexus. So that you can safely remove artifacts in dev which were not pushed to staging etc. 

 

5) Databases are mostly monitoring, audit trail and runtime config ( what jobs should run when, with what contexts )  useful for managing production environments, used by TAC to show job runtime history etc.

 

BTW I would thoroughly suggest moving to Talend Cloud. We moved across last year with help of Talend Professional services. Most jobs moved with very little change.Administering cloud much easier - TAC is a fairly complicated bit of engineering!

 

Anyway hope that helps,

AndrewB @ Harmoney

 

View solution in original post


All Replies
Highlighted
Five Stars

Re: In TAC Architecture , Nexus , Git , databases

Hi,

 

We have now moved to Talend Cloud - but from memory of using TAC the best answer I can think of to 1) and 2) would be

 

- GIT has the 'source code' ( xml, code snippets, contexts etc) and represents the editable version of the jobs in Studio

- Nexus has the built/compiled version of a job - akin to a deployable version ( as well as stores other dependencies/third party libraries etc) 

 

3) TAC has configuration to both the nexus server ( from our experience usually 1:1 with TAC instances ) and Via Workspaces/Projects to GIT - in our case we also have GIT configured on each user account - so we can keep audit trail of users updating jobs.

4) Removing Jobs from Nexus isn't ideal - suspect if you removed all references (running tasks/scheduled jobs etc it might be ok) but if you have jobs calling other jobs this could get messy. One option might be to maintain multiple nexus repositories ( i.e. dev / staging / preprod / prod ) probably a way of doing this - likely easier with paid version of nexus. So that you can safely remove artifacts in dev which were not pushed to staging etc. 

 

5) Databases are mostly monitoring, audit trail and runtime config ( what jobs should run when, with what contexts )  useful for managing production environments, used by TAC to show job runtime history etc.

 

BTW I would thoroughly suggest moving to Talend Cloud. We moved across last year with help of Talend Professional services. Most jobs moved with very little change.Administering cloud much easier - TAC is a fairly complicated bit of engineering!

 

Anyway hope that helps,

AndrewB @ Harmoney

 

View solution in original post

Highlighted
Five Stars

Re: In TAC Architecture , Nexus , Git , databases

Hi Andrew
Appreciated your help.
Just one query :
1. How TAC manage SVN /GIT when we publish the job ? because we publish deploy-able version of jobs to NEXUS repo right ?

Is it when we deploy job to NEXUS that time Studio also push source code etc to SVN/GIT ...?
Highlighted
Community Manager

Re: In TAC Architecture , Nexus , Git , databases

Hi
Deploy job to Nexus will not push source code to SVN/GIT.
To push source code to SVN/GIT, it depends on the commit model and lock model, these two options are
configured when you create a new project on TAC.

Regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
Highlighted
Five Stars

Re: In TAC Architecture , Nexus , Git , databases

Hi, 

Yes Studio only really interacts with GIT.

 

We then have a couple of alternative approaches.

 

- From GITHUB we then trigger a build pipeline ( in our case Jenkins ) which using Talend Command Line turns the source files ( XML + Code + Libraries + Magic) into a binary executable ( or JAR file ) which is stored in Nexus. We have also wrapped routes into a Spring Boot based docker instance for example.

 

- Talend Admin Console can also do this on the fly - we pick job to run by scanning Git repo/branches etc, if you try and deploy/run a Task it uses its own copy of Talend Studio  ( in headless - no GUI) mode to checkout code + generate the binary/runable version - if it hasn't already done so. In this case no build server required. I'm not sure if Nexus is required for this mode - I suspect it is because we tend to build on TAC server but run on runtimes ( so assume runtimes must pull built version out of nexus) 

 

As per previous - I'm a little bit rusty as been a while since I used TAC - Talend cloud abstracts most of this away!

 

Shong, hopefully thats somewhat correct - please correct as required!

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

The Forrester Wave: Strategic iPaaS and Hybrid Integration Platforms, Q1 2019

Find out how Forrester rate different iPaaS and Hybrid Integration Platforms

Read

Downloads and Trials

Test drive Talend's enterprise products.

Downloads