Auto generate TalenD jobs

Four Stars

Auto generate TalenD jobs

 Forgive my ignorance up front, I come from a background of using ETL/ELT tools such as Wherescape RED, SSIS and SAS DI Studio.


My question is around auto-generation of jobs.


SSIS is nothing more than BIML (Business Intelligence Markup Language) represented in a VisualStudio GUI and scheduled on something like Control-M or SQL Server Agent. The jobs do not need to be created by hand --- meaning dragging and dropping transformations and registered data sources and target databases. Instead the jobs (BIML) can be created by coding the BIML and automating the creation of jobs by extending the BIML with BIML Script (using VB .NET  or C#) .

Therefore a single BIML Script can generate hundred of jobs doing similar type work differentiated by parameters. Like loading source CSV files that are the same in structure but different by filename's suffix; example the date included in a filename like product_20180717.csv.


Anyway, I was wondering if the same level of automation can be achieved in TalenD? i.e. I write a script that generates TalenD jobs with the components I need.



Eleven Stars

Re: Auto generate TalenD jobs

Talend is a true ETL who can do a lot of thing around ETL like ftp rest managing file, connection oracle and ms sql………
on the enterprise version you can use dynamic schemas. When you have a lot of similar jobs just make one.
all you jobs (data jobs, orchestration jobs,....) can be called by an other one and use parameters so you can have one complex job called multiple time.

Last thing, it's not the best way but it's working.
Talend is a java generator. so if you still want to generate multiple similar jobs. you can copy and modify java code.


Francois Denis

Tag as "solved" for others! Kudos to thanks!