Four Stars

Dynamic job to load multiple tables with different structure in one talend job


Hi All,
I have one scenario,

I want to load all tables of data in to respective target tables through one talend job dynamically..please help me on this.

like

src1
-------
database--db1
schema---x
tables-----A,B,C,D,E,F,........X,Y,Z.                                                                     Tareget
all tables are different schema structure.                                                          ---------------------------
                                                                                                                          database----------db3
                                                                                                                           schema------------z
src2                                                                                                                    tables------------A,B,C,D,E,F,..........X,Y,Z.
---------                                                                               These tables schema structure same as src1,src2 tables structure
database---db2
schema--y
tables are same like src1---A,B,C,D,E,F,.............X,Y,Z.
all tables are diffrerent schema structure

 

i want load union the data of src1,src2 tables data then load to target respective tables in single talend job dynamically for all tables.

 

6 REPLIES
Four Stars

Re: Dynamic job to load multiple tables with different structure in one talend job

my source and target databases are MSSQL...I want to create one talend job instead of creating multiple jobs for all tables..

 

Please suggest me.Thanks..all

Four Stars

Re: Dynamic job to load multiple tables with different structure in one talend job

Hi guys the above scenario like as, 
I have 100 tables with different metadata and i need to develop a single job with loads all these tables to target system we Need to run 10 times for all tables or at a time if so please guide me out.

 

SRC----MSSQL

TARGET---MSSQL

Employee

Re: Dynamic job to load multiple tables with different structure in one talend job

Use dynamic column type.  However, this only works if you do not need to do transformation, since you do not know the schema and data type upfront.

 

Else use the Template functionality in the Studio to generate all your jobs https://help.talend.com/reader/itosd4jFDXxrm6qULuPrbg/rpFV93Uc4LKksvXQFtSdWg  It will generate many jobs, but it take 5 mins instead of several days.

 

Four Stars

Re: Dynamic job to load multiple tables with different structure in one talend job

Hello Iburatally Employe Thanks for you response.

 

I am using Talend Open Studio..So in this there is no option for "Create job from templates".

 

Please give some solutuion for TOS.

 

 

I created source tables passing iteratively.

 

Pleas find attached screen shot.

 

 

 

 

 

Six Stars

Re: Dynamic job to load multiple tables with different structure in one talend job

Hi ,

 

You can achieve this using dynamic column load. Following are the steps

1.  Create a list of your table names in one csv file 

2. Read that file in tFileInputDelimited

3. Use tFlowToIterate  and create global variable in with tablenames as value from file

4. Use tmysqlinput and set table name as global variable name and in schema select as built in and set only one row and select type as dynamic .

5. do same for another database input flow

6. use tunite to make union 

7. connect to target where specify table name as global variable from tFlowToIterate  and schema as built in and then create only one row in schema with type as dynamic.

 

Key here is to pass table names list using tFlowToIterate   and defining dynamic column  in built in schema

 

You can use following link for reference to dynamic column 

https://help.talend.com/reader/OnsdPcJL19xPfNBaWRUL~w/6uAF9tc63_edATsd7VMTwg

 

Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

 

Four Stars

Re: Dynamic job to load multiple tables with different structure in one talend job

Hi Shivanand,

 

Thanks for you response..

 

I am using Talend Open Studio(TOS)..In this there is no option dynamic rt?

 

How i can ..? Please