merge multiple databases into one

Highlighted
Four Stars

merge multiple databases into one

Hello, 

 

I worked previously on merging two mysal and oracle databases into one mysql, here is my basic schema that I used to do the job:

   myOracleDbInput  --------\    tUnit_1  ---->   mySqlDbOutput

 mySqlDbDbInput  ----------/

Now I'm wondering if I can merge several databases whose parameters are on text files for example located in a directory, for example:
       - Directory: Parameters of the databases:
                             -  database1.txt
                             -  database2.txt
                              ..

                             -  databaseN.txt
Is there an opportunity to iterate over the directory to achieve this Job?

 

 

Thank you.

Highlighted
Sixteen Stars
Sixteen Stars

Re: merge multiple databases into one

To keep your work simple you may have a main job using a tFileList to iterate over the file list and set the necessary context variables then, call a child job (transmit the whole context) to make the elementary task (for the current context).

TRF
Highlighted
Four Stars

Re: merge multiple databases into one

Thank you for your return, Can you explain me more about this method ? what do you mean by a child job !!!?

Highlighted
Sixteen Stars
Sixteen Stars

Re: merge multiple databases into one

A child is a job you start from another one using a tRunJob component.
This way you're able to keep the main task simple (no iteration) and delegate to the main (or parent) job the task to iterate over file list and set the context variables.
This is a very common design pattern.

TRF
Highlighted
Four Stars

Re: merge multiple databases into one

Can you provide an example please ?

Highlighted
Sixteen Stars
Sixteen Stars

Re: merge multiple databases into one

Currently on my mobile, you'll to wait for hours. However, you can get inspired by other topics with similare question.
Also, if you expect for someone to solve your case for you, you need to give the whole information. For example, what's in your text files? How are they structured?
Even if some people work for free, no one is a magician.

TRF
Highlighted
Four Stars

Re: merge multiple databases into one

I'm seeing the other posts, to see if I can do the work, and thank you for your help,

my repertory named "databasesParameters " structured as follows:
database1.txt:
            - host, xxxx
            - port, xxxx
            - data_base_name, xxxx
            - user, xxxx
            - password, xxxx
            - table_name, xxxx

 

database2.txt:
            - host, xxxx
            - port, xxxx
            - data_base_name, xxxx
            - user, xxxx
            - password, xxxx

            - table_name, xxxx

           ..

           ..

           ..

           ..

databaseN.txt:
            - host, xxxx
            - port, xxxx
            - data_base_name, xxxx
            - user, xxxx
            - password, xxxx

            - table_name, xxxx

 

 

Real Exemple : 

 

productsDb.txt

 

            - host,127.0.0.1
            - port, 5487
            - data_base_name, productsDb
            - user, free
            - password, pass

            - table_name, construction_products_holding

 

 

Thanks again for your help.

 

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Talend Cloud Developer Series – Defining Metadata

This video focuses on different methods of adding metadata to a job in Talend Cloud

Watch Now