merge multiple databases into one

Four Stars

merge multiple databases into one

Hello, 

 

I worked previously on merging two mysal and oracle databases into one mysql, here is my basic schema that I used to do the job:

   myOracleDbInput  --------\    tUnit_1  ---->   mySqlDbOutput

 mySqlDbDbInput  ----------/

Now I'm wondering if I can merge several databases whose parameters are on text files for example located in a directory, for example:
       - Directory: Parameters of the databases:
                             -  database1.txt
                             -  database2.txt
                              ..

                             -  databaseN.txt
Is there an opportunity to iterate over the directory to achieve this Job?

 

 

Thank you.

Highlighted
Fifteen Stars TRF
Fifteen Stars

Re: merge multiple databases into one

To keep your work simple you may have a main job using a tFileList to iterate over the file list and set the necessary context variables then, call a child job (transmit the whole context) to make the elementary task (for the current context).

TRF
Four Stars

Re: merge multiple databases into one

Thank you for your return, Can you explain me more about this method ? what do you mean by a child job !!!?

Fifteen Stars TRF
Fifteen Stars

Re: merge multiple databases into one

A child is a job you start from another one using a tRunJob component.
This way you're able to keep the main task simple (no iteration) and delegate to the main (or parent) job the task to iterate over file list and set the context variables.
This is a very common design pattern.

TRF
Four Stars

Re: merge multiple databases into one

Can you provide an example please ?

Fifteen Stars TRF
Fifteen Stars

Re: merge multiple databases into one

Currently on my mobile, you'll to wait for hours. However, you can get inspired by other topics with similare question.
Also, if you expect for someone to solve your case for you, you need to give the whole information. For example, what's in your text files? How are they structured?
Even if some people work for free, no one is a magician.

TRF
Four Stars

Re: merge multiple databases into one

I'm seeing the other posts, to see if I can do the work, and thank you for your help,

my repertory named "databasesParameters " structured as follows:
database1.txt:
            - host, xxxx
            - port, xxxx
            - data_base_name, xxxx
            - user, xxxx
            - password, xxxx
            - table_name, xxxx

 

database2.txt:
            - host, xxxx
            - port, xxxx
            - data_base_name, xxxx
            - user, xxxx
            - password, xxxx

            - table_name, xxxx

           ..

           ..

           ..

           ..

databaseN.txt:
            - host, xxxx
            - port, xxxx
            - data_base_name, xxxx
            - user, xxxx
            - password, xxxx

            - table_name, xxxx

 

 

Real Exemple : 

 

productsDb.txt

 

            - host,127.0.0.1
            - port, 5487
            - data_base_name, productsDb
            - user, free
            - password, pass

            - table_name, construction_products_holding

 

 

Thanks again for your help.

 

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Self-service Talend Migration: Moving from On-Premises to the Cloud

Move from On-Premises to the Cloud by following the advice of experts

Read Now

Modern Data Engineering in the Cloud

Learn about modern data engineering in the Cloud

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now