Four Stars

merge multiple databases into one

Hello, 

 

I worked previously on merging two mysal and oracle databases into one mysql, here is my basic schema that I used to do the job:

   myOracleDbInput  --------\    tUnit_1  ---->   mySqlDbOutput

 mySqlDbDbInput  ----------/

Now I'm wondering if I can merge several databases whose parameters are on text files for example located in a directory, for example:
       - Directory: Parameters of the databases:
                             -  database1.txt
                             -  database2.txt
                              ..

                             -  databaseN.txt
Is there an opportunity to iterate over the directory to achieve this Job?

 

 

Thank you.

6 REPLIES
Forteen Stars TRF
Forteen Stars

Re: merge multiple databases into one

To keep your work simple you may have a main job using a tFileList to iterate over the file list and set the necessary context variables then, call a child job (transmit the whole context) to make the elementary task (for the current context).

TRF
Four Stars

Re: merge multiple databases into one

Thank you for your return, Can you explain me more about this method ? what do you mean by a child job !!!?

Forteen Stars TRF
Forteen Stars

Re: merge multiple databases into one

A child is a job you start from another one using a tRunJob component.
This way you're able to keep the main task simple (no iteration) and delegate to the main (or parent) job the task to iterate over file list and set the context variables.
This is a very common design pattern.

TRF
Four Stars

Re: merge multiple databases into one

Can you provide an example please ?

Forteen Stars TRF
Forteen Stars

Re: merge multiple databases into one

Currently on my mobile, you'll to wait for hours. However, you can get inspired by other topics with similare question.
Also, if you expect for someone to solve your case for you, you need to give the whole information. For example, what's in your text files? How are they structured?
Even if some people work for free, no one is a magician.

TRF
Four Stars

Re: merge multiple databases into one

I'm seeing the other posts, to see if I can do the work, and thank you for your help,

my repertory named "databasesParameters " structured as follows:
database1.txt:
            - host, xxxx
            - port, xxxx
            - data_base_name, xxxx
            - user, xxxx
            - password, xxxx
            - table_name, xxxx

 

database2.txt:
            - host, xxxx
            - port, xxxx
            - data_base_name, xxxx
            - user, xxxx
            - password, xxxx

            - table_name, xxxx

           ..

           ..

           ..

           ..

databaseN.txt:
            - host, xxxx
            - port, xxxx
            - data_base_name, xxxx
            - user, xxxx
            - password, xxxx

            - table_name, xxxx

 

 

Real Exemple : 

 

productsDb.txt

 

            - host,127.0.0.1
            - port, 5487
            - data_base_name, productsDb
            - user, free
            - password, pass

            - table_name, construction_products_holding

 

 

Thanks again for your help.