How to load multiple tables at the same time from db1 to db2

Four Stars

How to load multiple tables at the same time from db1 to db2

Hi, 

   I am taking over an ETL job that is moving data from Oracle to HPE Vertica in Talend 6.3.1 free edition. In this job there are about 12 different tables being moved. The vertica cluster has 4 nodes and can easily handle parallel processing. When I kick off the talend job to load the 12 tables, I see that the job is executing the tOracleInput task one table at a time. Can anyone tell me if there is a way I can kick off all 12 tasks at the same time?  I'm fairly new to Talend and I can't seem to find anything in the talend forum that tell me how to do this. Thanks.


Accepted Solutions
Fifteen Stars TRF
Fifteen Stars

Re: How to load multiple tables at the same time from db1 to db2

Hi,

With the free edition you have 2 options:
1- have separated subjobs (in the same job) not linked one to each other + tick the option "Multi thread execution" in Extra tab of project
2- have 1 external subjob per table + tick the option "Multi thread execution" in Extra tab of project for parent and child jobs

If you choose 2, both operations "select from Oracle" and "insert into Vertical" should be realized in the same subjob. If you need to manage the order of insert operations, you need to get the result using a memory list or a file.
Think about tPreJob and tPostJob if you need some operations when the job starts or finish.

Hope this helps.

TRF

All Replies
Fifteen Stars TRF
Fifteen Stars

Re: How to load multiple tables at the same time from db1 to db2

Hi,

With the free edition you have 2 options:
1- have separated subjobs (in the same job) not linked one to each other + tick the option "Multi thread execution" in Extra tab of project
2- have 1 external subjob per table + tick the option "Multi thread execution" in Extra tab of project for parent and child jobs

If you choose 2, both operations "select from Oracle" and "insert into Vertical" should be realized in the same subjob. If you need to manage the order of insert operations, you need to get the result using a memory list or a file.
Think about tPreJob and tPostJob if you need some operations when the job starts or finish.

Hope this helps.

TRF
Highlighted
Four Stars

Re: How to load multiple tables at the same time from db1 to db2

Thank you! That is exactly what I was looking for.
Fifteen Stars TRF
Fifteen Stars

Re: How to load multiple tables at the same time from db1 to db2

Great!
Feel free to come back if you need more help.

TRF

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download