Incremental Loading of multiple Tables

One Star

Incremental Loading of multiple Tables

I am fairly new to Talend. My objective here is to load some incremental data (inserts/updates) from mysql source DB (8 or 10 tables) to a target mysql DB. In the process I am using a number of context variables to insert/update metadata into one or two reference tables that will hep track things like load job id, number of rows inserted or updated, load date etc. So I used one source table for now and it seems like I am able to successfully do a POC. I have created around 3 sub jobs for one table from start to end.
Now I have to the same thing for 8 more tables. What should be a good practice? Should I just use this one job to handle all the tables? Or Should I create separate jobs for each table? I still need to figure out best ways to handle the exceptional scenarios or any errors during the loading. Any guidance here will be appreciated.
Seventeen Stars

Re: Incremental Loading of multiple Tables

You should create one job per table.
You should track the progress of the loading an memorize the most recent (or latest) timestamp or identifier and start with it in the next job run.
You should limit the datasets you transfer from source to target to a reasonable limit (e.g. 1Mio) to avoid resource problems.
You should call the job as long as the job transfers datasets (if the limit will not be reached you have successfully transferred all data.
You should do one transfer within a transaction.
This way you can do jobs which are restartable and finish the task in predictable time.

Re: Incremental Loading of multiple Tables

To achieve Incremental Loading in talend , you can also use Talend CDC (Changed Data Capture) which can inform your job about change.
TalendHelpCenter:CDC architectural.
TalendHelpCenter:Change Data Capture CDC is easy with Talend Studio.
So far, Talend CDC (Changed Data Capture) is only available in Talend Enterprise Subscription Version.
Best regards
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.


Talend named a Leader.

Get your copy


Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables


Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables


Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema