How to append data in Destination DB

Highlighted
Six Stars

How to append data in Destination DB

Hello Everyone,

 

I have four jobs designed:
1) Job 1: Pull CSV files from FTP to Local Folder.
2) Job 2: Load CSV files from Local folder to MySQL source DB.
3) Job 3: Load dimension tables in destination DB from source DB.
4) Job 4: Load Fact Tables in destination DB from source DB.

After the initial data dump, I have the process set up to run once every month to receive CSV files regarding stats for each month. After Job 2, I have a truncate command for all the dimension tables and fact tables to reload the tables( so that they have the new data as well).

Instead of truncating I would like to append the new data received to the destination DB tables. Is there any way to do this?

Looking forward to your inputs.

Thanks and Regards,
Praneeth Reddy Varimalla

Highlighted
Sixteen Stars
Sixteen Stars

Re: How to append data in Destination DB

If records for dimension and fact tables never exist in the destination DB, Insert statement should be enough.

Else, if you have to update existing rows and insert new ones, have a look to Merge statement.


TRF

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog