I have four jobs designed:
1) Job 1: Pull CSV files from FTP to Local Folder.
2) Job 2: Load CSV files from Local folder to MySQL source DB.
3) Job 3: Load dimension tables in destination DB from source DB.
4) Job 4: Load Fact Tables in destination DB from source DB.
After the initial data dump, I have the process set up to run once every month to receive CSV files regarding stats for each month. After Job 2, I have a truncate command for all the dimension tables and fact tables to reload the tables( so that they have the new data as well).
Instead of truncating I would like to append the new data received to the destination DB tables. Is there any way to do this?
Looking forward to your inputs.
Thanks and Regards,
Praneeth Reddy Varimalla
If records for dimension and fact tables never exist in the destination DB, Insert statement should be enough.
Else, if you have to update existing rows and insert new ones, have a look to Merge statement.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema