Today i installed and configured talend to source from a CSV to mysql db.
My CSV file is updated daily but many of the records are the same as the previous day.
I would like to be able to source from my CSV file into my target mysql db adding a LastUpdateDate into the target when new records are added. The CSV does not have a date field.
I have been playing around trying to get this to work most of the day.
i used the TalendDate.getCurrentDate() in tmap_1 to a target column of LastUpdateDate but now its complaining about not having a key, which i do have configured.
A step by step best approach would be very helpful if anyone has been able to do this before.
Please post your initial design and tmap setting to enable others to help you better.
Also Post the error/ problem you receive.
There are a couple of ways of achieving this. It is dependent on a number of factors.
First What is the CSV being loaded into . Just a basic table a Dimensional table in a warehouse?
Do you need to process those records that are the same as the previous day or do you cut them from any further processing in the workflow?
How big is the CSV in terms of records ? This can affect how the UPSERT is done
Introduction to Talend Open Studio for Data Integration.
Practical steps to developing your data integration strategy.
Create systems and workflow to manage clean data ingestion and data transformation.