I read a large volume of files (which have the same format and the same columns) in order to insert them in sql server.
Each time I have to insert(load) new lines of files (each evening for example).
My question is: Can i use INSERT IF Not Exist(action in data) to do the trick or I have to create a CDC.
What is the best solution?
If the solution is CDC how can I create it?
Thanks in advance.
Talend Changed Data Capture(CDC) feature will be available in talend subscription solution.
If you want to capture the changed data and only load these changed data into target table to achieve table sync, you can compare tables by using tMap.
The work flow should be:
Target DB-->tMap(make inner join on your input and set the "Catch lookup inner join reject" as true)-->output
The output will be the changed data.
Let us know if it is Ok with you.
Could you please explain to me more.
My job is like this
tfile_list----->tSystem----->tExtractDelimetedFields----->tMap----->tMsSqlOutput_1------>tMsSqlOutput_2(for reject rows)
So i should to connet tMsSqlOutput_1 with an other tMap to consider it as a source?
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Pick up some tips and tricks with Context Variables
Learn how media organizations have achieved success with Data Integration
Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend