I read a large volume of files (which have the same format and the same columns) in order to insert them in sql server.
Each time I have to insert(load) new lines of files (each evening for example).
My question is: Can i use INSERT IF Not Exist(action in data) to do the trick or I have to create a CDC.
What is the best solution?
If the solution is CDC how can I create it?
Thanks in advance.
Talend Changed Data Capture(CDC) feature will be available in talend subscription solution.
If you want to capture the changed data and only load these changed data into target table to achieve table sync, you can compare tables by using tMap.
The work flow should be:
Target DB-->tMap(make inner join on your input and set the "Catch lookup inner join reject" as true)-->output
The output will be the changed data.
Let us know if it is Ok with you.
Could you please explain to me more.
My job is like this
tfile_list----->tSystem----->tExtractDelimetedFields----->tMap----->tMsSqlOutput_1------>tMsSqlOutput_2(for reject rows)
So i should to connet tMsSqlOutput_1 with an other tMap to consider it as a source?