ETL does not delete records that does not exist any more !!!

One Star

ETL does not delete records that does not exist any more !!!

Hi,
I have a dimension table of customers in my data warehouse, data's table are loaded with Talend ETL. With the option 'Update or insert', i can just load new records of new customers or update the existing ones, but i can not delete customers who do not exist any more (what is so importing for my project). what shoud i do in this case ? i need those 3 option toguether : insert or update or delete, that means in every job execution, i only want to find in my data warehouse tables the new output of ETL.
Thank you.
Sixteen Stars

Re: ETL does not delete records that does not exist any more !!!

The delete needs to be driven by ids that you know need to be deleted. Simply not having an id in the dataset, will not provide this on its own. You need to do a comparison between you source and target table. Where an id exists in your target but not in your source, you need to delete that record from your target. 
This can be implemented after your insert/update. Once that is done, create a recordset by joining the source and target tables. Return only rows which appear in your target and not your source. Then use the t{Database}Output with the "Action on data" set to "Delete". This will remove your records that no longer exist.
One Star

Re: ETL does not delete records that does not exist any more !!!

Thank you for your answer, but my jobs' execution will be sheduled, that means i should no longuer execute jobs manually, how can i apply your solution in this situation ? 
Five Stars

Re: ETL does not delete records that does not exist any more !!!

I think you're mis-understanding your issue and the proposed solution.
Setting aside Talend. how do you know what is to be deleted?
Will your source system tell you?
Do you need to perform some sort of comparison, as suggested, to determine the Ids of records to delete?
It doesn't matter whether or not your Job is to be scheduled, it needs a source of row IDs for deletion.