My title isn't very clear.
Well, I have a job consists in :
1) reading a file in a directory
2) feed the database
3) Many tMap to check the "business rules" (in french : règles de gestion, i don't know to translate in english, sorry) with :
- If the data are correct, they are accepted and loaded in the database
- If the data have an error, there are rejected and loaded in the database with an error message
Actually, when there is an error, we must correct the error directly in the file and relaunch the job. (and delete the lines in the database before relaunch the job).
But now, i have a new need :
-No more to correct the error in the file BUT directly in the database.
But the matter, when i relaunch the job, Talend start all over again : Reread the file, reload the database, etc.
The correction of the error in the database is useless....
I don't know how to correct the error in the database and to start again where it stopped and not since the beginning.
(Ps : I am not the expert of Talend....Not yet....)
Thank you for your help!!!!
You need to keep a logging table for each step to do this. For example, the first part of the process is loading the data from a file. Once that is successfully completed for that filename (I assume the filename is unique....it kind of has to be), you can move to the second step, which I gather is validating the data. Once that is done, then you can load the data into your success table. So there are 3 steps. If you keep a table to log the success or failure of each step (maybe handled by an individual job?), then when your process fails, you can restart from the job(process) that did not finish successfully.
For example, your logging table might look something like this....
You would drive the process from this table and have conditional logic in your job (assuming a Main job with 3 child jobs....one for each process). So when you restart, if the file has previously failed, it will restart the process that failed and continue from there.
I use this sort of framework a lot.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Pick up some tips and tricks with Context Variables
Learn how media organizations have achieved success with Data Integration
Accelerate your data lake projects with an agile approach