Incremental data load

One Star

Incremental data load

Hi all,
I am new using Talend DI and I have to made an incremental load from one Oracle Database to another Oracle Database. I have to compare the time of the last execution with a column call Last Modified Date to update the rows modified or insert the new ones.
Can anyone help me with this?  I just start with talend and i feel a bit lost with this.
Thanks very much!!
Seventeen Stars

Re: Incremental data load

There is no build in method in Talend.
The way you have to work goes like this:
Create a context variable to keep an Date object (call it start_date) and set it to a very early date (which does not exists in your data)
Tweak the query of the database input to restrict the data above this date.
Keep track of the highest last modified date in your data within the flow
Memories the youngest last modified date and write it e.g. in a control table or file
Next job run read the last last modified date and set the value in the mentioned context variable (mentioned above).
You can use a component suite which supports this method very well:
https://exchange.talend.com/#marketplaceproductoverview:gallery=marketplace%252F1&pi=marketplace%252...
 
One Star

Re: Incremental data load

Thanks a lot for your answer, I understand the most part and I will try to follow your steps.
Smiley Happy
One Star

Re: Incremental data load

Hi, 
Can you please provide the screen shots to implement incremental load.

Regards
Anand V
Seventeen Stars

Re: Incremental data load

Take a look at the component documentation I mentioned in my post.
One Star

Re: Incremental data load

Hi,
Looking for incremental load approach with some screen shots. Can any one help me please ?
Thanks,
Rama.