I want to implement an Incremental data load using Talend. how to identify incremental data using the updated_date column in the oracle server & how to load incremental data into the PostgreSQL server using Talend. how to process incremental data CDC (change data capture) logic using Talend.
@srinivask , check the bellow link.
Thanks for the given details.
1) Approach 1: how to store the last run date? where do I need to store the last_update column value & job details? In my case source is OracleDB & target PostgreSQL. could you plz provide any example.
2) Approach 2: how to fetch the last run date (or) max(load_date) from the target database. how to use "context.lastrun" variable value? could you plz provide any example.
@srinivask , 1) Last_Rundate that you can store on a file or table. which you need to take the current date from Oracle DB and nee to load.
2) what even you have loaded Last_Rundate table that you need to read using tDBinput and set as a global variable or assigned as context variable.
The approach I understand. I'm new to Talend ETL. could you please provide an example.
1) we don't have access to create a new table or file in Oracle for last_rundate. I need to upload it to PostgreSQL only. How to fetch the current date from OracleDB?
2) How to create a context variable and fetch context variable value?
@srinivask , create a table with last_rundate in postgresql. and read that table to get the last_rundate.
using the below design you wil get the lastrundate.
query should be like SELECT TO_CHAR(SYSDATE, 'DD-MON-YYYY HH:MIS') FROM dual
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Read about some useful Context Variable ideas