Incremental data load from Oracle database to PostgreSQL database

Highlighted
Five Stars

Incremental data load from Oracle database to PostgreSQL database

Hi,
I want to implement an Incremental data load using Talend. how to identify incremental data using the updated_date column in the oracle server & how to load incremental data into the PostgreSQL server using Talend.  how to process incremental data CDC (change data capture) logic using Talend. 

Highlighted
Moderator

Re: Incremental data load from Oracle database to PostgreSQL database

@srinivask , check the bellow link.

https://community.talend.com/t5/Design-and-Development/Incremental-Load/td-p/157128

 

Manohar B
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Highlighted
Five Stars

Re: Incremental data load from Oracle database to PostgreSQL database

Hi Manohar,

Thanks for the given details.

 

1) Approach 1: how to store the last run date? where do I need to store the last_update column value & job details? In my case source is OracleDB & target PostgreSQL.  could you plz provide any example. 

2)  Approach 2: how to fetch the last run date (or) max(load_date) from the target database. how to use "context.lastrun" variable value? could you plz provide any example.


Highlighted
Moderator

Re: Incremental data load from Oracle database to PostgreSQL database

@srinivask , 1) Last_Rundate that you can store on a file or table. which you need to take the current date from Oracle DB and nee to load.

 

2) what even you have loaded Last_Rundate table that you need to read using tDBinput and set as a global variable or assigned as context variable.

Manohar B
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Highlighted
Five Stars

Re: Incremental data load from Oracle database to PostgreSQL database

Hi 

The approach I understand.  I'm new to Talend ETL. could you please provide an example.  
1) we don't have access to create a new table or file in Oracle for last_rundate. I need to upload it to PostgreSQL only. How to fetch the current date from OracleDB?  
2) How to create a context variable and fetch context variable value? 

Highlighted
Moderator

Re: Incremental data load from Oracle database to PostgreSQL database

@srinivask , create a table with last_rundate in postgresql. and read that table to get the last_rundate.

 

using the below design you wil get the lastrundate.

 

tDBInput(Oracle)-->tDBoutput(postgressql)

 

query should be like SELECT TO_CHAR(SYSDATE, 'DD-MON-YYYY HH:MISmiley FrustratedS') FROM dual

Manohar B
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 3

Read about some useful Context Variable ideas

Blog