topic Re: Incremental data load from Oracle database to PostgreSQL database in Talend Studio

Incremental data load from Oracle database to PostgreSQL database

Anonymous — Sat, 16 Nov 2024 02:16:18 GMT

Hi,
I want to implement an Incremental data load using Talend. how to identify incremental data using the updated_date column in the oracle server & how to load incremental data into the PostgreSQL server using Talend. how to process incremental data CDC (change data capture) logic using Talend.

Re: Incremental data load from Oracle database to PostgreSQL database

manodwhb — Tue, 02 Jun 2020 05:26:59 GMT

@srinivask , check the bellow link.

https://community.talend.com/t5/Design-and-Development/Incremental-Load/td-p/157128

Re: Incremental data load from Oracle database to PostgreSQL database

Anonymous — Tue, 02 Jun 2020 05:54:35 GMT

Hi Manohar,

Thanks for the given details.

1) Approach 1: how to store the last run date? where do I need to store the last_update column value & job details? In my case source is OracleDB & target PostgreSQL. could you plz provide any example.

2) Approach 2: how to fetch the last run date (or) max(load_date) from the target database. how to use "context.lastrun" variable value? could you plz provide any example.

Re: Incremental data load from Oracle database to PostgreSQL database

manodwhb — Tue, 02 Jun 2020 06:12:07 GMT

@srinivask , 1) Last_Rundate that you can store on a file or table. which you need to take the current date from Oracle DB and nee to load.

2) what even you have loaded Last_Rundate table that you need to read using tDBinput and set as a global variable or assigned as context variable.

Re: Incremental data load from Oracle database to PostgreSQL database

Anonymous — Tue, 02 Jun 2020 08:02:00 GMT

The approach I understand. I'm new to Talend ETL. could you please provide an example.
1) we don't have access to create a new table or file in Oracle for last_rundate. I need to upload it to PostgreSQL only. How to fetch the current date from OracleDB?
2) How to create a context variable and fetch context variable value?

Re: Incremental data load from Oracle database to PostgreSQL database

manodwhb — Tue, 02 Jun 2020 08:31:19 GMT

@srinivask , create a table with last_rundate in postgresql. and read that table to get the last_rundate.

using the below design you wil get the lastrundate.

tDBInput(Oracle)-->tDBoutput(postgressql)

query should be like SELECT TO_CHAR(SYSDATE, 'DD-MON-YYYY HH:MIS') FROM dual