topic Re: How to set up CDC with two oracle databases in different servers ? in Talend Studio

How to set up CDC with two oracle databases in different servers ?

Anonymous — Wed, 31 Oct 2018 09:19:45 GMT

Hello, I have a mission to charge data from an oracle source into an oracle target and to keep that data in sync, considering that the source and target databases are in different servers.

For that purpose i chose Talend but i need confirmation that this can be done with this Talend.

can you please help me !

Re: How to set up CDC with two oracle databases in different servers ?

_Laurent_Conti_ — Wed, 31 Oct 2018 16:22:04 GMT

hello,

Yes of course. talend is made for that kind of jobs.

Talend (Data Integration) can extract from a source (database (oracle, mysql, ms sql server), web Services, files...), transform data (or clean, map ... ) and load the date to a target (database, file...).

During the transformation phase, you can check for new/update or delete to keep sync between source and target.

In case of huge volume of data, you can use CDC features based on triggers on source datasource but THIS IS ONLY AVALAIBLE with the subscription version of Talend.

Regards

_Laurent_

Re: How to set up CDC with two oracle databases in different servers ?

Anonymous — Fri, 02 Nov 2018 08:50:49 GMT

In my case i will deal with a huge amount of data, so i will need the CDC feature to apply between two distant databases indifferent servers. but i found this line wich confused me : "The CDC feature works only with the same database system running on the same server."

Re: How to set up CDC with two oracle databases in different servers ?

Anonymous — Fri, 02 Nov 2018 09:26:01 GMT

Hi,

Talend CDC is having two distinct stages. Stage 1 will capture all the records which are identified as change records in a separate schema under same database. Stage 2 will extract the CDC data based on the record identifier present in the interim table and will pass the data to downstream system (in your case different database in different server).

CDC Trigger mode

For details on Talend CDC, please refer the link below.

https://help.talend.com/reader/4UeRbZs9GU5n8b9nm3hUrQ/tucZCvCXQzs9sRUsmEJfsA

If the answer has helped you, could you please mark the topic as resolved? Kudos are also welcome 🙂

Warm Regards,

Nikhil Thampi

Re: How to set up CDC with two oracle databases in different servers ?

Anonymous — Fri, 02 Nov 2018 09:35:34 GMT

@nthampi Thank you for your reply, i m really gratefull but i need to ask 2 more questions :

1 - with two different servers for the source and the target batabases, where to install Talend ?

2- Does the CDC feature will have impact on the source database ?

Re: How to set up CDC with two oracle databases in different servers ?

Anonymous — Fri, 02 Nov 2018 09:48:33 GMT

Hi,

Please refer my updates below.

1 - with two different servers for the source and the target databases, where to install Talend ?

I would always recommend to install Talend on a different server. You should not install Talend in either source or target systems. Below link contain the reference architecture for Talend and it is also having the details of the Talend distribution in development and higher environments (which differ from Dev environment).

https://talendpnp.github.io/architecture-center/reference-architecture/summer-2018-7.0/index

2- Does the CDC feature will have impact on the source database ?

Yes. Since we are going to bring additional work load in the form of CDC extraction, there will be additional consumption of system resources. My personal view is that you should go for Redo log mode compared to Trigger mode as it is less resource intensive. However, the right methodology in promoting a CDC to production will be to do a proper performance testing and fine tune the resources at DB level and Talend to bring the optimum throughput.

If the answer has helped you, could you please mark the topic as resolved? Kudos are also welcome 🙂

Warm Regards,

Nikhil Thampi