Change Data Capture, how it works in Talend?

[Anonymous]

I have read Talend has change data capture. Can you please describe in details:
1) how it works
2) do you support Oracle in particular and may be a DB2 as a source?
3) Is it a log-based, meaning read transaction logs or have a trigger based mechanism
The reference to documentation would be helpful
Thanks,
Mike

[Anonymous]

Hi shong,
could you pls tell me how CDC(change data capture) works in talend?
Thanks in advance
Bappi

Hello

Do you know which SQL Sever version is needed to work with Talend CDC?

All version should be supported, if you have any problem, please report a bug and it is necessary for us to fix it.
Best regards
Shong

[Anonymous]

Hi shong,
could you pls tell me how CDC(change data capture) works in talend?

See gatigossou's response, he have explained how does CDC work in Talend, is it not enough?

[_AnonymousUser]

please refer the link http://www.talendforge.org/tutorials/tutorial.php?idTuto=41
thanks

[Anonymous]

Hello,
1) How it works :
Change Data Capture (CDC) is based on Publisher/Subscriber principle. The publisher captures the change data and makes it available to the subscribers. The subscribers can use the change data obtained from the publisher only once. To be able to use a given number of times the same change data, you should use as many differents subscribers. The informations about source tables from which the change data needs to be captured and the subscribers who can access to the change data are stored in a TSUBSCRIBER table. The TCDC_source_table contains the id of each change data. The triggers TCDC_TG_source_table_I, TCDC_TG_source_table_U and TCDC_TG_source_table_D are created to capture each Insert, Update and Delete on source_table. Each change row capture by the triggers is inserted into TCDC_source_table with the flag TALEND_CDC_STATE='0'. Before consuming the row, this flag is set to TALEND_CDC_STATE='1' and a view TCDC_VIEW_source_table is create or replace to fetch rows from source table and only rows with TALEND_CDC_STATE='1'. This allows you to lock the lines to consum and to delete it after consuming where SUBSCRIBER = the subscriber you have to consum the change data.
2) do you support Oracle in particular and may be a DB2 as a source?
CDC is available in Java for Oracle, MySQL, DB2, PostgreSQL, Sybase, and MS SQL Server databases.
3) Is it a log-based, meaning read transaction logs or have a trigger based mechanism
CDC have a trigger based mechanism
Thanks,

Does Oracle 11g Standard edition support log based CDC ?

[Anonymous]

Hi,
I'm using Talend Entreprise Edition 5.4.2. For my test i'm trying to import all data from my source DB to my target DB both in PostgreSQL. I can import one table an a time.
1) My DB contain 3651 tables. It's a very long job to do one table an a time.
Is their a way to import the whole database ?
2) Next I need to import each 24 the changed Data from my source to my target.
I need to monitor all my tables.
How can i do it ?
Thx.
Christina.

[Anonymous]

Hi
1)
If the source database and target database are both PostgreSQL, why don't you use the native import/export command or a special PostgreSQL client tool to migrate the whole database?
2)
For CDC , you need to configure each table one by one, there is no a way to configure CDC for all tables at a time.
Shong

[Anonymous]

Hi Christina,
Have you tried jdbc connection for postgress..
JDBC provides a component tJDBCTablesList and tJDBCColumnList
- use this and iterate through all tables and columns for respective table in database
- Create table and insert data into it using select * and insert into query.. using tJDBCRow component
try this approach
This could be received through master table as well.
Vaibhav

[Anonymous]

Thx Shong and Vaibhav.
1) My real DB is Caché Intersystem DB. I'm testing several Talend feature before working with my real DB.
The source DB is on Caché Intersystem, my DW will be on PostgreSQL.
I will test JDBC with TablesList and ColumnList...
Thx for ur support.

[_AnonymousUser]

Hello

Do you know which SQL Sever version is needed to work with Talend CDC?

All version should be supported, if you have any problem, please report a bug and it is necessary for us to fix it.
Best regards
Shong

Hi Shong,
we are having SQL Server 2012 and Talend 5.5.1 Enterprise Data Integration(Subscribed version) which is not supporting tMSSqlCDC for Change Data Capture where tOracleCDC is supporting.
would you please let me konw in which version of sql and Talend will support SQL CDC?
Regards,
Chendra Sekhar G.

[Anonymous]

Hi Chendra Sekhar G
Please have a look at document about TalendHelpCenteratabase support for CDC.
Best regards
Sabrina