topic Re: toraclecdc with dynamic schema in Talend Studio

toraclecdc with dynamic schema

ramnn — Tue, 05 Jun 2018 14:59:58 GMT

Hi,

My use case:

Using oracle cdc component in redo log mode to export data into CSV. Created a generic job instead of creating a separate job for each CDC table.

Get the table list (metadata) using Oracle Input component and extract the latest data (delta changes) for those table.

tOracleInput (Read config_table metadata) -> tFlowIterate (Iterate config_table) ==> tOracleCDC (Retrieve delta changes for those table records selected using tOracleInput component) -> tFileOutputDelimeted (Output the delta changes in CSV Output)

My Oracle metadata:

config_table data:
Table_name Active_flag
table1          "Y"
table2          "Y"

table1 schema: (CDC table)
id                    number
full_name       varchar2(50)
mobile            number

table2 schema: (CDC table)
id                          number
first_name            varchar2(30)
middle_name       varchar2(30)
last_name            varchar2(30)
prev_sur_name   varchar2(30)
address_ref         number

As you can see, my CDC tables schemas are different.

Configuration in tOracleCDC:
CDC mode: Log
Table using CDC: (String)globalMap.get("Table_name") (This Table_name attribute is passed from tOracleInputComponent)
Events to catch: Insert, Update, Delete (Selected all 3)
Schema using CDC: Built-In with attribute as "columnName" with "Dynamic" type (This is where the problem occurs. tOracleCDC doesn't support Dynamic type)

Please can someone help. is there any alternative solution to set the schema dynamically so that tOracleCDC component can use it.

Thanks in advance!

Re: toraclecdc with dynamic schema

Anonymous — Tue, 12 Jun 2018 10:53:30 GMT

Hello,

Here exists a new feature jira issue.

https://jira.talendforge.org/browse/TDI-17390

Best regards

Sabrina

Re: toraclecdc with dynamic schema

Anonymous — Wed, 13 Jun 2018 11:14:58 GMT

Hi,

There are 2 potential options to solve this 'industrialisation' requirement, namely:-

Add Dynamic schema support to existing tOracleCDC component
This will help create the opportunity for a single, generic job design template to build multiple CDC monitoring implementations
Create a JobScript template to read Oracle metadata and automate Job design
This will help create a repeatable process for automatically building the various individual Job designs to support CDC monitoring implementations

Both would typically require custom development / delivery work either from Talend Professional Services or Talend Engineering directly in my view.

Regards,

Darren