Missing few transactions ideally deletes

[nareshkumar]

We had Log streaming Task # source is RDS postgres  where as we had 2 different targets  1 is EDB  postgres and other is RDS aurora ..

We had Source table having 54 columns and target table having only 52 columns .. we had removed 2 columns in the Tasks...

Do we have any limitations ?

Some sample errors#

Stream component 'st_0_PG_TCDP_MDHP_Tgt' terminated
Stream component failed at subtask 0, component st_0_PG_TCDP_MDHP_Tgt
Error executing data handler
Failed to get update statement for table parts_master_raw.parts_maintenance_cd, stream position 345;638516567649827200;20240418151639010679B|00007CB3/8E010800.2.00007CB3/8E010800
Failed to prepare statement 'UPDATE "parts_master_raw"."parts_maintenance_cd"

Stream component 'st_0_PG_TCDP_MDHP_Tgt' terminated
Stream component failed at subtask 0, component st_0_PG_TCDP_MDHP_Tgt
Error executing data handler
Endpoint is disconnected
Connection error
Failed to execute insert statement, stream position is 331;638515507882112120;20240418151639010679B|00007C97/64B259D0.2.00007C97/64B259D0
RetCode: SQL_ERROR SqlState: 08S01 NativeError: 35 Message: The connection has been lost;
The connection has been lost
Failed (retcode -1) to execute statement: 'INSERT INTO

[DesmondWOO]

Hi @nareshkumar ,

• Does this problem occur on all records or few records only?
• If you don't remove those 2 columns from the task, does this problem occur?
• Are the removed columns part of the primary key columns or the unique index columns?
  
  

For your case, it would be better to have your table DDL and verbose task log for investigation. I recommend submitting a support ticket.

Regards,
Desmond

[Geetha]

Hi @nareshkumar ,

In addition to @DesmondWOO 's response, Provide further clarification about below statement while submitting a case: "We had Source table having 54 columns and target table having only 52 columns .. we had removed 2 columns in the Tasks"

Questions:

1. Are 2 columns not replicated to target (ever!) out of 54 selected from Source?
2. Did you notice this being auto deleted from the target when the selection is altered on task level?
3. After removing the columns did you perform reload? what is the task settings?

Best regards,
Geetha