Managing Kinesis streams for Replicate control tables

[NakulanR]

Hi Support,

When using Amazon Kinesis as a target endpoint in a Replicate task, there is a prerequisite to create a stream called 'attrep_apply_exceptions' so that the Apply Exceptions control table can be written to the target.

If the remaining control tables (attrep_status, attrep_history, etc.) are required, does a stream need to be created for each using the exact name of the control table? i.e. If you want to enable attrep_status, would you need to create a stream called 'attrep_status'?

Thanks,
Nak

[SushilKumar]

Hello @NakulanR 

That's Correct You need to create the "Create a stream named attrep_apply_exceptionsbefore starting the replication task."  its mandatory so that in case of an error it helps to troubleshoot the issue.

The reason Kenisis processes SQL in messages and without this we would not see what kind of SQLs break or cause the issue 

• The target streams must already exist before starting the replication task.

• Create a stream named attrep_apply_exceptionsbefore starting the replication task. Note that if this stream does not exist, the task will always fail when it encounters a data error, regardless of the error handling policy.

Regards,
Sushil Kumar

[NakulanR]

Hi @SushilKumar ,

So if we want to enable the Replication Status table (attrep_status) for the task, would a stream named 'attrep_status' need to be created?

Regards,

Nak

[NakulanR]

Hi @SushilKumar,

We've found that creating a Kinesis stream named 'attrep_status' doesn't mean the attrep_status table gets put into that newly created stream. Instead it gets put into the main data stream. Is there a way in which the attrep_status table can be put into it's own stream?

Regards,

Nak

[john_wang · Accepted solution]

Hello Nak, @NakulanR 

In the Amazon Kinesis Data Streams target endpoint settings, there are two options for publishing data to:

1. Specific Stream – All data, including user data messages and Qlik Replicate internal control messages (e.g., the attrep_status message), is published to a single stream.

   • Example: All messages are delivered to the stream APACKinesis.
   • 

2. Separate Stream for Each Table – Each table's data is published to a stream with the same name as the table by default. This behavior can be customized using $stream.

   • The task’s status messages are always published to the attrep_status stream.

Hope this helps.

John.