topic Re: Managing Kinesis streams for Replicate control tables in Qlik Replicate

Managing Kinesis streams for Replicate control tables

NakulanR — Tue, 04 Feb 2025 00:23:11 GMT

Hi Support,

When using Amazon Kinesis as a target endpoint in a Replicate task, there is a prerequisite to create a stream called 'attrep_apply_exceptions' so that the Apply Exceptions control table can be written to the target.

If the remaining control tables (attrep_status, attrep_history, etc.) are required, does a stream need to be created for each using the exact name of the control table? i.e. If you want to enable attrep_status, would you need to create a stream called 'attrep_status'?

Thanks,
Nak

Re: Managing Kinesis streams for Replicate control tables

SushilKumar — Tue, 04 Feb 2025 08:40:03 GMT

Hello @NakulanR

That's Correct You need to create the "Create a stream named attrep_apply_exceptionsbefore starting the replication task." its mandatory so that in case of an error it helps to troubleshoot the issue.

The reason Kenisis processes SQL in messages and without this we would not see what kind of SQLs break or cause the issue

The target streams must already exist before starting the replication task.
Create a stream named attrep_apply_exceptionsbefore starting the replication task. Note that if this stream does not exist, the task will always fail when it encounters a data error, regardless of the error handling policy.

Regards,
Sushil Kumar

Re: Managing Kinesis streams for Replicate control tables

NakulanR — Tue, 04 Feb 2025 21:33:21 GMT

Hi @SushilKumar ,

So if we want to enable the Replication Status table (attrep_status) for the task, would a stream named 'attrep_status' need to be created?

Regards,

Nak

Re: Managing Kinesis streams for Replicate control tables

NakulanR — Mon, 10 Feb 2025 23:14:09 GMT

Hi @SushilKumar,

We've found that creating a Kinesis stream named 'attrep_status' doesn't mean the attrep_status table gets put into that newly created stream. Instead it gets put into the main data stream. Is there a way in which the attrep_status table can be put into it's own stream?

Regards,

Nak

Re: Managing Kinesis streams for Replicate control tables

john_wang — Wed, 05 Mar 2025 01:28:37 GMT

Hello Nak, @NakulanR

In the Amazon Kinesis Data Streams target endpoint settings, there are two options for publishing data to:

Specific Stream – All data, including user data messages and Qlik Replicate internal control messages (e.g., the attrep_status message), is published to a single stream.
- Example: All messages are delivered to the stream APACKinesis.
Separate Stream for Each Table – Each table's data is published to a stream with the same name as the table by default. This behavior can be customized using $stream.
- The task’s status messages are always published to the attrep_status stream.

Hope this helps.

John.