Retrieving Number of Records Processed in Qlik Replicate

[Shaira]

Is there a way to retrieve the number of records processed in Qlik Replicate. Does Qlik provide access to this data via backend tables or logs? If so, where can we extract this information, and are there any restrictions on accessing it?

Thanks

[john_wang]

Hello @Shaira ,

Yes, we can control the number of records retrieved from the backend database tables. This is particularly relevant during the Full Load stage (initial load). The retrieval behavior can be managed using the Full Load Passthru Filter. For more details, refer to the summary in the article Fullload Passthru Filter.

During the CDC (Change Data Capture) stage, changes are retrieved from database transaction logs. While filters can be applied at this stage, they generally do not control the number of rows processed.

In a Qlik Replicate task, the CDC stage typically follows the Full Load stage automatically. This is the default setting.

Hope this helps.

John.

[Shaira]

Thanks for your response. You mentioned that the number of records retrieved can be controlled using the Full Load Passthru Filter and that CDC retrieves changes from transaction logs.

In our case:

We applied a record selection filter during processing.

The target is S3.

Given these conditions, is there a way to output the number of records processed into a local text file or another accessible format?

[DesmondWOO · Accepted solution]

Hi @Shaira ,

There is a control table called attrep_history that stores the number of records processed. However, this table is created on the target side. The count we generate is based on the changes that Replicate receives from the source side. For instance, if Replicate receives a change but the record is ignored due to a filter, it will still be counted.

Regards,
Desmond

[Dana_Baldwin]

Hi @Shaira 

Qlik Enterprise Manager has an Analytics feature that captures telemetry data regarding Qlik Replicate tasks, you might look into that: Analytics | Qlik Enterprise Manager Help

Thanks,

Dana

[john_wang]

Hello @Shaira ,

If you want to retrieve a specific rows number records, then you can use Fullload Passthru Filter, take note that:

1. it works for Full Load stage ONLY

2. The expression is various in different source databases. For example in Oracle you may use:

ROWNUM<20000

for PostgreSQL the syntax is like:

LIMIT 20000

you can press <Ctrl> key and at the same time double click the table to open the table setting window, paste the expression to the area of "Fullload Passthrough Filter" like:

Regards,

John.