Using "Record Selection Condition" in Qlik Replicate.

[user2828]

Hi team,

I have a table which has huge amount of data, which does not have a primary key. What I was doing earlier is I will truncate and load data whenever needed so that I can get latest data. One reason being the table size is huge and loading this takes significant time. I have identified a column which can be used as a filter column. 
For eg: I have ran the task today(02-08-2024) and Im loading the table again on (10-08-2024). I can use that column and I can filter it using the column identified, and get records which are newer. I want to know how can I use the same logic using Qlik Replicates "Record Selection Condition"

Regards,
Sreehari 

[kng · Accepted solution]

Hello Team,

Good Day and thanks for reaching Qlik Community.

You can use the passthrough filter to filter out the required data from the source DB during full load. Please find the below article which might help you!!

Filter for last 90 days of data in Qlik Replicate - Qlik Community - 1880761

Regards,

shivananda

[user2828]

Thank you @kng 

[suvbin]

can we add both pass thru filter and record selection during replication for full load + cdc task

[kng]

Hello Team,

Record Selection Condition: Impacts both Full Load and CDC

The "Record Selection Condition" supports complex expressions include dynamic input values however the performance is not as good as "Filter Conditions", because in latter filter mode, the rows will be retrieved from source DB, transferred to Replicate machine then being filtered in Replicate server, prior to apply to target side database. It may take more resources (CPU, and especially IO and network)

Regards,

shivananda

[john_wang · Accepted solution]

Hello @suvbin 

Yes, these filters can be used individually or in combination to suit your requirements.

However, please note that the Full Load Passthru Filter only affects the Full Load phase, even if the task is configured to run both Full Load and Change Data Capture (CDC).

You can find a more detailed explanation in your referenced article: Passthru Filter

Hope this helps.

John.

[suvbin]

Thank you for quick reply.