add filter to CDC

[david_lange]

Looking for recommendation, how to do an incremental load of a large table from oracle to snowflake.

We are using qlik replicate version 2022.11.0.208

I would like to performe a one-time full load with daily incremental.

The incremental load would use a WHERE clause to select changed rows.

Select'ing only rows that have changed or are new within the last day. I am using the Oracle coalesce function to return the first not null date, then checking if its new within the last day.

      where coalesce(last_updated, created_date) > sysdate -1

I am struggling to figure out how to enter this where clause in the:  "Table Settings" | "Filter" option on QLIK replicate.

Please advise on the best approach.

Thanks

-Dave

[Dana_Baldwin · Accepted solution]

Hi @David ,

Here is a link to a knowledge article on how to use filtering in Qlik Replicate. The pass through filter is for full load only, but the record selection method also works with CDC:

Filter for last 90 days of data in Qlik Replicate - Qlik Community - 1880761

Another option you might consider if it meets your use case, is to simply resume the task once per day when you want the new records applied. Qlik Replicate will read the source redo / archive logs and apply any updates/inserts/deletes since it last ran, this is the default behavior when change processing is enabled. One downside to this approach is it will apply changes until all are applied or until you stop the task again - so if you want it to only apply changes up to a certain point then this won't work for you.

Thanks,

Dana