topic Re: How do you know if Passthru filter is working? in Qlik Replicate

How do you know if Passthru filter is working?

JGRFCU — Wed, 07 Feb 2024 22:40:06 GMT

I have full load task with an Oracle source going to a Snowflake target, but I only want to load the last 7 days of data. I have updated the config file with "enable_passthrough_filter": true and my filter condition is EFFDATE >= trunc(CURRENT_DATE - 7 ). However, when I run the task the estimated count of records to load is the entire table (400 million+ rows), as if it is not being filtered. When I query the record count from the source using the same condition, it's only about 3 million rows.

How do I know that the passthru filter is working?

Re: How do you know if Passthru filter is working?

Michael_Litz — Wed, 07 Feb 2024 22:47:05 GMT

Hi @JGRFCU

The estimated count is just that - based on stats in the table. If you did run the where clause and got the real count then that will work in the task.

Once the task completes you should see the completed row count as equal to your direct query with the "filter" where clause.

Thanks,

MIchael

Re: How do you know if Passthru filter is working?

JGRFCU — Wed, 07 Feb 2024 23:02:26 GMT

Thank you Michael! I let the task run to completion and got the expected number of filtered rows.

Re: How do you know if Passthru filter is working?

Michael_Litz — Wed, 07 Feb 2024 23:45:33 GMT

Hi @JGRFCU ,

That is great. Good job enabling and using the Pass Through feature.

One other thing I though of is whether you will be doing CDC with this table. If so you need to add the equivalent filter on the record selection to filter any changes that come through - other wise you may get some record not found errors if source records are updated.

Thanks,
Michael

Re: How do you know if Passthru filter is working?

DesmondWOO — Thu, 08 Feb 2024 09:20:07 GMT

Hi @JGRFCU ,

You may also enable verbose logging on SOURCE_UNLOAD. The whole SQL statement will be printed in the task log.

Regards,
Desmond