How do you know if Passthru filter is working?

[JGRFCU]

I have full load task with an Oracle source going to a Snowflake target, but I only want to load the last 7 days of data.  I have updated the config file with  "enable_passthrough_filter": true and my filter condition is EFFDATE >= trunc(CURRENT_DATE - 7 ).  However, when I run the task the estimated count of records to load is the entire table (400 million+ rows), as if it is not being filtered.  When I query the record count from the source using the same condition, it's only about 3 million rows.   

How do I know that the passthru filter is working?

[Michael_Litz · Accepted solution]

Hi @JGRFCU 

The estimated count is just that - based on stats in the table. If you did run the where clause and got the real count then that will work in the task.

Once the task completes you should see the completed row count as equal to your direct query with the "filter" where clause.

Thanks,

MIchael

[JGRFCU]

Thank you Michael!  I let the task run to completion and got the expected number of filtered rows.

[Michael_Litz]

Hi @JGRFCU ,

That is great. Good job enabling and using the Pass Through feature.

One other thing I though of is whether you will be doing CDC with this table. If so you need to add the equivalent filter on the record selection to filter any changes that come through - other wise you may get some record not found errors if source records are updated.

Thanks,
Michael

[DesmondWOO]

Hi @JGRFCU ,

You may also enable verbose logging on SOURCE_UNLOAD. The whole SQL statement will be printed in the task log.

Regards,
Desmond