Skip to main content
Announcements
UPGRADE ADVISORY for Qlik Replicate 2024.5: Read More
cancel
Showing results for 
Search instead for 
Did you mean: 
JGRFCU
Contributor II
Contributor II

How do you know if Passthru filter is working?

I have full load task with an Oracle source going to a Snowflake target, but I only want to load the last 7 days of data.  I have updated the config file with  "enable_passthrough_filter": true and my filter condition is EFFDATE >= trunc(CURRENT_DATE - 7 ).  However, when I run the task the estimated count of records to load is the entire table (400 million+ rows), as if it is not being filtered.  When I query the record count from the source using the same condition, it's only about 3 million rows.   

How do I know that the passthru filter is working?

Labels (2)
1 Solution

Accepted Solutions
Michael_Litz
Support
Support

Hi @JGRFCU 

The estimated count is just that - based on stats in the table. If you did run the where clause and got the real count then that will work in the task.

Once the task completes you should see the completed row count as equal to your direct query with the "filter" where clause.

Thanks,

MIchael

View solution in original post

4 Replies
Michael_Litz
Support
Support

Hi @JGRFCU 

The estimated count is just that - based on stats in the table. If you did run the where clause and got the real count then that will work in the task.

Once the task completes you should see the completed row count as equal to your direct query with the "filter" where clause.

Thanks,

MIchael

JGRFCU
Contributor II
Contributor II
Author

Thank you Michael!  I let the task run to completion and got the expected number of filtered rows.

Michael_Litz
Support
Support

Hi @JGRFCU ,

 

That is great. Good job enabling and using the Pass Through feature.

One other thing I though of is whether you will be doing CDC with this table. If so you need to add the equivalent filter on the record selection to filter any changes that come through - other wise you may get some record not found errors if source records are updated.

Thanks,
Michael

DesmondWOO
Support
Support

Hi @JGRFCU ,

You may also enable verbose logging on SOURCE_UNLOAD. The whole SQL statement will be printed in the task log.

Regards,
Desmond

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!