Skip to main content
Announcements
UPGRADE ADVISORY for Qlik Replicate 2024.5: Read More
cancel
Showing results for 
Search instead for 
Did you mean: 
Eduardo_Carrijo
Contributor II
Contributor II

Filter the number of records in each table

How can I filter a number of records from each table when replicating a database? For example, the database has 10 thousand tables and each table has 1 million records. I would like to replicate only 100 records from each table.

Labels (1)
3 Replies
Dana_Baldwin
Support
Support

Hi @Eduardo_Carrijo 

Here are a couple of knowledge articles on how to filter data - but I'm not aware of a single place you can set up once to filter all the tables in your task - these would be per table:

Filter for last 90 days of data in Qlik Replicate - Qlik Community - 1880761

Filter for last 90 days of data in Qlik Replicate ... - Qlik Community - 1980009

Is this a one-time use case or a need you regularly encounter? You might consider entering a feature request here: https://community.qlik.com/t5/Ideas/idb-p/qlik-ideas

Thanks,

Dana

Heinvandenheuvel
Specialist III
Specialist III

You didn't mention the source endpoint type (nor target, not Replicate version or platform... Amateur!)

The only reasonable way to do this is using a 'pass thru filter' with "rownum < 101" or "Limit to 100 rows" or whatever your source endpoint requires - if it even supports that option. Search this forum for articles. Using that option puts the onus on the source, and only the required rows will be transferred and no work to be done by Replicate.

First you'll need to enable passthru and restart the service. Edit  ...\Replicate\bin\repctl.cfg and add or modify line line to read :   "enable_passthrough_filter": true,

Next you'll need to provide the filter for each table. I could not find a global transformation to provide this. 

Start with a single table. control-click the table name to open it and get the passthru section under filters. Export the task before and after and look at the difference. You find you need to provide the following for each table under the "manipulations": header :

 

 

 { "name":	"XXX.T123", "table_manipulation": {"owner": "XXX", "name": "T123", "fl_passthru_filter": "rownum < 101" },

 

 

The "name" must be the real schema.tablename and must match the owner and name in the sub-clause.

For the "10 thousand tables" mentioned this will be a challenge. Clearly you'll need to script the JSON file with some tool like SQL/Perl/Awk/Powershell. (my personal order of preference for this).

Of course you'd want the script to also generate your thousands of include lines under "explicit_included_tables" each looking like: 

{ "owner": "XXX", "name": "T123", "orig_db_id":0 },

Have Fun!

Hein.

 

 

 

 

 

john_wang
Support
Support

Besides @Heinvandenheuvel and @Dana_Baldwin ,

Maybe creating views for these tables (and adding filters in these views), then run Full Load ONLY task(s) in Qlik Replicate. So far all definitions can be done in scripts.

Regards,
John.

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!