Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
How can I filter a number of records from each table when replicating a database? For example, the database has 10 thousand tables and each table has 1 million records. I would like to replicate only 100 records from each table.
Here are a couple of knowledge articles on how to filter data - but I'm not aware of a single place you can set up once to filter all the tables in your task - these would be per table:
Filter for last 90 days of data in Qlik Replicate - Qlik Community - 1880761
Filter for last 90 days of data in Qlik Replicate ... - Qlik Community - 1980009
Is this a one-time use case or a need you regularly encounter? You might consider entering a feature request here: https://community.qlik.com/t5/Ideas/idb-p/qlik-ideas
Thanks,
Dana
You didn't mention the source endpoint type (nor target, not Replicate version or platform... Amateur!)
The only reasonable way to do this is using a 'pass thru filter' with "rownum < 101" or "Limit to 100 rows" or whatever your source endpoint requires - if it even supports that option. Search this forum for articles. Using that option puts the onus on the source, and only the required rows will be transferred and no work to be done by Replicate.
First you'll need to enable passthru and restart the service. Edit ...\Replicate\bin\repctl.cfg and add or modify line line to read : "enable_passthrough_filter": true,
Next you'll need to provide the filter for each table. I could not find a global transformation to provide this.
Start with a single table. control-click the table name to open it and get the passthru section under filters. Export the task before and after and look at the difference. You find you need to provide the following for each table under the "manipulations": header :
{ "name": "XXX.T123", "table_manipulation": {"owner": "XXX", "name": "T123", "fl_passthru_filter": "rownum < 101" },
The "name" must be the real schema.tablename and must match the owner and name in the sub-clause.
For the "10 thousand tables" mentioned this will be a challenge. Clearly you'll need to script the JSON file with some tool like SQL/Perl/Awk/Powershell. (my personal order of preference for this).
Of course you'd want the script to also generate your thousands of include lines under "explicit_included_tables" each looking like:
{ "owner": "XXX", "name": "T123", "orig_db_id":0 },
Have Fun!
Hein.
Besides @Heinvandenheuvel and @Dana_Baldwin ,
Maybe creating views for these tables (and adding filters in these views), then run Full Load ONLY task(s) in Qlik Replicate. So far all definitions can be done in scripts.
Regards,
John.