Dynamic Data Filtering

DeusExData — Sat, 16 Nov 2024 01:15:10 GMT

Hi,

I receive a series of flat files daily from an upstream system. I'm aiming to build a job in Talend that can use metadata from an external data store (DB or CSV) and filter down these flat files based on rules that I set in the metadata in order to help with data quality. These columns and values to filter upon can change daily, so there's no pattern which column it needs to be. I've looked into tFilterRow and tMap but haven't been able to pass in expressions as variables in order to get the right functionality.

Are there any existing capabilities within Talend that can achieve this? Would I be able to get this functionality through custom Java?

Thanks for your time.

Re: Dynamic Data Filtering

Anonymous — Fri, 23 Oct 2020 08:52:23 GMT

Hello,

What does your complicated filtering expression look like? Are you trying to use it in the advanced mode of tfilterRow?

You can create a user routine that takes a value as a parameter.

Here is online documentation about: TalendHelpCenter: Creating an user routine and call it in a Job

Best regards

Sabrina

topic Re: Dynamic Data Filtering in Talend Studio

Dynamic Data Filtering

Re: Dynamic Data Filtering