Dynamic Data Filtering

[DeusExData]

Hi,

I receive a series of flat files daily from an upstream system. I'm aiming to build a job in Talend that can use metadata from an external data store (DB or CSV) and filter down these flat files based on rules that I set in the metadata in order to help with data quality. These columns and values to filter upon can change daily, so there's no pattern which column it needs to be. I've looked into tFilterRow and tMap but haven't been able to pass in expressions as variables in order to get the right functionality.

Are there any existing capabilities within Talend that can achieve this? Would I be able to get this functionality through custom Java?

Thanks for your time.

[Anonymous]

Hello,

What does your complicated filtering expression look like? Are you trying to use it in the advanced mode of tfilterRow?

You can create a user routine that takes a value as a parameter.

Here is online documentation about: TalendHelpCenter: Creating an user routine and call it in a Job

Best regards

Sabrina