how to filter out columns with certain strings in the column name?

[Anonymous]

I've set up a File delimited Metadata, and I want to be able to automatically filter out any columns that have certain strings in the name, specifically "schedule" or "hist." And I cannot, for the life of me, figure out how to do this.
Its likely that the columns in this metadata will change, and I don't want to have to manually go in and remove these columns any time the metadata is updated.
I tried to use tFilterColumns to do this (since tFilterRow works for row filtering) but it just seems like a simpler version of tMap to me.

[Anonymous]

I surely don't know of any way to accomplish your task in the present environment. But, then again, I'm no "guru" when it comes to Talend. I've only been using it for a couple months. Perhaps there is a way - but it is beyond my means at present. Maybe one of the folks from Talend could offer some sort of solution. Sorry.

[Anonymous]

I am currently working on an ETL solution using Talend
The aim of this solution is to load a custom csv file (with randomed columns) into a Talend job with SQL database output (always the same table).
SMaz is right, Talend cannot load dynamic metadata while processing the job.
To go round this problem, I found this solution :
- First, you have to make sure that Talend is processing a file with a fixed metadata schema.
In your case, it could be all possible columns
- A custom application (web UI with ASP .NET in my case) load the input file and allow the user to manually map the input fields with the fixed output fields. At the end of this operation, a csv file is generated. This file contains always the same metadata schema with mapped data. This file is goeing to be treated by Talend job
In your case, this custom app could analyse the input file headers and map the input columns functions of your input columns name without any user intervention
Maybe it doesn't fit to you, but with this solution I can allow a user to load a document with randow columns into a efficient Talend job doeing many things.
I don't know if there is a more simple way to do this...

[Anonymous]

thats a possibility, but the # of possible columns is kind of enormous, 11648 at this point in time, and is subject to change at the data provider's whim

[Anonymous]

If you can get a database containing all possible columns list, you may write a program to generate the file and to enhance it when database is growing...
But it'll be a quite complex solution... hope for you that TOS's team will find something to help better ^^

[Anonymous]

I have a similar question. Is there any way to obtain a list or other enumeration of the names of the columns that are being provided in the result set?
For example, take the simple tRowGenerator component. I use it to generate 30 rows with a simple schema of 5 column. The rows coming from it are called row_1 and are piped into a tJavaFlex component that reads them. The tJavaFlex component uses hard-code references such as row_1.columnA and row_1.columnB. What I need is run-time access to the schema of row_1. Using a runtime argument, I need to determine whether or not to send the column values to the log.
Any suggestions?

[Anonymous]

Hi mquinz
This topic is rather old, can you please report a new topic for your question and give us an example to explain your request.
Thank you!
Shong