Extracting data from database mixed with hard-coded values using file containing the metadata

menorah84 — Fri, 07 Feb 2020 08:59:24 GMT

Our database connection is using JDBC (tDBInput). We have files containing metadata of the table/s we want to get data from. Something like this:

columnName	isIncluded	isHardCoded	HardCodeValue
col1	Y	N
col2	N	N
col3	Y	N
col4	Y	Y	"val4"
col5	Y	Y	"val5"

If isIncluded="Y", then the values of that column will be part of the resulting dataset (those rows in the metadata CSV file is ignored).

If isHardCoded="N", that column is included in the SELECT statement composed (in the Query section of the tDBInput).

If isHardCoded="Y", then the value of that column is not extracted from the database, but rather the columnNames together with the corresponding HardCodeValues are appended "horizontally" to the result of the query from the database.

Example, if the tDBInput outputs 100 rows (containing values for col1 and col3 only), each of these rows will have additional columns, col4 and col5, with values "val4" and "val5" respectively.

Based on my reading of some of the community posts (I am new to Talend), I can use tWriteDynamicFields, tSetDynamicSchema, tJavaRow, tDenormalize and/or combinations, but it is still hazy in my understanding. I have used tMap, but I can't put dynamic column name in the output.

So, what's the best solution / design for this? Here is the design of my initial experimentation (plus my ideas):

Re: Extracting data from database mixed with hard-coded values using file containing the metadata

Anonymous — Fri, 07 Feb 2020 14:18:48 GMT

Hi,

your problem is not clear to me,

Can't you just join the 2 flows?

Re: Extracting data from database mixed with hard-coded values using file containing the metadata

menorah84 — Mon, 10 Feb 2020 02:17:59 GMT

I'm not sure, I'm new to Talend, so my solution is like this. If you have one flow solution that is better, that would be great.

topic Re: Extracting data from database mixed with hard-coded values using file containing the metadata in Talend Studio

Extracting data from database mixed with hard-coded values using file containing the metadata

Re: Extracting data from database mixed with hard-coded values using file containing the metadata

Re: Extracting data from database mixed with hard-coded values using file containing the metadata