topic Reading delimited file with dynamic column in Talend Studio

Reading delimited file with dynamic column

AManohar — Fri, 15 Nov 2024 22:09:34 GMT

Hi,

I have a requirement to read a file with dynamic columns and load the mandatory fields into DB.

The dynamic columns are in the middle of the file.

But, based on the config we can figure out how many additional fields are added in this file.

sample:

col1|col2|col3|col4.1|col4.2|col4.3|col5|col6

I want to read this and store it as

col1|col2|col3|"

col4.1;col4.2;col4.3

"|col5|col6

Based on a config I would know there are 3 fields for col4, but some files might have 6 or even 1. That is the variable column.

How to read and process such files in talend.

Thank you.

Regards,

Anitha

Re: Reading delimited file with dynamic column

Anonymous — Mon, 30 Jan 2023 06:20:19 GMT

I think you need dynamic schema functionality which is only on subscription products. For more details, please refer to this documentation page.

https://help.talend.com/r/en-US/8.0/dynamic-schema

Access the schema properties and filter the columns based on the column name at runtime.

Regards

Shong

Re: Reading delimited file with dynamic column

AManohar — Tue, 31 Jan 2023 02:35:24 GMT

Hi Shong,

But in our case we dont have a column name.

Its just a pipe separated file.

eg : Name|language1|language2|language3|age

in this case, some file will have just one language, but some might have many. so we still need to store those additional languages in separate column.

But, name,language1 and age should be stored in a main table.

Re: Reading delimited file with dynamic column

Anonymous — Tue, 31 Jan 2023 03:49:11 GMT

Name|language1|language2|language3|age

this line is not the column name? There must have a header line when using a dynamic schema feature.