topic Re: Reading CSV files with different schemas in Talend Studio

Reading CSV files with different schemas

RMotta2408 — Fri, 15 Nov 2024 21:46:43 GMT

Hello everyone,

I want to develop a Job to read CSV files from a folder and display the results in the same order always.

The problem is, there's no fixed schema for these files.

I mean, the files may come with different schemas:

Column_A, Column_B, Column_C, Column_D
Column_B Column_D, Column_A, Column_C
Column_D, Column_B, Column_C, Column_A

... any possible combination.

My question is: can the Job be dynamic enough so that, no matter the order of the columns, the files always gets read and the results displayed in the right order (A, B, C, D)?

Thank you so much,

Rui

Re: Reading CSV files with different schemas

Anonymous — Wed, 24 May 2023 11:29:31 GMT

Hello

You need to use Dynamic schema, you have a similar requirement as described in this KB article.

Regards

Shong

Re: Reading CSV files with different schemas

RMotta2408 — Wed, 24 May 2023 11:32:41 GMT

Hi @Shicong Hong ,

I will have a look RIGHT NOW!

Thank you

Re: Reading CSV files with different schemas

RMotta2408 — Wed, 24 May 2023 11:34:12 GMT

The link is inaccessible. 😮

Re: Reading CSV files with different schemas

Anonymous — Wed, 24 May 2023 14:39:59 GMT

@Rui Motta , sorry, I didn't notice that the article is only visible for internal staffs, let me think about what I can do, or I can do my best to provide an example.

Re: Reading CSV files with different schemas

RMotta2408 — Wed, 24 May 2023 16:47:45 GMT

Hi there,

I've found a solution.

For each CSV file tFileList finds, I do this:

1- tFileInputDelimited

schema: column01 (dynamic).

2- tExtractDynamicFields

input schema: column01 (dynamic).
output schema: columnA, columnB, columnC, columnD.

3- tDBOutput

This way, no matter what comes in the CSV file, my Job will only consider what values come in these 4 columns. And if one or more of these columns are missing, the Job considers that column with the value "null".

Hope this helps someone.

Thank you.

Re: Reading CSV files with different schemas

Anonymous — Thu, 25 May 2023 02:36:31 GMT

It is a good solution! Thank you for your sharing! @Rui Motta

Regards

Shong

Re: Reading CSV files with different schemas

RMotta2408 — Thu, 25 May 2023 13:14:10 GMT

My pleasure.

Rui