topic Retrive Schema of a file Dynamically in Talend Studio

Retrive Schema of a file Dynamically

DSuryadevara — Thu, 08 Sep 2022 20:27:33 GMT

I have a requirement where we get 40 *.txt files with different schemas and we need to load it to respective tables by reading file after file.

is there a way i can pull schema of the particular file based on filename and run dynamically for each file in the same manner.

Can anyone suggest how i should proceed with this senario

Re: Retrive Schema of a file Dynamically

Anonymous — Fri, 09 Sep 2022 03:45:13 GMT

Hello,

Here is a feature "Dynamic Schema" in Talend Subscription solution which allows you to design schema with an unknown column structure (unknown name and number of columns).

For your use case, you can use tfilelist to fetch your multiple files with mask *.txt and try to load them into respective tables by using dynamic schema.

The work flow should be: tfilelist-->tfileinputdelimited(dynamicSchema)-->tDBOutput(dynamicSchema).

Please take a look at KB article about:

https://community.talend.com/s/article/How-to-process-changing-data-structure-ExK7j

Feel free to let us know if it is OK with you.

Best regards

Sabrina

Re: Retrive Schema of a file Dynamically

DSuryadevara — Fri, 09 Sep 2022 19:14:34 GMT

Hello Sabrina,

thanks for the response, this works but few columns coming from file are string or long timestamps like 20220909334455 but i need to load to tables in 09/09/2022 33:44:55 format. how will i be able to do it when running dynamically. will there be any way to convert long data type columns to Date data type for this particular job.

Re: Retrive Schema of a file Dynamically

Anonymous — Thu, 03 Nov 2022 07:54:22 GMT

Hello,

You are able to extract and map dynamic fields in tmap or tjavarow component.

Hope these online documentations help.

https://help.talend.com/r/en-US/8.0/processing/twritedynamicfields-tfixedflowinput-tjavarow-tlogrow-extracting-contents-of-dynamic-column-via-tjavarow-standard-component-enterprise-in-this

Best regards

Sabrina