topic Loading files with different schema in Talend Data Catalog

Loading files with different schema

Anonymous — Sat, 16 Nov 2024 05:26:34 GMT

Hi Team,

I have multiple file with different structure.

Is there a way you create one dynamic schema and load all files on the fly.

If not, can it be done using Spark? If yes, how do you do it?

Re: Loading files with different schema

Anonymous — Tue, 02 Jul 2019 09:23:50 GMT

Hello,

With talend subscription solution, here is a dynamic schema feature which allows you to design schema with an unknown column structure (unknown name and number of columns).

Where are your input files from?

Here is a community article:https://community.talend.com/t5/Design-and-Development/How-to-process-changing-data-structure/ta-p/21702?collapse_discussion=true&q=changing%20data&search_type=thread

Let us know if it is what you are looking for.

Best regards

Sabrina

Re: Loading files with different schema

Anonymous — Tue, 16 Jul 2019 13:32:20 GMT

Thanks for the reply! Is it possible to load multiple files to different targets?
Example only shows loading one file at a time to a db, and not the use case i'm looking at.