topic load data with dynamic schema in Talend Studio

load data with dynamic schema

Anonymous — Sat, 16 Nov 2024 07:44:17 GMT

Hi,

we have requirement where we need to load data file(.dat) into Oracle database. However data file structures will not same(no of columns and data type may differ) for all the files. All file will have same data delimiter. We need to design a standard graph which should process all input files even with different schema. Thus graph should read file and create dynamic schema based on input file and load into database. Looking for an assistance to design this graph in Talend.

Regards,

Vivek

Re: load data with dynamic schema

fdenis — Fri, 31 Aug 2018 14:42:22 GMT

read line 1 who must have columns names, create or alter table (All columns as String), then load your data.

Re: load data with dynamic schema

fdenis — Fri, 31 Aug 2018 14:45:23 GMT

use dynamic Types to load Data.

If you did not have dynamic type it's because you are in free version.

in this case you can load data using a dynamic insert query.

good luck.

Re: load data with dynamic schema

Anonymous — Mon, 03 Sep 2018 08:17:44 GMT

There is no column heading in input file. Can you please elaborate your solution so we can visualize better and look for possibility. We are using license version software.

Re: load data with dynamic schema

fdenis — Mon, 03 Sep 2018 09:25:45 GMT

if you have no header it's the same, but your columns name will be Col1, Col2....
read file line by line. by splitting first line, count the number of columns.
after you just have to generate Create Table ....
read the file line by line and générât one insert statement by line. Insert into (Col1,..) values ('val1...).

Good luck