topic Temporary store db extraction using dynamic schemas in files in Talend Studio

Temporary store db extraction using dynamic schemas in files

yoxler — Tue, 09 Jul 2019 09:11:39 GMT

Hi,

I want to mirror several database tables and only use one job and I use dynamic schemas since I do not know the structure of the tables.

Temorary storing it into a hash output works because hash input picks up the generic schema. Could this be done using files instead in any way?

Because once I put the data into the file the schema dissapears and I need to add a schema in the input file.

Is there a solution with files instead of using hash? some tables are quite large so I do not want so use hash.

Thanks

Andreas

Re: Temporary store db extraction using dynamic schemas in files

fdenis — Tue, 09 Jul 2019 09:29:36 GMT

why dont you start by loading data into work file.
then use it to load db then if there is error rename/move file?

Re: Temporary store db extraction using dynamic schemas in files

yoxler — Tue, 09 Jul 2019 10:31:01 GMT

The structure of the tables is unknown and will be changing over time, and thats why I want to use dynamic schema.

So the tDBOutput component has a "Drop table if exists and create" so it will always reflect the source table.

So I guess that if I would go with solution that you suggest it would end up in errors it there is a mismatch between the destination table and the file?

Re: Temporary store db extraction using dynamic schemas in files

fdenis — Tue, 09 Jul 2019 16:12:31 GMT

be carefull, dynamics link (orange) bring type only on direct way. if you store and load data (with file , type is only strings) breaking the link you will loose columns type's.

Re: Temporary store db extraction using dynamic schemas in files

yoxler — Wed, 10 Jul 2019 12:43:43 GMT

Ok, so there is no way to store the schema somehow?

I tried several approaches but non of them worked out.