topic Re: dynamic job to load multiple tables from multiple input files in Talend Studio

dynamic job to load multiple tables from multiple input files

Anonymous — Wed, 22 Mar 2017 07:48:19 GMT

Hi,
I have a scenario where i have to load like 200 target tables from 200 input files having one-to-one mapping. each file structure is different.
i also have an extra column in the input files having values as 'I'/'U'/'D' depending on which i will Insert/Update/Delete the records of the target tables.
now the simple solution is to write 200 different jobs but i'm trying to design this using one single dynamic job considering i have no transformation logic and all.
problems i am facing:
1. need to change the schema of tfileinputdelimited dynamically.
2. need to write a metadata file/table for the primary keys of different tables for update/delete.
3. these primary keys will also have to change dynamically.
being a novice in the world of talend i will really appreciate any help.
many thanks in advance.
poulami

Re: dynamic job to load multiple tables from multiple input files

Anonymous — Thu, 23 Mar 2017 07:36:36 GMT

Hi,
Here is dynamic schema feature which allows you to design schema with an unknown column structure (unknown name and number of columns).
Could you please take a look at this KB article:TalendHelpCenter: How to process changing data structure?
Best regards
Sabrina

Re: dynamic job to load multiple tables from multiple input files

Anonymous — Tue, 28 Mar 2017 10:50:57 GMT

Hi sabrina,
This one loads table from one single file only where only the schema type is selected as dynamic.
But in my scenario i have 200 source files and 200 respective target tables.
Also, i don't want to use 'Create table' as my table is already present in the database and my file contains two extra fields (dml_flag and datetime) as i mentioned earlier.
According to my understanding i need to loop through all source files , check dml_flag column for insert/update/delete and perform the operation accordingly for the respective target tables. And this whole process is to be dynamic as all the source and target schema are different.