How to create a dynamic schema mapping using Talend by reading schema.

[Anonymous]

Hi All,
i have to develop one such job which will be use to load all kind of file based on my distention table schema . i will have a XML file which has all the details of Input Type , File Type, Delimiter Type, Number of Columns, Columns Validation ( Trim, data Type, remove unwanted char), out put, table schema, columns data type, constraints. etc.
So the thing is whenever i am load file it should pic all the details from config. file, and then apply validation to convert or correct the data types with other validation to remove or format the data. then again read the destination table schema and map with source schema and then load to destination. this is sort of information. so please any one can tell how to and what to use to accomplish this task.

[Anonymous]

Hey,
Is it possible to accomplish the above task in Talend now?
Hey avdbrink,
Can you please guide how you accomplished the above task.
Thanks!

[Anonymous]

hi ksultania,
this post has beeb published a long time ago ...
Now you've got dynamic schema but only with 'TIS' version.
regards
laurent

[Anonymous]

Hey kzone,
I came across Dynamic schema feature, but it helps only in taking dynamic schema from input source. Is it possible to map this schema to a fixed output schema dynamically?

Also, can you suggest where to learn from and how to learn talend to be productive?

[Anonymous]

talend help is a great place to learn a lot about Talend.
for example with Dynamic schema : https://help.talend.com/pages/viewpage.action?pageId=5671283
regards
laurent

[Anonymous]

I came across Dynamic schema feature, but it helps only in taking dynamic schema from input source. Is it possible to map this schema to a fixed output schema dynamically? (that is making tMap feature dynamic)
Thanks for your suggestion.

[Anonymous]

Hi ksultania,
For your scenario, it need to change column value dynamically.
We are working on your anther topic https://community.talend.com/t5/Archive/Mapping-Dynamic-Schema-to-a-Fixed-Destination-Schema/td-p/19..., then come back to you asap.

Best regards
Sabrina

[Anonymous]

Thank you.
Appreciate the help.

[Anonymous]

Hi,
Any updates on the "Mapping dynamic schema to a Fixed Destination Schema" . we have a similar requirement, we are using dynamic schema feature for reading input files , now   need to convert dynamic schema to fixed destination schema so that we can do validation on the columns using tmap(we have validation like column1> column2, Column2 > sysdate + 7 etc.
We are using Talend BigData 6.0 Version ( Licensed Version)
Thanks,
Sanjay

[Anonymous]

Hi Sanjay,
Have you already checked the related forum: https://community.talend.com/t5/Design-and-Development/resolved-Generic-Talend-Job/td-p/92354
Best regards
Sabrina

[Anonymous]

Hi Sabrina,
Here is the more detailed requirement.
We are extracting data from multiple sources like SQL-Server, Teradata and performing some check on the extracted data,  the SQL is created dynamically based on the some preconditions for extracting the data. Suppose  the extracted data contains 3 columns say col1 date datatype , col2 date datatype, col3 integer etc. and query returned 10 rows, now I need to validate each column values, these validation rules are also dynamic.  we have different validation rules. For eg. validation rule is something like col1 > col 2
 
tTeradatainput---->tMap--->tfileoutputdelimited
    
tTeradatainput schema is defined as dynamic i,e we have created one column with dynamic data type. In tMap  is there a way to split the dynamic schema columns and do the validation.
Here Output file schema should also be dynamic in nature ( based on the number of columns and data types, is there a way to create output schema dynamically).
We are open to use any other component in place of tMap, if required.
How can we achieve this in Talend
Thanks,
Sanjay