topic Re: How to create a dynamic schema mapping using Talend by reading schema. in Talend Studio

How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Sat, 16 Nov 2024 12:28:55 GMT

Hi All,
i have to develop one such job which will be use to load all kind of file based on my distention table schema . i will have a XML file which has all the details of Input Type , File Type, Delimiter Type, Number of Columns, Columns Validation ( Trim, data Type, remove unwanted char), out put, table schema, columns data type, constraints. etc.
So the thing is whenever i am load file it should pic all the details from config. file, and then apply validation to convert or correct the data types with other validation to remove or format the data. then again read the destination table schema and map with source schema and then load to destination. this is sort of information. so please any one can tell how to and what to use to accomplish this task.

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Wed, 04 Jan 2012 08:08:51 GMT

Hi
Talend Open Studio doesn't support dynamically retrieve schema from an exist XML file.
I'm afraid you have to set all manually.
Besides, Talend Trade Version now supports dynamically retrieve schema from DB and files.
It is still impossible to retrieve from a XML file.
Best regards!
Pedro

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Thu, 05 Jan 2012 04:22:03 GMT

Thank you Pedro,
i am having enterprise version, and i can use DB also, so is it possible in enterprise version if yes then, can you please suggest any link or demo so i can move ahead.

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Thu, 05 Jan 2012 05:11:48 GMT

Hi
Even exterprise version can't support you retrieve schema from a XML file dynamically.
Best
Pedro

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Thu, 05 Jan 2012 06:31:55 GMT

Maybe an experienced developer can create a custom component for this purpose?

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Thu, 05 Jan 2012 08:59:46 GMT

Is there a sample job or any other sample available that shows us how this dynamically retrieval from DB schemes works?
Regards,
Arno

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Thu, 05 Jan 2012 09:04:02 GMT

Hi Arno
If you are the user of Commercial Version, you can get some explanation and scenarios in the document.
Serch the chapter 'Dynamic Schema'.
Best regards
Pedro

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Thu, 05 Jan 2012 09:22:12 GMT

Hi Pedro,
I already know about this possibility, but was hoping there were some new options which make it possible to have dynamic schema, on which you can perform transformations and such.
The Dynamic schema functionality offered by Talend right now does not match our requirements for dynamic schemes I suppose...
Thanks anyway!
Regards,
Arno

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Thu, 05 Jan 2012 09:27:34 GMT

I will communicate with our development team for this feature.
From user respect, creating issue at BugTracker for new feature will be more effective.
Regards,
Pedro

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Thu, 05 Jan 2012 09:56:49 GMT

Yeah, maybe we should create an issue... I'll reconsider that!
Thanks
Arno

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Wed, 28 May 2014 11:53:02 GMT

Hey,
Is it possible to accomplish the above task in Talend now?
Hey avdbrink,
Can you please guide how you accomplished the above task.
Thanks!

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Wed, 28 May 2014 13:41:46 GMT

hi ksultania,
this post has beeb published a long time ago ...
Now you've got dynamic schema but only with 'TIS' version.
regards
laurent

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Wed, 28 May 2014 13:54:08 GMT

Hey kzone,
I came across Dynamic schema feature, but it helps only in taking dynamic schema from input source. Is it possible to map this schema to a fixed output schema dynamically?

Also, can you suggest where to learn from and how to learn talend to be productive?

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Wed, 28 May 2014 14:18:43 GMT

talend help is a great place to learn a lot about Talend.
for example with Dynamic schema : https://help.talend.com/pages/viewpage.action?pageId=5671283
regards
laurent

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Thu, 29 May 2014 05:18:49 GMT

I came across Dynamic schema feature, but it helps only in taking dynamic schema from input source. Is it possible to map this schema to a fixed output schema dynamically? (that is making tMap feature dynamic)
Thanks for your suggestion.

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Thu, 29 May 2014 07:43:41 GMT

Hi ksultania,
For your scenario, it need to change column value dynamically.
We are working on your anther topic https://community.talend.com/t5/Archive/Mapping-Dynamic-Schema-to-a-Fixed-Destination-Schema/td-p/196017, then come back to you asap.

Best regards
Sabrina

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Thu, 29 May 2014 08:11:38 GMT

Thank you.
Appreciate the help.

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Mon, 28 Dec 2015 11:05:58 GMT

Hi,
Any updates on the "Mapping dynamic schema to a Fixed Destination Schema" . we have a similar requirement, we are using dynamic schema feature for reading input files , now need to convert dynamic schema to fixed destination schema so that we can do validation on the columns using tmap(we have validation like column1> column2, Column2 > sysdate + 7 etc.
We are using Talend BigData 6.0 Version ( Licensed Version)
Thanks,
Sanjay

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Tue, 29 Dec 2015 02:28:54 GMT

Hi Sanjay,
Have you already checked the related forum: https://community.talend.com/t5/Design-and-Development/resolved-Generic-Talend-Job/td-p/92354
Best regards
Sabrina

Re: How to create a dynamic schema mapping using Talend by reading schema.

Anonymous — Tue, 29 Dec 2015 11:09:49 GMT

Hi Sabrina,
Here is the more detailed requirement.
We are extracting data from multiple sources like SQL-Server, Teradata and performing some check on the extracted data, the SQL is created dynamically based on the some preconditions for extracting the data. Suppose the extracted data contains 3 columns say col1 date datatype , col2 date datatype, col3 integer etc. and query returned 10 rows, now I need to validate each column values, these validation rules are also dynamic. we have different validation rules. For eg. validation rule is something like col1 > col 2

tTeradatainput---->tMap--->tfileoutputdelimited

tTeradatainput schema is defined as dynamic i,e we have created one column with dynamic data type. In tMap is there a way to split the dynamic schema columns and do the validation.
Here Output file schema should also be dynamic in nature ( based on the number of columns and data types, is there a way to create output schema dynamically).
We are open to use any other component in place of tMap, if required.
How can we achieve this in Talend
Thanks,
Sanjay