topic Re: Dynamic Schemas in tXMLMap component in Talend Studio

Dynamic Schemas in tXMLMap component

Anonymous — Fri, 07 Jul 2017 20:05:41 GMT

Hi,

I am using Talend Enterprise edition. I realized that tXMLMap doesn't support dynamic schema. I am trying to see if there is a workaround for that. Here is my use case.

Use case: we pull data for multiple object types from a source system ( using REST) and store the data in csv file format.

Currently we have one talend job per object type, each talend job looks like below.

1. Repository metadata schema created for the object type from a sample response xml.

2. Job flow: tRESTClient pulls the data for a particular object type -> tXMLMap maps response body to the CSV file output.

In the tXMLMap configuration UI, we loaded the repository metadata for the object type. and mapped it to output schema manually.

we have high number of object types that, we want to have one single talend job serving all object types. Few questions on that.

1. Is there a way I can access repository metadata during runtime? within tJava component? so that I can parse the schema and map it to the target schema at runtime?

2. Does tXMLMap support loading/importing repository metadata for the response body at runtime ?

3. do you suggest writing a custom component to do this job? If so, does it require advanced talend knowledge?

4. do you know of any other workarounds to achieve this?

Thanks,

Sowjanya.

Re: Dynamic Schemas in tXMLMap component

Anonymous — Fri, 07 Jul 2017 21:24:09 GMT

Answer to your questions:

1) The metadata are actually stored in files. Hence, they can be read. But you can't do much more because you cannot inject XML like hierarchical schema dynamically in a job.

2) As far as I am aware, tXMLMap does not support loading/importing repository metadata dynamically at runtime

3) Depends on what you are trying to actually simplify. Making things dynamic just shift complexity to another area for debugging purposes.

4) It all depends how complex your response body are. Can you share some examples? Are they highly hierarchical? Or just flat?

Re: Dynamic Schemas in tXMLMap component

Anonymous — Sat, 08 Jul 2017 00:08:07 GMT

Thanks for the response.

Response bodies are pretty flat for all the objects. at max 2 levels.

we have 150 object types that we are trying to pull. all the jobs pretty much do same thing except that they are dealing with different object types and schemas.

Having one talend job instead of 150 jobs, would greatly reduce the development and maintenance efforts.

please suggest if there is any workaround for convering xml to csv using dynamic schemas.

1) you mentioned I cannot inject xml hierarchical schema dynamically in a job. I am thinking of mapping the output fields to input Xpath during runtime using java/groovy code components. similar to https://community.talend.com/t5/Troubleshooting-Configuration/Dynamic-Mapping-Getting-Groovy-with-Talend-at-Runtime/ta-p/21491

do you still think it is not possible to achieve this?

Re: Dynamic Schemas in tXMLMap component

Anonymous — Wed, 17 Jun 2020 08:38:02 GMT

@sowjanya Did you get any solution for this problem? Or still working with N jobs for N Schema?