topic Re: [resolved] Merge xml files in one with multiple loops in Talend Studio

[resolved] Merge xml files in one with multiple loops

Anonymous — Wed, 26 Nov 2014 13:56:37 GMT

Hi,
I have a job that generate a complex XML file using tAdvancedFileOutputXML in append mode, the result is correct, but this component is extremely slow (for 15MB file size is like 4hrs !) this is due to their use of DOM4J instead a SAX parser.
So i tried this :
step 1 : To write different xml files, one for each loop in sax mode, very fast
step 2 : Merge the xml files with the tXMLMap, but the tXMLMap doesn't allow more than one loop.

Here is the example on my output_xml file :

<root>
  <genericData>
    <name>test</name>
    <service>Service Name</service>
    <serviceDesc>serviceDesciption</serviceDesc>   
    <Resp1>
      <codeResp>aaa</codeResp>
      <nomResp>bbb</nomResp>
    </Resp1>
  </genericData>
  <survey> ---> Loop Element
	<A>blabla</A>
	<B>blingbling</B>
	<surveyNodeInfo>  ---> Loop Element
		<surveyNodeInfoX>xxxx</surveyNodeInfoX>
		<surveyNodeInfoY>yyyyy</surveyNodeInfoY>
	</surveyNodeInfo>
	<surveyNodeOtherInfo> ---> loop element
		<surveyNodeZ>zzzz</surveyNodeZ>
		<surveyNodeM>mmmmm</surveyNodeM>
	</surveyNodeOtherInfo> 
	<sampling> ---> Loop Element
		<C>blabla</C>
		<D>blingbling</D>
		<samplingNodeInfo> ---> Loop Element
			<samplingNodeInfoX>INFO</samplingNodeInfoX>
			<samplingNodeInfoY>OTHER INFO</samplingNodeInfoY>
		</samplingNodeInfo>
		<samplingNodeOtherInfo> ---> Loop Element
			<samplingNodeW></samplingNodeW>
			<samplingNodeU></samplingNodeU>
		</samplingNodeOtherInfo> 
		<E>indformation node</E>		
		<sample>---> loop element
				<F>blabla</F>
				<G>blingbling</G>
				<sampleNodeInfo> ---> Loop Element
					<sampleNodeInfoS>INFO</samplingNodeInfoS>
					<sampleNodeInfoT>OTHER INFO</samplingNodeInfoT>
				</sampleNodeInfo>
				<sampleNodeOtherInfo> ---> Loop Element
					<sampleNodeC>CCCC</sampleNodeC>
					<sampleNodeD>DDDDD</sampleNodeD>
				</sampleNodeOtherInfo> 
				<H>indformation node b</H>
				<sampleNodeJ>  ---> Loop Element
					<sampleNodeInfoJ>JJJJJJ</sampleNodeInfoJ>
				</sampleNodeJ>
		</sample>
		<samplingNodeJ>  ---> Loop Element
			<samplingNodeInfoJ>JJJJJJ</samplingNodeInfoJ>
		</samplingNodeJ>
	</sampling>
	<I>IIIII</I>
	<surveyNodeJ>  ---> Loop Element
		<surveyNodeInfoJ>JJJJJJ</surveyNodeInfoJ>
		<surveyNodeInfoJJ>KKKKKK</surveyNodeInfoJJ>
	</surveyNodeJ>
  </survey>
 </root>

Using TOS 5.5.1
How can a merge the xml files in a correct time ? or maybe Talend is not allow to do this ?
Thanks for your help.

Re: [resolved] Merge xml files in one with multiple loops

Anonymous — Thu, 27 Nov 2014 08:02:02 GMT

Hi,
Have you checked "all in one" feature in tXmlMap Settings which supports generating multi rows into one xml?
Best regards
Sabrina

Re: [resolved] Merge xml files in one with multiple loops

Anonymous — Thu, 27 Nov 2014 08:39:02 GMT

Hi,
Yes i already have this option
regards

Re: [resolved] Merge xml files in one with multiple loops

Anonymous — Sat, 29 Nov 2014 04:18:19 GMT

Talend Data Mapper can easily handle this.

Re: [resolved] Merge xml files in one with multiple loops

Anonymous — Mon, 01 Dec 2014 13:57:13 GMT

Hi,
The Talend Data Mapper is under licence ? can i istall it in TOS 5.5.1 ?

Re: [resolved] Merge xml files in one with multiple loops

Anonymous — Tue, 02 Dec 2014 07:47:16 GMT

Hi _mTs_,
Talend Data Mapper is only available in Talend Enterprise Subscription Version.
Please take a look at Talend Data Mapper User Guide: http://doc.talend.com/enterprise/talend_data_mapper/user_guide/5.5/.
Best regards
Sabrina

Re: [resolved] Merge xml files in one with multiple loops

Anonymous — Mon, 08 Dec 2014 09:56:22 GMT

Hi,
Thanks for the information.
I will like to know how fast is data mapper to generate a complex xml file 30 MB ? rows/s ?
Regards

Re: [resolved] Merge xml files in one with multiple loops

Anonymous — Wed, 10 Dec 2014 07:08:19 GMT

Hi,

I will like to know how fast is data mapper to generate a complex xml file 30 MB ? rows/s ?

Usually performance depends on job design.
Feel free to contact us for your requirment of Talend Data Mapper.
Best regards
Sabrina

Re: [resolved] Merge xml files in one with multiple loops

Anonymous — Tue, 13 Jan 2015 09:44:53 GMT

Hi again, i solve my problem using the tGroovy component.
This link help me to solve my probleme with TOS :
merge xml files with groovy
Regards
_mTs_

Re: [resolved] Merge xml files in one with multiple loops

Anonymous — Tue, 13 Jan 2015 09:56:18 GMT

Hi _mTs_,

What's your customized code in tGroovy component? Would you mind sharing it with us?
Best regards
Sabrina