topic Re: Talend Data Mapper - Memory Issue While Creating an XML from a huge volume Flat File in Talend Studio

Talend Data Mapper - Memory Issue While Creating an XML from a huge volume Flat File

Anonymous — Thu, 16 May 2019 23:29:24 GMT

Hi Talend Experts,

I am facing an issue while creating a complex multi-loop XML from a flat file of > 1GB size. The main issue is it is failing even after assigning 10+GB of JVM to the XML generation job. Is there any-way to improve the memory performance, say generating the xml with lesser memory in an efficient way.

Also is there any way to use the hard disk, instead of RAM, or to chose SAX parser or any other efficient way to reduce the memory usage.

Any urgent suggestion would be helpful.

Thanks in advance

Re: Talend Data Mapper - Memory Issue While Creating an XML from a huge volume Flat File

Anonymous — Fri, 17 May 2019 15:14:21 GMT

You can configure the buffer size

tFileOutputXml.PNG

Re: Talend Data Mapper - Memory Issue While Creating an XML from a huge volume Flat File

Anonymous — Fri, 17 May 2019 16:29:25 GMT

Thanks a lot for your reply. Actually I am creating the XML in Talend data mapper, as my requirement is to create a complex xml with multiple loops and other complexities.

Is there any way to manage the XML creation memory in Talend data mapper.

Re: Talend Data Mapper - Memory Issue While Creating an XML from a huge volume Flat File

Anonymous — Fri, 17 May 2019 16:37:31 GMT

Sorry, I don't have data mapper

Did you checked all the advance configuration?

Re: Talend Data Mapper - Memory Issue While Creating an XML from a huge volume Flat File

Anonymous — Fri, 17 May 2019 16:58:23 GMT

Hi Dgm,

I have done some check on this, but was unable to find any such settings. Appreciate if anyone can help to properly utilize the memory or any way to re-design the XML generation job.