TDM performance issue

[Rajender]

Hi Experts

I have generated XML using Talend Data mapper. XML of multi schema nested type.

There are 110 columns for a records. I have used your approach using 2 maps.

First Map convert flat file structure to XML structure with duplicates. Second Map use first map output as input and generate final XML.

When processing 200 records its working fine. But as volume increase First map file size also increase exponentially. After running for long hours job get failed.

Please suggest if any Idea. Thank you for your time and efforts.

Best Regards

Rajender Prasad

[Anonymous]

Hi,

    Could you please specify the memory parameters you are using for the job? Did you measure the performance difference for various memory values?

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[Rajender]

Hi Nikhil 

Job is failing due to out of memory Error. It seems JVM heap size full.

I have executed job for 8GB JVM(Run tab--> Advance setting(-Xmx4096)) but still job failed for 200 records only.

We have enterprise edition 7.1.1 and also raised a ticket for this issue to Talend. But still not resolved. 

Here I am looking for some community experts who can help.

Best Regards

Rajender Prasad

[Rajender]

Hi

Following error received when job failed.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
at java.lang.StringBuilder.append(StringBuilder.java:190)
at org.apache.commons.io.output.StringBuilderWriter.write(StringBuilderWriter.java:138)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2002)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1980)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1957)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1907)
at org.apache.commons.io.IOUtils.toString(IOUtils.java:778)
at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1712)
at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1734)

Best Regards

Rajender Prasad

[Anonymous]

Hi @Rajender ,

Please specify your -Xmx as 5120M or 5G.

If you miss the "M" the memory will not be calculated as MB.

In addition you can read the file as stream and give it a try.

Thanks and Regards,

Subhadip