topic Re: TDM performance issue in Talend Studio

TDM performance issue

Rajender — Sat, 16 Nov 2024 05:28:48 GMT

Hi Experts

I have generated XML using Talend Data mapper. XML of multi schema nested type.

There are 110 columns for a records. I have used your approach using 2 maps.

First Map convert flat file structure to XML structure with duplicates. Second Map use first map output as input and generate final XML.

When processing 200 records its working fine. But as volume increase First map file size also increase exponentially. After running for long hours job get failed.

Please suggest if any Idea. Thank you for your time and efforts.

Best Regards

Rajender Prasad

Re: TDM performance issue

Anonymous — Wed, 26 Jun 2019 14:29:53 GMT

Hi,

Could you please specify the memory parameters you are using for the job? Did you measure the performance difference for various memory values?

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: TDM performance issue

Rajender — Thu, 27 Jun 2019 19:26:52 GMT

Hi Nikhil

Job is failing due to out of memory Error. It seems JVM heap size full.

I have executed job for 8GB JVM(Run tab--> Advance setting(-Xmx4096)) but still job failed for 200 records only.

We have enterprise edition 7.1.1 and also raised a ticket for this issue to Talend. But still not resolved.

Here I am looking for some community experts who can help.

Best Regards

Rajender Prasad

Re: TDM performance issue

Rajender — Thu, 27 Jun 2019 19:31:01 GMT

Following error received when job failed.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
at java.lang.StringBuilder.append(StringBuilder.java:190)
at org.apache.commons.io.output.StringBuilderWriter.write(StringBuilderWriter.java:138)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2002)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1980)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1957)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1907)
at org.apache.commons.io.IOUtils.toString(IOUtils.java:778)
at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1712)
at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1734)

Best Regards

Rajender Prasad

Re: TDM performance issue

Anonymous — Thu, 27 Jun 2019 21:05:49 GMT

Hi @Rajender ,

Please specify your -Xmx as 5120M or 5G.

If you miss the "M" the memory will not be calculated as MB.

In addition you can read the file as stream and give it a try.

Thanks and Regards,

Subhadip