Replacing a tag in a big XML file efficiently

menorah84 — Sat, 16 Nov 2024 02:09:11 GMT

I am using tAdvancedFileOutputXML component to generate an XML from some DB table data.

The XML node Addresses encloses a repeated Address entry.

<Addresses>
    <Address>
      ...
    </Address>
    <Address>
      ...
    </Address>
<Addresses>

Each row of table data has maximum of two column groups that pertains to the addresses, like this:

So, in order for me to avoid doing more complex "pivoting", I just mapped the two column groups this way in tAdvancedFileOutputXML component:

<Addresses>
    <Address>
       [ad1_* columns goes here]
    </Address>
    <Address2>
       [ad2_* columns goes here]
    </Address2>
<Addresses>

So, the output is an XML file that has an inner node that looks like the above.

My next step now is to replace the tag Address2 in the file with Address, using tFileInputRaw and tMap components.

tFileInputRaw

tMap

My job would look like this:

However, when I run this job, I am getting an OutOfMemoryError on tFileInputRaw as the output XML

from tAdvancedFileOutputXML is pretty big (300MB - 1.5GB).

My question is, how do I replace those Address2 tags without getting this error? Do I need to parallelize the replace operation and how?

Re: Replacing a tag in a big XML file efficiently

vapukov — Mon, 22 Jun 2020 12:30:40 GMT

Hi, there several possible solutions:

Increase Java memory for the Job - Advaced setting on Run Job tab, or - https://community.talend.com/t5/Migration-Configuration-and/OutOfMemory-Exception/ta-p/21669

because your columns group is simple, change your query from select to

SELECT id, ad1_unit as ad_unit, ad1_st_name as ad_st_name, ..
UNION ALL
SELECT id, ad2_unit as ad_unit, ad2_st_name as ad_st_name, ..

and there you can have addresses in the same loop

if 2nd not a solution and you expect file sizes bigger than available memory, use command-line tools like perl or sed (call command with tSystem component)