topic Re: Large(1GB) XML file with 250+ columns in Talend Studio

Large(1GB) XML file with 250+ columns

Knowme — Sat, 16 Nov 2024 03:04:08 GMT

Hi,

I have a large XML file with 250+ columns.I need to load this into a sql table but below are the challenges i am facing.

1)I am not able to get the columns/metadata as the file is too large.

2)I dont know the columns and using doc as my column with xpath query "."

3)I am not able to split this file using tAdvancedOutputXMl since its throwing java heap space error.

4) i cannot convert it into csv since i dont have column names

How can i load this XML to my table.

Re: Large(1GB) XML file with 250+ columns

Anonymous — Thu, 05 Mar 2020 21:30:39 GMT

Hi,

You can build the schema automaticaly with a tXMLMap component.

Read your xml file with a tfileinputFullRow or tfileInputRegex or anything, just make sure all the XML is loaded in a single column of type document, then bind it to tXMLMap. You will be able to right click and import a xml file and Talend will generate a tree for you.

Re: Large(1GB) XML file with 250+ columns

Knowme — Fri, 06 Mar 2020 14:08:19 GMT

I did create a column doc and connected to tXMLMap.But when i right click to import the tree its taking forever since its huge file and ultimately gets locked.