topic Re: What will be the Maximum Size Can be taken into tFileInputXML in Talend Studio

What will be the Maximum Size Can be taken into tFileInputXML

Anonymous — Fri, 20 Jul 2012 11:57:53 GMT

Does tFileInputXML reda 5GB XML File???What will be the Maximum Size for XML file that Can be taken read through tFileInputXML....

Re: What will be the Maximum Size Can be taken into tFileInputXML

Anonymous — Fri, 20 Jul 2012 12:43:11 GMT

hi praveen,
ther is no just only answer
Do you want to read the entire file or some elements ?
For huge file use Sax Parser because DOM save a "snapshoot" of the document in 'live' memory available by Jvm.
So for entire file with Dom parser ,you 'll have to give a lot of memory for JVM
http://www.oracle.com/technetwork/articles/javase/monitoring-141801.html#Insufficient_Memory
so you'll have to manage -Xmx and -Xms size to have enought space jvm memory (heap)
regards
laurent

Re: What will be the Maximum Size Can be taken into tFileInputXML

Anonymous — Fri, 20 Jul 2012 13:06:10 GMT

I want read the entire file and it is of size 1 GB,i have increased the size aswell,still i am getting the same java heap space issue....for that purpose i want to check the max size of xml file with max memory consumption for 4GB RAM Machine.

Re: What will be the Maximum Size Can be taken into tFileInputXML

Anonymous — Fri, 20 Jul 2012 15:30:59 GMT

you can find some issue using jvm monitoring console
http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html
regards
laurent

Re: What will be the Maximum Size Can be taken into tFileInputXML

Anonymous — Fri, 17 Aug 2012 09:58:01 GMT

Can you please let me know the maximum size of the XML file that Talend Integration Suite can Process if i want to process entire file.
It will be great if somebody can give direct answer in numbers....like 5GB or 10 GB???...
Thanks,
KPK

Re: What will be the Maximum Size Can be taken into tFileInputXML

alevy — Mon, 20 Aug 2012 03:46:31 GMT

There is no direct answer. It depends on your job design: which parser you use, how many columns are in your schema, what processing is done with the file etc..

Re: What will be the Maximum Size Can be taken into tFileInputXML

Anonymous — Mon, 20 Aug 2012 11:44:44 GMT

i have a very simple xml file which is of size 1gb and iam trying to split using tfileinputxml---->Tfileadvancedoutputxml
very simple job and i used dom4 and SAX as well..still iam getting the same issue...will talend allow more than 1 GB files...why it is not working for even simple job..somebody put an end to my issue.....