Unable to process big file in tXmlMap

[Anonymous]

Hello,
I have a text file with more than 10 million records and trying to process it using the tXMLMap to output as xml file. But during this process, found that tXMLMap dumps all the data with in it and it causes the java outofmemoryerror. Could you someone please help me with alternative or any configurations can be done on this to resolve the issue? thanks in advance.




Thanks
Bala

[Anonymous]

No Problem and thank you for your detailed explanation. I have no issues in splitting the file into small XML's. Here the group element is member id. For example one member id can have multiple policy details. So the basic intention is we need to combine all the details of one member should present in the same XML file. I am relatively new to this Talend tool. Any detailed explanation would be great help.  thanks much

[Anonymous]

Alternatively, if you only have member ids you could do this. Lets say that your member ids go from 1 to 1000 (I know, an easy example, but stay with it and maybe you can extrapolate). If that is the case, then you could use a tMap component to split the data out to 10 files based on grouping the member ids. 1-99, 100-199, 200-299, 300-399, 400-499, 500-599, 600-699, 700-799, 800-899, 900-999 (or 1000). This can be achieved with a simple algorithm using the inbuilt filtering of the tMap output tables. Then you could ensure that you initial file is split into 10 relatively equally sized files, which are grouped as you need them for your final XML data.

Could you please explain the this approach bit? I tried this option and couldn't succeed. thanks

[Anonymous]

I have knocked up a very quick example of what I was talking about. It was put together in v5.6.2, so can be run in any version the same or higher than that.  TestSplitFile.zip

[Anonymous]

Thank you so much for your quick response. I will be out of station till 24th. Will test the same and get back to you once I am back. Thanks again 

[Anonymous]

Hello,
I have gone through the sample job. Actually you are splitting the file after every 10 records up to 100. But requirement is different from this. Actually each member should go to the separate xml file. Also be aware that one member can have many records  and it can present anywhere in the file. So we should collect all the information about the member and place into the one file. We should group based on the member id. thanks 
Bala