https://community.qlik.com/t5/Talend-Studio/How-to-divide-and-conquer-Huge-file-using-talend-compoents/m-p/2315695#M86321 <P>Hello,<BR />I have 2 billion record file, I want to process and get the result back using external Jar. But external jar will accept only file object.<BR />Prev. I used to store all 2 billion data in GlobalMap and system got hung because of huge JVM heap. So i got other idea which follows as&nbsp;<BR />tInputFile(Column:Record)---- Split the record into 'n' files ---- Using tJavaFlex read every file, call externalJar(it has method by passby parameter as file) then write back result values into file.<BR />Finally every input splits will interact with tJavaFlex and get n target files in disk.<BR />Then combine all n input records to target file.<BR />What component I can use to split the file into n files?<BR />If possible, What component I can use to combine n files into target files &nbsp;?<BR />If possible, How to append target data for every file?<BR /><BR />Please give me the brief design to proceed.&nbsp;</P> Fri, 27 Nov 2015 03:55:49 GMT Anonymous 2015-11-27T03:55:49Z https://community.qlik.com/t5/Talend-Studio/How-to-divide-and-conquer-Huge-file-using-talend-compoents/m-p/2315695#M86321 <P>Hello,<BR />I have 2 billion record file, I want to process and get the result back using external Jar. But external jar will accept only file object.<BR />Prev. I used to store all 2 billion data in GlobalMap and system got hung because of huge JVM heap. So i got other idea which follows as&nbsp;<BR />tInputFile(Column:Record)---- Split the record into 'n' files ---- Using tJavaFlex read every file, call externalJar(it has method by passby parameter as file) then write back result values into file.<BR />Finally every input splits will interact with tJavaFlex and get n target files in disk.<BR />Then combine all n input records to target file.<BR />What component I can use to split the file into n files?<BR />If possible, What component I can use to combine n files into target files &nbsp;?<BR />If possible, How to append target data for every file?<BR /><BR />Please give me the brief design to proceed.&nbsp;</P> Fri, 27 Nov 2015 03:55:49 GMT https://community.qlik.com/t5/Talend-Studio/How-to-divide-and-conquer-Huge-file-using-talend-compoents/m-p/2315695#M86321 Anonymous 2015-11-27T03:55:49Z https://community.qlik.com/t5/Talend-Studio/How-to-divide-and-conquer-Huge-file-using-talend-compoents/m-p/2315696#M86322 Hi,&nbsp;<BR /><BR />That's some file you have. Your solution would mainly depend on what file type your incoming, split and generated/appended files are. Fri, 27 Nov 2015 07:13:44 GMT https://community.qlik.com/t5/Talend-Studio/How-to-divide-and-conquer-Huge-file-using-talend-compoents/m-p/2315696#M86322 Anonymous 2015-11-27T07:13:44Z https://community.qlik.com/t5/Talend-Studio/How-to-divide-and-conquer-Huge-file-using-talend-compoents/m-p/2315697#M86323 It's txt files, split/ target files would be txt files aswell. can you elaborate the components to use please? Fri, 27 Nov 2015 07:23:18 GMT https://community.qlik.com/t5/Talend-Studio/How-to-divide-and-conquer-Huge-file-using-talend-compoents/m-p/2315697#M86323 Anonymous 2015-11-27T07:23:18Z