How to divide and conquer Huge file using talend compoents?
Hello, I have 2 billion record file, I want to process and get the result back using external Jar. But external jar will accept only file object. Prev. I used to store all 2 billion data in GlobalMap and system got hung because of huge JVM heap. So i got other idea which follows as tInputFile(Column:Record)---- Split the record into 'n' files ---- Using tJavaFlex read every file, call externalJar(it has method by passby parameter as file) then write back result values into file. Finally every input splits will interact with tJavaFlex and get n target files in disk. Then combine all n input records to target file. What component I can use to split the file into n files? If possible, What component I can use to combine n files into target files ? If possible, How to append target data for every file?
