read whole file at once

Anonymous — Wed, 07 Dec 2016 17:46:16 GMT

Hi

I'm having a input text file of 200 mb more or less with one single line data
And that input file contains a header...that header can exist at any point of the file...first position, middle position or at last
So due to memory issue can occur...I cant load whole file into talend in once
so is there any way I could read chunks of data from file and load into talend then match with my lookup file
and chunks will keep moving into talend till I dont find my header
once I find...next chunk should not come...!!

Is it fine If I use tfileinputraw component to read whole file(200 to 300 MB) in one go with enable input stream editor and then convert that object type to string and use that string with my lookups ?

Memory issue can occur with this much of file size having single line containing all data in talend ?

Re: read whole file at once

Anonymous — Thu, 08 Dec 2016 03:33:06 GMT

Hi,
Did you get "Outof memory" issue when use tfileinputraw component to read your whole file(200 to 300 MB)?
If so, have you tried to allocate more memory to your current job by editing the JVM parameter in the "Run " view to see if it is Ok with you?
Best regards
Sabrina

topic Re: read whole file at once in Talend Studio

read whole file at once

Re: read whole file at once