topic Re: processing large JSON file error: OutOfMemory in Talend Studio

processing large JSON file error: OutOfMemory

Anonymous — Mon, 28 Jan 2013 00:04:26 GMT

Hi, I am new to Talend Open Studio for Data Integration. I was able to manage creating few test jobs successfully. Now I am running into an issue: when the tInputFileJSON reads a big file (80M), it will run out of memory. I have made some JVM changes to increase heap size, but still running into the same problem. As a matter of fact, I am not surprise by that as I can only increase heap size so much, but potential input data is "unlimited".
Just wonder how I can control the job to read say 5000 rows and process the data before bringing next 5000 row? what is the best practice to process large file in Talend?

Re: processing large JSON file error: OutOfMemory

Anonymous — Mon, 28 Jan 2013 02:19:14 GMT

Hi,
In Talend, the input component such as tFileInputDelimited, tFileInputJson read all rows and cache them into memory, and then iterate one by one. So, it is impossible to process the first N rows, then next the N rows.
DO you have some other components like tMap, tFilterRow which consumes more memory in the job? For OutOfMemory exception, we are ususlly required to allocate more memory to the job,please see this KB article:
outOfMemory
Best regards
Sabrina

Re: processing large JSON file error: OutOfMemory

Anonymous — Mon, 28 Jan 2013 04:58:54 GMT

Thank you Sabrina! My JSON is pretty complicated and file size will keep increasing. So I don't think keeping allocating more memory is good for me in long run. Is there a way I can split a big file into multiple smaller files by the value of primary key? For example, I have a CSV with (device_id, billing_date, billing_amount), can I save all rows for device_id 111222 to file 111222.csv, and all rows for device_id 111333 to 111333.csv?

Re: processing large JSON file error: OutOfMemory

Anonymous — Mon, 28 Jan 2013 06:08:22 GMT

Hi,
For csv file, the answer is yes, use tFileInputFullRow to read the source file row by row and generate several files with the option 'Split the files into several files' on tFileOutputDelimited.
The job looks like:
tFileInputFullRow--main--tFileOutputDelimited
See pics for details
Best regards
Sabrina

Re: processing large JSON file error: OutOfMemory

karandama2006 — Sat, 23 Feb 2019 13:34:36 GMT

Has anyone solved this , I have the same issue .

JSON file is > 200mb and talend goes out of memory while parsing it