parsing massive json

Mohammed_Shevchenko — Thu, 24 Jul 2025 12:01:09 GMT

Hello
In order handle a large JSON file, I am using Java streaming instead of the tFileInputJSON component to avoid Java heap memory issues. However, I am encountering problems with special characters, specifically quotes ("), which neither the Java code nor the tExtractJsonFields component can handle properly. This is because my file contains fields with free-form text, for example:

{

"lien": "R0008",

"id": "0008",

"CATALOG_ID": "57",

"LAST_UPDATE": "17/05/2025 22:20:23",

"description": "hello, in order to "make it ok ", could u m'aider?Merci 7N0LYIG1\/\/\/xF9 class=\u0"

}

wich cause probleme :
javax.json.stream.JsonParsingException: Unexpected char 74 at (line no=1, column no=248, offset=247)
routines.system.JSONException: Expected a ',' or '}' at 248 [character 249 line 1]

Re: parsing massive json

jeoste — Fri, 01 Aug 2025 08:40:11 GMT

Hello,

To solve this problem you need first to read as "raw data" your json, like a text file.

Then you need custom java code in order to find the "description" field and then replace the special characters of description using replaceAll method with something like below

// code generated using AI as example // find "description" value Pattern p = Pattern.compile("(\"description\"\\s*:\\s*\")(.*?)(\")", Pattern.DOTALL); Matcher m = p.matcher(json); if (m.find()) { String desc = m.group(2); // data cleaning String descClean = desc.replaceAll("[^\\p{L}\\p{N} \\p{Punct}]", ""); String descReplaced = m.group(1) + descClean + m.group(3); json = json.replace(m.group(0), descReplaced); }

Once the file is cleaned, use JSON component to read and extract values (or use custom java code to extract json directly like below)

JSONObject jsonObj = new JSONObject(json); String description = jsonObj.getString("description");

topic parsing massive json in Talend Studio

parsing massive json

Re: parsing massive json