parsing massive json

[Mohammed_Shevchenko]

Hello
In order handle a large JSON file, I am using Java streaming instead of the tFileInputJSON component to avoid Java heap memory issues. However, I am encountering problems with special characters, specifically quotes ("), which neither the Java code nor the tExtractJsonFields component can handle properly. This is because my file contains fields with free-form text, for example:

{

  "lien": "R0008",

  "id": "0008",

  "CATALOG_ID": "57",

  "LAST_UPDATE": "17/05/2025 22:20:23",

  "description": "hello, in order to "make it ok ", could u m'aider?Merci 7N0LYIG1\/\/\/xF9  class=\u0"

}

 wich cause probleme :
javax.json.stream.JsonParsingException: Unexpected char 74 at (line no=1, column no=248, offset=247)
routines.system.JSONException: Expected a ',' or '}' at 248 [character 249 line 1]

[jeoste · Accepted solution]

Hello,

To solve this problem you need first to read as "raw data" your json, like a text file.

Then you need custom java code in order to find the "description" field and then replace the special characters of description using replaceAll method with something like below

// code generated using AI as example
// find "description" value
Pattern p = Pattern.compile("(\"description\"\\s*:\\s*\")(.*?)(\")", Pattern.DOTALL);
Matcher m = p.matcher(json);

if (m.find()) {
    String desc = m.group(2);
    // data cleaning
    String descClean = desc.replaceAll("[^\\p{L}\\p{N} \\p{Punct}]", "");
    String descReplaced = m.group(1) + descClean + m.group(3);
    json = json.replace(m.group(0), descReplaced);
}

  
Once the file is cleaned, use JSON component to read and extract values (or use custom java code to extract json directly like below)

JSONObject jsonObj = new JSONObject(json);
String description = jsonObj.getString("description");