LOAD using FROM_FIELD results in large logs

acgomes — Mon, 29 Jan 2024 19:37:42 GMT

Hello,

We are using the FROM_FIELD clause to parse some JSON documents. Those documents may amount to tens of thousands or maybe even a few millions of lines. We did some tests and the log files created looks something like this:

...
TABLE_NAME:
LOAD * FROM_FIELD(JSON_DOCUMENT, JSON_FIELD)(json, utf8, no labels)
X fields found: <list of fields>
1 lines fetched
TABLE_NAME:
LOAD * FROM_FIELD(JSON_DOCUMENT, JSON_FIELD)(json, utf8, no labels)
X fields found: <list of fields>
2 lines fetched
TABLE_NAME:
LOAD * FROM_FIELD(JSON_DOCUMENT, JSON_FIELD)(json, utf8, no labels)
X fields found: <list of fields>
3 lines fetched
TABLE_NAME:
LOAD * FROM_FIELD(JSON_DOCUMENT, JSON_FIELD)(json, utf8, no labels)
X fields found: <list of fields>
4 lines fetched
... (and it goes on, if the document has 1000, there will be 1000 lines like this one)

Our undestanding is that rows are handled one at a time and added to the script log. Because of this the logs become quite heavy. For a 40000 rows document we ended up with a 12MB txt file. Hence our problem.

We believe an anwser to any of those questions will solve or problem,
Is there:
A way to remove/hide those line generated from the log file for this specific app?
A better way to use the FROM_FIELD clause?
A better way to parse JSON documents?

Thanks,

topic LOAD using FROM_FIELD results in large logs in App Development

LOAD using FROM_FIELD results in large logs