Hello,
We are using the FROM_FIELD clause to parse some JSON documents. Those documents may amount to tens of thousands or maybe even a few millions of lines. We did some tests and the log files created looks something like this:
...
TABLE_NAME:
LOAD * FROM_FIELD(JSON_DOCUMENT, JSON_FIELD)(json, utf8, no labels)
X fields found: <list of fields>
1 lines fetched
TABLE_NAME:
LOAD * FROM_FIELD(JSON_DOCUMENT, JSON_FIELD)(json, utf8, no labels)
X fields found: <list of fields>
2 lines fetched
TABLE_NAME:
LOAD * FROM_FIELD(JSON_DOCUMENT, JSON_FIELD)(json, utf8, no labels)
X fields found: <list of fields>
3 lines fetched
TABLE_NAME:
LOAD * FROM_FIELD(JSON_DOCUMENT, JSON_FIELD)(json, utf8, no labels)
X fields found: <list of fields>
4 lines fetched
... (and it goes on, if the document has 1000, there will be 1000 lines like this one)
Our undestanding is that rows are handled one at a time and added to the script log. Because of this the logs become quite heavy. For a 40000 rows document we ended up with a 12MB txt file. Hence our problem.
We believe an anwser to any of those questions will solve or problem,
Is there:
A way to remove/hide those line generated from the log file for this specific app?
A better way to use the FROM_FIELD clause?
A better way to parse JSON documents?
Thanks,
