Hi.
Last time I faced a lot of issues regarding parsing and splitting a big json files. Big means let's say array of the 100 000 documents.
The requirement is to split one big file into a lot of small files (one file per one document) plus some not complecated transformations.
When I tried to implement it based on the tFileInputJSON/tFileOutputJSON component it took many hours because iteration link was there (splitted files need to be named based on the key value (id) ).
This is why I changed approach and implemented whole parsing and splitting process based on the java code (tJavaRow component).
I know that this is not elegant but it takes much less time (more less 3 minutes).
Here is the issue - 3 minutes for stakeholders is still too long.
I use the org.json library.
Can anyone propose some different approach or different java library to improve the performace?
I'm not a java developer so I'm afraid that very complicated java libraries can be chalenging for me.
Regards.
