[JSON files] Parsing and splitting huge json file

JackStrong — Sat, 16 Nov 2024 00:03:12 GMT

Hi.

Last time I faced a lot of issues regarding parsing and splitting a big json files. Big means let's say array of the 100 000 documents.

The requirement is to split one big file into a lot of small files (one file per one document) plus some not complecated transformations.

When I tried to implement it based on the tFileInputJSON/tFileOutputJSON component it took many hours because iteration link was there (splitted files need to be named based on the key value (id) ).

This is why I changed approach and implemented whole parsing and splitting process based on the java code (tJavaRow component).

I know that this is not elegant but it takes much less time (more less 3 minutes).

Here is the issue - 3 minutes for stakeholders is still too long.

I use the org.json library.

Can anyone propose some different approach or different java library to improve the performace?

I'm not a java developer so I'm afraid that very complicated java libraries can be chalenging for me.

Regards.

Re: [JSON files] Parsing and splitting huge json file

David_Beaty — Tue, 29 Jun 2021 10:19:17 GMT

Hiya

Hopefully it wont be too complex, but consider using the tJavaFlex component to split out the single big JSON into smaller/more manageable chunks.

This link might help you.

Thanks

David

Re: [JSON files] Parsing and splitting huge json file

Anjali3 — Wed, 08 Jun 2022 03:17:49 GMT

Please share the sample code of parsing and splitting the JSON file in java component.@Michal Swiniarski

topic Re: [JSON files] Parsing and splitting huge json file in Talend Studio

[JSON files] Parsing and splitting huge json file

Re: [JSON files] Parsing and splitting huge json file

Re: [JSON files] Parsing and splitting huge json file