topic Split Output JSON files based on size in Talend Studio

Split Output JSON files based on size

Sriram151515 — Sat, 16 Nov 2024 00:56:57 GMT

Hello,

I am quite new to the usage of Talend. I am trying to load large volume of data from Snowflake DB into Talend for conversion to JSON using Talend Data mapper in a specified format to be loaded to the downstream systems. The catch here is the output file size should not exceed 15MB. Hence need to split the output based on the size. Could you please guide how this could be implemented.

Thanks

Re: Split Output JSON files based on size

Anonymous — Wed, 16 Dec 2020 05:25:38 GMT

Hello,

Could you tell us how large this file is likely to be?

Is it possible to use tFileInputRaw to load it entirely in memory? The work flow should be: tFileInputRaw --> tExtractJSONFields.

You could try a route using cFile and streaming Splitter via Camel solution.

Best regards

Sabrina

Re: Split Output JSON files based on size

Sriram151515 — Thu, 17 Dec 2020 09:55:42 GMT

Hi Sabrina,

The overall file size is in 10 to 15 GB. But acceptable for the downstream system is only 15MB max. Could you please help with an example or some screenshots of the components.

Re: Split Output JSON files based on size

MdFazil — Wed, 22 Nov 2023 07:47:34 GMT

hello Sriram,

Do you get any solutions I too currently facing same problem. If there any solutions please let me know

Thanks in advance

regards

Fazil M