topic Re: Split large source file in talend in Talend Studio

Split large source file in talend

PMandrekar1684995754 — Thu, 25 May 2023 06:31:49 GMT

Hi Team,

If large source file want to split in source side itself how we can achieve this ?

Re: Split large source file in talend

anselmopeixoto — Fri, 26 May 2023 02:34:33 GMT

Hello @Priyanka Mandrekar

You might be able to achieve what you want starting from tfileInputFullRow component.

You'll probably need to get file's row count before (try using tFileRowCount) and create a loop (tLoop) to dynamically split file according with your needs.

Then you can set tFileInputFullRow's Limit parameter to read a specific number of rows and its Header parameter as a step.

Re: Split large source file in talend

Anonymous — Fri, 26 May 2023 04:46:50 GMT

Hello,

Could you tell us how large this file is likely to be? At what level should the file be split?

Due to "split" option will be available in tfileoutputdelimited component, If you want to get several delimited files, you can use tfileoututdelimited to achieve it.

Best regards

Sabrina