topic Re: Load a large file using tloop in Talend Studio

Load a large file using tloop

AJafri — Fri, 15 Nov 2024 21:07:24 GMT

I have file with million records need to load it in multiple iteration using tLoop. File have column with sequential value in it. We can load 10000 records in one iteration. I am stuck in the logic that has to be used in tLoop

Re: Load a large file using tloop

konvertilo — Thu, 11 Jan 2024 13:38:43 GMT

Hello

You need to give more context. Why do you want to use tloop ? there is a tFileInputDelimited that can read huge file, you don't have to build chunk of data.

What is the usage of the sequential value in this file ?

Re: Load a large file using tloop

Anonymous — Mon, 15 Jan 2024 05:15:28 GMT

Hello,

Could you please elaborate your case with an example with input and expected output values? Which will be helpful for us to understand your situation.

Best regards

Sabrina

Re: Load a large file using tloop

jeoste — Wed, 07 Feb 2024 16:13:36 GMT

Hello,

What you can do is from your tFileInput split it into X different files (of 10 000 lines as mentioned).

Then with tFileList, you can read each file and insert their content int DB. In this case no need to use tLoop, tFileList is replacing this behavior

More info here : https://stackoverflow.com/questions/10348570/split-file-to-more-files-in-talend