topic Re: Moving 300 files from folder containing 250000 to another folder in Talend Studio

Moving 300 files from folder containing 250000 to another folder

DSuryadevara — Fri, 15 Nov 2024 22:52:15 GMT

Hello,

I have a remote server which will be receiving 1000's of files on daily basis. And to process those files we have different jobs. i want to move 300 or 500 files to a different folder to only process those. i have tried tfilelist -> tfilecopy by giving if condition on ((Integer)globalMap.get("tFileList_4_NB_FILE")). it is working on folder having 7000 files, its taking less than a min. but if i give source folder where files are around 250000 then its taking forever to start and run.

Can anyone suggest the right approach. to move around 500 files from 250000 files folder in a faster way. when moved 500 the job need to terminate safely

Re: Moving 300 files from folder containing 250000 to another folder

Anonymous — Wed, 01 Jun 2022 08:12:39 GMT

@Deepika Suryadevara, my idea is to list the all files , add a sequence id to each row, filter the rows to have only 500 rows, and then copy these 500 files, the job design looks like:

tFileList--iterate--tFixedFlowInput-main--tUnite--main--tMap--tFlowToIterate--tFileCopy

on tFixedFlowInput, output the current file path, define one field and set it's value as:

((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))

on tMap, add a var and set it's expression as:

Numeric.sequence("s1",1,1)

Regards

Shong

Re: Moving 300 files from folder containing 250000 to another folder

DSuryadevara — Wed, 01 Jun 2022 15:43:45 GMT

Hello shong,

thanks for responding, but its still taking time to start the process. looks like tfilelist is loading and then iterating files. the job shows running but nothing is happening.