topic Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server in Talend Studio

What is the order on which tFTPFileList components decides to iterate the file from FTP server

Sanjay5 — Tue, 17 Apr 2018 22:11:04 GMT

Hi,

I am looking to get some info on the order on which tFTPFileList component decides to iterate the files from FTP server.

I have 3 files in the server with different timing.

File1 : Apr 17 11:20

File2 : Apr 17 13:20

File3 : Apr 17 14:20

Which file would be pulled first?

When i run my job, files are not getting downloaded based on timing.

I expect File1 to be pulled first. (First come first serve logic) But that is not happening.

I see "Order By" option in tFileList component but that is not there in tFTPFileList.

Any thoughts?

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

manodwhb — Wed, 18 Apr 2018 06:43:31 GMT

@stamrake,in tFTPFilelist you do not have "order by" Option,so first try to get the files in local directory and then list the files using tFileList,where you can specify the order by.

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

Sanjay5 — Wed, 18 Apr 2018 17:31:41 GMT

@manodwhb

Thanks Manohar for the idea,

It's a good workaround however still I am not sure how do i know that which file got pulled to local directory first.

If FILE3 gets pulled to local directory first and if I sort by time, it means I would be first processing FILE3 instead of FILE1.

Regards,

Sanjay T

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

manodwhb — Thu, 19 Apr 2018 05:19:01 GMT

@stamrake,can you sort by modified date on descending order then you may get File1.

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

manodwhb — Fri, 20 Apr 2018 12:37:00 GMT

@stamrake,still do you have an issue?

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

Sanjay5 — Mon, 23 Apr 2018 17:34:00 GMT

@manodwhb

Hi, Thanks for the idea.

This trick could give me FILE1 to be processed at first, but there is no guarantee for that, as we do not have sorting option in tFTPFileList.

For my use case i need to be certain of which file is getting processed first.

Regards,

Sanjay

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

manodwhb — Tue, 24 Apr 2018 06:45:42 GMT

@stamrake,in tFTPFile list we do not have option to sort, since you need to get the files in to local directory,from there you need to list using tFilelist. where in tFilelist you can specify the order by filename or file size or modified date or by default with acesnding or desending.

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

Sanjay5 — Tue, 24 Apr 2018 17:30:13 GMT

@manodwhb

Hi,

I agree with your suggestion for the work around.

Guess what, I have requested Talend Enterprise team to have a feature request on the component tFTPFileList to have "Order By" Option.

Feature request detail: (TDI-40435)