topic Re: Retrieving Latest Files From FTP (Java) in Talend Studio

Retrieving Latest Files From FTP (Java)

AndyBrown — Sat, 16 Nov 2024 13:36:45 GMT

I have an FTP account from which I must retrieve the most recent version for four different files. Every day our data source posts four different files with a datestamp appended to the end of the filename... Ex:
idns.txt.11310
lim.txt.11310
sbo.txt.11310
epf.txt.11310
NOTE: The datestamp is mmddyy where mm is a single or double digit, dd is the day, yy is the two-digit year.
If my process does not run for 5 days, I'll end up with 20 files in the directory.
I'm assuming I'll need to use a tFTPFileList and potentially tFileProperties to iterate over all files, but how do I compare datestamps to obtain the 'most recent' version of each of the four file types?

Re: Retrieving Latest Files From FTP (Java)

AndyBrown — Wed, 13 Jan 2010 20:43:48 GMT

I neglected to mention that I also need to truncate the ".mmddyy" extension upon retrieval to work on the ".txt" files.

Re: Retrieving Latest Files From FTP (Java)

Anonymous — Thu, 14 Jan 2010 05:19:02 GMT

Hello Andybrown
Here I show you a demo job, feel free to post your question if you have any trouble.
code on tJavaRow_1:

//Code generated according to input schema and output schema
if(input_row.date.length()==5){  //like: 11301
output_row.basename = input_row.basename;
output_row.date = TalendDate.parseDate("Mddyy",input_row.date);
}
else{ //the length of date is 6, like 111209
output_row.basename = input_row.basename;
output_row.date = TalendDate.parseDate("MMddyy",input_row.date);
}

the destination filename on tFIleCopy_1:

((String)globalMap.get("tFileList_1_CURRENT_FILE")).substring(0,((String)globalMap.get("tFileList_1_CURRENT_FILE")).lastIndexOf("."))

About usage of tFlowToIterate component, see 5827
Best regards

shong

Re: Retrieving Latest Files From FTP (Java)

AndyBrown — Thu, 14 Jan 2010 14:10:32 GMT

Hi Shong,
I greatly appreciate your time in explaining this to me. It's now perfectly understandable. However, I'm getting the following exception on tFTPFileProperties_1...
======
Starting job Get_Feed_Files at 06:06 14/01/2010.
connecting to socket on port 3334
connected
Exception in component tFTPFileProperties_1
com.enterprisedt.net.ftp.FTPException: ./lim.txt.11310: file too large for SIZE when TYPE is ASCII.
at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:999)
at com.enterprisedt.net.ftp.FTPClient.size(FTPClient.java:1953)
at mfr.get_feed_files_0_1.Get_Feed_Files.tFTPFileList_1Process(Get_Feed_Files.java:1792)
at mfr.get_feed_files_0_1.Get_Feed_Files.tFTPConnection_1Process(Get_Feed_Files.java:482)
at mfr.get_feed_files_0_1.Get_Feed_Files.runJobInTOS(Get_Feed_Files.java:2686)
at mfr.get_feed_files_0_1.Get_Feed_Files.main(Get_Feed_Files.java:2522)
578 milliseconds
disconnected
Job Get_Feed_Files ended at 06:06 14/01/2010.
======
The actual size of the file is just 41,045 bytes (yes, just 41.0 KB). This is happening on both active and passive modes. Any ideas?
Thanks,
-Andy

Re: Retrieving Latest Files From FTP (Java)

AndyBrown — Thu, 14 Jan 2010 14:16:49 GMT

Another note... after changing all connection modes to "active" from the initial connection on down, I'm getting this exception:
=====
Starting job Get_Feed_Files at 06:14 14/01/2010.
connecting to socket on port 3742
connected
Exception in component tFTPFileList_1
java.io.IOException: Unexpected null reply received
at com.enterprisedt.net.ftp.FTPControlSocket.readReply(FTPControlSocket.java:855)
at com.enterprisedt.net.ftp.FTPClient.dir(FTPClient.java:3089)
at mfr.get_feed_files_0_1.Get_Feed_Files.tFTPFileList_1Process(Get_Feed_Files.java:1642)
at mfrget_feed_files_0_1.Get_Feed_Files.tFTPConnection_1Process(Get_Feed_Files.java:482)
at mfr.get_feed_files_0_1.Get_Feed_Files.runJobInTOS(Get_Feed_Files.java:2686)
at mfr.get_feed_files_0_1.Get_Feed_Files.main(Get_Feed_Files.java:2522)
1360 milliseconds
disconnected
Job Get_Feed_Files ended at 06:14 14/01/2010.
=======

Re: Retrieving Latest Files From FTP (Java)

jkrfs — Thu, 14 Jan 2010 16:31:38 GMT

You have to change the file type in the components setting to Binary from Ascii...

Re: Retrieving Latest Files From FTP (Java)

AndyBrown — Thu, 14 Jan 2010 22:02:54 GMT

You have to change the file type in the components setting to Binary from Ascii...

Thanks for the response...
That was my first guess based on the exception, however there are no options for "Binary | Ascii" in any of the Java FTP controls. There is only 'mode' (passive vs. active).

Re: Retrieving Latest Files From FTP (Java)

Anonymous — Fri, 15 Jan 2010 02:48:33 GMT

Hello

however there are no options for "Binary | Ascii" in any of the Java FTP controls. There is only 'mode' (passive vs. active).

The option is on tFTPGet component, so now only use tFTPGet to test if you can get the file to local machine.
Best regards
shong

Re: Retrieving Latest Files From FTP (Java)

AndyBrown — Fri, 15 Jan 2010 15:36:06 GMT

Hello

however there are no options for "Binary | Ascii" in any of the Java FTP controls. There is only 'mode' (passive vs. active).

The option is on tFTPGet component, so now only use tFTPGet to test if you can get the file to local machine.
Best regards
shong

Unfortunately, I am having major issues with the tFTPGet component in retrieving files from my FTP server. I have a ticket open and a WebEx with Tim Drees next week about this.
In the meantime, is there any other way to set the mode to binary in the tFTPFileProperties component, since this is the component raising the exception?

Re: Retrieving Latest Files From FTP (Java)

jkrfs — Fri, 15 Jan 2010 16:37:09 GMT

A quick work around would be to use tFtpGet to download the files, do the file check locally (using tFileList) and then send back to another ftpGet which files you want to download.

Re: Retrieving Latest Files From FTP (Java)

AndyBrown — Fri, 15 Jan 2010 16:40:33 GMT

What I ended up doing is to bring the files down to a temp directory using a tFTPFileList and a tSystem (using a curl command for each file), then iterating over the local files, moving them to my inbound directory. I'll delete the source files both locally and on the FTP server so I only have to process a handful each time. Until my FTP issue is identified, this shall be it.
Thanks for the assist.

Re: Retrieving Latest Files From FTP (Java)

jkrfs — Fri, 15 Jan 2010 19:12:00 GMT

Would love to check your job out if you're willing to share. Sounds like a good work around...

Re: Retrieving Latest Files From FTP (Java)

Anonymous — Wed, 19 Jan 2011 10:01:17 GMT

hey,
please i have problem
my team is working at BI prject with talend to retrieve data from datasource using ftp servers to another place, please if anyone can help me and tell me how this processus work ??