Retrieving Latest Files From FTP (Java)

[AndyBrown]

I have an FTP account from which I must retrieve the most recent version for four different files. Every day our data source posts four different files with a datestamp appended to the end of the filename... Ex:
idns.txt.11310
lim.txt.11310
sbo.txt.11310
epf.txt.11310
NOTE: The datestamp is mmddyy where mm is a single or double digit, dd is the day, yy is the two-digit year.
If my process does not run for 5 days, I'll end up with 20 files in the directory.
I'm assuming I'll need to use a tFTPFileList and potentially tFileProperties to iterate over all files, but how do I compare datestamps to obtain the 'most recent' version of each of the four file types?

[AndyBrown]

I neglected to mention that I also need to truncate the ".mmddyy" extension upon retrieval to work on the ".txt" files.

[Anonymous]

Hello Andybrown
Here I show you a demo job, feel free to post your question if you have any trouble.
code on tJavaRow_1:

//Code generated according to input schema and output schema
if(input_row.date.length()==5){  //like: 11301
output_row.basename = input_row.basename;
output_row.date = TalendDate.parseDate("Mddyy",input_row.date);
}
else{ //the length of date is 6, like 111209
output_row.basename = input_row.basename;
output_row.date = TalendDate.parseDate("MMddyy",input_row.date);
}

the destination filename on tFIleCopy_1:

((String)globalMap.get("tFileList_1_CURRENT_FILE")).substring(0,((String)globalMap.get("tFileList_1_CURRENT_FILE")).lastIndexOf("."))

About usage of tFlowToIterate component, see 5827
Best regards

shong

[AndyBrown]

Hi Shong,
I greatly appreciate your time in explaining this to me. It's now perfectly understandable. However, I'm getting the following exception on tFTPFileProperties_1...
======
Starting job Get_Feed_Files at 06:06 14/01/2010.
connecting to socket on port 3334
connected
Exception in component tFTPFileProperties_1
com.enterprisedt.net.ftp.FTPException: ./lim.txt.11310: file too large for SIZE when TYPE is ASCII.
at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:999)
at com.enterprisedt.net.ftp.FTPClient.size(FTPClient.java:1953)
at mfr.get_feed_files_0_1.Get_Feed_Files.tFTPFileList_1Process(Get_Feed_Files.java:1792)
at mfr.get_feed_files_0_1.Get_Feed_Files.tFTPConnection_1Process(Get_Feed_Files.java:482)
at mfr.get_feed_files_0_1.Get_Feed_Files.runJobInTOS(Get_Feed_Files.java:2686)
at mfr.get_feed_files_0_1.Get_Feed_Files.main(Get_Feed_Files.java:2522)
578 milliseconds
disconnected
Job Get_Feed_Files ended at 06:06 14/01/2010.
======
The actual size of the file is just 41,045 bytes (yes, just 41.0 KB). This is happening on both active and passive modes. Any ideas?
Thanks,
-Andy

[AndyBrown]

Another note... after changing all connection modes to "active" from the initial connection on down, I'm getting this exception:
=====
Starting job Get_Feed_Files at 06:14 14/01/2010.
connecting to socket on port 3742
connected
Exception in component tFTPFileList_1
java.io.IOException: Unexpected null reply received
at com.enterprisedt.net.ftp.FTPControlSocket.readReply(FTPControlSocket.java:855)
at com.enterprisedt.net.ftp.FTPClient.dir(FTPClient.java:3089)
at mfr.get_feed_files_0_1.Get_Feed_Files.tFTPFileList_1Process(Get_Feed_Files.java:1642)
at mfrget_feed_files_0_1.Get_Feed_Files.tFTPConnection_1Process(Get_Feed_Files.java:482)
at mfr.get_feed_files_0_1.Get_Feed_Files.runJobInTOS(Get_Feed_Files.java:2686)
at mfr.get_feed_files_0_1.Get_Feed_Files.main(Get_Feed_Files.java:2522)
1360 milliseconds
disconnected
Job Get_Feed_Files ended at 06:14 14/01/2010.
=======

[jkrfs]

You have to change the file type in the components setting to Binary from Ascii...

[AndyBrown]

You have to change the file type in the components setting to Binary from Ascii...

Thanks for the response...
That was my first guess based on the exception, however there are no options for "Binary | Ascii" in any of the Java FTP controls. There is only 'mode' (passive vs. active).

[Anonymous]

Hello

however there are no options for "Binary | Ascii" in any of the Java FTP controls. There is only 'mode' (passive vs. active).

The option is on tFTPGet component, so now only use tFTPGet to test if you can get the file to local machine.
Best regards
shong

[AndyBrown]

Hello

however there are no options for "Binary | Ascii" in any of the Java FTP controls. There is only 'mode' (passive vs. active).

The option is on tFTPGet component, so now only use tFTPGet to test if you can get the file to local machine.
Best regards
shong

Unfortunately, I am having major issues with the tFTPGet component in retrieving files from my FTP server. I have a ticket open and a WebEx with Tim Drees next week about this.
In the meantime, is there any other way to set the mode to binary in the tFTPFileProperties component, since this is the component raising the exception?

[jkrfs]

A quick work around would be to use tFtpGet to download the files, do the file check locally (using tFileList) and then send back to another ftpGet which files you want to download.