[resolved] FTP timeout

[Anonymous]

Dear all,
I download files from a ftp server. See my Job design attached.
What I want to do in detail:
On FTP following structure is given:
dir1
-file1
-file2
dir2
-file4
-flie5
I want to copy the whole directory structure. file 1 and file 2 should be deposit in dir1 on my local server etc. That is the reason why I use two FTP FileLists. FtpFileList_1 reads dir1, dir2 etc. and FtpFileList_2 reads the files within the directories. So I'm able to re-use the component parameter to determine the destination directory (see GetZipsFromIn).
Second, I want to make sure the upload of the files is done - that's why I use FTPFileProperty (mtime) and filter for it.
Furthermore I use FtpRename (RenameToPROGRESS) to rename the file before I download it (regard iteration order). This makes sure that a second job (I use cronjob) won't download a file that is currently downloaded by the first job (file mask in FtpFileList_2). Anyway, everything works perfect.
The job downloaded up to 90 files. But whenever the download took more than 5 minutes, I got error messages (see below). I tested it out with one huge file and lots of small/normal files. So too many connections won't be a problem.
Strangely the error occurs after the file(s) has been downloaded even when it took more than 5 minutes! But it leads to an aboration of the job!
Parallel I tested it out with FileZilla (same server, same user). There I didn't have any trouble. So FTP server don't have any restrictions.
Also I modfied the job several times - I got rid of the flowtoiterate component or modified iterations to flows. But every time, I got the same error. I cannot use the ftp connection component, because each component needs a "direct" connection.
Any ideas? It might be a problem to use the component parameter too often..
Error message (1 big file):
Exception in component tFTPFileProperties_1
com.enterprisedt.net.ftp.FTPConnectionClosedException: Idle timeout (300 seconds): closing control connection
at com.enterprisedt.net.ftp.FTPControlSocket.validateReplyCode(FTPControlSocket.java:1217)
at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:1167)
at com.enterprisedt.net.ftp.FTPClient.quit(FTPClient.java:3949)
at submissionchecker.tesdsg4_0_1.tesdsg4.tFTPFileList_1Process(tesdsg4.java:1806)
at submissionchecker.tesdsg4_0_1.tesdsg4.runJobInTOS(tesdsg4.java:2503)
at submissionchecker.tesdsg4_0_1.tesdsg4.main(tesdsg4.java:2368)
Error message (multiple files with various file size; 0,5 MB - 40 MB):
Exception in component tFTPFileList_1
com.enterprisedt.net.ftp.FTPConnectionClosedException: No transfer timeout (300 seconds): closing control connection
at com.enterprisedt.net.ftp.FTPControlSocket.validateReplyCode(FTPControlSocket.java:1217)
at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:1167)
at com.enterprisedt.net.ftp.FTPClient.quit(FTPClient.java:3949)
at submissionchecker.executetranspect_0_3.ExecuteTranspect.tFTPFileList_2Process(ExecuteTranspect.java:4714)
at submissionchecker.executetranspect_0_3.ExecuteTranspect.tFixedFlowInput_1Process(ExecuteTranspect.java:3747)
at submissionchecker.executetranspect_0_3.ExecuteTranspect.runJobInTOS(ExecuteTranspect.java:6090)
at submissionchecker.executetranspect_0_3.ExecuteTranspect.runJob(ExecuteTranspect.java:5936)
at submissionchecker.master_0_1.Master.tRunJob_1Process(Master.java:726)
at submissionchecker.master_0_1.Master.tRunJob_3Process(Master.java:572)
at submissionchecker.master_0_1.Master.runJobInTOS(Master.java:995)
at submissionchecker.master_0_1.Master.main(Master.java:839)

[Anonymous]

I found out the reason why the connection was terminated. The reason is the FTP server (TransferTimout, TransferIdle). I increased the time to more than five minutes and it worked.
For me this inidicates that a not-in-use ftp connection is all the time open (it might be the ftp file lists components). This would not be a problem for the download itself, but for the job (because I cannot capture the error message even when I use the subjob error or component error link).

[Anonymous]

Hi _OZ_
thats because I have recently published a tFTPClose component because the tFTPConnection component or an other FTP components does not close the connection. Only the death of the job closes the connection.

[Anonymous]

Dear jlolling,
thanks for your reply.
I downloaded the ftpclose component too. But I only can use it, when i pick up a ftpconnection component. But using a ftpconnection component doesn't make sense, because I have to open a connection for each component (ftpfilelist, ftpfileproperty etc.) separately.
Or does it mean ftpclose closes all open connection even when it is a connection opened with other ftp components?

[Anonymous]

For your luck, the tFTPget/put/list connections closes it internal connections for it self. Take a look at the source code and scroll to the end part and you will find the close/disconnected call.
The tFTPClose closes ONLY the connection which are established from a tFTPConnection component, anything else does not make sense.
BUT:
If you start transfers within a loop or iteration take care you do not establish to much connections:
tFileList --(iteration)--> tFTPPut (creates it own connection for every file!)
this will cause probably a kind of problem you have. You can adapt this example also for the get operation.

[Anonymous]

Make sense that each component closes the ftp connection for itself.
I guess the problem is the FTPFileList. It will keep the connection alive until all list objects (files/directories) are processed respectively until the subjob is done (which takes in my case a while). Therefore the only solution is to increase the server params... I can live with that solution, but honestly it would be better to capture/to get rid of the error message or the redesign the job (but honestly I have no idea how to design it without a FtpFileList)...
Thanks again for ideas/help. I appreciate it.

[Anonymous]

Hi,
You can store the list (from FTPFileList) in a:
- temporary file
- temporary buffer (using the "technical" tHashOutput/tHashInput components)
So your FTP connection will be closed faster
regards

[Anonymous]

@svert. Thanks for this hint. A good idea to use the hash components. When I have time, I give it a trial.

[Anonymous]

Hi ,
does the tftp get close its connection?

[Anonymous · Accepted solution]

Hi twincam16,
So far, there is no tFTPClose component in talend. You can vote for this feature jira issue https://jira.talendforge.org/browse/TDI-20134 or have a look at custom component TalendExchange:tFTPClose written by jlolling to see if it is OK with you.
Best regards
Sabrina