topic Re: How to close a FTP Connection in Talend in Talend Studio

How to close a FTP Connection in Talend

_AnonymousUser — Wed, 07 Nov 2012 07:54:03 GMT

Hi,
I have an ETL Job where i download a file from FTP (in Passive mode), process it and then load it to DB.
The source FTP has a limitation that it allows only 5 connections from the same IP.
When the files are small, the earlier connections don't timeout. So, when i download the file for 6th day, it fails with the Exception: "com.enterprisedt.net.ftp.FTPConnectionClosedException:Not logged in, only 5 sessions from same IP allowed concurrently."
I cannot reuse the same FTP connection for all days of load as file processing can take time.
So, is there any way we can close the FTP connection after each day's file is processed?
Let me know if there are any other ways of doing it.
Thanks in advance,
Diwakar

Re: How to close a FTP Connection in Talend

Anonymous — Wed, 07 Nov 2012 11:54:04 GMT

Hi
Can you please upload a screenshot of job? How do you schedule the job now?

Re: How to close a FTP Connection in Talend

_AnonymousUser — Thu, 08 Nov 2012 07:38:31 GMT

Hi
Can you please upload a screenshot of job? How do you schedule the job now?

The complete job is quite big, but the psuedo code for the FTP logic is :
tJavaFlex --------------> tFTPConnection -----------------> tFTPGet --------------->Process and load file to DB
(generate iterate OnComponentOk OnComponentOk
ETL Dates)
The FTP Server doesn't allow more than 5 passive connections at a time from the same IP. The connection times out in around 5 mins. But if the File downloading and processing is quick, then the connection limit of 5 is reached before the earlier ones time out.
For initial load, i need to run the job for 30 days but the job fails at 6th day due to the above mentioned issue.
Let me know if there are any confusions/questions?
Thanks,
Diwakar..

Re: How to close a FTP Connection in Talend

Anonymous — Thu, 13 Dec 2012 19:27:49 GMT

Does anyone have a solution for this? I have the exact same problem. Which FTP component issues a quit or bye command to close the FTP connection when done?

Re: How to close a FTP Connection in Talend

Anonymous — Fri, 14 Dec 2012 04:27:12 GMT

tJavaFlex --------------> tFTPConnection -----------------> tFTPGet --------------->Process and load file to DB
(generate iterate OnComponentOk OnComponentOk
ETL Dates)
.

Re: How to close a FTP Connection in Talend

Anonymous — Sat, 15 Dec 2012 06:56:07 GMT

I have created approx. ~20 FTP accounts at InMotionHosting ISP (one for each of our business partners). Our partners put an Excel file on their respective FTP accounts. I have a Talend job that iterates through each of the 20 FTP accounts and downloads the Excel files from each partner. The ISP only supports 8 open FTP sessions from a single IP address.
To simplify my job has: FTP connection (using variable for connection info) --> FTPGetFile --> FTPDeleteFile then iterates to the next partner FTP connection passed to FTP connection variable.
After FTPDeleteFile is completed, how do I ensure the current FTP connection is closed?

Re: How to close a FTP Connection in Talend

Anonymous — Sat, 15 Dec 2012 07:00:43 GMT

OK, i opened a new topic. Thanks in advance for any advice on how an FTP connection is forced closed in Talend.

Re: How to close a FTP Connection in Talend

_AnonymousUser — Mon, 17 Dec 2012 10:38:01 GMT

tJavaFlex --------------> tFTPConnection -----------------> tFTPGet --------------->Process and load file to DB
(generate iterate OnComponentOk OnComponentOk
ETL Dates)
.

To avoid this problem, you can re-design the job as below, create only FTP connection at the beginning of job, and all the ftp component use the same connection.
tFTPConnection
|
onsubjobok
|
tJavaFlex--iterate--tFTPGet--oncomponentok--process and load file to db.
@cocheese, what's your problem? Can you please report a new topic for your problem with details?
Shong
Hi,
I can't create a connection at the beginning and reuse it, as the step "process and load file to db" is not always quick, sometimes if the the files are big, this process can take a lot of time due to which the connection times out.
Thanks,
Diwakar

Re: How to close a FTP Connection in Talend

Anonymous — Mon, 17 Dec 2012 13:34:10 GMT

Hi Diwakar
See my answer in another topic:
http://www.talendforge.org/forum/viewtopic.php?pid=99879#p99879
Shong