Hi,
I have an ETL Job where i download a file from FTP (in Passive mode), process it and then load it to DB.
The source FTP has a limitation that it allows only 5 connections from the same IP.
When the files are small, the earlier connections don't timeout. So, when i download the file for 6th day, it fails with the Exception: "com.enterprisedt.net.ftp.FTPConnectionClosedException:Not logged in, only 5 sessions from same IP allowed concurrently."
I cannot reuse the same FTP connection for all days of load as file processing can take time.
So, is there any way we can close the FTP connection after each day's file is processed?
Let me know if there are any other ways of doing it.
Thanks in advance,
Diwakar
Hi Can you please upload a screenshot of job? How do you schedule the job now?
The complete job is quite big, but the psuedo code for the FTP logic is :
tJavaFlex --------------> tFTPConnection -----------------> tFTPGet --------------->Process and load file to DB
(generate iterate OnComponentOk OnComponentOk
ETL Dates)
The FTP Server doesn't allow more than 5 passive connections at a time from the same IP. The connection times out in around 5 mins. But if the File downloading and processing is quick, then the connection limit of 5 is reached before the earlier ones time out.
For initial load, i need to run the job for 30 days but the job fails at 6th day due to the above mentioned issue.
Let me know if there are any confusions/questions?
Thanks,
Diwakar..
Does anyone have a solution for this? I have the exact same problem. Which FTP component issues a quit or bye command to close the FTP connection when done?
tJavaFlex --------------> tFTPConnection -----------------> tFTPGet --------------->Process and load file to DB (generate iterate OnComponentOk OnComponentOk ETL Dates) .
To avoid this problem, you can re-design the job as below, create only FTP connection at the beginning of job, and all the ftp component use the same connection.
tFTPConnection
|
onsubjobok
|
tJavaFlex--iterate--tFTPGet--oncomponentok--process and load file to db.
@cocheese, what's your problem? Can you please report a new topic for your problem with details?
Shong
I have created approx. ~20 FTP accounts at InMotionHosting ISP (one for each of our business partners). Our partners put an Excel file on their respective FTP accounts. I have a Talend job that iterates through each of the 20 FTP accounts and downloads the Excel files from each partner. The ISP only supports 8 open FTP sessions from a single IP address.
To simplify my job has: FTP connection (using variable for connection info) --> FTPGetFile --> FTPDeleteFile then iterates to the next partner FTP connection passed to FTP connection variable.
After FTPDeleteFile is completed, how do I ensure the current FTP connection is closed?
tJavaFlex --------------> tFTPConnection -----------------> tFTPGet --------------->Process and load file to DB (generate iterate OnComponentOk OnComponentOk ETL Dates) .
To avoid this problem, you can re-design the job as below, create only FTP connection at the beginning of job, and all the ftp component use the same connection.
tFTPConnection
|
onsubjobok
|
tJavaFlex--iterate--tFTPGet--oncomponentok--process and load file to db.
@cocheese, what's your problem? Can you please report a new topic for your problem with details?
Shong
Hi,
I can't create a connection at the beginning and reuse it, as the step "process and load file to db" is not always quick, sometimes if the the files are big, this process can take a lot of time due to which the connection times out.
Thanks,
Diwakar
