topic Re: tFTPGet - Super Slow in Talend Studio

tFTPGet - Super Slow

talendtester — Sat, 16 Nov 2024 06:44:56 GMT

1. Is there a component faster than tFTPGet?

2. Is there some setting which can make the tFTPGet transfer speed faster?

3. Is there some config setting which will improve the transfer speed?

I am using TOS Version: 7.1.1

We have daily files which are 34GB compressed to FTP from an AWS external server which I am told is not limited.

Using the tFTPGet component I can get the file, the issue is the transfer speed is less than 1MB per second.

When I transfer the files via CyberDuck I am seeing speeds of 7.8MB per second, so it takes around an hour per file.

Re: tFTPGet - Super Slow

vapukov — Sat, 26 Jan 2019 12:07:20 GMT

hi,

there are no special settings for this component

you could test with command line scp (winscp in case of windows), if it will be also faster - then use tSystem for call command line utility

Re: tFTPGet - Super Slow

talendtester — Mon, 28 Jan 2019 19:03:47 GMT

With winSCP I am getting between 1.5MB and 2MB per second.

Re: tFTPGet - Super Slow

vapukov — Mon, 28 Jan 2019 22:26:03 GMT

as I understand CyberDuck support multithread for downloading (multiple-connections-for-file-transfers )

winscp - not

so you can check by searching command line clients like - https://aria2.github.io/

or test command line version of CyberDuck - https://duck.sh/

Re: tFTPGet - Super Slow

talendtester — Thu, 31 Jan 2019 21:26:29 GMT

I used tLibraryLoad to load edtftpj-2.0.4.jar then in tJava:

((com.enterprisedt.net.ftp.FTPClient)globalMap.get("conn_tFTPConnection_1")).setNetworkBufferSize(65536);
((com.enterprisedt.net.ftp.FTPClient)globalMap.get("conn_tFTPConnection_1")).setTransferBufferSize(65536);

Error:

Detail Message: The method setNetworkBufferSize(int) is undefined for the type FTPClient

Re: tFTPGet - Super Slow

vapukov — Thu, 31 Jan 2019 21:54:42 GMT

why you manually load library, if it already used by component?

and why you are use libray from 2009?!!

Version 2.0.4
(13 February 2009)

Re: tFTPGet - Super Slow

talendtester — Thu, 31 Jan 2019 22:55:08 GMT

If I don't I am getting this error:

Detail Message: com.enterprisedt cannot be resolved to a type

I downloaded edtftpj-2.5.0 to use with tLLibraryLoad, now I get:

java.lang.ClassCastException: com.jcraft.jsch.ChannelSftp cannot be cast to com.enterprisedt.net.ftp.FTPClient

Re: tFTPGet - Super Slow

vapukov — Fri, 01 Feb 2019 00:44:31 GMT

my question was - why you are use outdated library?

in any case starting from begin of the thread, I not sure increasing only this parameter resolve your problem

Duck work fast because run several threads in parallel, in description of library I not found nothing about this

Re: tFTPGet - Super Slow

talendtester — Fri, 01 Feb 2019 01:01:22 GMT

The duck.sh website is blocked by our firewall.

Is this what you are suggesting?
https://blog.cyberduck.io/2015/06/24/scripting-cloud-storage-copy-using-the-command-line-interface-cli/

Re: tFTPGet - Super Slow

vapukov — Fri, 01 Feb 2019 01:15:03 GMT

yes, but just as variant

search google - multithread FTP command line client and test what better for you