Transferring the entire file from sFTP Server.

[Sanjay5]

Hi,

Is there a way to download the entire file from sFTP server instead of downloading it record by record.

The tFTPGet component which I used in my job to download the file from SFTP server is downloading record by record instead of entire file.

If possible, I would like to download the entire file.

[Anonymous]

Can you demonstrate what you mean please? The tFTPGet component acts like a normal FTP Get. It simply downloads the file. The contents of the file are ignored. How is this not downloading the entire file?

[Anonymous]

Hi,

Do you want to move your files from SFTP server to your local disk?

Best regards

Sabrina

[Sanjay5]

I am trying to transfer a file using tFTPGet to Talend Job Server.

File size is 65GB.

Total transfer time is around 4 hours.

I wanted to reduce the time duration, so I split the file into two chunks of 32.5 GB each.

I started transfer process in parallel, I thought the time duration would reduce to around 2 hours.

But file transfer duration remains 4 hours even for parallel processing.

I realized that transfer process is record by record, and parallelisms does not give me any benefit.

So, I wanted to what should I do, If I want to reduce the time duration of File transfer from FTP server to my Job Server.

If there is a way to transfer the entire file (not record by record) that would reduce the time maybe. 

[Anonymous · Accepted solution]

OK, where to start?

1) Splitting a file in two and expecting to increase the performance of a transfer over the same pipe when sending both files is not going to work. Its essentially like splitting water into two buckets and trying to pour them both through the same pipe in order to improve performance. What is your "pipe's" bandwidth? You can only go as fast as that will allow.
2) The files are not coming across "record by record". That is not how FTP works. You *may* be seeing the files build up incrementally from beginning to end, but that does not mean it is "record by record". FTP does not work that way.

The only way you will improve the speed is to either increase the bandwidth or reduce the size of the file. I would attempt to zip the file before transfer and then unzip the other side. Another way of improving performance is to chop the files up into much smaller chunks and send them over in chunk order (if you need to process the file in an order). You can then start processing the data as soon as the first file arrives.