How to process a file stored on remote server, without copying it?

[Anonymous]

Hi

Is it possible to process a file stored on a remote server without copying it?

I'm working on very large files and would prefer not to store the data locally.

Thanks

[TRF]

If you can reach the file using a remote folder, it's possible, else you need to transfer it (using a ftp transfer or other).

[Anonymous]

Why do you not want to copy the file? If it is for performance reasons, trying to process a file remotely will not necessarily improve the situation. It would be far safer and more efficient to copy and process locally. 

[Anonymous]

Hi,

Can you detail a little bit more your use case? What's your further processing on your large file which is on remote server? FTP server?

Best regards

Sabrina

[Anonymous]

---

@TRF wrote:
If you can reach the file using a remote folder, it's possible, else you need to transfer it (using a ftp transfer or other).

---

Hi @TRF

Sorry I've been away. I haven't tried to add it as a remote folder I'll try and let you know.

[Anonymous]

---

@rhall wrote:

Why do you not want to copy the file? If it is for performance reasons, trying to process a file remotely will not necessarily improve the situation. It would be far safer and more efficient to copy and process locally. 

---

Hi @rhall.

Sorry for the delayed response. Yes, its the performance that concerns me, as well as the local storage. I'd like to run jobs concurrently which would mean filling up the local machine quite quickly even if I delete the file at the end of each run.

[Anonymous]

---

@xdshi wrote:

Hi,

Can you detail a little bit more your use case? What's your further processing on your large file which is on remote server? FTP server?

Best regards

Sabrina

---

Hi Sabrina

One case is where I'm comparing a file to its equivalent database table after a load, I just want to get the file value, compare and forget about it.

[Anonymous]

---

@accuracie wrote:

---

@TRF wrote:
If you can reach the file using a remote folder, it's possible, else you need to transfer it (using a ftp transfer or other).

---

Hi @TRF

Sorry I've been away. I haven't tried to add it as a remote folder I'll try and let you know.

---

I just realized the server I'm accessing is an SFTP server not FTP, to my knowledge this isn't possible for SFTP?

Please advise on how to map an SFTP server on windows, or a point in the right direction please.

[Anonymous]

You will have more hard drive space than RAM. The computer running the job's RAM will be used even if you do not download the file and leave it where it is when processing. Processing the data on the local machine will be quicker and (as you said) the file can just be deleted after the job is done.

There is another way you could approach this though. Would it be an option to send the job to the machine with the files and then start it on that machine using SSH? This would potentially achieve your requirements and is certainly very possible with Talend.

[Anonymous]

---

@rhall wrote:

You will have more hard drive space than RAM. The computer running the job's RAM will be used even if you do not download the file and leave it where it is when processing. Processing the data on the local machine will be quicker and (as you said) the file can just be deleted after the job is done.

There is another way you could approach this though. Would it be an option to send the job to the machine with the files and then start it on that machine using SSH? This would potentially achieve your requirements and is certainly very possible with Talend.

---

I've seen a few examples where people "send" the job, I'll explore this as well but do you mind explaining how that works exactly. I haven't gotten a grip on the concept just yet.

Also, I really don't feel comfortable with copying the data but its about logic more than my feelings so would you choose copying the file over sending the job to the server, if possible that is?

Thanks