topic Re: SFTP multiple files using tFTPPut in Talend Studio

SFTP multiple files using tFTPPut

iamabhishek — Sun, 06 May 2018 18:54:52 GMT

Trying to build a solution to transfer files to an FTP server from local on-premise system. Currently the process is only transferring the last file of the lot found from the directory. Tried debugging the job but couldn't understand where the actual problem is.

What I have tried so far:

Used tFileList to get the list of files for a given directory, given "*.txt" as FileMask.

Used tIterateToFlow to iterate through the list of files. The Schema is of a single column "fileName" with the value set as - ((String)globalMap.get("tFileList_1_CURRENT_FILE"))

Finally used tFTPPut to SFTP the files to remote directory. Local Directory is set as - ((String)globalMap.get("tFileList_1_CURRENT_FILEDIRECTORY")) & FileMask as - ((String)globalMap.get("tFileList_1_CURRENT_FILE")). Overwrite is the option chosen.

Screenshot of the job -

Thanks in advance for any kind of help and let me know if more information is required for this.

Platform: Talend Open Studio for Data Integration Version: 6.5.1

Re: SFTP multiple files using tFTPPut

manodwhb — Mon, 07 May 2018 03:45:58 GMT

@iamabhishek,you no need to use tIteratetoFlow,since you can directly connect Iterate flow from tFilelist to tFTPPut and in between you cna use tJava componet to print the current file name.

Re: SFTP multiple files using tFTPPut

iamabhishek — Mon, 07 May 2018 05:36:07 GMT

@manodwhb Have tried that too. If we don't use tIterateFlow and directly pass the iteration from tFileList to tFTPPut then only the first file would be retrieved and sent and for the second file there would be en error - "No File Present".

This is the output when debugged using tJava -

[statistics] connecting to socket on port 3583
[statistics] connected
[trace] connecting to socket on port 5031
[trace] connected
Output.txt
C:/FTP/dir
Sample.txt
C:/FTP/dir
No File Present
[statistics] disconnected
[trace] disconnected

Re: SFTP multiple files using tFTPPut

manodwhb — Mon, 07 May 2018 06:07:48 GMT

@iamabhishek,i have used below way and working for me.check your desing based on this.

Re: SFTP multiple files using tFTPPut

manodwhb — Mon, 07 May 2018 06:18:34 GMT

@iamabhishek,below settings also working as expected.

Re: SFTP multiple files using tFTPPut

iamabhishek — Mon, 07 May 2018 06:25:37 GMT

@manodwhb Thank you for the solution.

Noticed that you have used "*.txt" as FileMask in the tFTPPut which ensures that it will pull all the files from the CURRENT_FILEDIRECTORY. In that case we don't even need to iterate the files in the directory and we don't need tFileList too.

But the desired approach should have been like get the files using tFileList and iterate over the list and using tFTPPut transfer the files. The FileMask should be used in tFileList and whatever the filename is passed in that iteration tFTPPut should use that preferably using - CURRENT_FILE

Re: SFTP multiple files using tFTPPut

manodwhb — Mon, 07 May 2018 06:28:21 GMT

@iamabhishek,you can find my other sceernshot i have used ((String)globalMap.get("tFileList_1_CURRENT_FILE")) as file mask in tFTPPut and working fine.

Re: SFTP multiple files using tFTPPut

manodwhb — Mon, 07 May 2018 06:31:07 GMT

@iamabhishek

Re: SFTP multiple files using tFTPPut

iamabhishek — Mon, 07 May 2018 06:38:13 GMT

Thank you @manodwhb, sorry I might have overlooked your reply.

Yes, your solution is working perfectly for the list of files in local directory much appreciated.

One thing I noticed here is and the reason for my process not working was -

I was using tFTPPut --> (on component ok) --> tFTPClose , wherein you have used

tFileList --> (on subjob ok) --> tFTPClose which has clicked.

Any idea on this.

Re: SFTP multiple files using tFTPPut

manodwhb — Mon, 07 May 2018 06:47:06 GMT

@iamabhishek,since you need to design job as i shared in screenshot, other wise in your design first file will be placed and for second file tFTPPUT will fail that connection alreday closed.that you have used on component ok of tFTPPUt to tFTPCLOSE

Re: SFTP multiple files using tFTPPut

manodwhb — Mon, 07 May 2018 06:47:40 GMT

@iamabhishek,please accept solution.

Re: SFTP multiple files using tFTPPut

iamabhishek — Mon, 07 May 2018 09:05:13 GMT

Thanks for the explanation, makes sense now.