topic Re: Talend Open Studio FTP Server - FTP Server File Transfer in Talend Studio

Talend Open Studio FTP Server - FTP Server File Transfer

gcasey — Mon, 04 Jun 2018 20:35:27 GMT

Hi,

My predecessor built some scripts to transfer files between two different FTP servers. I need to recreate or modify them to do the same thing, but with a different server/subdirectory on the GET side and the same server/different subdirectory on the PUT side. I've tried reverse engineering the old scripts and modifying them to do this in TOS, but I keep running into errors. Essentially, here's what my script does:
1. tFTPConnect to local FTP server

2. tFTPConnect to remote FTP server

3. tFTPFileList to remote FTP server subdirectory

4. Iterate tFTPGet to copy files from remote FTP server subdirectory and save a copy to a local network share

5. Iterate tFTPPut to place files from the local network share in the local FTP server subdirectory

6. Iterate tFTPDelete files from the remote FTP server

7. tIterateToFlow and tFileOutputDelimited to log the results

I have repositories setup for both FTP server connections and I have setup several string contexts for the subdirectory paths, but I can't get Talend to see the file subdirectory paths in the contexts as valid. I also have a log catcher on some of the subjobs and email notifications on subjob errors, but I still can't debug this. I'm not a programmer, so I'm no good at debugging on the code side. Can anyone provide some guidance on the config side to make this work?

Re: Talend Open Studio FTP Server - FTP Server File Transfer

vapukov — Mon, 04 Jun 2018 23:08:45 GMT

logic could be different for each project, but You can combine components, for example:
- list folders

- use folder name as part for tFTPGet

- You can use tSCPFileList, which is little more intellectual:

Re: Talend Open Studio FTP Server - FTP Server File Transfer

gcasey — Tue, 05 Jun 2018 19:30:26 GMT

Hi,

Thanks for the reply. This is what I have setup right now in Designer.
What I'm doing now (or trying to do) is get the list of files (*.*) from the source FTP server using tFTPFileList. Then I'm going to use tFTPGet to download them to a local network directory. So far, this is working fine. Then I use tFTPPut to take those files from the local network directory and put them on the destination FTP server. Lastly, I delete the files from the source FTP server. Here is my problem:

1. I need to run this about every hour and I will constantly have a random number of new files coming in from the source FTP server. When tFTPPut executes, I need it to just grab the new files from the local directory that have imported since the last script run and move them to the destination FTP server. I think I can do this with Filemasking, but I don't know what to use. Can anyone recommend a solution for this?

Re: Talend Open Studio FTP Server - FTP Server File Transfer

gcasey — Tue, 05 Jun 2018 19:35:39 GMT

Another issue I have is with my context variables. I created a few string contexts for my folder paths to the FTP subdirectories and the local network path. When I add the context to the component attributes and run the script it gives me an FTP 550 error on the context.contextname saying there is no such file or folder. I have confirmed my FTP creds have permissions to the folders. I'm not sure why it does this. I've had to put the filepath in the component directory fields as a workaround. It works fine this way, but I'm not sure why the contexts I've created don't work. I've searched the community and web, but can't find any resolution. Anyone have any suggestions?

Re: Talend Open Studio FTP Server - FTP Server File Transfer

gcasey — Tue, 05 Jun 2018 21:03:00 GMT

Just realized I should probably have posted this in Design and Development. Closing this thread and moving it there.