Need to push Files from an FTP Server to AWS S3 bucket directly using Talend Cloud without change in target filename

[Paar]

I am trying to push files from FTP to AWS S3 bucket directly using Talend Pipeline Designer , it is working but the target filename is like "part-00-....". I intend to get the target filename same as source filename. Even tried using tftpget in Talend cloud installed in local I can only specify local directory in tftpget. Any idea how to overcome this issue?

[Anonymous]

The Pipeline Designer method will be using multipart upload functionality with S3. This is why you get the part-00... names. Can you share a screenshot of your pipeline so that I can get a better idea of what you are doing?

I would say that moving files is probably better handled by a job in Talend Cloud. You can specify filenames using the filemask settings....

https://help.talend.com/r/en-US/8.0/ftp/tftpfilelist-tftpconnection-tlogrow-listing-all-files-folders-on-ftp-root-directory-standard-component

[Paar]

Hi Rhall ,

Thanks for the reply , I tried to implement using Talend Cloud Job but using tftpget component I can transfer files only from FTP to Local Directory. I intend to transfer files from FTP to AWS S3 Directly without using local Directory.

I have tried this operation both in Pipeline Designer and as Talend Cloud Job.

In Pipeline Designer I am getting Target filename as "Part-00-..."

In Job it is not possible to Transfer files without using local Directory.

So is there anyway to implement this operation in Talend Cloud Pipeline Designer or as Talend Cloud Job.

I have also attached the Screenshot of Talend Cloud Pipeline that I have implemented.

[Afoster1677824530]

Hello,

It is our pleasure to see you query hope so our suggestion will helpful for you now let see

As you asked ....Yes, you can overcome this issue by using a combination of Talend components that are specifically designed for this purpose.

Here is an outline of the steps you can follow to achieve this:

1. Use the "tFTPGet" component to download the files from FTP server to a temporary local directory.
2. Use the "tFileList" component to retrieve the list of files from the temporary directory.
3. Use a loop (for-each) to iterate over the list of files retrieved by the "tFileList" component.
4. For each file, extract the filename and path using the "tFileProperties" component.
5. Use the "tS3Put" component to upload the file to S3, using the extracted filename as the target filename.
6. Finally, use the "tFileDelete" component to delete the file from the temporary directory.

Here's a quick example of what the Talend Pipeline Designer job might look like:

scss

tFTPGet ---(onSubjobOk)---> tFileList ---(iterate)---> tFileProperties ---(main)---> tS3Put ---(onSubjobOk)---> tFileDelete

You can customize this job by setting the appropriate properties of each component, such as the FTP server details, the S3 bucket details, and the temporary directory path.

I hope this helps! https://www.checkmyrota.net/