topic How to get file from ftp and load to HDFS? in Talend Studio

How to get file from ftp and load to HDFS?

Anonymous — Sat, 16 Nov 2024 04:27:32 GMT

Hi,
I want to get files from FTP and load into HDFS, but the component tFTPGet just can move files into a local directory. How can I move into the HDFS? Thank you

Re: How to get file from ftp and load to HDFS?

TRF — Mon, 07 Oct 2019 06:56:11 GMT

You need to proceed in 2 steps:
- tFtpGet to the file locally
- tHDFSOutput to push from local to HDFS

Re: How to get file from ftp and load to HDFS?

manodwhb — Mon, 07 Oct 2019 08:20:22 GMT

@fransiskakd ,you can design in below way also.

Re: How to get file from ftp and load to HDFS?

Anonymous — Mon, 07 Oct 2019 08:23:28 GMT

Thanks for your response , but I do not want to store the data in local repository because the talend job will upload in talend server and run automatically without having to open my local directory. Is there any other way?

Re: How to get file from ftp and load to HDFS?

TRF — Mon, 07 Oct 2019 08:33:21 GMT

@fransiskakd you don't have choice.

tHDFSOutput component requires an input flow so you have to get the file locally (locally mining where the job runs) then read the file to push data into tHDFSOutput and finally, delete the local file.