topic Re: transfer folder from local file system to HDFS in Talend Studio

transfer folder from local file system to HDFS

Anonymous — Sat, 16 Nov 2024 08:30:19 GMT

hello all..

I need to transfer folder including many sub folders and files to hdfs.

tHdfsput is used to transfer a file to but i want to transfer whole folder having same hierarchical structure of sub folders and files.

i found a connector tHDFSOutputRaw but how to use it means what will be the input source for tHDFSOutputRaw .

please reply. I need this on urgent basis . I am new to talend.

please help

Re: transfer folder from local file system to HDFS

Anonymous — Mon, 26 Mar 2018 08:31:39 GMT

Hi Shalla,

If u want to transfer the folder and all the sub folders inside HDFS, then u can put an thdfslist(check the box include subdirectories) and then iterate it to thdfsput.

Re: transfer folder from local file system to HDFS

Anonymous — Mon, 26 Mar 2018 09:30:47 GMT

thanku @Ameesh

thdfsList take the files from hdfs directory.

folders are on my local machine and through talend i want to move complete folder as it is.

Please reply as soon as possible

Re: transfer folder from local file system to HDFS

Anonymous — Mon, 26 Mar 2018 09:32:33 GMT

tfilelist sorry!

Re: transfer folder from local file system to HDFS

Anonymous — Mon, 26 Mar 2018 09:35:30 GMT

Hello,

tFileList will take the files from your local machine and iterate it to thdfsput where the files will get moved to hdfs.

. tFileList-->tHDFSPut

Best regards

Sabrina

Re: transfer folder from local file system to HDFS

Anonymous — Mon, 26 Mar 2018 11:11:50 GMT

@xdshi

tFileList is not getting connect to tHDFSOutput so i used

tFileList-tIterateToFlow-tHDFSOutput

and create a mapping in tIterateToFlow like key = (String)globalMap.get("tFileList_1_CURRENT_FILEPATH")

but it showing error ""can not create a path from an empty string)

I have a folder structure like this at my local machine

Folder A

Folder B (having many xml file)

Folder C (having many pdf file)

Folder T (having images)

and i want the same folder structure to be transferred to hdfs-(all folders and sub folders and files in the same way as it is)

please reply soon

Re: transfer folder from local file system to HDFS

Anonymous — Mon, 26 Mar 2018 11:18:12 GMT

Hello,

Why don't you try to use tfilelist-->tHDFSPut?

Best regards

Sabrina

Re: transfer folder from local file system to HDFS

Anonymous — Mon, 26 Mar 2018 11:22:09 GMT

@xdshi

A link can not be drown between tFileList and tHDFSOutput.

tFileList is not getting connct directly to tHDFSOutput

Re: transfer folder from local file system to HDFS

Anonymous — Mon, 26 Mar 2018 13:43:34 GMT

Please select iterate in the link, its working fine.

tfilelist--->thdfsput

Re: transfer folder from local file system to HDFS

Anonymous — Tue, 27 Mar 2018 04:30:04 GMT

Hello,

tHDFSOutput component runs standalone and does not generate input or output flow for the other components.

You need to use Iterate connection instead of Row connection.

For more information about connection type, please refer to this online document:TalendHelpCenter:Connection types.

Best regards

Sabrina

Re: transfer folder from local file system to HDFS

Anonymous — Wed, 04 Dec 2019 05:36:02 GMT

Hi,

Can u please show the data integration between the DI and BD job in talend?