transfer folder from local file system to HDFS

[Anonymous]

hello all..

I need to transfer folder including many sub folders and files to hdfs. 

tHdfsput is used to transfer a file to but i want to transfer whole folder having same hierarchical structure of sub folders and files.

i found a connector tHDFSOutputRaw but how to use it means what will be the input source for  tHDFSOutputRaw .

please reply. I need this on urgent basis . I am new to talend.

please help

[Anonymous]

hello all..

I need to transfer folder including many sub folders and files to hdfs. 

tHdfsput is used to transfer a file to but i want to transfer whole folder having same hierarchical structure of sub folders and files.

i found a connector tHDFSOutputRaw but how to use it means what will be the input source for  tHDFSOutputRaw .

please reply. I need this on urgent basis . I am new to talend.

please help

[Anonymous]

Hi Shalla,

If u want to transfer the folder and all the sub folders inside HDFS, then u can put an thdfslist(check the box include subdirectories) and then iterate it to thdfsput. 

[Anonymous]

thanku @Ameesh

thdfsList take the files from hdfs directory.

folders  are on my local machine and through talend i want to move complete folder as it is.

Please reply as soon as possible

[Anonymous]

tfilelist sorry!

[Anonymous · Accepted solution]

Hello,

tFileList will take the files from your local machine and iterate it to thdfsput where the files will get moved to hdfs.

. tFileList-->tHDFSPut

Best regards

Sabrina

[Anonymous]

@xdshi

tFileList is not getting connect to tHDFSOutput so i used

tFileList-tIterateToFlow-tHDFSOutput 

and create a mapping in tIterateToFlow like key = (String)globalMap.get("tFileList_1_CURRENT_FILEPATH")

but it showing error ""can not create a path from an empty string)

I have a folder structure like this at my local machine

Folder A

        Folder B (having many xml file)

        Folder C  (having many pdf file)

Folder T (having images)

and i want the same folder structure to be transferred to hdfs-(all folders and sub folders and files in the same way as it is)

please reply soon

[Anonymous]

Hello,

Why don't you try to use tfilelist-->tHDFSPut?

Best regards

Sabrina

[Anonymous]

@xdshi

A link can not be drown between tFileList and tHDFSOutput.

tFileList is not getting connct directly to tHDFSOutput

[Anonymous]

Please select iterate in the link, its working fine.

tfilelist--->thdfsput

[Anonymous]

Hello,

tHDFSOutput component runs standalone and does not generate input or output flow for the other components.

You need to use Iterate connection instead of Row connection.

For more information about connection type, please refer to this online document:TalendHelpCenter:Connection types.

Best regards

Sabrina