Five Stars

transfer folder from local file system to HDFS

 

hello all..

I need to transfer folder including many sub folders and files to hdfs. 

tHdfsput is used to transfer a file to but i want to transfer whole folder having same hierarchical structure of sub folders and files.

i found a connector tHDFSOutputRaw but how to use it means what will be the input source for  tHDFSOutputRaw .

please reply. I need this on urgent basis . I am new to talend.

please help

1 ACCEPTED SOLUTION

Accepted Solutions
Moderator

Re: transfer folder from local file system to HDFS

Hello,

tFileList will take the files from your local machine and iterate it to thdfsput where the files will get moved to hdfs.

. tFileList-->tHDFSPut

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
9 REPLIES
Four Stars

Re: transfer folder from local file system to HDFS

Hi Shalla,

If u want to transfer the folder and all the sub folders inside HDFS, then u can put an thdfslist(check the box include subdirectories) and then iterate it to thdfsput. 

Five Stars

Re: transfer folder from local file system to HDFS

thanku @Ameesh

thdfsList take the files from hdfs directory.

folders  are on my local machine and through talend i want to move complete folder as it is.

Please reply as soon as possible

 

 

Four Stars

Re: transfer folder from local file system to HDFS

tfilelist sorry!
Moderator

Re: transfer folder from local file system to HDFS

Hello,

tFileList will take the files from your local machine and iterate it to thdfsput where the files will get moved to hdfs.

. tFileList-->tHDFSPut

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: transfer folder from local file system to HDFS

@xdshi

tFileList is not getting connect to tHDFSOutput so i used

tFileList-tIterateToFlow-tHDFSOutput 

and create a mapping in tIterateToFlow like key = (String)globalMap.get("tFileList_1_CURRENT_FILEPATH")

but it showing error ""can not create a path from an empty string)

I have a folder structure like this at my local machine

Folder A

        Folder B (having many xml file)

        Folder C  (having many pdf file)

Folder T (having images)

and i want the same folder structure to be transferred to hdfs-(all folders and sub folders and files in the same way as it is)

please reply soon

Moderator

Re: transfer folder from local file system to HDFS

Hello,

Why don't you try to use tfilelist-->tHDFSPut?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: transfer folder from local file system to HDFS

@xdshi

 

A link can not be drown between tFileList and tHDFSOutput.

tFileList is not getting connct directly to tHDFSOutput

Four Stars

Re: transfer folder from local file system to HDFS

Please select iterate in the link, its working fine.

tfilelist--->thdfsput

 

Moderator

Re: transfer folder from local file system to HDFS

Hello,

tHDFSOutput component runs standalone and does not generate input or output flow for the other components.

You need to use Iterate connection instead of Row connection.

For more information about connection type, please refer to this online document:TalendHelpCenter:Connection types.

Best regards

Sabrina

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.