HDFSConnection to external hadoop cluster with password

One Star

HDFSConnection to external hadoop cluster with password

This seems a very basic thing, but I cannot get it to work, nor find youtubes, webinars, forum posts, ... that seem to cover this.
I've got a hadoop cluster running on different systems, and from another PC with Talend Big Data installed I would like to connect to it and do stuff.
I use the HDFSConnection component, and fill in the correct details (and yes, I'm sure of them: I can access via other channels, including website, I've configured core-site.xml myself, ...). The thing is, the user I want to use has a password (evidently), which I can't seem to enter in the HDFSConnection.
Do I need to add it to the properties ?
Also very strange is that when I run the job (with some other stuff in there, a.o. DHFSPut (using the existing connection)) I do NOT get an error. The job seems to run fine, but no output is put on HDFS, which is normal, because somewhere I should get a authentication error.
info:
hadoop 1.4.0
TOS BD: 5.2.0
Any thoughts ?
Employee

Re: HDFSConnection to external hadoop cluster with password

Hello,
First, I have a question.
The first is: How can you use Hadoop in version 1.4.0 ? As far as I know, this version doesn't exist. Did you maybe mean Hadoop 1.0.4 ?
Regarding the security, you don't need to add a password property. You just have to put a username. This username will be used as the user ID to work on HDFS (read/write/execute).
Let me know,
Rémy.
One Star

Re: HDFSConnection to external hadoop cluster with password

Rémy,
Yes, you're right, it's hadoop 1.0.4
And after a lot of experimenting (I find the help - forum, youtube, tutorials, ... - on this topic extremely limited, but maybe I haven't figured out where to look yet) it turned out that I wasn't configuring my tHDFSPut component correctly.
When I find some time, I'll write my experience down in a blog or something to share with others that experience the same problem.
Thanks for the help,
Carlo
Seven Stars

Re: HDFSConnection to external hadoop cluster with password

Hi rdubois,
I was wondering about this.... how come none of hadoop components have password requirement???
you don't need to add a password property. You just have to put a username. This username will be used as the user ID to work on HDFS (read/write/execute).
Could you please elaborate how come HDFS / Hadoop authenticates the user for Read/Write/Execute operation request coming from Talend?

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now