topic HDFSConnection to external hadoop cluster with password in Talend Studio

HDFSConnection to external hadoop cluster with password

Anonymous — Sat, 16 Nov 2024 12:08:56 GMT

This seems a very basic thing, but I cannot get it to work, nor find youtubes, webinars, forum posts, ... that seem to cover this.
I've got a hadoop cluster running on different systems, and from another PC with Talend Big Data installed I would like to connect to it and do stuff.
I use the HDFSConnection component, and fill in the correct details (and yes, I'm sure of them: I can access via other channels, including website, I've configured core-site.xml myself, ...). The thing is, the user I want to use has a password (evidently), which I can't seem to enter in the HDFSConnection.
Do I need to add it to the properties ?
Also very strange is that when I run the job (with some other stuff in there, a.o. DHFSPut (using the existing connection)) I do NOT get an error. The job seems to run fine, but no output is put on HDFS, which is normal, because somewhere I should get a authentication error.
info:
hadoop 1.4.0
TOS BD: 5.2.0
Any thoughts ?

Re: HDFSConnection to external hadoop cluster with password

Anonymous — Sun, 18 Nov 2012 11:28:42 GMT

Hello,
First, I have a question.
The first is: How can you use Hadoop in version 1.4.0 ? As far as I know, this version doesn't exist. Did you maybe mean Hadoop 1.0.4 ?
Regarding the security, you don't need to add a password property. You just have to put a username. This username will be used as the user ID to work on HDFS (read/write/execute).
Let me know,
Rémy.

Re: HDFSConnection to external hadoop cluster with password

Anonymous — Fri, 14 Dec 2012 21:35:46 GMT

Rémy,
Yes, you're right, it's hadoop 1.0.4
And after a lot of experimenting (I find the help - forum, youtube, tutorials, ... - on this topic extremely limited, but maybe I haven't figured out where to look yet) it turned out that I wasn't configuring my tHDFSPut component correctly.
When I find some time, I'll write my experience down in a blog or something to share with others that experience the same problem.
Thanks for the help,
Carlo

Re: HDFSConnection to external hadoop cluster with password

Anonymous — Mon, 14 Apr 2014 06:30:39 GMT

Hi rdubois,
I was wondering about this.... how come none of hadoop components have password requirement???
you don't need to add a password property. You just have to put a username. This username will be used as the user ID to work on HDFS (read/write/execute).
Could you please elaborate how come HDFS / Hadoop authenticates the user for Read/Write/Execute operation request coming from Talend?