topic Re: Connecting to MapR Hadoop DFS in Talend Studio

Connecting to MapR Hadoop DFS

Anonymous — Sat, 16 Nov 2024 12:13:32 GMT

Hi,
I've created a data integration job to connect to MapR distribution as following,
1. Create tHDFSConnection (Big Data -> tHDFSConnection)
=> Hadoop version = MapR
=> URI "maprfs:///:9000/"
=> Username "hadoop"
=> Group "hadoop"
2. Running this job was working fine and closed.
3. Added tHDFSGet (Big Data -> tHDFSGet)
=> Use an existing connection: Selected tHDFSConnection_1 created in step#1
=> HDFS Directory "vz/" (It is available in my hadoop setup)
=> Local Directory "D:/ETL/AWS-MapR-TOS"
--- Run Details ---
Starting job demo at 14:16 06/08/2012.

connecting to socket on port 4059
connected
java.lang.UnsatisfiedLinkError: no MapRClient in java.library.path
at java.lang.ClassLoader.loadLibrary(Unknown Source)
...
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:115)
at verizontest.demo_0_1.demo.tHDFSGet_1Process(demo.java:363)
at verizontest.demo_0_1.demo.runJobInTOS(demo.java:737)
disconnected
at verizontest.demo_0_1.demo.main(demo.java:592)
Unable to load libMapRClient.so native library
Exception in thread "main" java.lang.Error: java.lang.UnsatisfiedLinkError: no MapRClient in java.library.path
at verizontest.demo_0_1.demo.tHDFSGet_1Process(demo.java:546)
at verizontest.demo_0_1.demo.runJobInTOS(demo.java:737)
at verizontest.demo_0_1.demo.main(demo.java:592)
Caused by: java.lang.UnsatisfiedLinkError: no MapRClient in java.library.path
at java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.lang.Runtime.loadLibrary0(Unknown Source)
...
...
at verizontest.demo_0_1.demo.tHDFSGet_1Process(demo.java:363)
... 2 more
Job demo ended at 14:16 06/08/2012.
Please let me know the inputs to resolve this and able to connect to MapR instance. Thanks.

Re: Connecting to MapR Hadoop DFS

Anonymous — Thu, 09 Aug 2012 09:36:48 GMT

Refresh. Please let me know if there are possibilities to connect to MapR from Talend.
Moderators, Please move this thread accordingly. Thanks.

Re: Connecting to MapR Hadoop DFS

Anonymous — Wed, 22 Aug 2012 02:16:09 GMT

In order to connect to Mapr you need to install the MapR Client and add the location to your PATH environment variable.
http://www.mapr.com/doc/display/MapR/Setting+Up+the+Client
I am currently working through this as well.

Re: Connecting to MapR Hadoop DFS

Anonymous — Fri, 09 Nov 2012 16:44:47 GMT

Hi,
Were you able to resolve this issue. I have the same issue. The MapR client is installed correctly and i am able to connect to HDFS via cmd - client.

Re: Connecting to MapR Hadoop DFS

Anonymous — Fri, 16 Nov 2012 15:48:37 GMT

Hi harry6,
If you are on windows, you have to put a DLL in the PATH (in the System32 folder for example.)
Rémy.

Re: Connecting to MapR Hadoop DFS

Anonymous — Fri, 26 Apr 2013 17:44:10 GMT

Hi Remy,
I just installed a MapR VM and played around a little bit with it.
So what is the current "installation guide" for using MapR with Talend 5.2 / 5.3 beta? Do you still need to install a MapR client, or is this integrated into the Studio (maybe at least with 5.3)?
Besides, the only other difference to Hortonworks and Cloudera is to use "maprfs" instead of "hdfs" in the connection url?
Thanks.
Kai

Re: Connecting to MapR Hadoop DFS

_AnonymousUser — Wed, 25 Nov 2015 00:57:12 GMT

I have a MapR cluster version 4.0.1. And I have installed Talend Studio on windows. On the HDFSConnection, What is the namenodeURI?
what is the authentication user name? can I just use username=mapr group=mapr by default?

Re: Connecting to MapR Hadoop DFS

Anonymous — Thu, 26 May 2016 00:10:05 GMT

Hi Team ,
I have created the job which have jdbcinput and writes data to hdfs file on mapr system .

While running the job I am getting error . Can u tell me how to point to mapr client path .
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See for more info.
==========Unable to find library on native path due to Exception. ==============
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)

To see the whole post, download it here
OriginalPost.pdf

Re: Connecting to MapR Hadoop DFS

Anonymous — Wed, 01 Jun 2016 08:41:11 GMT

Hi
We have replied to your another topic: https://community.talend.com/t5/Design-and-Development/Mapr-Connection/td-p/102430
Could you please take a look at it?
Best regards
Sabrina

Re: Connecting to MapR Hadoop DFS

Anonymous — Wed, 02 Aug 2017 05:26:21 GMT

By refering this link we could successfully connect to Map-R.so want to give a detailed steps on connection.

-As sabrina mentioned,the main important thing install MAPR client on the machine u use talend studio.In our case it is Windows,we installed map R client on windows machine.

-Set the environment variables and the HADOOP_HOME & MAPR_HOME with the installed path.this can be set in the cluster configuration in talend only.

please find below screenshot.

mapr_connectivity.PNG

Re: Connecting to MapR Hadoop DFS

Anonymous — Wed, 23 Aug 2017 11:39:47 GMT

Hi sabrina,

I have Mapr fs connection running successfully wiht above setup.But while settin gup that connection it asked for a jar file Protobuf 2.6 .jar version .So tht time we have used the tlibraryload component to make the conenction successful.But now to make this ocnnection enabled in metadata it is throwing the same error .So how can i let talend knw about this Jar to load it automatically as i have to use this metadataconnection further on in my many jobs

Re: Connecting to MapR Hadoop DFS

Anonymous — Thu, 24 Aug 2017 04:37:34 GMT

Hello,

Could you please indicate on which talend build version you are using?

Best regards

Sabrina

Re: Connecting to MapR Hadoop DFS

Anonymous — Mon, 28 Aug 2017 07:36:59 GMT

I am on talend 6.3 enterprise version