One Star

Connecting to MapR Hadoop DFS

Hi,
I've created a data integration job to connect to MapR distribution as following,
1. Create tHDFSConnection (Big Data -> tHDFSConnection)
=> Hadoop version = MapR
=> URI "maprfs:///:9000/"
=> Username "hadoop"
=> Group "hadoop"
2. Running this job was working fine and closed.
3. Added tHDFSGet (Big Data -> tHDFSGet)
=> Use an existing connection: Selected tHDFSConnection_1 created in step#1
=> HDFS Directory "vz/" (It is available in my hadoop setup)
=> Local Directory "D:/ETL/AWS-MapR-TOS"
--- Run Details ---
Starting job demo at 14:16 06/08/2012.

connecting to socket on port 4059
connected
java.lang.UnsatisfiedLinkError: no MapRClient in java.library.path
at java.lang.ClassLoader.loadLibrary(Unknown Source)
...
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:115)
at verizontest.demo_0_1.demo.tHDFSGet_1Process(demo.java:363)
at verizontest.demo_0_1.demo.runJobInTOS(demo.java:737)
disconnected
at verizontest.demo_0_1.demo.main(demo.java:592)
Unable to load libMapRClient.so native library
Exception in thread "main" java.lang.Error: java.lang.UnsatisfiedLinkError: no MapRClient in java.library.path
at verizontest.demo_0_1.demo.tHDFSGet_1Process(demo.java:546)
at verizontest.demo_0_1.demo.runJobInTOS(demo.java:737)
at verizontest.demo_0_1.demo.main(demo.java:592)
Caused by: java.lang.UnsatisfiedLinkError: no MapRClient in java.library.path
at java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.lang.Runtime.loadLibrary0(Unknown Source)
...
...
at verizontest.demo_0_1.demo.tHDFSGet_1Process(demo.java:363)
... 2 more
Job demo ended at 14:16 06/08/2012.
Please let me know the inputs to resolve this and able to connect to MapR instance. Thanks.

  • Big Data
8 REPLIES
One Star

Re: Connecting to MapR Hadoop DFS

Refresh. Please let me know if there are possibilities to connect to MapR from Talend.
Moderators, Please move this thread accordingly. Thanks.
Employee

Re: Connecting to MapR Hadoop DFS

In order to connect to Mapr you need to install the MapR Client and add the location to your PATH environment variable.
http://www.mapr.com/doc/display/MapR/Setting+Up+the+Client
I am currently working through this as well.
One Star

Re: Connecting to MapR Hadoop DFS

Hi,
Were you able to resolve this issue. I have the same issue. The MapR client is installed correctly and i am able to connect to HDFS via cmd - client.
Employee

Re: Connecting to MapR Hadoop DFS

Hi harry6,
If you are on windows, you have to put a DLL in the PATH (in the System32 folder for example.)
Rémy.
Employee

Re: Connecting to MapR Hadoop DFS

Hi Remy,
I just installed a MapR VM and played around a little bit with it.
So what is the current "installation guide" for using MapR with Talend 5.2 / 5.3 beta? Do you still need to install a MapR client, or is this integrated into the Studio (maybe at least with 5.3)?
Besides, the only other difference to Hortonworks and Cloudera is to use "maprfs" instead of "hdfs" in the connection url?
Thanks.
Kai
One Star

Re: Connecting to MapR Hadoop DFS

I have a MapR cluster version 4.0.1. And I have installed Talend Studio on windows. On the HDFSConnection, What is the namenodeURI?
what is the authentication user name? can I just use username=mapr group=mapr by default?
One Star

Re: Connecting to MapR Hadoop DFS

Hi Team ,
I have created the job which have jdbcinput and writes data to hdfs file on mapr system . 

While running the job I am getting error . Can u tell me how to point to mapr client path . 
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See for more info.
==========Unable to find library on native path due to Exception. ==============
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.mapr.fs.ShimLoader.loadNativeLibrary(ShimLoader.java:308)
        at com.mapr.fs.ShimLoader.load(ShimLoader.java:197)
        at org.apache.hadoop.conf.CoreDefaultProperties.(CoreDefaultProperties.java:60)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:278)
        at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1847)
        at org.apache.hadoop.conf.Configuration.getProperties(Configuration.java:2062)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2272)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2224)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2141)
        at org.apache.hadoop.conf.Configuration.get(Configuration.java:1081)
        at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:177)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:392)
        at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:375)
        at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:392)
        at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:288)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:479)
        at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:170)
        at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
        at demo.test2_0_1.test2.runJobInTOS(test2.java:1566)
        at demo.test2_0_1.test2.main(test2.java:1551)
Caused by: java.lang.UnsatisfiedLinkError: no MapRClient in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
        at java.lang.Runtime.loadLibrary0(Runtime.java:849)
        at java.lang.System.loadLibrary(System.java:1088)
        at com.mapr.fs.shim.LibraryLoader.loadLibrary(LibraryLoader.java:41)
        ... 26 more
==========Unable to find library in jar due to exception. ==============
java.lang.RuntimeException: no native library is found for os.name=Linux and os.arch=x86_64
        at com.mapr.fs.ShimLoader.findNativeLibrary(ShimLoader.java:496)
        at com.mapr.fs.ShimLoader.loadNativeLibrary(ShimLoader.java:318)
        at com.mapr.fs.ShimLoader.load(ShimLoader.java:197)
        at org.apache.hadoop.conf.CoreDefaultProperties.(CoreDefaultProperties.java:60)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:278)
        at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1847)
        at org.apache.hadoop.conf.Configuration.getProperties(Configuration.java:2062)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2272)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2224)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2141)
        at org.apache.hadoop.conf.Configuration.get(Configuration.java:1081)
        at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:177)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:392)
        at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:375)
        at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:392)
        at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:288)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:479)
        at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:170)
        at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
        at demo.test2_0_1.test2.runJobInTOS(test2.java:1566)
        at demo.test2_0_1.test2.main(test2.java:1551)
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.mapr.fs.ShimLoader.loadNativeLibrary(ShimLoader.java:308)
        at com.mapr.fs.ShimLoader.load(ShimLoader.java:197)
        at org.apache.hadoop.conf.CoreDefaultProperties.(CoreDefaultProperties.java:60)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:278)
        at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1847)
        at org.apache.hadoop.conf.Configuration.getProperties(Configuration.java:2062)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2272)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2224)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2141)
        at org.apache.hadoop.conf.Configuration.get(Configuration.java:1081)
        at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:177)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:392)
        at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:375)
        at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:392)
        at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:288)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:479)
        at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:170)
        at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
        at demo.test2_0_1.test2.runJobInTOS(test2.java:1566)
        at demo.test2_0_1.test2.main(test2.java:1551)
Caused by: java.lang.UnsatisfiedLinkError: no MapRClient in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
        at java.lang.Runtime.loadLibrary0(Runtime.java:849)
        at java.lang.System.loadLibrary(System.java:1088)
        at com.mapr.fs.shim.LibraryLoader.loadLibrary(LibraryLoader.java:41)
        ... 26 more
Exception in thread "main" java.lang.ExceptionInInitializerError
        at com.mapr.fs.ShimLoader.load(ShimLoader.java:214)
        at org.apache.hadoop.conf.CoreDefaultProperties.(CoreDefaultProperties.java:60)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:278)
        at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1847)
        at org.apache.hadoop.conf.Configuration.getProperties(Configuration.java:2062)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2272)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2224)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2141)
        at org.apache.hadoop.conf.Configuration.get(Configuration.java:1081)
        at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:177)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:392)
        at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:375)
        at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:392)
        at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:288)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:479)
        at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:170)
        at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
        at demo.test2_0_1.test2.runJobInTOS(test2.java:1566)
        at demo.test2_0_1.test2.main(test2.java:1551)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.mapr.fs.ShimLoader.loadNativeLibrary(ShimLoader.java:308)
        at com.mapr.fs.ShimLoader.load(ShimLoader.java:197)
        ... 20 more
Caused by: java.lang.UnsatisfiedLinkError: no MapRClient in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
        at java.lang.Runtime.loadLibrary0(Runtime.java:849)
        at java.lang.System.loadLibrary(System.java:1088)
        at com.mapr.fs.shim.LibraryLoader.loadLibrary(LibraryLoader.java:41)
        ... 26 more
Moderator

Re: Connecting to MapR Hadoop DFS

Hi 
We have replied to your another topic:https://www.talendforge.org/forum/viewtopic.php?id=50287
Could you please take a look at it?
Best regards
Sabrina