WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform

konakona 发布于 2014/03/19 22:45
阅读 50K+
收藏 1

sbin/start-all.sh 主要错误为:

 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

然而完整的运行返回数据为:

MacdeMacBook-Pro-2:2.2.0 mac$ sbin/start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

14/03/19 22:34:33 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[Rate of successful kerberos logins and latency (milliseconds)], always=false, type=DEFAULT, sampleName=Ops)

14/03/19 22:34:33 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[Rate of failed kerberos logins and latency (milliseconds)], always=false, type=DEFAULT, sampleName=Ops)

14/03/19 22:34:33 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics

14/03/19 22:34:33 DEBUG security.Groups:  Creating new Groups object

14/03/19 22:34:33 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...

14/03/19 22:34:33 DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path

14/03/19 22:34:33 DEBUG util.NativeCodeLoader: java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java

14/03/19 22:34:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

14/03/19 22:34:33 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Falling back to shell based

14/03/19 22:34:33 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping

14/03/19 22:34:33 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000

14/03/19 22:34:33 DEBUG security.UserGroupInformation: hadoop login

14/03/19 22:34:33 DEBUG security.UserGroupInformation: hadoop login commit

14/03/19 22:34:33 DEBUG security.UserGroupInformation: using local user:UnixPrincipal?? mac

14/03/19 22:34:33 DEBUG security.UserGroupInformation: UGI loginUser:mac (auth:SIMPLE)

14/03/19 22:34:33 DEBUG security.UserGroupInformation: PrivilegedAction as:mac (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:314)

14/03/19 22:34:33 DEBUG impl.MetricsSystemImpl: StartupProgress, NameNode startup progress

Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.

Starting namenodes on []

localhost: ssh: connect to host localhost port 22: Connection refused

localhost: ssh: connect to host localhost port 22: Connection refused

Starting secondary namenodes [0.0.0.0]

0.0.0.0: ssh: connect to host 0.0.0.0 port 22: Connection refused

14/03/19 22:34:37 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[Rate of successful kerberos logins and latency (milliseconds)], always=false, type=DEFAULT, sampleName=Ops)

14/03/19 22:34:37 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[Rate of failed kerberos logins and latency (milliseconds)], always=false, type=DEFAULT, sampleName=Ops)

14/03/19 22:34:37 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics

14/03/19 22:34:37 DEBUG security.Groups:  Creating new Groups object

14/03/19 22:34:37 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...

14/03/19 22:34:37 DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path

14/03/19 22:34:37 DEBUG util.NativeCodeLoader: java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java

14/03/19 22:34:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

14/03/19 22:34:37 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Falling back to shell based

14/03/19 22:34:37 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping

14/03/19 22:34:37 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000

14/03/19 22:34:37 DEBUG security.UserGroupInformation: hadoop login

14/03/19 22:34:37 DEBUG security.UserGroupInformation: hadoop login commit

14/03/19 22:34:37 DEBUG security.UserGroupInformation: using local user:UnixPrincipal?? mac

14/03/19 22:34:37 DEBUG security.UserGroupInformation: UGI loginUser:mac (auth:SIMPLE)

14/03/19 22:34:37 DEBUG security.UserGroupInformation: PrivilegedAction as:mac (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:314)

starting yarn daemons

starting resourcemanager, logging to /usr/local/Cellar/hadoop/2.2.0/libexec/logs/yarn-mac-resourcemanager-MacdeMacBook-Pro-2.local.out

localhost: ssh: connect to host localhost port 22: Connection refused

这是在网上找到的最贴切的解决方法
Run the example with an argument like this -Djava.library.path={YOUR_HADOOP_HOME}/lib/native/Linux-amd64-64/ will probably fix the problem on the condition that Mac has something native under native directory
大概的意思和解决思路我已经差不多了,因为我是用Brew安装的Hadoop,可能存在库不同的情况,但是是可以解决的嘛。


问题在于我hadoop下没有lib目录,更没有native这样的目录,里面的文件是从哪来的呢?没有可以借鉴的地方呀,哪位高手知道呢?我本人是MAC,我好去download...

这个问题困扰我一晚了,呵呵。

加载中
0
konakona
konakona

另外我在官方找到了native_libraries的相关文案:https://hadoop.apache.org/docs/r0.18.3/native_libraries.html#Supported+Platforms

需要的2个东东我也已经用brew装好了,但还是不行。

0
TavenLi
TavenLi

这个原因是因为官方提供的lib目录中.so文件是在32位系统下编译的,如果是64位系统,需要自己下载源码在64位上重新编译

0
TavenLi
TavenLi

官方详细文档请参阅:http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/NativeLibraries.html

konakona
konakona
谢谢啊!英语好就是不一样,我晚点研究下看看怎么折腾。
0
konakona
konakona

引用来自“TavenLi”的评论

官方详细文档请参阅:http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/NativeLibraries.html

不对呀 =, =我 OSX 貌似不分32 64的吧……

=,= 话说怎么看出hadoop是位的版本呢?

MacdeMacBook-Pro-2:2.2.0 mac$ hadoop version

Hadoop 2.2.0

Subversion https://svn.apache.org/repos/asf/hadoop/common -r 1529768

Compiled by hortonmu on 2013-10-07T06:28Z

Compiled with protoc 2.5.0

From source with checksum 79e53ce7994d1628b240f09af91e1af4

This command was run using /usr/local/Cellar/hadoop/2.2.0/libexec/share/hadoop/common/hadoop-common-2.2.0.jar



-------------------------

重新安装的2.2.0(使用brew)

现在所有配置、大部分问题都解决,只有在启动第一步(start-dfs.sh) 时,出现了一个错误:


14/04/22 08:55:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

0
sealte
sealte
http://www.cnblogs.com/gpcuster/archive/2011/02/17/1957042.html
0
ROCKTHEEARTH
ROCKTHEEARTH

官网上说“The native hadoop library is supported on *nix platforms only. The library does not to work with Cygwin or the Mac OS X platform.

不支持Mac平台哦。

0
追梦的青春
追梦的青春

14/03/19 22:34:33 DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in  java.library.path

环境变量配置缺失

返回顶部
顶部