mapreduce操作hbase报错NoClassDefFoundError

skysky 发布于 2015/05/27 16:21
阅读 3K+
收藏 0

下面是具体的报错信息,执行方式:/opt/cloudera/parcels/CDH/bin/hadoop  jar mapreduce.jar 

查了官方文档,可以通过制定hadoop_classpath实现,但是我效,操作如下:

第一步:指定环境变量 HADOOP_CLASSPATH=`/opt/cloudera/parcels/CDH/bin/hbase classpath`:` /opt/cloudera/parcels/CDH/bin/hadoop classpath`

第二步:执行/opt/cloudera/parcels/CDH/bin/hadoop  jar mapreduce.jar 

指不指定HADOOP_CLASSPATH都会出错,结果应该是jar包还是没有引入进去,但是我通过echo $HADOOP_CLASSPATH |grep high-scale-lib-1.1.1.jar 是可以查到该包的,奇怪,求大神支招

直接用mapreduce是OK的,就是操作hbase会出错,http://hbase.apache.org/book.html#hbase.mapreduce.classpath 这是官方解释

44. HBase, MapReduce, and the CLASSPATH

By default, MapReduce jobs deployed to a MapReduce cluster do not have access to either the HBase configuration under $HBASE_CONF_DIR or the HBase classes.

To give the MapReduce jobs the access they need, you could add hbase-site.xml to the $HADOOP_HOME/conf/directory and add the HBase JARs to the HADOOP_HOME/conf/ directory, then copy these changes across your cluster. You could add hbase-site.xml to $HADOOP_HOME/conf and add HBase jars to the $HADOOP_HOME/libdirectory. You would then need to copy these changes across your cluster or edit $HADOOP_HOMEconf/hadoop-env.sh and add them to the HADOOP_CLASSPATH variable. However, this approach is not recommended because it will pollute your Hadoop install with HBase references. It also requires you to restart the Hadoop cluster before Hadoop can use the HBase data.

Since HBase 0.90.x, HBase adds its dependency JARs to the job configuration itself. The dependencies only need to be available on the local CLASSPATH. The following example runs the bundled HBase RowCounter MapReduce job against a table named usertable If you have not set the environment variables expected in the command (the parts prefixed by a $ sign and curly braces), you can use the actual system paths instead. Be sure to use the correct version of the HBase JAR for your system. The backticks (` symbols) cause ths shell to execute the sub-commands, setting the CLASSPATH as part of the command. This example assumes you use a BASH-compatible shell.

$ HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-server-VERSION.jar rowcounter usertable



错误日志:

Exception in thread "main" java.lang.NoClassDefFoundError: org/cliffc/high_scale_lib/Counter

at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addHBaseDependencyJars(TableMapReduceUtil.java:707)
at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addDependencyJars(TableMapReduceUtil.java:752)
at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:192)
at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:148)
at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:271)
at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:101)
at com.yjhh.www.MapReduce.main(MapReduce.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.ClassNotFoundException: org.cliffc.high_scale_lib.Counter
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
加载中
0
张伟康
张伟康
HBASE_CLASSPATH改为JAVA_CLASSPATH,配置如下
# Extra Java CLASSPATH elements. Optional.

export JAVA_CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

你试试看

skysky
skysky
问题解决了,我把hbase目录下的lib复制到hadoop下了, 虽然比较扯的方法,但是解决了,还有一种方法就是打成可执行的jar包,直接打进包里
0
skysky
skysky

复制hbase下lib中的jar包到hadoop下的脚本

#!/bin/sh



base=/opt/cloudera/parcels/CDH-5.2.5-1.cdh5.2.5.p0.3/lib/hbase/lib
find=/opt/cloudera/parcels/CDH-5.2.5-1.cdh5.2.5.p0.3/lib/hadoop/lib/


for i in  `ls $base`
do
        if [ -d $base/$i ];then
                echo "dir $i"
        else
                #echo $i
                val=`find $find -name $i`
                #echo $val
                if [ -z $val ]; then
#                       echo "Not Find $val"
                        echo "cp $base/$i $find/$i"
                        cp $base/$i $find/$i
                else
                        echo "Find $val"
                fi
        fi
done
返回顶部
顶部