安装一个单节点的 Hadoop 分布式系统 已翻译 100%

构建 MapReduce Tar包

你可以获取 MapReduce 的tar包，也可以从源码中构建这个tar包

$ mvn clean install -DskipTests
$ cd hadoop-mapreduce-project
$ mvn clean install assembly:assembly -Pnative

注意: 你需要使用 2.4.1 或者更新版本的 protoc

如果你想忽略 mapreduce 内置的构建，可以忽略 maven 的 -Pnative 参数。编译后的tar包在 target 目录下。

环境设置

假设你已经安装 hadoop-common/hadoop-hdfs 并设置好环境变量 $HADOOP_COMMON_HOME/$HADOOP_HDFS_HOME, 解压 hadoop mapreduce 的 tar 包，然后设置环境变量 $HADOOP_MAPRED_HOME 到解压后的目录。设置 $HADOOP_YARN_HOME 变量和 $HADOOP_MAPRED_HOME 一样。

注意：下面的指令假设你已经在运行 hdfs 了。

配置

为了启动 ResourceManager 和 NodeManager, 你需要更新配置。假设环境变量 $HADOOP_CONF_DIR 为配置的目录，并且已经有了 HDFS 的配置和 core-site.xml ，你需要设置两个配置文件：mapred-site.xml 和 yarn-site.xml.

mapred-site.xml

添加下面内容到 mapred-site.xml.

  <property>
    <name>mapreduce.cluster.temp.dir</name>
    <value></value>
    <description>No description</description>
    <final>true</final>
  </property>

  <property>
    <name>mapreduce.cluster.local.dir</name>
    <value></value>
    <description>No description</description>
    <final>true</final>
  </property>

yarn-site.xml

添加下面内容到 yarn-site.xml

  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>host:port</value>
    <description>host is the hostname of the resource manager and 
    port is the port on which the NodeManagers contact the Resource Manager.
    </description>
  </property>

  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>host:port</value>
    <description>host is the hostname of the resourcemanager and port is the port
    on which the Applications in the cluster talk to the Resource Manager.
    </description>
  </property>

  <property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
    <description>In case you do not want to use the default scheduler</description>
  </property>

  <property>
    <name>yarn.resourcemanager.address</name>
    <value>host:port</value>
    <description>the host is the hostname of the ResourceManager and the port is the port on
    which the clients can talk to the Resource Manager. </description>
  </property>

  <property>
    <name>yarn.nodemanager.local-dirs</name>
    <value></value>
    <description>the local directories used by the nodemanager</description>
  </property>

  <property>
    <name>yarn.nodemanager.address</name>
    <value>0.0.0.0:port</value>
    <description>the nodemanagers bind to this port</description>
  </property>  

  <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>10240</value>
    <description>the amount of memory on the NodeManager in GB</description>
  </property>
 
  <property>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/app-logs</value>
    <description>directory on hdfs where the application logs are moved to </description>
  </property>

   <property>
    <name>yarn.nodemanager.log-dirs</name>
    <value></value>
    <description>the directories used by Nodemanagers as log directories</description>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce.shuffle</value>
    <description>shuffle service that needs to be set for Map Reduce to run </description>
  </property>

capacity-scheduler.xml

  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>unfunded,default</value>
  </property>
  
  <property>
    <name>yarn.scheduler.capacity.root.capacity</name>
    <value>100</value>
  </property>
  
  <property>
    <name>yarn.scheduler.capacity.root.unfunded.capacity</name>
    <value>50</value>
  </property>
  
  <property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>50</value>
  </property>

运行守护程序

假设环境变量 $HADOOP_COMMON_HOME, $HADOOP_HDFS_HOME, $HADOO_MAPRED_HOME, $HADOOP_YARN_HOME, $JAVA_HOME and $HADOOP_CONF_DIR 已经设置正确，设置$YARN_CONF_DIR 的值跟 $HADOOP_CONF_DIR 相同。

使用如下命令允许 ResourceManager 和 NodeManager：

$ cd $HADOOP_MAPRED_HOME
$ sbin/yarn-daemon.sh start resourcemanager
$ sbin/yarn-daemon.sh start nodemanager

程序启动后，可允许 randomwriter 进行测试：

$ $HADOOP_COMMON_HOME/bin/hadoop jar hadoop-examples.jar randomwriter out