Hadoop2.7.1分布式安装配置过程

本篇内容介绍了"Hadoop2.7.1分布式安装配置过程"的有关知识，在实际案例的操作过程中，不少人都会遇到这样的困境，接下来就让小编带领大家学习一下如何处理这些情况吧！希望大家仔细阅读，能够学有所成！

环境说明

VirtualBox5(三台)，CentOS7，Hadoop2.7.1

1、基础安装及配置

先完成Hadoop2.7.1分布式安装-准备篇

注意：Hadoop-2.5.1开始终于不用再编译64位的libhadoop.so.1.0.0了，再早版本的hadoop自带的是32位的，如需64位需要自省编译，具体见Hadoop2.4.1分布式安装。

2、下载并解压hadoop2.7.1压缩包

从apache hadoop官网下载并解压hadoop压缩包到本地目录/home/wukong/local/hadoop-2.7.1/

在打算做namenode的机器上，wget或其他方式下载hadoop的压缩包，并解压到本地指定目录。下载解压命令参考Linux常用命令。

3、需要做的各种配置

共有七个文件，位于/home/wukong/local/hadoop-2.7.1/etc/hadoop，以下分别描述：

• hadoop-env.sh

# 必配# The java implementation to use.export JAVA_HOME=/opt/jdk1.7.0_79# 选配。考虑是虚拟机，所以少配一点# The maximum amount of heap to use, in MB. Default is 1000.export HADOOP_HEAPSIZE=500export HADOOP_NAMENODE_INIT_HEAPSIZE="100"

• yarn-env.sh

# some Java parametersexport JAVA_HOME=/opt/jdk1.7.0_79if [ "$JAVA_HOME" != "" ]; then  #echo "run java in $JAVA_HOME"  JAVA_HOME=$JAVA_HOMEfiif [ "$JAVA_HOME" = "" ]; then  echo "Error: JAVA_HOME is not set."  exit 1fiJAVA=$JAVA_HOME/bin/javaJAVA_HEAP_MAX=-Xmx600m # 默认的heap_max是1000m，我的虚拟机没这么大内存，所以改小了

• slaves

#写入你slave的节点。如果是多个就每行一个，写入host名bd02bd03

• core-site.xml

      fs.defaultFS    hdfs://bd01:9000        io.file.buffer.size    131072        hadoop.tmp.dir    file:/home/wukong/local/hdp-data/tmp    Abase for other temporary directories.        hadoop.proxyuser.hduser.hosts    *        hadoop.proxyuser.hduser.groups    *  

其中hdp-data目录是原来没有的，需要自行创建

• hdfs-site.xml

      dfs.namenode.secondary.http-address    bd01:9001        dfs.namenode.name.dir    file:/home/wukong/local/hdp-data/name        dfs.datanode.data.dir    file:/home/wukong/a_usr/hdp-data/data        dfs.replication    3        dfs.webhdfs.enabled    true  

• mapred-site.xml

      mapreduce.framework.name    yarn        mapreduce.jobhistory.address    bd01:10020        mapreduce.jobhistory.webapp.address    bd01.19888  

• yarn-site.xml

      yarn.nodemanager.aux-services    mapreduce_shuffle        yarn.nodemanager.aux-services.mapreduce.shuffle.class    org.apache.hadoop.mapred.ShuffleHandler        yarn.resourcemanager.address    bd01:8032        yarn.resourcemanager.scheduler.address    bd01:8030        yarn.resourcemanager.resource-tracker.address    bd01:8031        yarn.resourcemanager.admin.address    bd01:8033        yarn.resourcemanager.webapp.address    bd01:8088  

4、复制hadoop到所有节点

远程拷贝的方法见Linux常用命令

5、格式化HDFS

[wukong@bd01 hadoop-2.7.1]$ hdfs namenode -format

当执行完毕，没有抛异常，并且看到这一句时，就是成功了

15/07/31 10:51:09 INFO common.Storage: Storage directory /home/wukong/local/hdp-data/name has been successfully formatted.

6、启动DFS

[wukong@bd01 ~]$ start-dfs.sh Starting namenodes on [bd01]bd01: starting namenode, logging to /home/wukong/local/hadoop-2.7.1/logs/hadoop-wukong-namenode-bd01.outbd02: starting datanode, logging to /home/wukong/local/hadoop-2.7.1/logs/hadoop-wukong-datanode-bd02.outbd03: starting datanode, logging to /home/wukong/local/hadoop-2.7.1/logs/hadoop-wukong-datanode-bd03.outStarting secondary namenodes [bd01]bd01: starting secondarynamenode, logging to /home/wukong/local/hadoop-2.7.1/logs/hadoop-wukong-secondarynamenode-bd01.out[wukong@bd01 ~]

通过jps和日志看是否启动成功。jps查看机器启动的进程情况。正常情况下master上应该有namenode和sencondarynamenode。slave上有datanode。

[wukong@bd01 hadoop]$ jps5224 Jps5074 SecondaryNameNode4923 NameNode[wukong@bd02 ~]$ jps2307 Jps2206 DataNode[wukong@bd03 ~]$ jps2298 Jps2198 DataNode

7、启动YARN

[wukong@bd01 ~]$ start-yarn.shstarting yarn daemonsstarting resourcemanager, logging to /home/wukong/local/hadoop-2.7.1/logs/yarn-wukong-resourcemanager-bd01.outbd03: starting nodemanager, logging to /home/wukong/local/hadoop-2.7.1/logs/yarn-wukong-nodemanager-bd03.outbd02: starting nodemanager, logging to /home/wukong/local/hadoop-2.7.1/logs/yarn-wukong-nodemanager-bd02.out[wukong@bd01 ~]$

通过jps和日志验证启动是否成功

[wukong@bd01 ~]$ jps5830 ResourceManager6106 Jps5074 SecondaryNameNode4923 NameNode[wukong@bd01 ~]$ [wukong@bd02 ~]$ jps4615 Jps2206 DataNode4502 NodeManager[wukong@bd02 ~]$ [wukong@bd03 ~]$ jps 4608 Jps4495 NodeManager2198 DataNode[wukong@bd03 ~]$

8、可能遇到的问题

8.1、启动DFS时报JAVA_HOME没找到

[wukong@bd01 ~]$ start-dfs.shStarting namenodes on [bd01]The authenticity of host 'bd01 (192.168.1.21)' can't be established.ECDSA key fingerprint is af:96:74:e1:41:ec:af:ec:d8:8e:df:cd:99:61:33:0d.Are you sure you want to continue connecting (yes/no)? yesbd01: Warning: Permanently added 'bd01,192.168.1.21' (ECDSA) to the list of known hosts.bd01: Error: JAVA_HOME is not set and could not be found.bd03: Error: JAVA_HOME is not set and could not be found.bd02: Error: JAVA_HOME is not set and could not be found.Starting secondary namenodes [bd01]bd01: Error: JAVA_HOME is not set and could not be found.[wukong@bd01 some_log]$ java -versionjava version "1.7.0_79"Java(TM) SE Runtime Environment (build 1.7.0_79-b15)Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)[wukong@bd01 ~]

8.2、配置hadoop相关的环境变量

# .bash_profile# Get the aliases and functionsif [ -f ~/.bashrc ]; then        . ~/.bashrcfi# 把hadoop可执行文件和脚本的路径都配进来PATH=$PATH:$HOME/local/hadoop-2.7.1/bin:$HOME/local/hadoop-2.7.1/sbinexport PATH~                                                                               ~                                                 ~                                                                               ".bash_profile" 16L, 267C

"Hadoop2.7.1分布式安装配置过程"的内容就介绍到这里了，感谢大家的阅读。如果想了解更多行业相关的知识可以关注网站，小编将为大家输出更多高质量的实用文章！