hadoop&spark安装（下）

上一遍文章中其实最主要的就是JAVA环境变量以及hadoop环境变量的设置，这两个设置好了的话，运行hadoop基本上不会出问题。

在hadoop的基础上安装spark好简单。

安装Spark之前需要先安装Hadoop集群，因为之前已经安装了hadoop，所以我直接在之前的hadoop集群上安装spark。

硬件环境：

hddcluster1 10.0.0.197 redhat7

hddcluster2 10.0.0.228 centos7 这台作为master

hddcluster3 10.0.0.202 redhat7

hddcluster4 10.0.0.181 centos7

软件环境：

scala-2.11.7

spark-2.0.2-bin-hadoop2.7.tgz

#所有操作用hadoop

基本流程：

1、master解压scala-2.11.7和spark-2.0.2-bin-hadoop2.7.tgz到相应的目录

2、配置scala和spark环境变量

3、修改配置文件

4、拷贝scala和spark到各个节点，授权

5、启动spark集群

#hadoop用户下操作，下载scala,安装wget http://downloads.lightbend.com/scala/2.11.7/scala-2.11.7.tgztar -zxvf scala-2.11.7.tgz mv scala-2.11.7 /usr/local/scalasudo mv scala-2.11.7 /usr/local/scalavim .bashrc     #添加export SCALA_HOME=/usr/local/scalaexport PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$SCALA_HOME/binsource .bashrc[hadoop@hddcluster2 ~]$ scala -versionScala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL

#在官网下载spark-2.0.2-bin-hadoop2.7.tgztar -zxvf spark-2.0.2-bin-hadoop2.7.tgzmv  spark-2.0.2-bin-hadoop2.7 sparksudo mv spark /usr/local/vim .bashrc   #添加export SPARK_HOME=/usr/local/sparkexport PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$SCALA_HOME/bin:$SPARK_HOME

#修改spark配置文件cd /usr/local/spark/confcp spark-env.sh.template spark-env.shvi spark-env.sh   #添加下面内容###jdk direxport JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64###scala direxport SCALA_HOME=/usr/local/scala###the ip of master node of sparkexport SPARK_MASTER_IP=10.0.0.228###the max memory size of workerexport SPARK_WORKER_MEMORY=8G###hadoop configuration file direxport HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop/#修改slavecp slaves.template slavesvi slaves   #把localhost改为下面内容hddcluster1hddcluster2hddcluster3hddcluster4

#把/usr/local/spark 和 /usr/local/scala打包，然后复制到slave节点、cd /usr/localtar -zcf ~/master.spark.tar.gz  ./sparktar -zcf ~/master.scala.tar.gz  ./scalascp master.spark.tar.gz hddcluster1:~scp master.scala.tar.gz hddcluster1:~#登录各个节点进行解压到/usr/localtar -zxf master.spark.tar.gz -C /usr/local/tar -zxf master.scala.tar.gz -C /usr/local/chown -R hadoop:hadoop /usr/local/sparkchown -R hadoop:hadoop /usr/local/scala再配置.bashrc环境变量和master的一样。加上hadoop上一篇的.bashrc内容是这样子：#scalaexport SCALA_HOME=/usr/local/scala#sparkexport SPARK_HOME=/usr/local/spark#java and hadoopexport JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64export HADOOP_HOME=/usr/local/hadoopexport HADOOP_INSTALL=$HADOOP_HOMEexport HADOOP_MAPRED_HOME=$HADOOP_HOMEexport HADOOP_COMMON_HOME=$HADOOP_HOMEexport HADOOP_HDFS_HOME=$HADOOP_HOMEexport YARN_HOME=$HADOOP_HOMEexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexport PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$SCALA_HOME/bin:$SPARK_HOMEexport HADOOP_PREFIX=$HADOOP_HOMEexport HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib:$HADOOP_PREFIX/lib/native"到此Spark集群搭建完毕

.启动Spark集群：启动Spark之前需要先将hadoop的dfs以及yarn启动。/usr/local/spark/sbin/start-all.sh 启动所有服务之后，在命令行输入jps:[hadoop@hddcluster2 ~]$ jps29601 ResourceManager32098 SparkSubmit29188 DataNode29364 SecondaryNameNode29062 NameNode29915 NodeManager30251 Master30380 Worker30062 JobHistoryServer18767 Jps 比hadoop集群启动时多了Master和worker/usr/local/spark/bin/spark-shell.sh出现scala>时说明成功。在浏览器中输入10.0.0.228:8080时，会看到如下图，有4个Worker