1. 环境介绍
1.1 节点与服务映射关系
ip host 服务
192.168.40.132 master Namenode、NodeManager、DataNode、zookeeper 192.168.40.133 slave1 ResurceManager、NodeManager、DataNode、zookeeper 192.168.40.134 slave2 NodeManager、DataNode、zookeeper
1.2 jdk版本
[root
@slave2 ~]# java -
version
java
version "1.7.0_79"
Java(TM) SE Runtime Environment (build
1.7.0_79-b15)
Java HotSpot(TM)
64-Bit Server VM (build
24.79-b02, mixed mode)
1234
1234
1.3 hadoop版本
[root@slave2 ~]# hadoop version
Hadoop
2.6.
0
Subversion https:
Compiled
by jenkins
on 2014-
11-
13T21:
10Z
Compiled
with protoc
2.5.
0
From source
with checksum
18e43357c8f927c0695f1e9522859d6a
This command was run
using /application/hadoop-
2.6.
0/share/hadoop/common/hadoop-common-
2.6.
0.jar
1234567
1234567
1.4 各节点进程 jps
2. 安装maven
2.1 下载、解压maven
wget http://mirrors
.tuna.tsinghua.edu.cn/apache/maven/maven-
3/
3.3.9/binaries/apache-maven-
3.3.9-bin
.tar.gz
tar -xzvf apache-maven-
3.3.9-bin
.tar.gz
ln -s apache-maven-
3.3.9/ maven
12345
12345
2.2 配置maven环境变量
export MAVEN_HOME=/application/maven
export PATH=.:
$MAVEN_HOME/bin:
$JAVA_HOME/bin:
$HADOOP_HOME/bin:
$HADOOP_HOME/sbin:
$PATH
source /etc/profile
1234
1234
2.3 验证，查看maven版本号
mvn -v
Apache Maven
3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
2015-
11-
11T00:
41:
47+
08:
00)
Maven home: /
application/maven
Java
version:
1.7.0_79, vendor: Oracle Corporation
Java home: /
application/jdk1
.7.0_79/jre
Default locale: en_US, platform encoding: UTF-
8
OS
name:
"linux",
version:
"2.6.32-504.el6.x86_64", arch:
"amd64", family:
"unix"
1234567
1234567
3. 安装storm-yarn
3.1 下载storm-yarn
wget
https:/
/github.com/yahoo/storm-yarn/archive/master.zip
1
1
重命名下载的压缩包为storm-yarn.zip
mv master
.zip storm-yarn
.zip
1
1
解压
unzip storm-yarn
.zip
1
1
3.2 修改pom.xml
根据Hadoop的版本号修改storm-yarn-master/pom.xml
<properties>
<storm.version>0.9.6
</storm.version>
<hadoop.version>2.6.0
</hadoop.version>
</properties>
1234
1234
因为我的环境中hadoop的版本是2.6.0，没有安装storm（也不用安装），所以pom.xml没有修改。
3.3 编译打包storm-yarn-master
mvn
package -DskipTests
1
1
打包成功
[INFO]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total
time:
08:
41 min
[INFO] Finished
at:
2016-
08-
26T10:
16:
27+
08:
00
[INFO] Final Memory:
14M/
34M
[INFO]
12345678
12345678
3.4 解压lib/storm.zip并移动解压后的目录
解压storm-yarn-master/lib/storm.zip
unzip storm-yarn-master/lib/storm
.zip
[root@master lib]
# ls
storm-
0.9.0-wip21 storm-
0.9.0-wip21
.zip storm-starter-
0.0.1-SNAPSHOT
.jar storm
.zip
1234
1234
将解压后生成的storm-0.9.0-wip21移到和storm-yarn-master同级的目录
[root
@master lib]mv storm-
0.
9.0-wip21 ../..
[root
@master lib]
[root
@master application]
maven storm-yarn.zip apache-maven-
3.3.
9 tools hadoop zookeeper hadoop-
2.6.
0 java storm-
0.
9.0-wip21 zookeeper-
3.3.
6 jdk1.
7.0_79 storm-yarn-master
1234
1234
3.5 配置环境变量
export
PATH=.
:/application/storm-0.
9.0-wip21/
bin:/application/storm-yarn-master/
bin:$MAVEN_HOME/
bin::$JAVA_HOME/
bin:$HADOOP_HOME/
bin:$HADOOP_HOME/
sbin:$PATH
1
1
使用source /etc/profile使之生效
3.6 编辑 storm-0.9.0-wip21/conf/storm.yaml
修改zookeeper地址
storm.zookeeper.servers:
-
"master"
-
"slave1"
-
"slave2"
1234
1234
3.7 压缩storm.zip并上传到hdfs上
压缩storm-0.9.0-wip21
zip
-r storm
.zip storm
-0.9.0-wip21
1
1
将压缩后的文件storm.zip上传到hdfs上的目录/lib/storm/0.9.0-wip21/
hadoop fs
-mkdir -p /lib/storm/
0.9.0-wip21
hdfs dfs
-put storm
.zip /lib/storm/
0.9.0-wip21/
hdfs dfs
-ls /lib/storm/
0.9.0-wip21/
Found
1 items
-rw-r--r
-- 3 root supergroup
17141097 2016-08-26 10:
27 /lib/storm/
0.9.0-wip21/storm
.zip
12345
12345
4. 运行storm-yarn
4.1 启动storm-yarn
[root@master application]
# storm-yarn launch storm-0.9.0-wip21/conf/storm.yaml
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding
in [jar:file:/application/hadoop-
2.6.0/share/hadoop/common/lib/slf4j-log4j12-
1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder
.class]
SLF4J: Found binding
in [jar:file:/application/storm-
0.9.0-wip21/lib/logback-classic-
1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder
.class]
SLF4J: See http://www
.slf4j
.org/codes
.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org
.slf4j
.impl.Log4jLoggerFactory]
16/
08/
26 10:
28:
28 INFO client
.RMProxy: Connecting to ResourceManager at slave1/
192.168.40.133:
8032
16/
08/
26 10:
28:
29 WARN util
.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/
08/
26 10:
28:
30 INFO yarn
.StormOnYarn: Copy App Master jar from local filesystem
and add to local environment
16/
08/
26 10:
28:
32 INFO yarn
.StormOnYarn:
Set the environment for the application master
16/
08/
26 10:
28:
32 INFO yarn
.StormOnYarn: YARN CLASSPATH COMMAND = [[yarn, classpath]]
16/
08/
26 10:
28:
32 INFO yarn
.StormOnYarn: Using JAVA_HOME = [/application/java]
16/
08/
26 10:
28:
32 INFO yarn
.StormOnYarn: Setting up app master command:[/application/java/bin/java, -Dstorm
.home=./storm/storm-
0.9.0-wip21/, -Dlogfile
.name=<LOG_DIR>/master
.log,
com.yahoo.storm.yarn.MasterServer,
1><LOG_DIR>/stdout,
2><LOG_DIR>/stderr]
16/
08/
26 10:
28:
32 INFO impl
.YarnClientImpl: Submitted application application_1472175545791_0002
application_1472175545791_0002
123456789101112131415
123456789101112131415
通过http://slave2:8088查看yarn的appplication运行情况
4.2 获取nimbus地址
得到storm的属性信息，并保存到/root/.storm/storm.yaml中
[root@master application]
# mkdir -p /root/.storm/
[root@master application]
# storm-yarn getStormConfig -appId application_1472175545791_0002 -output ~/.storm/storm.yaml
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding
in [jar:file:/application/hadoop-
2.6.0/share/hadoop/common/lib/slf4j-log4j12-
1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder
.class]
SLF4J: Found binding
in [jar:file:/application/storm-
0.9.0-wip21/lib/logback-classic-
1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder
.class]
SLF4J: See http://www
.slf4j
.org/codes
.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org
.slf4j
.impl.Log4jLoggerFactory]
16/
08/
26 10:
30:
13 INFO client
.RMProxy: Connecting to ResourceManager at slave1/
192.168.40.133:
8032
16/
08/
26 10:
30:
13 WARN util
.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/
08/
26 10:
30:
14 INFO yarn
.StormOnYarn: application report for application_1472175545791_0002 :slave2:
9000
16/
08/
26 10:
30:
14 INFO yarn
.StormOnYarn: Attaching to slave2:
9000 to talk to app master application_1472175545791_0002
{
10:
35:
18}
16/
08/
26 10:
30:
14 INFO yarn
.StormMasterCommand: storm
.yaml downloaded into /root/
.storm/storm
.yaml
123456789101112
123456789101112
根据/root/.storm/storm.yaml获取nimbus地址
[root@master
.storm]
# cat ~/.storm/storm.yaml | grep nimbus.host
nimbus.host: 192.168.40.134
12
12
4.3 通过nimbus访问storm UI
通过访问http://192.168.40.134:7070访问storm UI
4.4 提交任务wordcount
[root@master storm-yarn-master]
# storm jar lib/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.WordCountTopology WordCountTopology -c nimbus.host=192.168.40.134
561 [main] INFO backtype
.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar...
573 [main] INFO backtype
.storm.StormSubmitter - Uploading topology jar lib/storm-starter-
0.0.1-SNAPSHOT
.jar to assigned location: storm-local/nimbus/inbox/stormjar-
12a737f5-
82e9-
43a6-b422-c1740cc36635
.jar
596 [main] INFO backtype
.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: storm-local/nimbus/inbox/stormjar-
12a737f5-
82e9-
43a6-b422-c1740cc36635
.jar
596 [main] INFO backtype
.storm.StormSubmitter - Submitting topology WordCountTopology
in distributed mode with conf {
"nimbus.host":
"192.168.40.134",
"topology.workers":
3,
"topology.debug":true}
948 [main] INFO backtype
.storm.StormSubmitter - Finished submitting topology: WordCountTopology
123456
123456
查看Storm UI
查看topology wordcount的运行情况
查看各节点jps
5. 关闭
5.1 关闭storm 提交的任务
[root@master ~]
# storm kill WordCountTopology
2277 [main] INFO backtype
.storm.thrift - Connecting to Nimbus at
192.168.40.134:
6627
2387 [main] INFO backtype
.storm.command.kill-topology - Killed topology: WordCountTopology
123
123
5.2 关闭storm on yarn
[root
@master ~]
Killing application application_1472175545791_0002
