storm on yarn 部署

    xiaoxiao2021-12-15  32

    1. 环境介绍

    1.1 节点与服务映射关系

    ip host 服务 192.168.40.132 master Namenode、NodeManager、DataNode、zookeeper 192.168.40.133 slave1 ResurceManager、NodeManager、DataNode、zookeeper 192.168.40.134 slave2 NodeManager、DataNode、zookeeper

    1.2 jdk版本

    [root@slave2 ~]# java -version java version "1.7.0_79" Java(TM) SE Runtime Environment (build 1.7.0_79-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode) 1234 1234

    1.3 hadoop版本

    [root@slave2 ~]# hadoop version Hadoop 2.6.0 Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1 Compiled by jenkins on 2014-11-13T21:10Z Compiled with protoc 2.5.0 From source with checksum 18e43357c8f927c0695f1e9522859d6a This command was run using /application/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar 1234567 1234567

    1.4 各节点进程 jps

    2. 安装maven

    2.1 下载、解压maven

    wget http://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz tar -xzvf apache-maven-3.3.9-bin.tar.gz ln -s apache-maven-3.3.9/ maven 12345 12345

    2.2 配置maven环境变量

    export MAVEN_HOME=/application/maven export PATH=.:$MAVEN_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH source /etc/profile 1234 1234

    2.3 验证,查看maven版本号

    mvn -v Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00) Maven home: /application/maven Java version: 1.7.0_79, vendor: Oracle Corporation Java home: /application/jdk1.7.0_79/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "2.6.32-504.el6.x86_64", arch: "amd64", family: "unix" 1234567 1234567

    3. 安装storm-yarn

    3.1 下载storm-yarn

    wget https://github.com/yahoo/storm-yarn/archive/master.zip 1 1

    重命名下载的压缩包为storm-yarn.zip

    mv master.zip storm-yarn.zip 1 1

    解压

    unzip storm-yarn.zip 1 1

    3.2 修改pom.xml

    根据Hadoop的版本号修改storm-yarn-master/pom.xml

    <properties> <storm.version>0.9.6</storm.version> <hadoop.version>2.6.0</hadoop.version> </properties> 1234 1234

    因为我的环境中hadoop的版本是2.6.0,没有安装storm(也不用安装),所以pom.xml没有修改。

    3.3 编译打包storm-yarn-master

    mvn package -DskipTests 1 1

    打包成功

    [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ storm-yarn --- [INFO] ---------------------------------------------------------------------- [INFO] BUILD SUCCESS [INFO] ---------------------------------------------------------------------- [INFO] Total time: 08:41 min [INFO] Finished at: 2016-08-26T10:16:27+08:00 [INFO] Final Memory: 14M/34M [INFO] ---------------------------------------------------------------------- 12345678 12345678

    3.4 解压lib/storm.zip并移动解压后的目录

    解压storm-yarn-master/lib/storm.zip

    unzip storm-yarn-master/lib/storm.zip [root@master lib]# ls storm-0.9.0-wip21 storm-0.9.0-wip21.zip storm-starter-0.0.1-SNAPSHOT.jar storm.zip 1234 1234

    将解压后生成的storm-0.9.0-wip21移到和storm-yarn-master同级的目录

    [root@master lib]mv storm-0.9.0-wip21 ../.. [root@master lib]# cd ../.. [root@master application]# ls maven storm-yarn.zip apache-maven-3.3.9 tools hadoop zookeeper hadoop-2.6.0 java storm-0.9.0-wip21 zookeeper-3.3.6 jdk1.7.0_79 storm-yarn-master 1234 1234

    3.5 配置环境变量

    export PATH=.:/application/storm-0.9.0-wip21/bin:/application/storm-yarn-master/bin:$MAVEN_HOME/bin::$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH 1 1

    使用source /etc/profile使之生效

    3.6 编辑 storm-0.9.0-wip21/conf/storm.yaml

    修改zookeeper地址

    storm.zookeeper.servers: - "master" - "slave1" - "slave2" 1234 1234

    3.7 压缩storm.zip并上传到hdfs上

    压缩storm-0.9.0-wip21

    zip -r storm.zip storm-0.9.0-wip21 1 1

    将压缩后的文件storm.zip上传到hdfs上的目录/lib/storm/0.9.0-wip21/

    hadoop fs -mkdir -p /lib/storm/0.9.0-wip21 hdfs dfs -put storm.zip /lib/storm/0.9.0-wip21/ hdfs dfs -ls /lib/storm/0.9.0-wip21/ Found 1 items -rw-r--r-- 3 root supergroup 17141097 2016-08-26 10:27 /lib/storm/0.9.0-wip21/storm.zip 12345 12345

    4. 运行storm-yarn

    4.1 启动storm-yarn

    [root@master application]# storm-yarn launch storm-0.9.0-wip21/conf/storm.yaml SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/application/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/application/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 16/08/26 10:28:28 INFO client.RMProxy: Connecting to ResourceManager at slave1/192.168.40.133:8032 16/08/26 10:28:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/08/26 10:28:30 INFO yarn.StormOnYarn: Copy App Master jar from local filesystem and add to local environment 16/08/26 10:28:32 INFO yarn.StormOnYarn: Set the environment for the application master 16/08/26 10:28:32 INFO yarn.StormOnYarn: YARN CLASSPATH COMMAND = [[yarn, classpath]] 16/08/26 10:28:32 INFO yarn.StormOnYarn: Using JAVA_HOME = [/application/java] 16/08/26 10:28:32 INFO yarn.StormOnYarn: Setting up app master command:[/application/java/bin/java, -Dstorm.home=./storm/storm-0.9.0-wip21/, -Dlogfile.name=<LOG_DIR>/master.log, com.yahoo.storm.yarn.MasterServer, 1><LOG_DIR>/stdout, 2><LOG_DIR>/stderr] 16/08/26 10:28:32 INFO impl.YarnClientImpl: Submitted application application_1472175545791_0002 application_1472175545791_0002 123456789101112131415 123456789101112131415

    通过http://slave2:8088查看yarn的appplication运行情况

    4.2 获取nimbus地址

    得到storm的属性信息,并保存到/root/.storm/storm.yaml中

    [root@master application]# mkdir -p /root/.storm/ [root@master application]# storm-yarn getStormConfig -appId application_1472175545791_0002 -output ~/.storm/storm.yaml SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/application/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/application/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 16/08/26 10:30:13 INFO client.RMProxy: Connecting to ResourceManager at slave1/192.168.40.133:8032 16/08/26 10:30:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/08/26 10:30:14 INFO yarn.StormOnYarn: application report for application_1472175545791_0002 :slave2:9000 16/08/26 10:30:14 INFO yarn.StormOnYarn: Attaching to slave2:9000 to talk to app master application_1472175545791_0002 {10:35:18}16/08/26 10:30:14 INFO yarn.StormMasterCommand: storm.yaml downloaded into /root/.storm/storm.yaml 123456789101112 123456789101112
    根据/root/.storm/storm.yaml获取nimbus地址
    [root@master .storm]# cat ~/.storm/storm.yaml | grep nimbus.host nimbus.host: 192.168.40.134 12 12

    4.3 通过nimbus访问storm UI

    通过访问http://192.168.40.134:7070访问storm UI

    4.4 提交任务wordcount

    [root@master storm-yarn-master]# storm jar lib/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.WordCountTopology WordCountTopology -c nimbus.host=192.168.40.134 561 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar... 573 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar lib/storm-starter-0.0.1-SNAPSHOT.jar to assigned location: storm-local/nimbus/inbox/stormjar-12a737f5-82e9-43a6-b422-c1740cc36635.jar 596 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: storm-local/nimbus/inbox/stormjar-12a737f5-82e9-43a6-b422-c1740cc36635.jar 596 [main] INFO backtype.storm.StormSubmitter - Submitting topology WordCountTopology in distributed mode with conf {"nimbus.host":"192.168.40.134","topology.workers":3,"topology.debug":true} 948 [main] INFO backtype.storm.StormSubmitter - Finished submitting topology: WordCountTopology 123456 123456
    查看Storm UI

    查看topology wordcount的运行情况

    查看各节点jps

    5. 关闭

    5.1 关闭storm 提交的任务

    [root@master ~]# storm kill WordCountTopology 2277 [main] INFO backtype.storm.thrift - Connecting to Nimbus at 192.168.40.134:6627 2387 [main] INFO backtype.storm.command.kill-topology - Killed topology: WordCountTopology 123 123

    5.2 关闭storm on yarn

    [root@master ~]# yarn application -kill application_1472175545791_0002 Killing application application_1472175545791_0002
    转载请注明原文地址: https://ju.6miu.com/read-1000196.html

    最新回复(0)