Ubuntu12.04 Hadoop2.7.3伪分布式配置

    xiaoxiao2021-03-26  6

    下载地址:http://mirror.bit.edu.cn/apache/hadoop/common/

    HDFS的配置

    1. etc/hadoop/core-site.xml

    fs.defaultFS:指定HDFS中NameNode地址

    hadoop.tmp.dir:指定Hadoop运行时产生文件的存储目录

    <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/software/hadoop-2.7.3/data</value> </property> </configuration>

    默认8020端口

     

    2.etc/hadoop/hdfs-site.xml:

    dfs.replication:指定HDFS副本的数量

    <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>

    MapReduce的配置

    1.etc/hadoop/mapred-site.xml

    mapreduce.framework.name:指定MR运行在YARN上

    <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>

    2.etc/hadoop/yarn-site.xml

    yarn.resourcemanager.hostname:指定YARN的ResourceManager 的地址

    yarn.nodemanager.aux-services: Reducer获取数据的方式

    <configuration> <property> <name>yarn.resourcemanager.hostname</name> <value>localhost</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>

    1、格式化Hadoop文件系统

    $ hdfs namenode -format

    2、配置免密登录

    $ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys $ chmod 0600 ~/.ssh/authorized_keys

    3、启动hdfs 和 yarn

    启动namenode:hadoop-daemon.sh start namenode

    启动datanode:hadoop-daemon.sh start  datanode

    jps 查看进程

    访问 http://localhost:50070 查看

    启动ResourceManager:yarn-daemon.sh start resourcemanager

    启动NodeManager:yarn-daemon.sh start nodemanager

    访问 http://localhost:8088 查看

    或者启动所有:$ start-all.sh

    启动过程可能会出现错误:Error: JAVA_HOME is not set and could not be found.

    需要修改hadoop配置hadoop/etc/hadoop/hadoop-env.sh 中的export JAVA_HOME 指定JDK的绝对路径

     

    关闭从节点上的namenode

    $sbin/hadoop-daemon.sh stop datanode

    创建软连接:ln -s hadoop-2.7.3 hadoop

     

     

    转载请注明原文地址: https://ju.6miu.com/read-500147.html

    最新回复(0)