1.1 配置Java环境变量
命令:
<span class="token function">tar</span> -zxvf jdk-8u211-linux-x64.tar.gz -C /home/user/tools<span class="token function">tar</span> -zxvf jdk-8u211-linux-x64.tar.gz -C /home/user/toolstar -zxvf jdk-8u211-linux-x64.tar.gz -C /home/user/tools
命令:
<span class="token function">vim</span> .bashrc<span class="token function">vim</span> .bashrcvim .bashrc
在.bashrc文件中添加以下内容:
<span class="token builtin class-name">export</span> <span class="token assign-left variable">JAVA_HOME</span><span class="token operator">=</span>/home/user/tools/jdk1.8.0_211 <span class="token builtin class-name">export</span> <span class="token assign-left variable">JRE_HOME</span><span class="token operator">=</span>/home/user/tools/jdk1.8.0_211/jre <span class="token builtin class-name">export</span> <span class="token assign-left variable">CLASSPATH</span><span class="token operator">=</span>.:<span class="token variable">$JAVA_HOME</span>/lib:<span class="token variable">$JRE_HOME</span>/lib:<span class="token variable">$CLASSPATH</span> <span class="token builtin class-name">export</span> <span class="token assign-left variable"><span class="token environment constant">PATH</span></span><span class="token operator">=</span><span class="token variable">$JAVA_HOME</span>/bin:<span class="token variable">$JRE_HOME</span>/bin:<span class="token environment constant">$PATH</span><span class="token builtin class-name">export</span> <span class="token assign-left variable">JAVA_HOME</span><span class="token operator">=</span>/home/user/tools/jdk1.8.0_211 <span class="token builtin class-name">export</span> <span class="token assign-left variable">JRE_HOME</span><span class="token operator">=</span>/home/user/tools/jdk1.8.0_211/jre <span class="token builtin class-name">export</span> <span class="token assign-left variable">CLASSPATH</span><span class="token operator">=</span>.:<span class="token variable">$JAVA_HOME</span>/lib:<span class="token variable">$JRE_HOME</span>/lib:<span class="token variable">$CLASSPATH</span> <span class="token builtin class-name">export</span> <span class="token assign-left variable"><span class="token environment constant">PATH</span></span><span class="token operator">=</span><span class="token variable">$JAVA_HOME</span>/bin:<span class="token variable">$JRE_HOME</span>/bin:<span class="token environment constant">$PATH</span>export JAVA_HOME=/home/user/tools/jdk1.8.0_211 export JRE_HOME=/home/user/tools/jdk1.8.0_211/jre export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
命令:
<span class="token builtin class-name">source</span> .bashrc<span class="token builtin class-name">source</span> .bashrcsource .bashrc
注意:所有节点都要配置
1.2 配置Hadoop环境变量
修改.bashrc
文件:
<span class="token builtin class-name">export</span> <span class="token assign-left variable">JAVA_HOME</span><span class="token operator">=</span>/home/user/tools/jdk1.8.0_211 <span class="token builtin class-name">export</span> <span class="token assign-left variable">JRE_HOME</span><span class="token operator">=</span>/home/user/tools/jdk1.8.0_211/jre <span class="token builtin class-name">export</span> <span class="token assign-left variable">HADOOP_HOME</span><span class="token operator">=</span>/home/user/tools/hadoop-2.10.1 <span class="token builtin class-name">export</span> <span class="token assign-left variable">HADOOP_CONF_DIR</span><span class="token operator">=</span><span class="token variable">$HADOOP_HOME</span>/etc/hadoop <span class="token builtin class-name">export</span> <span class="token assign-left variable">YARN_CONF_DIR</span><span class="token operator">=</span><span class="token variable">$HADOOP_HOME</span>/etc/hadoop <span class="token builtin class-name">export</span> <span class="token assign-left variable">CLASSPATH</span><span class="token operator">=</span>.:<span class="token variable">$JAVA_HOME</span>/lib:<span class="token variable">$JRE_HOME</span>/lib:<span class="token variable">$CLASSPATH</span> <span class="token builtin class-name">export</span> <span class="token assign-left variable"><span class="token environment constant">PATH</span></span><span class="token operator">=</span><span class="token variable">$JAVA_HOME</span>/bin:<span class="token variable">$JRE_HOME</span>/bin:<span class="token environment constant">$PATH</span><span class="token builtin class-name">:</span><span class="token variable">$HADOOP_HOME</span>/bin<span class="token builtin class-name">export</span> <span class="token assign-left variable">JAVA_HOME</span><span class="token operator">=</span>/home/user/tools/jdk1.8.0_211 <span class="token builtin class-name">export</span> <span class="token assign-left variable">JRE_HOME</span><span class="token operator">=</span>/home/user/tools/jdk1.8.0_211/jre <span class="token builtin class-name">export</span> <span class="token assign-left variable">HADOOP_HOME</span><span class="token operator">=</span>/home/user/tools/hadoop-2.10.1 <span class="token builtin class-name">export</span> <span class="token assign-left variable">HADOOP_CONF_DIR</span><span class="token operator">=</span><span class="token variable">$HADOOP_HOME</span>/etc/hadoop <span class="token builtin class-name">export</span> <span class="token assign-left variable">YARN_CONF_DIR</span><span class="token operator">=</span><span class="token variable">$HADOOP_HOME</span>/etc/hadoop <span class="token builtin class-name">export</span> <span class="token assign-left variable">CLASSPATH</span><span class="token operator">=</span>.:<span class="token variable">$JAVA_HOME</span>/lib:<span class="token variable">$JRE_HOME</span>/lib:<span class="token variable">$CLASSPATH</span> <span class="token builtin class-name">export</span> <span class="token assign-left variable"><span class="token environment constant">PATH</span></span><span class="token operator">=</span><span class="token variable">$JAVA_HOME</span>/bin:<span class="token variable">$JRE_HOME</span>/bin:<span class="token environment constant">$PATH</span><span class="token builtin class-name">:</span><span class="token variable">$HADOOP_HOME</span>/binexport JAVA_HOME=/home/user/tools/jdk1.8.0_211 export JRE_HOME=/home/user/tools/jdk1.8.0_211/jre export HADOOP_HOME=/home/user/tools/hadoop-2.10.1 export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH:$HADOOP_HOME/bin
使环境变量生效,
命令:
<span class="token builtin class-name">source</span> .bashrc<span class="token builtin class-name">source</span> .bashrcsource .bashrc
1.3 配置hostname
修改3个节点的/etc/hostname
文件,分别命名为hadoop01
、hadoop02
、hadoop03
1.4 配置hosts
修改3个节点的/etc/hosts
文件,将IP地址与hostname相对应
<span class="token number">172.16</span>.70.165 hadoop01 <span class="token number">172.16</span>.70.166 hadoop02 <span class="token number">172.16</span>.70.167 hadoop03<span class="token number">172.16</span>.70.165 hadoop01 <span class="token number">172.16</span>.70.166 hadoop02 <span class="token number">172.16</span>.70.167 hadoop03172.16.70.165 hadoop01 172.16.70.166 hadoop02 172.16.70.167 hadoop03
1.5 设置服务器间免密登录
命令:
ssh-keygen -t rsassh-keygen -t rsassh-keygen -t rsa
命令:
ssh-copy-id -i ~/.ssh/id_rsa.pub user@<span class="token operator"><</span>B服务器的IP<span class="token operator">></span>ssh-copy-id -i ~/.ssh/id_rsa.pub user@<span class="token operator"><</span>B服务器的IP<span class="token operator">></span>ssh-copy-id -i ~/.ssh/id_rsa.pub user@<B服务器的IP>
1.6 创建存储目录
将一块较大的磁盘挂载到/data/
路径下,并修改/data/
目录的权限,
命令:
<span class="token function">chown</span> –R user:user /data<span class="token function">chown</span> –R user:user /datachown –R user:user /data
在全部3个节点上,在/data/
路径下,
命令:
<span class="token function">sudo</span> <span class="token function">mkdir</span> tmp <span class="token function">sudo</span> <span class="token function">chmod</span> <span class="token number">777</span> -R tmp/<span class="token function">sudo</span> <span class="token function">mkdir</span> tmp <span class="token function">sudo</span> <span class="token function">chmod</span> <span class="token number">777</span> -R tmp/sudo mkdir tmp sudo chmod 777 -R tmp/
在全部3个节点上,在/data/
路径下,
命令:
<span class="token function">sudo</span> <span class="token function">mkdir</span> hadoop-logs <span class="token function">sudo</span> <span class="token function">chmod</span> <span class="token number">777</span> -R hadoop-logs/<span class="token function">sudo</span> <span class="token function">mkdir</span> hadoop-logs <span class="token function">sudo</span> <span class="token function">chmod</span> <span class="token number">777</span> -R hadoop-logs/sudo mkdir hadoop-logs sudo chmod 777 -R hadoop-logs/
在全部3个节点上,在/data/
路径下,
命令:
<span class="token function">sudo</span> <span class="token function">mkdir</span> yarn-logs <span class="token function">sudo</span> <span class="token function">chmod</span> <span class="token number">777</span> -R yarn-logs/<span class="token function">sudo</span> <span class="token function">mkdir</span> yarn-logs <span class="token function">sudo</span> <span class="token function">chmod</span> <span class="token number">777</span> -R yarn-logs/sudo mkdir yarn-logs sudo chmod 777 -R yarn-logs/
2.1 官网下载
2.2 解压
在全部3个节点上,
命令:
<span class="token function">tar</span> -zxvf hadoop-2.10.1.tar.gz -C /home/user/tools/hadoop-2.10.1/<span class="token function">tar</span> -zxvf hadoop-2.10.1.tar.gz -C /home/user/tools/hadoop-2.10.1/tar -zxvf hadoop-2.10.1.tar.gz -C /home/user/tools/hadoop-2.10.1/
3.1 core-site.xml文件
在主节点上,
修改~/hadoop-2.10.1/etc/hadoop/core-site.xml
文件:
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>configuration</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>fs.defaultFS<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>hdfs://master:9000<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>hadoop.tmp.dir<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>/data/tmp<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>configuration</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>configuration</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>fs.defaultFS<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>hdfs://master:9000<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>hadoop.tmp.dir<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>/data/tmp<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>configuration</span><span class="token punctuation">></span></span><configuration> <property> <name>fs.defaultFS</name> <value>hdfs://master:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/data/tmp</value> </property> </configuration>
3.2 hadoop-env.sh脚本
在主节点上,
修改~/hadoop-2.10.1/etc/hadoop/hadoop-env.sh
脚本,添加环境变量:
<span class="token builtin class-name">export</span> <span class="token assign-left variable">JAVA_HOME</span><span class="token operator">=</span>/home/user/tools/jdk1.8.0_211(即JDK的安装路径) <span class="token builtin class-name">export</span> <span class="token assign-left variable">HADOOP_LOG_DIR</span><span class="token operator">=</span>/data/hadoop-logs<span class="token builtin class-name">export</span> <span class="token assign-left variable">JAVA_HOME</span><span class="token operator">=</span>/home/user/tools/jdk1.8.0_211(即JDK的安装路径) <span class="token builtin class-name">export</span> <span class="token assign-left variable">HADOOP_LOG_DIR</span><span class="token operator">=</span>/data/hadoop-logsexport JAVA_HOME=/home/user/tools/jdk1.8.0_211(即JDK的安装路径) export HADOOP_LOG_DIR=/data/hadoop-logs
3.3 hdfs-site.xml文件
在主节点上,
修改~/hadoop-2.10.1/etc/hadoop/hdfs-site.xml
文件:
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>configuration</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>dfs.replication<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>2<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>dfs.namenode.secondary.http-address<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>172.16.80.21:50090<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>configuration</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>configuration</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>dfs.replication<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>2<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>dfs.namenode.secondary.http-address<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>172.16.80.21:50090<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>configuration</span><span class="token punctuation">></span></span><configuration> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>172.16.80.21:50090</value> </property> </configuration>
3.4 mapred-env.sh脚本
在主节点上,
修改~/hadoop-2.10.1/etc/hadoop/mapred-env.sh
脚本,添加环境变量:
<span class="token builtin class-name">export</span> <span class="token assign-left variable">JAVA_HOME</span><span class="token operator">=</span>/home/user/tools/jdk1.8.0_211(即JDK的安装路径)<span class="token builtin class-name">export</span> <span class="token assign-left variable">JAVA_HOME</span><span class="token operator">=</span>/home/user/tools/jdk1.8.0_211(即JDK的安装路径)export JAVA_HOME=/home/user/tools/jdk1.8.0_211(即JDK的安装路径)
3.5 mapred-site.xml文件
在主节点上的~/hadoop-2.10.1/etc/hadoop/
路径下,
命令:
<span class="token function">cp</span> mapred-site.xml.template mapred-site.xml<span class="token function">cp</span> mapred-site.xml.template mapred-site.xmlcp mapred-site.xml.template mapred-site.xml
修改mapred-site.xml
文件:
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>configuration</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>mapreduce.framework.name<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>yarn<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>configuration</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>configuration</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>mapreduce.framework.name<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>yarn<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>configuration</span><span class="token punctuation">></span></span><configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
3.6 yarn-env.sh脚本
在主节点上,
修改~/hadoop-2.10.1/etc/hadoop/yarn-env.sh
脚本,添加环境变量:
<span class="token builtin class-name">export</span> <span class="token assign-left variable">JAVA_HOME</span><span class="token operator">=</span>/home/user/tools/jdk1.8.0_211(即JDK的安装路径) <span class="token assign-left variable">YARN_LOG_DIR</span><span class="token operator">=</span><span class="token string">"/data/yarn-logs"</span><span class="token builtin class-name">export</span> <span class="token assign-left variable">JAVA_HOME</span><span class="token operator">=</span>/home/user/tools/jdk1.8.0_211(即JDK的安装路径) <span class="token assign-left variable">YARN_LOG_DIR</span><span class="token operator">=</span><span class="token string">"/data/yarn-logs"</span>export JAVA_HOME=/home/user/tools/jdk1.8.0_211(即JDK的安装路径) YARN_LOG_DIR="/data/yarn-logs"
3.7 yarn-site.xml文件
在主节点上,
修改~/hadoop-2.10.1/etc/hadoop/yarn-site.xml
文件:
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>configuration</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.nodemanager.aux-services<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>mapreduce_shuffle<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.resourcemanager.hostname<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>master<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.nodemanager.vmem-check-enabled<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>false<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.nodemanager.resource.cpu-vcores<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>6<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.scheduler.maximum-allocation-vcores<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>6<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.nodemanager.resource.memory-mb<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>25600<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.scheduler.minimum-allocation-mb<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>1024<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.scheduler.maximum-allocation-mb<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>8192<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.app.mapreduce.am.resource.mb<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>1024<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>mapreduce.map.memory.mb<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>1536<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>mapreduce.map.java.opts<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>-Xmx1024M<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>mapreduce.reduce.memory.mb<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>3072<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>mapreduce.reduce.java.opts<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>-Xmx2560M<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.log-aggregation-enable<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>true<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.log-aggregation.retain-seconds<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>604800<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>configuration</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>configuration</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.nodemanager.aux-services<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>mapreduce_shuffle<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.resourcemanager.hostname<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>master<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.nodemanager.vmem-check-enabled<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>false<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.nodemanager.resource.cpu-vcores<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>6<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.scheduler.maximum-allocation-vcores<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>6<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.nodemanager.resource.memory-mb<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>25600<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.scheduler.minimum-allocation-mb<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>1024<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.scheduler.maximum-allocation-mb<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>8192<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.app.mapreduce.am.resource.mb<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>1024<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>mapreduce.map.memory.mb<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>1536<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>mapreduce.map.java.opts<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>-Xmx1024M<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>mapreduce.reduce.memory.mb<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>3072<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>mapreduce.reduce.java.opts<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>-Xmx2560M<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.log-aggregation-enable<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>true<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>name</span><span class="token punctuation">></span></span>yarn.log-aggregation.retain-seconds<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>name</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>value</span><span class="token punctuation">></span></span>604800<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>value</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>property</span><span class="token punctuation">></span></span> <span class="token tag"><span class="token tag"><span class="token punctuation"></</span>configuration</span><span class="token punctuation">></span></span><configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.hostname</name> <value>master</value> </property> <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property> <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>6</value> </property> <property> <name>yarn.scheduler.maximum-allocation-vcores</name> <value>6</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>25600</value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>8192</value> </property> <property> <name>yarn.app.mapreduce.am.resource.mb</name> <value>1024</value> </property> <property> <name>mapreduce.map.memory.mb</name> <value>1536</value> </property> <property> <name>mapreduce.map.java.opts</name> <value>-Xmx1024M</value> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>3072</value> </property> <property> <name>mapreduce.reduce.java.opts</name> <value>-Xmx2560M</value> </property> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <property> <name>yarn.log-aggregation.retain-seconds</name> <value>604800</value> </property> </configuration>
3.8 slaves文件
在主节点上,
修改~/hadoop-2.10.1/etc/hadoop/slaves
文件,添加其它节点的IP地址:
<span class="token number">172.16</span>.70.166 <span class="token number">172.16</span>.70.167<span class="token number">172.16</span>.70.166 <span class="token number">172.16</span>.70.167172.16.70.166 172.16.70.167
注意:需要删除slaves文件中原来localhost那一行
将配置文件复制到其它2个节点上,
命令:
<span class="token function">scp</span> -r /home/user/tools/hadoop-2.10.1/ <span class="token number">172.16</span>.70.166:<span class="token variable"><span class="token variable">`</span><span class="token builtin class-name">pwd</span><span class="token variable">`</span></span> <span class="token function">scp</span> -r /home/user/tools/hadoop-2.10.1/ <span class="token number">172.16</span>.70.167:<span class="token variable"><span class="token variable">`</span><span class="token builtin class-name">pwd</span><span class="token variable">`</span></span><span class="token function">scp</span> -r /home/user/tools/hadoop-2.10.1/ <span class="token number">172.16</span>.70.166:<span class="token variable"><span class="token variable">`</span><span class="token builtin class-name">pwd</span><span class="token variable">`</span></span> <span class="token function">scp</span> -r /home/user/tools/hadoop-2.10.1/ <span class="token number">172.16</span>.70.167:<span class="token variable"><span class="token variable">`</span><span class="token builtin class-name">pwd</span><span class="token variable">`</span></span>scp -r /home/user/tools/hadoop-2.10.1/ 172.16.70.166:`pwd` scp -r /home/user/tools/hadoop-2.10.1/ 172.16.70.167:`pwd`
注意:此步骤仅需在第一次安装时执行
在主节点的~/hadoop-2.10.1/bin/
路径下:
命令:
./hadoop namenode -format./hadoop namenode -format./hadoop namenode -format
./hdfs namenode –format./hdfs namenode –format./hdfs namenode –format
在主节点上,在~/hadoop-2.10.1/sbin/
路径下,
命令:
./start-all.sh./start-all.sh./start-all.sh
在主节点上,在~/hadoop-2.10.1/sbin/
路径下,
命令:
./stop-all.sh./stop-all.sh./stop-all.sh
- Hadoop:http://<节点IP>:50070/
- YARN:http://<节点IP>:8088/
原文链接:https://blog.csdn.net/michael_f2008/article/details/130739216