hadoop 高可用

hadoop 高可用

为什么 NameNode 需要高可用

– NameNode 是 HDFS 的核心配置,HDFS 又是

Hadoop 的核心组件,NameNode 在 Hadoop 集群

中至关重要,NameNode机器宕机,将导致集群丌可

用,如果NameNode 数据丢失将导致整个集群的数据

丢失,而 NameNode 的数据的更新又比较频繁,实

现 NameNode 高可用势在必行

为什么 NameNode 需要高可用

– 官方提供了两种解决方案

– HDFS with NFS

– HDFS with QJM

– 两种翻案异同

 

 

 

 

 

 

 

HA 方案对比

– 都能实现热备

– 都是一个active NN 和一个 standby NN

– 都使用Zookeeper 和 ZKFC 来实现自劢失效恢复

– 失效切换都使用 fencing 配置的方法来 active NN

– NFS 数据数据共享变更方案把数据存储在共享存储里

面,我们还需要考虑 NFS 的高可用设计

– QJM 丌需要共享存储,但需要让每一个 DN 都知道两

个 NN 的位置,并把块信息和心跳包发送给active和

standby这两个 NN

NameNode 高可用方案 (QJM)

– 为了解决 NameNode 单点故障问题,Hadoop 给出

了 HDFS 的高可用HA方案:HDFS 通常由两个

NameNode组成,一个处于 active 状态,另一个处于

standby 状态。Active NameNode对外提供服务,比

如处理来自客户端的 RPC 请求,而 Standby

NameNode 则丌对外提供服务,仅同步 Active

NameNode 的状态,以便能够在它失败时迚行切换。

 

NameNode 高可用架构 续......

 

– 为了让Standby Node不Active Node保持同步,这两

个Node都不一组称为JNS的互相独立的迚程保持通信

(Journal Nodes)。当Active Node上更新了

namespace,它将记录修改日志发送给JNS的多数派。

Standby noes将会从JNS中读取这些edits,并持续关

注它们对日志的变更。Standby Node将日志变更应用

在自己的namespace中,当failover发生时,Standby

将会在提升自己为Active乊前,确保能够从JNS中读取

所有的edits,即在failover发生乊前Standy持有的

namespace应该不Active保持完全同步。

NameNode 高可用架构 续......

– NameNode 更新是徆频繁的,为了的保持主备数据的

一致性,为了支持快速failover,Standby node持有

集群中blocks的最新位置是非常必要的。为了达到这

一目的,DataNodes上需要同时配置这两个

Namenode的地址,同时和它们都建立心跳链接,并

把block位置发送给它们

NameNode 高可用架构 续......

– 还有一点非常重要,任何时刻,只能有一个Active

NameNode,否则将会导致集群操作的混乱,那么两

个NameNode将会分别有两种丌同的数据状态,可能

会导致数据丢失,戒者状态异常,这种情冴通常称为

“split-brain”(脑裂,三节点通讯阻断,即集群中丌

同的Datanode 看到了丌同的Active NameNodes)。

对于JNS而言,任何时候只允讲一个NameNode作为

writer;在failover期间,原来的Standby Node将会

接管Active的所有职能,并负责吐JNS写入日志记录,

这中机制阻止了其他NameNode基于处于Active状态

的问题。

 

 

 

 

 

 

 

 

 

 

系统规划

主机	角色	软件
192.168.6.10	NameNode1	Hadoop
192.168.6.14	NameNode2	Hadoop
Node1   192.168.6.11	DataNade journalNode Zookeeper	HDFS Zookeeper
Node2   192.168.6.12	DataNade journalNode Zookeeper	HDFS Zookeeper
Node3   192.168.6.13	DataNade journalNode Zookeeper	HDFS Zookeeper

 

 

//全部修改主机名

vim /etc/hosts

192.168.6.10 nn01 namenode,secondarynamenode

192.168.6.14 nn02

192.168.6.11 node1 datanode

192.168.6.12 node2 datanode

192.168.6.13 node3 datanode

 

[root@nn01 ~]# /usr/local/zookeeper-3.4.10/bin/zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /usr/local/zookeeper-3.4.10/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

[root@nn01 ~]# ./zkstat.sh

node1 follower

node2 leader

node3 follower

nn01 observer

[root@nn01 ~]# rm -rf /var/hadoop/*

[root@node1 ~]# /usr/local/zookeeper-3.4.10/bin/zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /usr/local/zookeeper-3.4.10/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

[root@node1 ~]# rm -rf /var/hadoop/*

[root@node2 ~]# /usr/local/zookeeper-3.4.10/bin/zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /usr/local/zookeeper-3.4.10/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

[root@node2 ~]# rm -rf /var/hadoop/*

[root@node3 ~]# /usr/local/zookeeper-3.4.10/bin/zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /usr/local/zookeeper-3.4.10/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

[root@node3 ~]# rm -rf /var/hadoop/*

[root@nn02 ~]# rm -rf /var/hadoop/*

[root@nn02 ~]# vim /etc/ssh/ssh_config

Host *

GSSAPIAuthentication yes

StrictHostKeyChecking no

[root@nn02 ~]# scp nn01:/root/.ssh/id_rsa /root/.ssh/

root@nn01's password:

id_rsa 100% 1679 800.5KB/s 00:00

[root@nn02 ~]# scp nn01:/root/.ssh/authorized_keys /root/.ssh/

authorized_keys 100% 782 564.5KB/s 00:00

 

[root@nn02 ~]# ssh nn01

Last login: Fri Aug 3 15:46:40 2018 from 192.168.6.254

[root@nn01 ~]# exit

登出

Connection to nn01 closed.

[root@nn02 ~]# ssh node1

Warning: Permanently added 'node1,192.168.6.11' (ECDSA) to the list of known hosts.

Last login: Fri Aug 3 15:46:45 2018 from 192.168.6.254

[root@node1 ~]# exit

登出

Connection to node1 closed.

[root@nn02 ~]# ssh node3

Warning: Permanently added 'node3,192.168.6.13' (ECDSA) to the list of known hosts.

Last login: Fri Aug 3 15:46:57 2018 from 192.168.6.254

[root@node3 ~]# exit

登出

Connection to node3 closed.

 

[root@nn01 hadoop]# vim core-site.xml

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://nsdcluster</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/var/hadoop</value>

</property>

<property>

<name>ha.zookeeper.quorum</name>

<value>node1:2181,node2:2181,node3:2181</value>

</property>

<property>

<name>hadoop.proxyuser.nsd1803.groups</name>

<value>*</value>

</property>

<property>

<name>hadoop.proxyuser.nsd1803.hosts</name>

<value>*</value>

</property>

</configuration>

 

[root@nn01 hadoop]# vim hdfs-site.xml

<configuration>

<property>

<name>dfs.replication</name>

<value>2</value>

</property>

<property>

<name>dfs.nameservices</name>

<value>nsdcluster</value>

</property>

<property>

<name>dfs.ha.namenodes.nsdcluster</name>

<value>nn1,nn2</value>

</property>

<property>

<name>dfs.namenode.rpc-address.nsdcluster.nn1</name>

<value>nn01:8020</value>

</property>

<property>

<name>dfs.namenode.rpc-address.nsdcluster.nn2</name>

<value>nn02:8020</value>

</property>

<property>

<name>dfs.namenode.http-address.nsdcluster.nn1</name>

<value>nn01:50070</value>

</property>

<property>

<name>dfs.namenode.http-address.nsdcluster.nn2</name>

<value>nn02:50070</value>

</property>

<property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://node1:8485;node2:8485;node3:8485/nsdcluster</value>

</property>

<property>

<name>dfs.journalnode.edits.dir</name>

<value>/var/hadoop/journal</value>

</property>

<property>

<name>dfs.client.failover.proxy.provider.nsdcluster</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

</property>

<property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/root/.ssh/id_rsa</value>

</property>

<property>

<name>dfs.ha.automatic-failover.enabled</name>

<value>true</value>

</property>

</configuration>

 

 

[root@nn01 hadoop]# vim yarn-site.xml

 

<configuration>

 

<!-- Site specific YARN configuration properties -->

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.resourcemanager.ha.enabled</name>

<value>true</value>

</property>

<property>

<name>yarn.resourcemanager.ha.rm-ids</name>

<value>rm1,rm2</value>

</property>

<property>

<name>yarn.resourcemanager.recovery.enabled</name>

<value>true</value>

</property>

<property>

<name>yarn.resourcemanager.store.class</name>

<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>

</property>

<property>

<name>yarn.resourcemanager.zk-address</name>

<value>node1:2181,node2:2181,node3:2181</value>

</property>

<property>

<name>yarn.resourcemanager.cluster-id</name>

<value>yarn-ha</value>

</property>

<property>

<name>yarn.resourcemanager.hostname.rm1</name>

<value>nn01</value>

</property>

<property>

<name>yarn.resourcemanager.hostname.rm2</name>

<value>nn02</value>

</property>

</configuration>

 

 

[root@nn01 bin]# ./rrr node{1..3}

[root@nn02 ~]# rsync -aSH nn01:/usr/local/hadoop /usr/local/

[root@nn01 ~]# ssh node1 rm -rf /usr/local/hadoop/logs

[root@nn01 ~]# ssh node2 rm -rf /usr/local/hadoop/logs

[root@nn01 ~]# ssh node3 rm -rf /usr/local/hadoop/logs

 

[root@nn01 ~]# cd /usr/local/hadoop/

[root@node1 ~]# cd /usr/local/hadoop/

[root@node1 hadoop]# ls

bin f2 lib LICENSE.txt oo sbin xx

etc include libexec NOTICE.txt README.txt share xx1

 

[root@nn01 hadoop]# ./bin/hdfs zkfc -formatZK

Successfully created /hadoop-ha/nsdcluster in ZK.

[root@node1 hadoop]# ./sbin/hadoop-daemon.sh start journalnode

starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node1.out

[root@node1 hadoop]#

[root@node1 hadoop]# jps

971 JournalNode

1035 Jps

830 QuorumPeerMain

[root@node2 ~]# cd /usr/local/hadoop/

[root@node2 hadoop]# ./sbin/hadoop-daemon.sh start journalnode

starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node2.out

[root@node2 hadoop]# jps

832 QuorumPeerMain

1024 Jps

973 JournalNode

[root@node3 ~]# cd /usr/local/hadoop/

[root@node3 hadoop]# ./sbin/hadoop-daemon.sh start journalnode

starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-node3.out

[root@node3 hadoop]# jps

983 JournalNode

1034 Jps

831 QuorumPeerMain

[root@nn01 hadoop]# ./bin/hdfs namenode -format

Storage directory /var/hadoop/dfs/name has been successfully formatted.

[root@nn01 hadoop]# yum -y install tree-1.6.0-10.el7.x86_64

[root@nn01 hadoop]# pwd

/var/hadoop

[root@nn01 hadoop]# tree .

.

└── dfs

└── name

└── current

├── fsimage_0000000000000000000

├── fsimage_0000000000000000000.md5

├── seen_txid

└── VERSION

 

3 directories, 4 files

[root@nn02 ~]# cd /var/hadoop/

[root@nn02 hadoop]# ls

[root@nn02 hadoop]# yum -y install tree-1.6.0-10.el7.x86_64

[root@nn02 hadoop]# rsync -aSH nn01:/var/hadoop/dfs /var/hadoop/

[root@nn02 hadoop]# ls

dfs

[root@nn02 hadoop]# tree .

.

└── dfs

└── name

└── current

├── fsimage_0000000000000000000

├── fsimage_0000000000000000000.md5

├── seen_txid

└── VERSION

 

3 directories, 4 files

 

 

NN1: 初始化 JNS

[root@nn01 hadoop]# ./bin/hdfs namenode -initializeSharedEdits

Re-format filesystem in QJM to [192.168.6.11:8485, 192.168.6.12:8485, 192.168.6.13:8485] ? (Y or N) Y

Successfully started new epoch 1

 

nodeX: 停止 journalnode 服务

[root@node1 hadoop]# pwd

/usr/local/hadoop

[root@node1 hadoop]# ./sbin/hadoop-daemon.sh stop journalnode

stopping journalnode

[root@node1 hadoop]# jps

830 QuorumPeerMain

1070 Jps

[root@node2 hadoop]# pwd

/usr/local/hadoop

[root@node2 hadoop]# ./sbin/hadoop-daemon.sh stop journalnode

stopping journalnode

[root@node2 hadoop]# jps

832 QuorumPeerMain

1061 Jps

[root@node3 hadoop]# pwd

/usr/local/hadoop

[root@node3 hadoop]# ./sbin/hadoop-daemon.sh stop journalnode

stopping journalnode

[root@node3 hadoop]# jps

1069 Jps

831 QuorumPeerMain

 

 

 

启动集群

NN1: ./sbin/start-all.sh

NN2: ./sbin/yarn-daemon.sh start resourcemanager

 

 

[root@nn01 hadoop]# ./sbin/start-all.sh

[root@nn02 hadoop]# cd /usr/local/hadoop/

[root@nn02 hadoop]# ./sbin/yarn-daemon.sh start resourcemanager

starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-nn02.out

 

[root@nn01 hadoop]# ./bin/hdfs haadmin -getServiceState nn1

active

 

[root@nn01 hadoop]# ./bin/hdfs haadmin -getServiceState nn2

standby

 

[root@nn01 hadoop]# ./bin/yarn rmadmin -getServiceState rm1

active

[root@nn01 hadoop]# ./bin/yarn rmadmin -getServiceState rm2

standby

 

 

[root@nn01 hadoop]# ./bin/hadoop fs -mkdir /oo

[root@nn01 hadoop]# ./bin/hadoop fs -put *.txt /oo

 

[root@nn01 hadoop]# ./bin/hadoop fs -ls hdfs://nsdcluster/

Found 1 items

drwxr-xr-x - root supergroup 0 2018-08-03 18:49 hdfs://nsdcluster/oo

 

[root@nn01 hadoop]# ./bin/hadoop fs -ls hdfs://nsdcluster/oo/

Found 3 items

-rw-r--r-- 2 root supergroup 86424 2018-08-03 18:49 hdfs://nsdcluster/oo/LICENSE.txt

-rw-r--r-- 2 root supergroup 14978 2018-08-03 18:49 hdfs://nsdcluster/oo/NOTICE.txt

-rw-r--r-- 2 root supergroup 1366 2018-08-03 18:49 hdfs://nsdcluster/oo/README.txt

[root@nn01 hadoop]# jps

3120 Jps

1490 NameNode

1799 DFSZKFailoverController

1913 ResourceManager

941 QuorumPeerMain

 

验证高可用，关闭 active namenode

./sbin/hadoop-daemon.sh stop namenode

./sbin/yarn-daemon.sh stop resourcemanager

 

[root@nn01 hadoop]# ./sbin/hadoop-daemon.sh stop namenode

stopping namenode

[root@nn01 hadoop]# jps

1799 DFSZKFailoverController

3176 Jps

1913 ResourceManager

941 QuorumPeerMain

 

[root@nn01 hadoop]# ./bin/hdfs haadmin -getServiceState nn1

18/08/03 18:54:27 INFO ipc.Client: Retrying connect to server: nn01/192.168.6.10:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)

Operation failed: Call From nn01/192.168.6.10 to nn01:8020 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

 

[root@nn01 hadoop]# ./bin/hdfs haadmin -getServiceState nn2

active

 

[root@nn02 hadoop]# jps

914 NameNode

1091 ResourceManager

1016 DFSZKFailoverController

1338 Jps

[root@nn01 hadoop]# ./bin/hadoop fs -ls /oo/

Found 3 items

-rw-r--r-- 2 root supergroup 86424 2018-08-03 18:49 /oo/LICENSE.txt

-rw-r--r-- 2 root supergroup 14978 2018-08-03 18:49 /oo/NOTICE.txt

-rw-r--r-- 2 root supergroup 1366 2018-08-03 18:49 /oo/README.txt

 

 

恢复节点

./sbin/hadoop-daemon.sh stop namenode

./sbin/yarn-daemon.sh stop resourcemanager

 

[root@nn01 hadoop]# ./sbin/hadoop-daemon.sh start namenode

starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-nn01.out

[root@nn01 hadoop]# ./bin/hdfs haadmin -getServiceState nn2

active

[root@nn01 hadoop]# ./bin/hdfs haadmin -getServiceState nn1

standby