NameNode 与 SecondaryNameNode 的工作机制

NameNode：在内存中储存 HDFS 文件的元数据信息（目录）

如果节点故障或断电，存在内存中的数据会丢失，显然只在内存中保存是不可靠的
实际在磁盘当中也有保存：Fsimage 和 Edits，一个 NameNode 节点在重启后会根据这磁盘上的这两个文件来恢复到之前的状态

Fsimage（镜像文件）和 Edits（编辑日志）：记录内存中的元数据

如果每次对 HDFS 的操作都实时的把内存中的元数据信息往磁盘上传输，这样显然效率不够高，也不稳定
这时就出现了 Edits 文件，用来记录每次对 HDFS 的操作，这样在磁盘上每次就只用做很小改动（只进行追加操作）
当  Edits 文件达到了一定大小或过了一定的时间，就需要把 Edits 文件转化 Fsimage 文件，然后清空 Edits
这样的 Fsimage 文件不会和内存中的元数据实时同步，需要加上 Edits 文件才相等

SecondaryNameNode：负责 Edits 转化成 Fsimage

SecondaryNameNode 不是 NameNode 的备份
SecondaryNameNode 会定时定量的把集群中的 Edits 文件转化为 Fsimage 文件，来保证 NameNode 中数据的可靠性

二、NameNode 工作机制

NameNode 与 SecondaryNameNode 的工作机制

磁盘上的元数据文件

NameNode 与 SecondaryNameNode 的工作机制

三、查看 Edits 与 Fsimage

Edits 和 Fsimage 并非明文存储，需要转换后才能查看，使用 hdfs 命令进行转换

# 部分命令参数
Usage: hdfs [--config confdir] [--loglevel loglevel] COMMAND
       where COMMAND is one of:

# 转换 fsimage 文件
  oiv                  apply the offline fsimage viewer to an fsimage
# 转换旧版 fsimage 文件
  oiv_legacy           apply the offline fsimage viewer to an legacy fsimage
# 转换 edits 文件
  oev                  apply the offline edits viewer to an edits file

1.转换 Fsimage 文件

hdfs oiv：查看参数

部分参数说明

Usage: bin/hdfs oiv [OPTIONS] -i INPUTFILE -o OUTPUTFILE

-i 要转换的文件
-o 转换后文件路径
-p 转换格式 (XML|FileDistribution|ReverseXML|Web|Delimited)

转换文件

hdfs oiv -p XML -i ./fsimage_0000000000000000326 -o ./fsimage.xml

转换后的文件，记录了文件信息

<?xml version="1.0"?>
<fsimage>
    <version>
        <layoutVersion>-63</layoutVersion>
        <onDiskVersion>1</onDiskVersion>
        <oivRevision>Unknown</oivRevision>
    </version>
    <NameSection>
        <namespaceId>1803101069</namespaceId>
        <genstampV1>1000</genstampV1>
        <genstampV2>1015</genstampV2>
        <genstampV1Limit>0</genstampV1Limit>
        <lastAllocatedBlockId>1073741839</lastAllocatedBlockId>
        <txid>326</txid>
    </NameSection>
    <INodeSection>
        <lastInodeId>16400</lastInodeId>
        <numInodes>11</numInodes>
        <inode>
            <id>16385</id>
            <type>DIRECTORY</type>
            <name></name>
            <mtime>1555218314075</mtime>
            <permission>root:supergroup:0755</permission>
            <nsquota>9223372036854775807</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16388</id>
            <type>FILE</type>
            <name>javafx-src.zip</name>
            <replication>2</replication>
            <mtime>1554996304962</mtime>
            <atime>15550 84059617</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>root:supergroup:0644</permission>
            <blocks>
                <block>
                    <id>107 3741829</id>
                    <genstamp>1005</genstamp>
                    <numBytes>5211154</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16389</id>
            <type>DIRECTORY</type>
            <name>0529</name>
            <mtime>1554999968472</mtime>
            <permission>root:supergroup:0755</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16390</id>
            <type>DIRECTORY</type>
            <name>mkmk</name>
            <mtime>1555000275046</mtime>
            <permission>root:supergroup:0755</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16391</id>
            <type>DIRECTORY</type>
            <name>10088</name>
            <mtime>1555000763405</mtime>
            <permission>root:supergroup:0755</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16394</id>
            <type>FILE</type>
            <name>Writage-1.12.msi</name>
            <replication>3</replication>
            <mtime>1555086090236</mtime>
            <atime>155 5086085824</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>root:supergroup:0644</permission>
            <blocks>
                <block>
                    <id>1 073741832</id>
                    <genstamp>1008</genstamp>
                    <numBytes>12771328</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16395</id>
            <type>FILE</type>
            <name>hadoop-2.9.2-win10-64.tar.gz</name>
            <replication>3</replication>
            <mtime>1555086717860</mtime>
            <atime>1555086612935</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>root:supergroup:0644</permission>
            <blocks>
                <block>
                    <id>1073741833</id>
                    <genstamp>1009</genstamp>
                    <numBytes>134217728</numBytes>
                </block>
                <block>
                    <id>1073741834</id>
                    <genstamp>1010</genstamp>
                    <numBytes>134217728</numBytes>
                </block>
                <block>
                    <id>1073741835</id>
                    <genstamp>1011</genstamp>
                    <numBytes>43080138</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16396</id>
            <type>DIRECTORY</type>
            <name>Priority</name>
            <mtime>1555219450567</mtime>
            <permission>root:supergroup:0755</permission>
            <nsquota>-1</nsquota>
            <dsquota>-1</dsquota>
        </inode>
        <inode>
            <id>16398</id>
            <type>FILE</type>
            <name>AAA.msi</name>
            <replication>3</replication>
            <mtime>1555219070073</mtime>
            <atime>155521906501 6</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>root:supergroup:0644</permission>
            <blocks>
                <block>
                    <id>1073741837</id>
                    <genstamp>1013</genstamp>
                    <numBytes>12771328</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16399</id>
            <type>FILE</type>
            <name>BBB.msi</name>
            <replication>1</replication>
            <mtime>1555219285187</mtime>
            <atime>155521928012 9</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>root:supergroup:0644</permission>
            <blocks>
                <block>
                    <id>1073741838</id>
                    <genstamp>1014</genstamp>
                    <numBytes>12771328</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
        <inode>
            <id>16400</id>
            <type>FILE</type>
            <name>CCC.msi</name>
            <replication>2</replication>
            <mtime>1555219455139</mtime>
            <atime>155521945056 7</atime>
            <preferredBlockSize>134217728</preferredBlockSize>
            <permission>root:supergroup:0644</permission>
            <blocks>
                <block>
                    <id>1073741839</id>
                    <genstamp>1015</genstamp>
                    <numBytes>12771328</numBytes>
                </block>
            </blocks>
            <storagePolicyId>0</storagePolicyId>
        </inode>
    </INodeSection>
    <INodeReferenceSection></INodeReferenceSection>
    <SnapshotSection>
        <snapshotCounter>0</snapshotCounter>
        <numSnapshots>0</numSnapshots>
    </SnapshotSection>
    <INodeDirectorySection>
        <directory>
            <parent>16385</parent>
            <child>16389</child>
            <child>16391</child>
            <child>16396</child>
            <child>16394</child>
            <child>16395</child>
            <child>16388</child>
            <child>16390</child>
        </directory>
        <directory>
            <parent>16396</parent>
            <child>16398</child>
            <child>16399</child>
            <child>16400</child>
        </directory>
    </INodeDirectorySection>
    <FileUnderConstructionSection></FileUnderConstructionSection>
    <SecretManagerSection>
        <currentId>0</currentId>
        <tokenSequenceNumber>0</tokenSequenceNumber>
        <numDelegationKeys>0</numDelegationKeys>
        <numTokens>0</numTokens>
    </SecretManagerSection>
    <CacheManagerSection>
        <nextDirectiveId>1</nextDirectiveId>
        <numDirectives>0</numDirectives>
        <numPools>0</numPools>
    </CacheManagerSection>
</fsimage>

View Code