意外删除表时恢复 cassandra 集群数据答案

【问题标题】：Restore cassandra cluster data when acccidentally drop table意外删除表时恢复 cassandra 集群数据
【发布时间】：2014-07-01 06:20:19
【问题描述】：

如您所知，Cassandra 集群具有复制功能，即使集群中的某个节点出现故障，也可以防止数据丢失。但是如果管理员不小心删除了一个包含大量数据的表，并且该命令已经被集群中的所有副本执行，这是否意味着您丢失了该表并且无法恢复它？有什么建议可以在服务器停机时间短的情况下应对这种灾难？

【问题讨论】：

标签： cassandra backup restore

【解决方案1】：

来自卡桑德拉docs：

自动快照（默认值： true ）启用或禁用是否在键空间截断或删除表之前对数据进行快照。阻止数据丢失，强烈建议使用默认设置。如果你设置为 false，您将在截断或丢弃时丢失数据。

【讨论】：

感谢您的回答。海量数据恢复时间不是很长吗？
在恢复时不会复制实际数据，因此速度非常快。但我也建议在此之后运行相当慢的nodetool repair 任务，以确保恢复的数据完全一致。

【解决方案2】：

如果管理员已删除数据并在所有节点中进行了复制，则如果没有一致的备份，则很难恢复数据。

也许考虑到 cassandra 中的删除不会立即执行，您可以恢复数据。当您删除数据时，cassandra 会使用 tombstone 替换数据。然后可以将 tombstone 传播到错过初始删除请求的副本。

见http://wiki.apache.org/cassandra/DistributedDeletes

标记有 tombstone 的列在配置的时间段内存在（由列族上设置的 gc_grace_seconds 值定义），然后在该时间到期后被压缩过程永久删除。默认值为 10 天。

按照About Deletes 中的说明，如果您关闭某些节点并等待压缩成功并且数据从 SSTables 中完全删除，然后再次打开节点，数据可能会再次出现。但这只有在您不对节点进行定期修复操作时才会发生。

我以前从未尝试过，这只是我阅读 cassandra 文档的一个想法。

【讨论】：

谢谢。我的问题包括在所有节点中复制删除的情况。并且假设删除还没有传播，我们怎么知道哪个节点还保存着数据，即使知道了，还是要等到compaction成功才能恢复数据，这样会造成很大的经济损失。

【解决方案3】：

第 1 步：我使用以下命令创建了一个表

CREATE TABLE Cricket (
    PlayerID uuid,
    LastName varchar,
    FirstName varchar,  
    City varchar,
    State varchar,
    PRIMARY KEY (PlayerID));

步骤2：使用以下命令插入3条记录

INSERT INTO Cricket (PlayerID, LastName, FirstName, City, State)
VALUES (now(), 'Pendulkar', 'Sachin', 'Mumbai','Maharastra');
INSERT INTO Cricket (PlayerID, LastName, FirstName, City, State)
VALUES (now(), 'Vholi', 'Virat', 'Delhi','New Delhi');
INSERT INTO Cricket (PlayerID, LastName, FirstName, City, State)
VALUES (now(), 'Sharma', 'Rohit', 'Berhampur','Odisha');

第 3 步：我不小心删除了 Cricket 表

drop table Cricket;

步骤4：需要使用自动快照备份恢复该表注意：auto_snapshot（默认值：true）启用或禁用是否在键空间截断或删除表之前对数据进行快照。为防止数据丢失，强烈建议使用默认设置。

第 5 步：查找快照位置和文件

cassandra@node1:~/data/students_details$ cd cricket-88128dc0960d11ea947b39646348bb4f
cassandra@node1:~/data/students_details/cricket-88128dc0960d11ea947b39646348bb4f$ ls -lrth
total 0
drwxrwxr-x 2 cassandra cassandra  6 May 14 18:05 backups
drwxrwxr-x 3 cassandra cassandra 43 May 14 18:06 snapshots
Step-6: You will get one .cql file in that snapshot location which having tables DDL.

cassandra@node1:~/data/students_details/cricket-88128dc0960d11ea947b39646348bb4f/snapshots/dropped-1589479603749-cricket$ ls -lrth
total 44K
-rw-rw-r-- 1 cassandra cassandra   92 May 14 18:06 md-1-big-Summary.db
-rw-rw-r-- 1 cassandra cassandra   61 May 14 18:06 md-1-big-Index.db
-rw-rw-r-- 1 cassandra cassandra   16 May 14 18:06 md-1-big-Filter.db
-rw-rw-r-- 1 cassandra cassandra  179 May 14 18:06 md-1-big-Data.db
-rw-rw-r-- 1 cassandra cassandra   92 May 14 18:06 md-1-big-TOC.txt
-rw-rw-r-- 1 cassandra cassandra 4.7K May 14 18:06 md-1-big-Statistics.db
-rw-rw-r-- 1 cassandra cassandra    9 May 14 18:06 md-1-big-Digest.crc32
-rw-rw-r-- 1 cassandra cassandra   43 May 14 18:06 md-1-big-CompressionInfo.db
-rw-rw-r-- 1 cassandra cassandra  891 May 14 18:06 schema.cql
-rw-rw-r-- 1 cassandra cassandra   31 May 14 18:06 manifest.json
cassandra@node1:~/data/students_details/cricket-88128dc0960d11ea947b39646348bb4f/snapshots/dropped-1589479603749-cricket$

more schema.cql
cassandra@node1:~/data/students_details/cricket-88128dc0960d11ea947b39646348bb4f/snapshots/dropped-1589479603749-cricket$ more schema.cql
CREATE TABLE IF NOT EXISTS students_details.cricket (
        playerid uuid PRIMARY KEY,
        city text,
        firstname text,
        lastname text,
        state text)
        WITH ID = 88128dc0-960d-11ea-947b-39646348bb4f
        AND bloom_filter_fp_chance = 0.01
        AND dclocal_read_repair_chance = 0.1
        AND crc_check_chance = 1.0
        AND default_time_to_live = 0
        AND gc_grace_seconds = 864000
        AND min_index_interval = 128
        AND max_index_interval = 2048
        AND memtable_flush_period_in_ms = 0
        AND read_repair_chance = 0.0
        AND speculative_retry = '99PERCENTILE'
        AND comment = ''
        AND caching = { 'keys': 'ALL', 'rows_per_partition': 'NONE' }
        AND compaction = { 'max_threshold': '32', 'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy' }
        AND compression = { 'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor' }
        AND cdc = false
        AND extensions = {  };

第 7 步：登录数据库并使用该 DDL 创建表。

    apiadmin@cqlsh:coopersdev> use students_details;
    apiadmin@cqlsh:students_details> CREATE TABLE IF NOT EXISTS students_details.cricket (
                        ...         playerid uuid PRIMARY KEY,
                        ...         city text,
                        ...         firstname text,
                        ...         lastname text,
                        ...         state text)
                        ...         WITH ID = 88128dc0-960d-11ea-947b-39646348bb4f
                        ...         AND bloom_filter_fp_chance = 0.01
                        ...         AND dclocal_read_repair_chance = 0.1
                        ...         AND crc_check_chance = 1.0
                        ...         AND default_time_to_live = 0
                        ...         AND gc_grace_seconds = 864000
                        ...         AND min_index_interval = 128
                        ...         AND max_index_interval = 2048
                        ...         AND memtable_flush_period_in_ms = 0
                        ...         AND read_repair_chance = 0.0
                        ...         AND speculative_retry = '99PERCENTILE'
                        ...         AND comment = ''
                        ...         AND caching = { 'keys': 'ALL', 'rows_per_partition': 'NONE' }
                        ...         AND compaction = { 'max_threshold': '32', 'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy' }
                        ...         AND compression = { 'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor' }
                        ...         AND cdc = false
                        ...         AND extensions = {  };
apiadmin@cqlsh:students_details>

步骤 8：将快照文件夹中的所有文件复制到现有的板球桌文件夹

cassandra@node1:~/data/students_details/cricket-88128dc0960d11ea947b39646348bb4f/snapshots/dropped-1589479603749-cricket$ pwd
/home/cassandra/data/students_details/cricket-88128dc0960d11ea947b39646348bb4f/snapshots/dropped-1589479603749-cricket
cassandra@node1:~/data/students_details/cricket-88128dc0960d11ea947b39646348bb4f/snapshots/dropped-1589479603749-cricket$ cp * /home/cassandra/data/students_details/cricket-88128dc0960d11ea947b39646348bb4f
cassandra@node1:~/data/students_details/cricket-88128dc0960d11ea947b39646348bb4f/snapshots/dropped-1589479603749-cricket$ cd /home/cassandra/data/students_details/cricket-88128dc0960d11ea947b39646348bb4f
cassandra@node1:~/data/students_details/cricket-88128dc0960d11ea947b39646348bb4f$ ls -lrth
total 44K
drwxrwxr-x 2 cassandra cassandra    6 May 14 18:05 backups
drwxrwxr-x 3 cassandra cassandra   43 May 14 18:06 snapshots
-rw-rw-r-- 1 cassandra cassandra  891 May 14 18:11 schema.cql
-rw-rw-r-- 1 cassandra cassandra   92 May 14 18:11 md-1-big-TOC.txt
-rw-rw-r-- 1 cassandra cassandra   92 May 14 18:11 md-1-big-Summary.db
-rw-rw-r-- 1 cassandra cassandra 4.7K May 14 18:11 md-1-big-Statistics.db
-rw-rw-r-- 1 cassandra cassandra   61 May 14 18:11 md-1-big-Index.db
-rw-rw-r-- 1 cassandra cassandra   16 May 14 18:11 md-1-big-Filter.db
-rw-rw-r-- 1 cassandra cassandra    9 May 14 18:11 md-1-big-Digest.crc32
-rw-rw-r-- 1 cassandra cassandra  179 May 14 18:11 md-1-big-Data.db
-rw-rw-r-- 1 cassandra cassandra   43 May 14 18:11 md-1-big-CompressionInfo.db
-rw-rw-r-- 1 cassandra cassandra   31 May 14 18:11 manifest.json
cassandra@node1:~/data/students_details/cricket-88128dc0960d11ea947b39646348bb4f$

步骤 9：使用以下命令使用 sstableloader 开始恢复表数据

cassandra@node1:~$ sstableloader -d 10.213.61.21 -username cassandra --password cassandra /home/cassandra/data/students_details/cricket-d3576f60960f11ea947b39646348bb4f/snapshots
    Established connection to initial hosts
    Opening sstables and calculating sections to stream

    Summary statistics:
       Connections per host    : 1
       Total files transferred : 0
       Total bytes transferred : 0.000KiB
       Total duration          : 2920 ms
       Average transfer rate   : 0.000KiB/s
       Peak transfer rate      : 0.000KiB/s

Step-10：表恢复成功。请验证。

 playerid                             | city      | firstname | lastname  | state
--------------------------------------+-----------+-----------+-----------+------------
 d7b12c90-960f-11ea-947b-39646348bb4f | Berhampur |     Rohit |    Sharma |     Odisha
 d7594890-960f-11ea-947b-39646348bb4f |     Delhi |     Virat |     Vholi |  New Delhi
 d7588540-960f-11ea-947b-39646348bb4f |    Mumbai |    Sachin | Pendulkar | Maharastra

【讨论】：