不断将 docker 数据上的 elasticsearch 推送到 S3

Question

我有一个 AWS EC2 实例，它有一个运行 elasticsearch 的 docker 容器。每天，都有一些进程将新文档推送到 elasticsearch 中。我想在文档进入弹性搜索后的第二天将这些文档推送到 S3 存储桶。

我在网上找到了这个解决方案：

https://github.com/AckeeDevOps/elasticsearch-backup-to-s3

我是 elasticsearch、docker 和 aws 的新手，所以 git 存储库中提供的步骤不够详细，我不知道该怎么做，所以我想知道是否有人可以验证我对我需要采取的步骤的理解，如下：

0) 克隆上面的git仓库并设置如下参数：

ELASTICSEARCH_URL url with port where your elasticsearch runs, for example localhost:9200
S3_URL contains address in S3 where to store backups
    bucket-name/directory
S3_ACCESS_KEY
S3_SECRET_KEY
CRON_SCHEDULE cron schedule string, default '0 2 * * *'

1) 在我的 EC2 实例上，执行

docker build https://github.com/AckeeDevOps/elasticsearch-backup-to-s3

2) 然后，我需要更改正在运行的现有 docker 容器以具有其他选项

/var/backup/elasticsearch -p 9200:9200 -d elasticsearch -Des.path.repo=/var/backup/elasticsearch

3) 我不明白这一行是做什么的：

docker run --link elasticsearch:elasticsearch -e ELASTICSEARCH_URL="elasticsearch:9200" -e SNAPSHOT_VOLUME="/var/backup/elasticsearch" -e S3_URL="your S3 url" -e S3_ACCESS_KEY="your S3 access key" -e S3_SECRET_KEY="your S3 secret key"

是否有更可接受/记录的方式将弹性搜索数据连续推送到外部存储桶？我们将不胜感激任何建议和/或说明使用上述存储库所需采取的步骤。

Answer 1

引用的文档

elasticsearch-backup-snapshot-and-restore-on-aws-s3

opendistro

Dockerfile

FROM docker.elastic.co/elasticsearch/elasticsearch:7.6.2

#this is to tell that we are passing these arguments at runtime
ARG ENV_VAR_AWS_ACCESS_KEY_ID
ARG ENV_VAR_AWS_SECRET_ACCESS_KEY

ENV AWS_ACCESS_KEY_ID ${ENV_VAR_AWS_ACCESS_KEY_ID}
ENV AWS_SECRET_ACCESS_KEY ${ENV_VAR_AWS_SECRET_ACCESS_KEY}
ENV xpack.security.enabled 'false'
ENV  xpack.monitoring.enabled 'false'
ENV xpack.graph.enabled 'false'
ENV xpack.watcher.enabled 'false'
ENV discovery.type 'single-node'
ENV bootstrap.memory_lock 'true'
ENV indices.memory.index_buffer_size '30%'

RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3
RUN /usr/share/elasticsearch/bin/elasticsearch-keystore create
RUN echo $AWS_ACCESS_KEY_ID | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin s3.client.default.access_key
RUN echo $AWS_SECRET_ACCESS_KEY | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin s3.client.default.secret_key

register.json

{
  "type": "s3",
  "settings": {
    "bucket": "elk-backup-codeaprendiz"
  }
}

恢复快照.json

{
  "indices": "kibana*,my-index*",
  "ignore_unavailable": true,
  "include_global_state": false,
  "include_aliases": false,
  "partial": false,
  "rename_pattern": "kibana(.+)",
  "rename_replacement": "restored-kibana$1",
  "index_settings": {
    "index.blocks.read_only": false
  },
  "ignore_index_settings": [
    "index.refresh_interval"
  ]
}

snapshotsetting.json

{
  "indices": "kibana*,my-index*,-my-index-2016",
  "ignore_unavailable": true,
  "include_global_state": false,
  "partial": false
}

构建镜像并部署

• 构建新映像

docker build  \
--build-arg ENV_VAR_AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID> \
--build-arg ENV_VAR_AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY> \
--tag=codeaprendiz/elasticsearch .

• 检查存在的图像

$ docker images | grep "codeaprendiz/elasticsearch"
codeaprendiz/elasticsearch                      latest              f06a06d5fd8a        36 seconds ago      796MB

• 运行映像

docker run -p 9200:9200 -p 9600:9600  codeaprendiz/elasticsearch

注册

• 在 S3 注册你的 repo

$ curl -X PUT -H "Content-Type: application/json" -d @register.json "http://localhost:9200/_snapshot/my-s3-repository"
{"acknowledged":true}

拍摄快照

• 拍摄快照

$ curl -X PUT -H "Content-Type: application/json" -d @snapshotsetting.json "http://localhost:9200/_snapshot/my-s3-repository/firstsnap?wait_for_completion=true"
{"snapshot":{"snapshot":"firstsnap","uuid":"VpRaTS-eRr6TLqIOi9Zw2w","version_id":7060299,"version":"7.6.2","indices":[],"include_global_state":false,"state":"SUCCESS","start_time":"2020-05-16T14:13:06.219Z","start_time_in_millis":1589638386219,"end_time":"2020-05-16T14:13:06.624Z","end_time_in_millis":1589638386624,"duration_in_millis":405,"failures":[],"shards":{"total":0,"failed":0,"successful":0}}}

• 相应的 docker 日志

$ docker logs -f friendly_fermi
{"type": "server", "timestamp": "2020-05-16T14:13:06,646Z", "level": "INFO", "component": "o.e.s.SnapshotsService", "cluster.name": "docker-cluster", "node.name": "676c35dac6af", "message": "snapshot [my-s3-repository:firstsnap/VpRaTS-eRr6TLqIOi9Zw2w] started", "cluster.uuid": "Crq-wvoIQmuzm920sZr8MA", "node.id": "Q_xnc6qyRxy-BbvRLQNwlg"  }
{"type": "server", "timestamp": "2020-05-16T14:13:08,852Z", "level": "INFO", "component": "o.e.s.SnapshotsService", "cluster.name": "docker-cluster", "node.name": "676c35dac6af", "message": "snapshot [my-s3-repository:firstsnap/VpRaTS-eRr6TLqIOi9Zw2w] completed with state [SUCCESS]", "cluster.uuid": "Crq-wvoIQmuzm920sZr8MA", "node.id": "Q_xnc6qyRxy-BbvRLQNwlg"  }

• 请求您的快照

$ curl -X GET  "http://localhost:9200/_snapshot/my-s3-repository/firstsnap"
{"snapshots":[{"snapshot":"firstsnap","uuid":"VpRaTS-eRr6TLqIOi9Zw2w","version_id":7060299,"version":"7.6.2","indices":[],"include_global_state":false,"state":"SUCCESS","start_time":"2020-05-16T14:13:06.219Z","start_time_in_millis":1589638386219,"end_time":"2020-05-16T14:13:06.624Z","end_time_in_millis":1589638386624,"duration_in_millis":405,"failures":[],"shards":{"total":0,"failed":0,"successful":0}}]}

• 检查快照状态

$ curl -X GET "http://localhost:9200/_snapshot/_status"                
{"snapshots":[]}

• 检查所有快照

$ curl -X GET "http://localhost:9200/_snapshot/_all"
{"my-s3-repository":{"type":"s3","settings":{"bucket":"elk-backup-codeaprendiz"}}}

• 查看存储库中的所有快照

$ curl -X GET "http://localhost:9200/_snapshot/my-s3-repository/_all"
{"snapshots":[{"snapshot":"firstsnap","uuid":"VwFvTv3nSKOD5K8J3EBE2A","version_id":7060299,"version":"7.6.2","indices":[],"include_global_state":false,"state":"SUCCESS","start_time":"2020-05-14T14:45:46.358Z","start_time_in_millis":1589467546358,"end_time":"2020-05-14T14:45:46.561Z","end_time_in_millis":1589467546561,"duration_in_millis":203,"failures":[],"shards":{"total":0,"failed":0,"successful":0}}]}

恢复您的快照

• 恢复快照

$ curl -X POST -H "Content-Type: application/json" -d @restoresnapshot.json "http://localhost:9200/_snapshot/my-s3-repository/firstsnap/_restore"
{"snapshot":{"snapshot":"firstsnap","indices":[],"shards":{"total":0,"failed":0,"successful":0}}}

Link to git - basic

Link to git - intermediate