CEPH 集群部署答案

【问题标题】：CEPH cluster deploymentCEPH 集群部署
【发布时间】：2021-09-28 13:35:22
【问题描述】：

我正在尝试部署具有 4 个节点的 CEPH 集群。管理员，节点1，节点2，客户端我正在使用 Oracle VM VIrtual Box 部署 centos 8 节点。

我正在使用 CEPH Octopus。

我正在使用 ansible-playbook 来部署集群

我按照此博客中列出的步骤来部署我的集群： https://computingforgeeks.com/install-and-configure-ceph-storage-cluster-on-centos-linux/?amp#ex1

[星期一] admin monitor_address=192.168.1.75

[mdss]
admin

[osds]
node1
node2

[clients]
node3

[mgrs]
admin

[grafana-server]
node2

但每次进程停止时，我都会收到以下错误：

  TASK [ceph-osd : use ceph-volume lvm batch to create bluestore osds] ***********
Wednesday 21 July 2021  11:29:51 +0530 (0:00:00.085)       0:02:06.609 ***********
fatal: [node2]: FAILED! => changed=true
  cmd:
  - ceph-volume
  - --cluster
  - ceph
  - lvm
  - batch
  - --bluestore
  - --yes
  - --osds-per-device
  - '4'
  - /dev/sdb
  delta: '0:50:01.301186'
  end: '2021-06-07 16:19:53.554904'
  msg: non-zero return code
  rc: 1
  start: '2021-06-07 15:29:52.253718'
  stderr: |-
    --> DEPRECATION NOTICE
    --> You are using the legacy automatic disk sorting behavior
    --> The Pacific release will change the default to --no-auto
    --> passed data devices: 1 physical, 0 LVM
    --> relative data size: 0.25
    Running command: /bin/ceph-authtool --gen-print-key
    Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4f1d0cc7-0363-4d06-84e8-c45be914159b
     stderr: 2021-06-07T15:34:53.415+0530 7f018bcde700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T15:39:53.418+0530 7f018bcde700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T15:44:53.420+0530 7f018bcde700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T15:49:53.422+0530 7f018bcde700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T15:54:53.423+0530 7f018bcde700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T15:59:53.425+0530 7f018bcde700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T16:04:53.427+0530 7f018bcde700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T16:09:53.428+0530 7f018bcde700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T16:14:53.430+0530 7f018bcde700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T16:19:53.432+0530 7f018bcde700  0 monclient(hunting): authenticate timed out after 300
     stderr: [errno 110] RADOS timed out (error connecting to the cluster)
    Traceback (most recent call last):
      File "/sbin/ceph-volume", line 11, in <module>
        load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
      File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in __init__
        self.main(self.argv)
      File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
        return f(*a, **kw)
      File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main
        terminal.dispatch(self.mapper, subcommand_args)
      File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
        instance.main()
      File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 42, in main
        terminal.dispatch(self.mapper, self.argv)
      File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
        instance.main()
      File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
        return func(*a, **kw)
      File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 415, in main
        self._execute(plan)
      File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute
        c.create(argparse.Namespace(**args))
      File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
        return func(*a, **kw)
      File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create
        prepare_step.safe_prepare(args)
      File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare
        self.prepare()
      File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
        return func(*a, **kw)
      File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare
        self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id)
      File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id
        raise RuntimeError('Unable to create a new OSD id')
    RuntimeError: Unable to create a new OSD id
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
fatal: [node1]: FAILED! => changed=true
  cmd:
  - ceph-volume
  - --cluster
  - ceph
  - lvm
  - batch
  - --bluestore
  - --yes
  - --osds-per-device
  - '4'
  - /dev/sdb
  delta: '0:50:02.352739'
  end: '2021-06-07 16:19:54.591371'
  msg: non-zero return code
  rc: 1
  start: '2021-06-07 15:29:52.238632'
  stderr: |-
    --> DEPRECATION NOTICE
    --> You are using the legacy automatic disk sorting behavior
    --> The Pacific release will change the default to --no-auto
    --> passed data devices: 1 physical, 0 LVM
    --> relative data size: 0.25
    Running command: /bin/ceph-authtool --gen-print-key
    Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8b324b40-b7d4-4e97-9178-e0f987fd3b67
     stderr: 2021-06-07T15:34:53.504+0530 7fbccc9f2700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T15:39:53.531+0530 7fbccc9f2700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T15:44:53.560+0530 7fbccc9f2700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T15:49:53.775+0530 7fbccc9f2700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T15:54:53.799+0530 7fbccc9f2700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T15:59:53.820+0530 7fbccc9f2700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T16:04:53.846+0530 7fbccc9f2700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T16:09:53.849+0530 7fbccc9f2700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T16:14:54.327+0530 7fbccc9f2700  0 monclient(hunting): authenticate timed out after 300
     stderr: 2021-06-07T16:19:54.476+0530 7fbccc9f2700  0 monclient(hunting): authenticate timed out after 300
     stderr: [errno 110] RADOS timed out (error connecting to the cluster)
    Traceback (most recent call last):
      File "/sbin/ceph-volume", line 11, in <module>
        load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
      File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in __init__
        self.main(self.argv)
      File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
        return f(*a, **kw)
      File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main
        terminal.dispatch(self.mapper, subcommand_args)
      File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
        instance.main()
      File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 42, in main
        terminal.dispatch(self.mapper, self.argv)
      File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
        instance.main()
      File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
        return func(*a, **kw)
      File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 415, in main
        self._execute(plan)
      File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute
        c.create(argparse.Namespace(**args))
      File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
        return func(*a, **kw)
      File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create
        prepare_step.safe_prepare(args)
      File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare
        self.prepare()
      File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
        return func(*a, **kw)
      File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare
        self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id)
      File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id
        raise RuntimeError('Unable to create a new OSD id')
    RuntimeError: Unable to create a new OSD id
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

请帮我解决这个问题。在我尝试的 10 次中，我只成功了 2 次。我每次都使用相同的配置。

【问题讨论】：

我正在使用 ceph 章鱼。
您可以添加您关注的文档吗？我不熟悉 ansible 但error connecting to the cluster 通常意味着该节点缺少 ceph.conf 和/或适当的密钥环。
@eblock：我按照这个博客上列出的步骤，computingforgeeks.com/…
请用其他信息编辑您的问题，而不是评论它们，以便所有信息都集中在一个地方。你能验证一下ceph.conf和/var/lib/ceph/bootstrap-osd/ceph.keyring已经在osd节点上成功创建了吗？

标签： ceph

【解决方案1】：

将 OSD 节点更改为 3 个节点，然后再试一次，因为 quorum，或者您可以从此配置文件激活测试模式： group_vars/all.yml ---> 将 ceph_test: false 更改为 ceph_test: true

【讨论】：