kubernetes ceph-rbd挂载步骤 类型storageClass 参考https://github.com/nacos-group/nacos-k8s/tree/master/deploy

https://blog.csdn.net/mailjoin/article/details/79686919

https://github.com/nacos-group/nacos-k8s/blob/master/README-ceph.md

 

由于kubelet本身并不支持rbd的命令，所以需要添加一个kube系统插件:

 下载插件 quay.io/external_storage/rbd-provisioner

下载地址： 

https://quay.io/repository/external_storage/rbd-provisioner?tag=latest&tab=tags

在k8s集群的node上面下载 docker pull quay.io/external_storage/rbd-provisioner:latest

只安装插件本身会报错:需要安装kube的角色和权限 以下是下载地址：

https://github.com/kubernetes-incubator/external-storage  

https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/rbd/deploy/rbac #下载kube的role的yaml文件

下载rbac文件夹：

使用：  kubectl  apply  -f rbac/

运行rbd-provisioner

如果报错：

报错因为rbd-provisioner的镜像中不能找到ceph的key和conf，需要把集群中key和conf拷贝进rbd-provisioner的镜像。

找到rbd-provisioner的镜像运行节点

docker cp  /etc/ceph/ceph.client.admin.keyring  <镜像名>：/etc/ceph/

docker cp  /etc/ceph/ceph.conf  <镜像名>：/etc/ceph/

如果又报错：

一直处于Pending，因为linux内核不支持 image format 1，所以我们要在sc中加入新建镜像时给他规定镜像的格式为2

在stroageclass中添加：

                                    imageFormat: "2"

                                    imageFeatures: "layering"

这样pvc就创建成功：

 

安装插件及角色（rbac）：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

#clusterrole.yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata:   name: rbd-provisioner rules:   - apiGroups: [""]     resources: ["persistentvolumes"]     verbs: ["get", "list", "watch", "create", "delete"]   - apiGroups: [""]     resources: ["persistentvolumeclaims"]     verbs: ["get", "list", "watch", "update"]   - apiGroups: ["storage.k8s.io"]     resources: ["storageclasses"]     verbs: ["get", "list", "watch"]   - apiGroups: [""]     resources: ["events"]     verbs: ["list", "watch", "create", "update", "patch"]   - apiGroups: [""]     resources: ["services"]     resourceNames: ["kube-dns"]     verbs: ["list", "get"] #clusterrolebinding.yaml kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata:   name: rbd-provisioner subjects:   - kind: ServiceAccount     name: rbd-provisioner     namespace: default roleRef:   kind: ClusterRole   name: rbd-provisioner   apiGroup: rbac.authorization.k8s.io #deployment.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata:   name: rbd-provisioner spec:   replicas: 1   strategy:     type: Recreate   template:     metadata:       labels:         app: rbd-provisioner     spec:       containers:       - name: rbd-provisioner         image: "quay.io/external_storage/rbd-provisioner:latest"         env:         - name: PROVISIONER_NAME           value: ceph.com/rbd    #定义插件的名字       serviceAccount: rbd-provisioner #role.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata:   name: rbd-provisioner rules: - apiGroups: [""]   resources: ["secrets"]   verbs: ["get"] #rolebinding.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata:   name: rbd-provisioner roleRef:   apiGroup: rbac.authorization.k8s.io   kind: Role   name: rbd-provisioner subjects: - kind: ServiceAccount   name: rbd-provisioner   namespace: default #serviceaccount.yaml apiVersion: v1 kind: ServiceAccount metadata:   name: rbd-provisioner

　　

创建storageClass：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

kind: StorageClass apiVersion: storage.k8s.io/v1 metadata:   name: rbd provisioner: ceph.com/rbd    #使用插件来生成sc parameters:   monitors: 10.101.3.9:6789,10.101.3.11:6789,10.101.3.12:6789   adminId: admin   adminSecretName: ceph-k-secret   adminSecretNamespace: default  #这里使用default 如果使用其他就要修改还要修改插件中的   pool: rbd   userId: admin   userSecretName: ceph-k-secret   fsType: ext4   imageFormat: "2"   imageFeatures: "layering"

　　

参照官方文档做法如下：
1. 生成ceph-secret:

[root@walker-2 ~]# cat /etc/ceph/admin.secret | base64
QVFEdmRsTlpTSGJ0QUJBQUprUXh4SEV1ZGZ5VGNVa1U5cmdWdHc9PQo=
[root@walker-2 ~]# kubectl create secret generic ceph-secret --type="kubernetes.io/rbd" --from-literal=key='QVFEdmRsTlpTSGJ0QUJBQUprUXh4SEV1ZGZ5VGNVa1U5cmdWdHc9PQo=' --namespace=kube-system

StorageClass 配置：
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: axiba
provisioner: kubernetes.io/rbd
parameters:
monitors: 172.16.18.5:6789,172.16.18.6:6789,172.16.18.7:6789
adminId: admin
adminSecretName: ceph-secret
adminSecretNamespace: hehe
pool: rbd
userId: admin
userSecretName: ceph-secret
fsType: ext4
imageFormat: "2"
imageFeatures: "layering"

adminId: ceph 客户端ID，能够在pool中创建image, 默认是admin
adminSecretName: ceph客户端秘钥名(第一步中创建的秘钥)
adminSecretNamespace: 秘钥所在的namespace
pool: ceph集群中的pool, 可以通过命令 ceph osd pool ls查看
userId: 使用rbd的用户ID，默认admin
userSecretName: 同上
imageFormat: ceph RBD 格式。”1” 或者 “2”，默认为”1”. 2支持更多rbd特性
imageFeatures: 只有格式为”2”时，才有用。默认为空值(“”)
---------------------

创建PVC：

1 2 3 4 5 6 7 8 9 10 11

apiVersion: v1 kind: PersistentVolumeClaim metadata:   name: ceph-rbd-dyn-pv-claim spec:   accessModes:     - ReadWriteOnce   storageClassName: rbd   resources:     requests:       storage: 1Gi

由于kubelet本身并不支持rbd的命令，所以需要添加一个kube系统插件:

 下载插件 quay.io/external_storage/rbd-provisioner

下载地址： 

https://quay.io/repository/external_storage/rbd-provisioner?tag=latest&tab=tags

在k8s集群的node上面下载 docker pull quay.io/external_storage/rbd-provisioner:latest

只安装插件本身会报错:需要安装kube的角色和权限 以下是下载地址：

https://github.com/kubernetes-incubator/external-storage  

https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/rbd/deploy/rbac #下载kube的role的yaml文件

下载rbac文件夹：

使用：  kubectl  apply  -f rbac/

运行rbd-provisioner

如果报错：

报错因为rbd-provisioner的镜像中不能找到ceph的key和conf，需要把集群中key和conf拷贝进rbd-provisioner的镜像。

找到rbd-provisioner的镜像运行节点

docker cp  /etc/ceph/ceph.client.admin.keyring  <镜像名>：/etc/ceph/

docker cp  /etc/ceph/ceph.conf  <镜像名>：/etc/ceph/

如果又报错：

一直处于Pending，因为linux内核不支持 image format 1，所以我们要在sc中加入新建镜像时给他规定镜像的格式为2

在stroageclass中添加：

                                    imageFormat: "2"

                                    imageFeatures: "layering"

这样pvc就创建成功：

 

安装插件及角色（rbac）：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

#clusterrole.yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata:   name: rbd-provisioner rules:   - apiGroups: [""]     resources: ["persistentvolumes"]     verbs: ["get", "list", "watch", "create", "delete"]   - apiGroups: [""]     resources: ["persistentvolumeclaims"]     verbs: ["get", "list", "watch", "update"]   - apiGroups: ["storage.k8s.io"]     resources: ["storageclasses"]     verbs: ["get", "list", "watch"]   - apiGroups: [""]     resources: ["events"]     verbs: ["list", "watch", "create", "update", "patch"]   - apiGroups: [""]     resources: ["services"]     resourceNames: ["kube-dns"]     verbs: ["list", "get"] #clusterrolebinding.yaml kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata:   name: rbd-provisioner subjects:   - kind: ServiceAccount     name: rbd-provisioner     namespace: default roleRef:   kind: ClusterRole   name: rbd-provisioner   apiGroup: rbac.authorization.k8s.io #deployment.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata:   name: rbd-provisioner spec:   replicas: 1   strategy:     type: Recreate   template:     metadata:       labels:         app: rbd-provisioner     spec:       containers:       - name: rbd-provisioner         image: "quay.io/external_storage/rbd-provisioner:latest"         env:         - name: PROVISIONER_NAME           value: ceph.com/rbd    #定义插件的名字       serviceAccount: rbd-provisioner #role.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata:   name: rbd-provisioner rules: - apiGroups: [""]   resources: ["secrets"]   verbs: ["get"] #rolebinding.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata:   name: rbd-provisioner roleRef:   apiGroup: rbac.authorization.k8s.io   kind: Role   name: rbd-provisioner subjects: - kind: ServiceAccount   name: rbd-provisioner   namespace: default #serviceaccount.yaml apiVersion: v1 kind: ServiceAccount metadata:   name: rbd-provisioner

　　

创建storageClass：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

kind: StorageClass apiVersion: storage.k8s.io/v1 metadata:   name: rbd provisioner: ceph.com/rbd    #使用插件来生成sc parameters:   monitors: 10.101.3.9:6789,10.101.3.11:6789,10.101.3.12:6789   adminId: admin   adminSecretName: ceph-k-secret   adminSecretNamespace: default  #这里使用default 如果使用其他就要修改还要修改插件中的   pool: rbd   userId: admin   userSecretName: ceph-k-secret   fsType: ext4   imageFormat: "2"   imageFeatures: "layering"

　　

参照官方文档做法如下：
1. 生成ceph-secret:

[root@walker-2 ~]# cat /etc/ceph/admin.secret | base64
QVFEdmRsTlpTSGJ0QUJBQUprUXh4SEV1ZGZ5VGNVa1U5cmdWdHc9PQo=
[root@walker-2 ~]# kubectl create secret generic ceph-secret --type="kubernetes.io/rbd" --from-literal=key='QVFEdmRsTlpTSGJ0QUJBQUprUXh4SEV1ZGZ5VGNVa1U5cmdWdHc9PQo=' --namespace=kube-system

StorageClass 配置：
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: axiba
provisioner: kubernetes.io/rbd
parameters:
monitors: 172.16.18.5:6789,172.16.18.6:6789,172.16.18.7:6789
adminId: admin
adminSecretName: ceph-secret
adminSecretNamespace: hehe
pool: rbd
userId: admin
userSecretName: ceph-secret
fsType: ext4
imageFormat: "2"
imageFeatures: "layering"

adminId: ceph 客户端ID，能够在pool中创建image, 默认是admin
adminSecretName: ceph客户端秘钥名(第一步中创建的秘钥)
adminSecretNamespace: 秘钥所在的namespace
pool: ceph集群中的pool, 可以通过命令 ceph osd pool ls查看
userId: 使用rbd的用户ID，默认admin
userSecretName: 同上
imageFormat: ceph RBD 格式。”1” 或者 “2”，默认为”1”. 2支持更多rbd特性
imageFeatures: 只有格式为”2”时，才有用。默认为空值(“”)
---------------------

创建PVC：

1 2 3 4 5 6 7 8 9 10 11

apiVersion: v1 kind: PersistentVolumeClaim metadata:   name: ceph-rbd-dyn-pv-claim spec:   accessModes:     - ReadWriteOnce   storageClassName: rbd   resources:     requests:       storage: 1Gi