kubernetes 接 ceph 當 storage backend

kubernetes 版本:1.13.0
不同版本的設定可能會不一樣,我在201903測試過可以用
未來可能會有些許不同
不過原則應該都一樣

原則:
kubernetes 支援用 rbd 當 storage backend。但是 kube-controller-manager container 未必會包含 rbd 指令。如果 kube-controller-manager container 未包含 rbd 指令,你建立 storage 時就會看到 『failed to create rbd image: executable file not found in $PATH, command output:』的錯誤。這時候,你就要提供新的 container ,讓 kubernetes 可以透過你提供的 container (之後都叫 provider )來執行 rbd 相關操作。該 provider 放在 kube-system 下。

腳本簡易說明:

一開始建立名為 rbd-provisioner 的 serviceAccount ,之後在建立 secret 用來放 ceph keyring。最後建立 storage class ( 裡面放 ceph 相關資訊 )

https://gist.github.com/kjelly/97ebc4133b1293eff9135eada974d670

完整腳本如下:

---
# kubectl -n kube-system create -f create-kube-ceph-pbrc.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-provisioner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["kube-dns","coredns"]
    verbs: ["list", "get"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-provisioner
subjects:
  - kind: ServiceAccount
    name: rbd-provisioner
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: rbd-provisioner
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: rbd-provisioner
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: rbd-provisioner
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rbd-provisioner
subjects:
- kind: ServiceAccount
  name: rbd-provisioner
  namespace: kube-system

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rbd-provisioner
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: rbd-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: rbd-provisioner
    spec:
      containers:
      - name: rbd-provisioner
        image: "quay.io/external_storage/rbd-provisioner:latest"
        env:
        - name: PROVISIONER_NAME
          value: ceph.com/rbd
      serviceAccount: rbd-provisioner

---
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret-admin
  namespace: kube-system
data:
  # change the value
  # sudo ceph auth get-key client.admin | base64
  key: QVFDYlFubGNVZjVzRkJBQUtKanN1R1FuWUpvNlRweWZQT0E0d3c9PQ==
type: kubernetes.io/rbd


---
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret-user
  namespace: kube-system
data:
  # change the value
  # sudo ceph auth get-key client.admin | base64
  key: QVFDYlFubGNVZjVzRkJBQUtKanN1R1FuWUpvNlRweWZQT0E0d3c9PQ==
type: kubernetes.io/rbd


---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ceph-dynamic
  annotations:
     storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: ceph.com/rbd
parameters:
  monitors: 10.20.0.5:6789 # change it if needed.
  adminId: admin # change it if needed.
  adminSecretName: ceph-secret-admin # change it if needed.
  adminSecretNamespace: kube-system # change it if needed.
  pool: kube # change it if needed.
  userId: admin # change it if needed.
  userSecretName: ceph-secret-user # change it if needed.
  userSecretNamespace: kube-system # change it if needed.
  imageFormat: "2"
  imageFeatures: "layering"

You'll only receive email when they publish something new.

More from kjelly