Ceph分布式存储与K8s集成

2019/10/6 by Qin Zhiqiang in 云原生 Ceph, CephFS, K8s, 分布式存储

在K8S集群中，比较麻烦的是：有状态服务持久化，并且保证高可用。面临的问题：

有状态服务POD死亡时，持久化数据能够立刻转移到新拉起的POD。
持久化数据本身读写的高可用。

Ceph分布式存储就是解决问题2；同时使用独立的分布式存储集群，分离PV与工作节点的耦合，确保PVC能够转移给新的POD，解决问题1。

本文将演练：部署一个Ceph分布式存储单例，通过CephFS集成到K8s。

1. 部署Ceph分布式存储

Ceph高可用集群，请参考：https://computingforgeeks.com/how-to-deploy-ceph-storage-cluster-on-ubuntu-18-04-lts/ ，与下文的单例部署的差异主要在$HOST变量的位置。

下文中只练习Ceph单例与K8S的配合工作。新建独立的VM：Ubuntu Server 18.04，内存：2G，IP：192.168.80.94， Host name：k8s-storage。

第一步，在Ubuntu Server 18.04中安装ceph-deploy：

sudo apt install gnupg2 -y

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
#echo deb https://download.ceph.com/debian-mimic/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
echo deb http://mirrors.ustc.edu.cn/ceph/debian-mimic/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list

sudo apt update
sudo apt install ceph-deploy -y

第二步，部署Ceph单例：

先在虚拟机中添加两块磁盘，之后其检查结果如下：

xxx@k8s-storage:~$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   127G  0 disk
├─sda1   8:1    0   512M  0 part /boot/efi
└─sda2   8:2    0 126.5G  0 part /
sdb      8:16   0    40G  0 disk
sdc      8:32   0    40G  0 disk
sr0     11:0    1  1024M  0 rom

xxx@k8s-storage:~$ sudo fdisk -l
Disk /dev/sdb: 40 GiB, 42949672960 bytes, 83886080 sectors
...
Disk /dev/sdc: 40 GiB, 42949672960 bytes, 83886080 sectors
...

我们将使用/dev/sdb和/dev/sdc做Ceph存储磁盘。

部署命令如下：

export DATA_DEV=/dev/sdb
export JRNL_DEV=/dev/sdc
export HOST=k8s-storage

mkdir ceph-deploy
cd ceph-deploy

# Install packages
sudo ceph-deploy new $HOST
sudo ceph-deploy install --no-adjust-repos $HOST

# Deploy a monitor
echo "osd crush chooseleaf type = 0" | sudo tee -a ceph.conf
echo "osd pool default size = 1" | sudo tee -a ceph.conf
sudo ceph-deploy mon create-initial

# Copy admin key to the nodes
sudo ceph-deploy admin $HOST

# Deploy a manager daemon
sudo ceph-deploy mgr create $HOST

# Deploy a metadata server
sudo ceph-deploy mds create $HOST

# list disks
sudo ceph-deploy disk list $HOST

# destroy existed data
sudo ceph-deploy disk zap $HOST $DATA_DEV $JRNL_DEV

# create new Ceph OSD daemon
sudo ceph-deploy osd create --data $DATA_DEV $HOST
sudo ceph-deploy osd create --data $JRNL_DEV $HOST

# show disk info
lsblk

# check health
sudo ceph health
  # HEALTH_OK

查看单例集群状态：

xxx@k8s-storage:~/ceph-deploy$ sudo ceph status
  cluster:
    id:     46ef2e01-f338-4aee-a8cc-3fef7ad39ecb
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum k8s-storage
    mgr: k8s-storage(active)
    osd: 2 osds: 2 up, 2 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   2.0 GiB used, 78 GiB / 80 GiB avail
    pgs:

2. 开启CephFS服务

要通过CephFS与K8s集成，Ceph需要开启CephFS服务，需要 Linux Kernel >= 4.0。

开启CephFS服务，在k8s-storage上执行：

# Create the pools
sudo ceph osd pool create fs_data 128
sudo ceph osd pool create fs_metadata 128
sudo ceph osd lspools

# Create CephFS
sudo ceph fs new cephfs fs_metadata fs_data

# Check
sudo ceph fs ls
sudo ceph mds stat

# enable application
sudo ceph osd pool application enable fs_data cephfs
sudo ceph osd pool application enable fs_metadata cephfs

# change max_mds
sudo ceph fs set cephfs max_mds 1

# check health
sudo ceph health
# HEALTH_OK

3. 集成到K8s

使用CephFS模式与with RBAC roles部署到K8S。

获取Ceph管理密码，在k8s-storage上执行：

# Find the secret
sudo ceph auth get-key client.admin > /tmp/secret

部署cephfs-provisioner，在k8s-master上执行：

# Follow: https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/cephfs#test-instruction
nano /tmp/secret # copy `k8s-storage` /tmp/secret to `k8s-master`
  # ...

kubectl create ns cephfs
kubectl create secret generic ceph-secret-admin --from-file=/tmp/secret --namespace=cephfs

# check secret
kubectl get secret ceph-secret-admin -n cephfs -o yaml

# Deploy cephfs-provisioner, follow: https://github.com/kubernetes-incubator/external-storage/blob/master/ceph/cephfs/deploy/README.md
sudo apt install unzip
wget -O external-storage-master.zip https://codeload.github.com/kubernetes-incubator/external-storage/zip/master

unzip external-storage-master.zip
cd external-storage-master/ceph/cephfs/deploy
kubectl -n cephfs apply -f ./rbac


# Make StorageClass
mkdir ~/cephfs
cd ~/cephfs

cat >storageclass-cephfs.yaml<<EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: cephfs
provisioner: ceph.com/cephfs
parameters:
    monitors: 192.168.80.94:6789
    adminId: admin
    adminSecretName: ceph-secret-admin
    adminSecretNamespace: "cephfs"
    claimRoot: /volumes/kubernetes
EOF

kubectl apply -f storageclass-cephfs.yaml


# check the cephfs-provisioner
kubectl get pods -n cephfs -o wide

4. 执行测试样例

使用Nginx显示修改的index.html做测试，在k8s-master上执行：

cat >test-cephfs-claim.yaml<<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-cephfs-claim
spec:
  storageClassName: cephfs
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

kubectl apply -f test-cephfs-claim.yaml

# check
kubectl get pvc
kubectl get pv

cat >test-cephfs.yaml<<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-cephfs
  labels:
    k8s-app: test-cephfs
spec:
  revisionHistoryLimit: 1
  replicas: 1
  selector:
    matchLabels:
      k8s-app: test-cephfs
  template:
    metadata:
      labels:
        k8s-app: test-cephfs
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
          name: http
        volumeMounts:
        - name: cephfs
          mountPath: /usr/share/nginx/html
      volumes:
      - name: cephfs
        persistentVolumeClaim:
          claimName: test-cephfs-claim
---
apiVersion: v1
kind: Service
metadata:
  name: test-cephfs
  labels:
    k8s-app: test-cephfs
spec:
  type: NodePort
  ports:
  - port: 80
    name: http
  selector:
    k8s-app: test-cephfs
EOF

kubectl apply -f test-cephfs.yaml

检查PVC与POD是否正常：

xxx@k8s-master:~/cephfs$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS   REASON   AGE
pvc-e0883f6e-45de-41bb-8a2c-06ae1ef4e533   1Gi        RWO            Delete           Bound    default/test-cephfs-claim   cephfs                  3m20s

xxx@k8s-master:~/cephfs$ kubectl get pvc
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-cephfs-claim   Bound    pvc-e0883f6e-45de-41bb-8a2c-06ae1ef4e533   1Gi        RWO            cephfs         5s

xxx@k8s-master:~/cephfs$ kubectl get pods -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
test-cephfs-67469b6fdd-mnhbs   1/1     Running   0          63m   10.244.2.2   k8s-worker-2   <none>           <none>

xxx@k8s-master:~/cephfs$ kubectl get svc
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP        23h
test-cephfs   NodePort    10.100.45.185   <none>        80:31314/TCP   64m

可以看到PODtest-cephfs-xxx已经正常启动。如果POD无法正常启动，说明k8s-storage节点的CephFS服务有异常；修复之后，先执行kubectl delete -f test-cephfs-claim.yaml，再重新执行kubectl apply -f test-cephfs-claim.yaml，触发PVC。

现在使用NodePort去看k8s-worker-2节点的Nginx服务返回的内容：

xxx@k8s-master:~$ curl 192.168.80.92:31314
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.17.4</center>
</body>
</html

当前POD中的/usr/share/nginx/html没有文件，返回403正常。

写入文件测试：

xxx@k8s-master:~$ kubectl exec -ti test-cephfs-67469b6fdd-mnhbs -- /bin/sh -c 'echo Hello World from CephFS!!! > /usr/share/nginx/html/index.html'

xxx@k8s-master:~$ curl 192.168.80.92:31314
Hello World from CephFS!!!

可以看得文件写入成功。我们再删除该POD，再试试到另外一个节点。

xxx@k8s-master:~$ kubectl delete pods test-cephfs-67469b6fdd-mnhbs
pod "test-cephfs-67469b6fdd-mnhbs" deleted

xxx@k8s-master:~$ kubectl get pods -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
test-cephfs-67469b6fdd-8x2s9   1/1     Running   0          31s   10.244.1.4   k8s-worker-1   <none>           <none>

xxx@k8s-master:~$ curl 192.168.80.91:31314
Hello World from CephFS!!!

可以看到，通过k8s-worker-1节点新拉起的POD去访问，也获取到刚才写入的内容。至此，CephFS集成到K8S演练完成。