k8s上用ceph-rbd存储

发表于 | 分类于 | 浏览 次

k8s默认使用的本地存储，集群容灾性差，ceph作为开源的分布式存储系统，与openstack环境搭配使用，已经很多云计算公司运用于生产环境，可靠性得到验证。这里介绍一下在k8s环境下ceph如何使用.

Kubernetes支持后两种存储接口,支持的接入模式如下图:

ceph端

新建pool

新建一个pool pool_1包含90个pg 

1
ceph osd pool create pool_1 90

RBD块设备

在ceph集群中新建1个rbd块设备，lun1 

1
rbd create pool_1/lun1 --size 10G

ceph权限控制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
使用ceph-deploy --overwrite-conf admin部署的keyring权限太大，可以自己创建一个keyring client.rdb给块设备客户端node用


# ceph auth get-or-create client.rbd mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=pool_1'  > ceph.client.rbd.keyring

k8s节点需要安装ceph
yum install ceph-common
echo 'rbd' > /etc/modules-load.d/rbd.conf
modprobe rbd
lsmod | grep rbd
rbd                    83640  0 
libceph               306625  1 rbd

配置文件秘钥传到k8s上
[root@ceph ceph]# scp ceph.client.rbd.keyring 192.168.6.102:/etc/ceph/
root@192.168.6.102's password: 
ceph.client.rdb.keyring                                                                                                       100%   63     8.5KB/s   00:00    
[root@ceph ceph]# scp ceph.conf 192.168.6.102:/etc/ceph/                
root@192.168.6.102's password: 
ceph.conf                                                                                                                       100%  310    25.1KB/s   00:00    
[root@ceph ceph]#

k8s的node上操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@node1 ceph]# ceph -s --name client.rdb
  cluster:
    id:     cbc04385-1cdf-4512-a3f5-a5b3e8686a05
    health: HEALTH_WARN
            application not enabled on 1 pool(s)
 
  services:
    mon: 1 daemons, quorum ceph
    mgr: ceph(active)
    osd: 1 osds: 1 up, 1 in
 
  data:
    pools:   1 pools, 90 pgs
    objects: 5 objects, 709B
    usage:   1.00GiB used, 19.0GiB / 20.0GiB avail
    pgs:     90 active+clean
    
警告解决办法：
ceph health detail
ceph osd pool application enable pool_1 rbd

map设备

1
2
3
4
5
6
7
8
9
# rbd map pool_1/lun1 --name client.rbd
rbd: sysfs write failed
RBD image feature set mismatch. Try disabling features unsupported by the kernel with "rbd feature disable".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address

解决办法：
在ceph节点上
rbd feature disable pool_1/lun1 exclusive-lock, object-map, fast-diff, deep-flatten

将块设备挂载在操作系统中进行格式化

1
2
rbd map pool_1/lun1 --name client.rbd
mkfs.ext4 /dev/rbd0

创建pv、pvc

1
2
3
对ceph.client.admin.keyring 的内容进行base64编码
[root@node1 ceph]# ceph auth get-key client.rbd | base64
QVFCTktERmRzeXpKQUJBQVVvVGVvWVYyamxhRi8zNU1hZ2R2dFE9PQ==
1
2
3
4
5
6
7
8
9
10
11
12
根据上面的输出，创建secret ceph-client-rbd
[root@node1 ceph]# cat ceph-secret.yml 
apiVersion: v1
kind: Secret
metadata:
  name: ceph-client-rbd
type: "kubernetes.io/rbd"  
data:
  key: QVFCTktERmRzeXpKQUJBQVVvVGVvWVYyamxhRi8zNU1hZ2R2dFE9PQ==
  
  
kubectl apply -f ceph-secret.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
创建pv，注意： 这里是user：rbd 而不是user: client.rbd

[root@node1 ceph]# cat pv.yml 
kind: PersistentVolume
apiVersion: v1
metadata:
  name: ceph-pool1-lun1
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce          
  rbd:
    fsType: ext4
    image: lun1
    monitors:
      - '192.168.6.101:6789'
    pool: pool_1
    readOnly: false
    secretRef:
      name: ceph-client-rbd
      namespace: default
    user: rbd
    
[root@node1 ceph]# kubectl apply -f pv.yml 
persistentvolume/ceph-pool1-lun1 created
[root@node1 ceph]# kubectl get pv
NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
ceph-pool1-lun1   10Gi       RWO            Retain           Available           manual                  4s
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
创建pvc

[root@node1 ceph]# cat pvc.yml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc1
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
      
 
 
[root@node1 ceph]# kubectl apply -f pvc.yml 
persistentvolumeclaim/pvc1 created
[root@node1 ceph]# kubectl get pvc
NAME   STATUS   VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc1   Bound    ceph-pool1-lun1   10Gi       RWO            manual         7s
Donate