ceph对象存储

发表于 | 分类于 | 浏览 次

Ceph RGW简介

Ceph RGW(即RADOS Gateway)是Ceph对象存储网关服务，是基于LIBRADOS接口封装实现的FastCGI服务，对外提供存储和管理对象数据的Restful API。 对象存储适用于图片、视频等各类文件的上传下载，可以设置相应的访问权限。目前Ceph RGW兼容常见的对象存储API，例如兼容绝大部分Amazon S3 API，兼容OpenStack Swift API。

部署 RGW 服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@ceph1 ceph]# ceph-deploy install --rgw ceph1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy install --rgw ceph1
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  testing                       : None
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fa3faca5e60>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  dev_commit                    : None
[ceph_deploy.cli][INFO  ]  install_mds                   : False
[ceph_deploy.cli][INFO  ]  stable                        : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  adjust_repos                  : True
[ceph_deploy.cli][INFO  ]  func                          : <function install at 0x7fa3fbb955f0>
[ceph_deploy.cli][INFO  ]  install_mgr                   : False
[ceph_deploy.cli][INFO  ]  install_all                   : False
[ceph_deploy.cli][INFO  ]  repo                          : False
[ceph_deploy.cli][INFO  ]  host                          : ['ceph1']
[ceph_deploy.cli][INFO  ]  install_rgw                   : True

将配置文件、密钥文件同步到 ceph1：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[root@ceph1 ceph]# ceph-deploy admin ceph1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy admin ceph1
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fe0e152d3b0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['ceph1']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x7fe0e1dc0230>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph1
[ceph1][DEBUG ] connected to host: ceph1 
[ceph1][DEBUG ] detect platform information from remote host
[ceph1][DEBUG ] detect machine type
[ceph1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

启动一个RGW服务 ：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
先将ceph.conf加一个参数配置
[root@ceph1 ceph]# vim ceph.conf 
[global]
fsid = cde3244e-89e0-4630-84d5-bf08c0e33b24
mon_initial_members = ceph1
mon_host = 192.168.6.101
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd_pool_default_size = 2
[mgr]
mgr modules = dashboard
[mon]
mon allow pool delete = true   ####有这个配置，生成的pool才可以被删除



[root@ceph1 ceph]#  ceph-deploy rgw create ceph1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy rgw create ceph1
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  rgw                           : [('ceph1', 'rgw.ceph1')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fda85404ab8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function rgw at 0x7fda85a53050>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts ceph1:rgw.ceph1
[ceph1][DEBUG ] connected to host: ceph1 
[ceph1][DEBUG ] detect platform information from remote host
[ceph1][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: CentOS Linux 7.6.1810 Core
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph1
[ceph1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph1][DEBUG ] create path recursively if it doesn't exist
[ceph1][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph1 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph1/keyring
[ceph1][INFO  ] Running command: systemctl enable ceph-radosgw@rgw.ceph1
[ceph1][INFO  ] Running command: systemctl start ceph-radosgw@rgw.ceph1
[ceph1][INFO  ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph1 and default port 7480

验证：
[root@ceph1 ceph]# systemctl status ceph-radosgw@rgw.ceph1
● ceph-radosgw@rgw.ceph1.service - Ceph rados gateway
   Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-07-11 15:03:24 CST; 9s ago
 Main PID: 21057 (radosgw)
   CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw@rgw.ceph1.service
           └─21057 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph1 --setuser ceph --setgroup ceph

Jul 11 15:03:24 ceph1 systemd[1]: Started Ceph rados gateway.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@ceph1 ceph]# ceph -s
  cluster:
    id:     cde3244e-89e0-4630-84d5-bf08c0e33b24
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum ceph1
    mgr: ceph1(active)
    osd: 2 osds: 2 up, 2 in
    rgw: 1 daemon active
 
  data:
    pools:   4 pools, 32 pgs
    objects: 187 objects, 1.09KiB
    usage:   2.01GiB used, 30.0GiB / 32.0GiB avail
    pgs:     32 active+clean

这时看下danshboard图：

使用亚马逊 s3 客户端进行访问

用户

创建用户 ：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
[root@ceph1 ceph]# radosgw-admin user create --uid="radosgw" --display-name="First User"
{
    "user_id": "radosgw",
    "display_name": "First User",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "radosgw",
            "access_key": "CQE7E6ZDVA74KVJ0077A",
            "secret_key": "wnRy76RM2s85ozWvKwTBezrBU0RwcCTQJf1HFAM8"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}

这个是后续需要的账户信息

1
2
3
"user": "radosgw",
            "access_key": "CQE7E6ZDVA74KVJ0077A",
            "secret_key": "wnRy76RM2s85ozWvKwTBezrBU0RwcCTQJf1HFAM8"

授权用户，允许 radosgw 读写 users 信息： 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

[root@ceph1 ceph]# radosgw-admin caps add --uid=radosgw --caps="users=*"
{
    "user_id": "radosgw",
    "display_name": "First User",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "radosgw",
            "access_key": "CQE7E6ZDVA74KVJ0077A",
            "secret_key": "wnRy76RM2s85ozWvKwTBezrBU0RwcCTQJf1HFAM8"
        }
    ],
    "swift_keys": [],
    "caps": [
        {
            "type": "users",
            "perm": "*"
        }
    ],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}

允许 radosgw 读写所有的usage信息： 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
[root@ceph1 ceph]#  radosgw-admin caps add --uid=radosgw --caps="usage=read,write"
{
    "user_id": "radosgw",
    "display_name": "First User",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "radosgw",
            "access_key": "CQE7E6ZDVA74KVJ0077A",
            "secret_key": "wnRy76RM2s85ozWvKwTBezrBU0RwcCTQJf1HFAM8"
        }
    ],
    "swift_keys": [],
    "caps": [
        {
            "type": "usage",
            "perm": "*"
        },
        {
            "type": "users",
            "perm": "*"
        }
    ],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}

创建子用户，做为后面 swift 客户端访问时使用：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
[root@ceph1 ceph]#  radosgw-admin subuser create --uid=radosgw --subuser=radosgw:swift --access=full
{
    "user_id": "radosgw",
    "display_name": "First User",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [
        {
            "id": "radosgw:swift",
            "permissions": "full-control"
        }
    ],
    "keys": [
        {
            "user": "radosgw",
            "access_key": "CQE7E6ZDVA74KVJ0077A",
            "secret_key": "wnRy76RM2s85ozWvKwTBezrBU0RwcCTQJf1HFAM8"
        }
    ],
    "swift_keys": [
        {
            "user": "radosgw:swift",
            "secret_key": "A3GDj2yjkGJahkCM6YJS4QKQlGz2zd65GXvCkiwV"
        }
    ],
    "caps": [
        {
            "type": "usage",
            "perm": "*"
        },
        {
            "type": "users",
            "perm": "*"
        }
    ],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}

创建密钥 ：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
[root@ceph1 ceph]# radosgw-admin key create --subuser=radosgw:swift --key-type=swift --gen-secret
{
    "user_id": "radosgw",
    "display_name": "First User",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [
        {
            "id": "radosgw:swift",
            "permissions": "full-control"
        }
    ],
    "keys": [
        {
            "user": "radosgw",
            "access_key": "CQE7E6ZDVA74KVJ0077A",
            "secret_key": "wnRy76RM2s85ozWvKwTBezrBU0RwcCTQJf1HFAM8"
        }
    ],
    "swift_keys": [
        {
            "user": "radosgw:swift",
            "secret_key": "CZ2F5lqOfdmTCLSPyzqcw5pPNeetruAL4HjHkiAh"
        }
    ],
    "caps": [
        {
            "type": "usage",
            "perm": "*"
        },
        {
            "type": "users",
            "perm": "*"
        }
    ],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}

安装 s3 客户端软件

1
[root@ceph1 ceph]# yum -y install s3cmd.noarch

对 s3 进行配置

1
[root@ceph1 ceph]# s3cmd --configure
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
New settings:
  Access Key: CQE7E6ZDVA74KVJ0077A
  Secret Key: wnRy76RM2s85ozWvKwTBezrBU0RwcCTQJf1HFAM8
  Default Region: US
  S3 Endpoint: 192.168.6.101:7480
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.192.168.6.101:7480 bucket
  Encryption password: 123456
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: 
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

格式是这样：

1
2
3
4
5
6
7
8
Default Region [US]:                        #这里一定不要修改，否则后面会报错
S3 Endpoint [s3.amazonaws.com]: 192.168.6.101:7480
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: %(bucket)s.192.168.20.148:7480 bucket     #相当于百度网盘的创建文件夹，这里是固定格式
Path to GPG program [/usr/bin/gpg]:                 #保持默认
Use HTTPS protocol [Yes]: no                    #这里写 no ，因为没有提供 https 端口
HTTP Proxy server name:                       #这里不用写，因为没有代理
Test access with supplied credentials? [Y/n] y
Save settings? [y/N] y

由于我没把端口改成80，所以需要带端口访问的，后续可以nginx代理

创建存储数据的 bucket

1
2
3
4
5
6
7
[root@ceph1 ~]# s3cmd mb s3://cephdir
[root@ceph1 ~]# s3cmd put /etc/hosts s3://ceph_dir
upload: '/etc/hosts' -> 's3://ceph_dir/hosts'  [1 of 1]
 200 of 200   100% in    1s   133.14 B/s  done
 
 [root@ceph1 ~]# s3cmd ls s3://ceph_dir
2019-07-11 08:41       200   s3://ceph_dir/hosts

s3 的测试脚本：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@ceph-f ~]# yum -y install python-boto
[root@ceph-f ~]# vim s3test.py
import boto.s3.connection
access_key = 'N6ALEK0KS0ISYCIM5JBG'
secret_key = 'qK9hrpX2uwna4elPP1VsuErmAHBw3So40fE2K4yM'
conn = boto.connect_s3(
         aws_access_key_id=access_key,         
         aws_secret_access_key=secret_key,         
         host='ceph1', port=7480,         
         is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(),        
         )
bucket = conn.create_bucket('xxx_yyy')
for bucket in conn.get_all_buckets():
     print "{name} {created}".format(         
     name=bucket.name,         
     created=bucket.creation_date,     
     )

在使用时，请替换自己的 access_key、secret_key、主机名和端口

swift 接口测试

1
2
3
4
yum install python-setuptools
easy_install pip
pip install --upgrade setuptools
pip install --upgrade python-swiftclient

命令行访问 ：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@ceph1 ~]# swift -A http://192.168.6.101:7480/auth/v1.0 -U radosgw:swift -K CZ2F5lqOfdmTCLSPyzqcw5pPNeetruAL4HjHkiAh list
ceph_dir


swift -A http://192.168.6.101:7480/auth/v1.0 -U radosgw:swift -K CZ2F5lqOfdmTCLSPyzqcw5pPNeetruAL4HjHkiAh post sy-yt

swift -A http://192.168.6.101:7480/auth/v1.0 -U radosgw:swift -K CZ2F5lqOfdmTCLSPyzqcw5pPNeetruAL4HjHkiAh post sy_yt



[root@ceph1 ~]# swift -A http://192.168.6.101:7480/auth/v1.0 -U radosgw:swift -K CZ2F5lqOfdmTCLSPyzqcw5pPNeetruAL4HjHkiAh list
ceph_dir
sy-yt
sy_yt

这里提供 swift 的测试脚本：

1
2
3
4
5
6
7
8
9
10
11
12
[root@ceph-f ~]# vim swift.py
import swiftclient
user = 'radosgw:swift'
key = 'CZ2F5lqOfdmTCLSPyzqcw5pPNeetruAL4HjHkiAh'

conn = swiftclient.Connection(
         user=user,         
         key=key,         
         authurl='http://192.168.6.101:7480/auth/v1.0',
         )
for container in conn.get_account()[1]:
         print container['name']

在使用时，请替换自己的 access_key、secret_key、authurl

Donate