扫一扫微信二维码

开源技术实践分享:Manila + Cephfs 调研 发布时间:2017-11-24


本文章主要介绍如何在Openstack Pike + kolla 集成Manila项目提供NAS服务。主要是Manila驱动为Cephfs的情况,利用Ganesha把Guest NFS 协议转换为libcephfs协议存储数据。

\

简单介绍
什么是Manila?
Manila项目全称是File Share Service,文件共享即服务。是OpenStack大帐篷模式下的子项目之一,用来提供云上的文件共享服务,支持CIFS协议和NFS协议。目前Manila项目中集成了众多驱动实现共享服务,使用驱动最多还是Cephfs, 所以值得看看。

驱动分类
本文讲述的manila cephfs驱动分为两种,其它众多驱动请参考官方手册。

1、cephfs native share
相当于租户虚机直接连接cephfs集群,存在安全问题,生产环境中很难推动。
① ceph-common包安装在manila-share中
② ceph-common/ceph-fuse 软件安装在租户虚机Guest里
③ ceph集群与manila-share网络打通
④ ceph集群与租户虚机网络打通

2、cephfs nfs share
利用Ganesha作中转,把Guest NFS 协议转换为libcephfs协议存储数据,隐藏后端ceph集群,更适合云的业务场景。

官网说明
① Pike or later versions of manila.
② Kraken or later versions of Ceph.
③ 2.5 or later versions of NFS-Ganesha.
④ A Ceph cluster with a filesystem configured
⑤ ceph-common package installed in the servers running themanila-share service.
⑥ NFS client installed in the guest.
⑦ Network connectivity between your Ceph cluster’s public network andthe servers running the manila-share service.
⑧ Network connectivity between your Ceph cluster’s public network andNFS-Ganesha server.
⑨ Network connectivity between your NFS-Ganesha server and the manilaguest.

环境准备
1、系统配置(hosts、时间同步等)
sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
setenforce 0
systemctl stop firewalld
systemctl disable firewalld
systemctl  stop NetworkManager
systemctl  disable NetworkManager

2、openstack pike + ceph 环境
跟据官网所说准备一套openstackpike + ceph +kolla环境,其中cephfs配置参照cephfs补丁,跑出一个cephfs集群。
[root@control ~]# /opt/kolla-ansible/tools/kolla-ansible  deploy

安装nfs-ganesha
1、配源nfs-ganesha


2、安装nfs-ganesha
由于没有单独制作nfs-ganesha的docker镜像, 所以直接在主控上面安装nfs-ganesha作测试。
[root@ control ]# yum -y install nfs-ganesha-ceph nfs-ganesha-2.6 ceph-common

3、配置nfs-ganesha
编辑配置文件
[root@control ~]# cat  ganesha.repo
[nfs-ganesha]
name=nfs-ganesha packages for \$basearch
baseurl=https://4.chacra.ceph.com/r/nfs-ganesha/next/1feb192e01e838c16e1b3c01ba3995c37f55e985/centos/7/flavors/ceph_luminous/\$basearch
enabled=1
gpgcheck=0
type=rpm-md
[nfs-ganesha-noarch]
name=nfs-ganesha noarch packages
baseurl=https://4.chacra.ceph.com/r/nfs-ganesha/next/1feb192e01e838c16e1b3c01ba3995c37f55e985/centos/7/flavors/ceph_luminous/noarch
enabled=1
gpgcheck=0
type=rpm-md
[nfs-ganesha-source]
name=nfs-ganesha source packages
baseurl=https://4.chacra.ceph.com/r/nfs-ganesha/next/1feb192e01e838c16e1b3c01ba3995c37f55e985/centos/7/flavors/ceph_luminous/SRPMS
enabled=1
gpgcheck=0
type=rpm-md

4、服务启动
启动服务
[root@ control  ]# systemctl  start nfs-ganesha

安装Manila
1、安装Manila
注意:kolla中global.yml有关Manila配置
[root@ control ]# cat /etc/kolla/globals.yml| grep manila
#enable_horizon_manila: "{{ enable_manila | bool }}"
enable_manila: "yes"
enable_manila_backend_generic: "yes"
#enable_manila_backend_hnas: "no

2、授权Manila
如下命令会生成manila.keyring文件,一并复制到 manila-share的/etc/ceph目录下:
read -d '' MON_CAPS << EOF
allow r,
allow command "auth del",
allow command "auth caps",
allow command "auth get",
allow command "auth get-or-create"
EOF
ceph auth get-or-create client.manila -o manila.keyring \
mds 'allow *' \
osd 'allow rw' \
mon "$MON_CAPS"

3、配置Manila
manila-share基础包安装
[root@ control ]#  yum -y install ceph ceph-common libcephfs2-devel  librgw2-devel

manila-share 关于ceph的配置
 [root@ control ]# docker  cp  /etc/kolla/ceph-mon/*  manila_share:/etc/ceph

编辑manila-share的/etc/ceph/ceph.conf 添加如下配置
[client.manila]
client mount uid = 0
client mount gid = 0
log file = /var/logs/ceph-client.manila.log

支持快照
ceph mds set allow_new_snaps true --yes-i-really-mean-it

编辑manila-share的/etc/manila/manila.conf 添加如下配置:
[DEFAULT]
...
enabled_share_backends = generic, cephfsnfs1
default_share_type = default_share_type
enabled_share_protocols = NFS,CIFS,CEPHFS
[cephfsnfs1]
driver_handles_share_servers = False
share_backend_name = CEPHFSNFS1
share_driver = manila.share.drivers.cephfs.driver.CephFSDriver
cephfs_protocol_helper_type = NFS
cephfs_conf_path = /etc/ceph/ceph.conf
cephfs_auth_id = manila
cephfs_cluster_name = ceph
cephfs_enable_snapshots = False
cephfs_ganesha_server_is_remote= True
cephfs_ganesha_server_username=root
cephfs_ganesha_server_password=99cloud
# cephfs_ganesha_path_to_private_key =/path/to/key   #或者用key方式
cephfs_ganesha_server_ip = 172.18.23.17   # 指明nfs-ganesha服务器位置

注意:manila-share去调nfs-ganesha的协议是ssh,所以有必要配置ssh访问.如果nfs-ganesha与manila-share服务在一台机器上不用配ssh的。并且cephfs_ganesha_server_is_remote= False 让其调用本地ganesha即可。

4、验证服务
重启各manila/ceph容器或服务
(manila-share)[root@control /]# manila service-list
+----+------------------+--------------------+------+---------+-------+----------------------------+

| Id | Binary           | Host               | Zone | Status  | State | Updated_at                 |

+----+------------------+--------------------+------+---------+-------+----------------------------+

| 1  | manila-data      | control            | nova | enabled | up    | 2017-11-15T10:14:37.000000 |

| 2  | manila-scheduler | control            | nova | enabled | up    | 2017-11-15T10:14:38.000000 |

| 3  | manila-share     | control@generic    | nova | enabled | up    | 2017-11-15T10:14:40.000000 |

| 4  | manila-share     | control@cephfsnfs1 | nova | enabled | up    | 2017-11-15T10:14:40.000000 |

+----+------------------+--------------------+------+---------+-------+----------------------------+

Manila测试
1、创建实例
创建manila实例
manila type-create cephfsnfstype false
manila type-key cephfsnfstype set vendor_name=Ceph storage_protocol=NFS
manila create --share-type cephfsnfstype --name cephnfsshare1 nfs 1

2、查看共享路径
查看共享路径
 (manila-share)[root@control /]# manila share-export-location-list cephnfsshare1

+--------------------------------------+---------------------------------------------------------------------+-----------+

| ID                                   | Path                                                                | Preferred |

+--------------------------------------+---------------------------------------------------------------------+-----------+

| a5e7b14e-62e7-4d10-a9d4-7bf2208b440e | 172.18.23.17:/volumes/_nogroup/cd753328-a37b-4715-b847-3add77824a0f | False     |

+--------------------------------------+---------------------------------------------------------------------+-----------+

3、权限管理
\

4、vm 挂载
在己授权IP地址的guest vm 上挂载(其实是任意支持NFS协议客户端包括云以外的主机)
[root@client-2 ~]# mount  172.18.23.17:/volumes/_nogroup/cd753328-a37b-4715-b847-3add77824a0f /data/
[root@client-2 ~]# df -hT | grep data$
172.18.23.17:/volumes/_nogroup/cd753328-a37b-4715-b847-3add77824a0f nfs4      1.0G     0  1.0G   0% /data

5、容量管理
最大只能写入创建共享实例定义的容量空间
dashborad视图
\

手动dd测试写入超过先前定义容量时将返回错误
[root@client-2 data]# dd if=/dev/zero  of=test.img bs=512M count=3
dd: error writing ‘test.img’: Disk quota exceeded
3+0 records in
2+0 records out
1073741824 bytes (1.1 GB) copied, 24.5038 s, 43.8 MB/s

当业务数据量大时可扩容Manila共享实例
\
\

参考链接
官方手册:https://docs.openstack.org/manila/pike/contributor/
开发文档:https://docs.openstack.org/manila/latest/


© 2012-2017 九州云信息科技有限公司 99Cloud 版权所有 咨询热线:400 006 0472 售后服务热线:400 670 7810 培训咨询热线:400 826 0070   ICP证:浙ICP备12032350号-1

网站建设:信达互联

北京网站建设公司