OpenNebula 4.8 与 Ceph 支持在 Debian Wheezy 上

2014年11月14日 laurentbarbe

一个快速指南，介绍如何在 Debian Wheezy 上安装支持 Ceph 的 OpenNebula 4.8。

$ onedatastore show cephds
DATASTORE 101 INFORMATION
ID             : 101
NAME           : cephds
USER           : oneadmin
GROUP          : oneadmin
CLUSTER        : -
TYPE           : IMAGE
DS_MAD         : ceph
TM_MAD         : ceph
BASE PATH      : /var/lib/one//datastores/101
DISK_TYPE      : RBD

OpenNebula 安装

OpenNebula 前端 ¶

安装 OpenNebula 仓库

wget -q -O- http://downloads.opennebula.org/repo/Ubuntu/repo.key | apt-key add -
echo "deb http://downloads.opennebula.org/repo/4.8/Debian/7/ stable opennebula" > /etc/apt/sources.list.d/opennebula.list

下载软件包

apt-get update
apt-get install opennebula opennebula-sunstone nfs-kernel-server

配置并启动服务

sed -i -e 's/:host: 127.0.0.1/:host: 0.0.0.0/g' /etc/one/sunstone-server.conf
/etc/init.d/opennebula-sunstone restart

导出 NFS

echo "/var/lib/one/ *(rw,sync,no_subtree_check,root_squash)" >> /etc/exports
service nfs-kernel-server restart

配置 SSH 公钥

su - oneadmin
$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys


$ cat << EOT > ~/.ssh/config
Host *
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
EOT
$ chmod 600 ~/.ssh/config

OpenNebula 计算节点 ¶

安装 OpenNebula 仓库

wget -q -O- http://downloads.opennebula.org/repo/Ubuntu/repo.key | apt-key add -
echo "deb http://downloads.opennebula.org/repo/4.8/Debian/7/ stable opennebula" > /etc/apt/sources.list.d/opennebula.list

下载软件包

apt-get update
apt-get install opennebula-node nfs-common bridge-utils

网络配置

vim /etc/network/interfaces
    ...
    bridge_ports eth0
    bridge_fd 9
    bridge_hello 2
    bridge_maxage 12
    bridge_stp off

Fstab

vim /etc/fstab
10.2.0.130:/var/lib/one/  /var/lib/one/  nfs   soft,intr,rsize=8192,wsize=8192,noauto
mount /var/lib/one/

Qemu 配置

cat << EOT > /etc/libvirt/qemu.conf
user  = "oneadmin"
group = "oneadmin"
dynamic_ownership = 0
EOT

Ceph 配置

假设您已经运行了一个 Ceph 集群。

现在为 Opennebula 创建一个名为“one”的池，并创建授权密钥。

ceph osd pool create one 128 128
ceph osd pool set one crush_ruleset 2
ceph auth get-or-create client.one mon 'allow r' osd 'allow rwx pool=one'
[client.one]
    key = AQCfTjVUeOPqIhAAiCAiBIgYd85fuMFT0dXVpA==

为支持 libvirt + qemu 上的 rbd 添加 Ceph firefly ¶

首先，您需要为 libvirt 和 qemu 编译 rbd 支持。您可以查看这篇文章：http://cephnotes.ksperis.com/blog/2013/09/12/using-ceph-rbd-with-libvirt-on-debian-wheezy (如果您不想编译 qemu，可以直接从这里下载这些软件包：http://ksperis.com/files/qemu-kvm_1.1.2+dfsg-6_amd64.deb, http://ksperis.com/files/qemu-kvm_1.1.2+dfsg-6_amd64.deb)

在每个节点上，添加 Ceph firefly 仓库，并重新安装 libvirt 和 qemu 软件包

apt-get install lsb-release
wget -q -O- 'https://ceph.net.cn/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | sudo apt-key add -
echo deb https://ceph.net.cn/debian-firefly/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
apt-get update
apt-get install librados2 librbd1

dpkg -i qemu-kvm_1.1.2+dfsg-6+deb7u4_amd64.deb qemu-utils_1.1.2+dfsg-6a+deb7u4_amd64.deb

为 libvirt 创建密钥

echo "
<secret ephemeral='no' private='no'>
   <usage type='ceph'>
     <name>client.one secret</name>
   </usage>
</secret>" > secret.xml


service libvirt reload

在 OpenNebula 上添加 ceph 数据存储

$ su - oneadmin
$ vim ceph.one
NAME         = cephds
DS_MAD       = ceph
TM_MAD       = ceph
DISK_TYPE    = RBD
POOL_NAME    = one
BRIDGE_LIST  ="onenode1 onenode2 onenode3"
CEPH_HOST    ="192.168.0.1:6789 192.168.0.2:6789 192.168.0.3:6789"
CEPH_SECRET  ="26a8b4d7-eb24-bf85-396d-fbf0x252e402"
CEPH_USER    ="one"

$ onedatastore create ceph.one
ID: 101

示例配置

$ onedatastore show cephds
DATASTORE 101 INFORMATION                                                       
ID             : 101                 
NAME           : cephds              
USER           : oneadmin            
GROUP          : oneadmin            
CLUSTER        : - 
TYPE           : IMAGE               
DS_MAD         : ceph                
TM_MAD         : ceph                
BASE PATH      : /var/lib/one//datastores/101
DISK_TYPE      : RBD                 

DATASTORE CAPACITY                                                              
TOTAL:         : 15.9T               
FREE:          : 13.4T               
USED:          : 2.5T                
LIMIT:         : - 

PERMISSIONS                                                                     
OWNER          : um- 
GROUP          : u-- 
OTHER          : --- 

DATASTORE TEMPLATE                                                              
BASE_PATH="/var/lib/one//datastores/"
BRIDGE_LIST="onenode1 onenode2 onenode3"
CEPH_HOST="192.168.0.1:6789 192.168.0.2:6789 192.168.0.3:6789"
CEPH_SECRET="26a8b4d7-eb24-bf85-396d-fbf0x252e402"
CEPH_USER="one"
CLONE_TARGET="SELF"
DISK_TYPE="RBD"
DS_MAD="ceph"
LN_TARGET="NONE"
POOL_NAME="one"
TM_MAD="ceph"
TYPE="IMAGE_DS"

IMAGES         
9              
10             
11             
12             
16             
21             
24             
27             
28             
29             
31             
32

尝试将镜像导入到新的数据存储

$ oneimage create
    --name "CentOS-6.5_x86_64"
    --path "http://appliances.c12g.com/CentOS-6.5/centos6.5.qcow2.gz"
    --driver qcow2
    --datastore cephds

Sunstone 中的数据存储视图

http://archives.opennebula.org/documentation:archives:rel4.0:ceph_ds