这是一个简单的联合网关配置示例，用于在两个 Ceph 集群之间进行异步复制。
( 此配置基于 Ceph 文档：https://ceph.net.cn/docs/master/radosgw/federated-config/ )
这里我只使用一个区域 (“default”) 和两个区域 (“main” 和 “fallback”)，每个集群一个。
请注意，在此示例中，我使用 3 个放置目标 (default, hot, cold)，它们分别对应于池 .main.rgw.buckets, .main.rgw.hot.buckets, .main.rgw.cold.buckets。请务必将标签 {MAIN_USER_ACCESS}, {MAIN_USER_SECRET}, {FALLBACK_USER_ACESS}, {FALLBACK_USER_SECRET} 替换为相应的值。
首先，我创建了区域和区域文件，这些文件将在 2 个集群上需要
区域文件 “region.conf.json”
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
| { "name": "default",
"api_name": "default",
"is_master": "true",
"endpoints": [
"http:\/\/s3.mydomain.com:80\/"],
"master_zone": "main",
"zones": [
{ "name": "main",
"endpoints": [
"http:\/\/s3.mydomain.com:80\/"],
"log_meta": "true",
"log_data": "true"},
{ "name": "fallback",
"endpoints": [
"http:\/\/s3-fallback.mydomain.com:80\/"],
"log_meta": "true",
"log_data": "true"}],
"placement_targets": [
{ "name": "default-placement",
"tags": []},
{ "name": "cold-placement",
"tags": []},
{ "name": "hot-placement",
"tags": []}],
"default_placement": "default-placement"}
|
一个区域文件 “zone-main.conf.json”
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
| { "domain_root": ".main.domain.rgw",
"control_pool": ".main.rgw.control",
"gc_pool": ".main.rgw.gc",
"log_pool": ".main.log",
"intent_log_pool": ".main.intent-log",
"usage_log_pool": ".main.usage",
"user_keys_pool": ".main.users",
"user_email_pool": ".main.users.email",
"user_swift_pool": ".main.users.swift",
"user_uid_pool": ".main.users.uid",
"system_key": {
"access_key": "{MAIN_USER_ACCESS}",
"secret_key": "{MAIN_USER_SECRET}"},
"placement_pools": [
{ "key": "default-placement",
"val": { "index_pool": ".main.rgw.buckets.index",
"data_pool": ".main.rgw.buckets",
"data_extra_pool": ".main.rgw.buckets.extra"}},
{ "key": "cold-placement",
"val": { "index_pool": ".main.rgw.buckets.index",
"data_pool": ".main.rgw.cold.buckets",
"data_extra_pool": ".main.rgw.buckets.extra"}},
{ "key": "hot-placement",
"val": { "index_pool": ".main.rgw.buckets.index",
"data_pool": ".main.rgw.hot.buckets",
"data_extra_pool": ".main.rgw.buckets.extra"}}]}
|
和一个区域文件 “zone-fallback.conf.json”
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
| { "domain_root": ".fallback.domain.rgw",
"control_pool": ".fallback.rgw.control",
"gc_pool": ".fallback.rgw.gc",
"log_pool": ".fallback.log",
"intent_log_pool": ".fallback.intent-log",
"usage_log_pool": ".fallback.usage",
"user_keys_pool": ".fallback.users",
"user_email_pool": ".fallback.users.email",
"user_swift_pool": ".fallback.users.swift",
"user_uid_pool": ".fallback.users.uid",
"system_key": {
"access_key": "{FALLBACK_USER_ACESS}",
"secret_key": "{FALLBACK_USER_SECRET}"
},
"placement_pools": [
{ "key": "default-placement",
"val": { "index_pool": ".fallback.rgw.buckets.index",
"data_pool": ".fallback.rgw.buckets",
"data_extra_pool": ".fallback.rgw.buckets.extra"}},
{ "key": "cold-placement",
"val": { "index_pool": ".fallback.rgw.buckets.index",
"data_pool": ".fallback.rgw.cold.buckets",
"data_extra_pool": ".fallback.rgw.buckets.extra"}},
{ "key": "hot-placement",
"val": { "index_pool": ".fallback.rgw.buckets.index",
"data_pool": ".fallback.rgw.hot.buckets",
"data_extra_pool": ".fallback.rgw.buckets.extra"}}]}
|
在第一个集群 (MAIN) ¶
我创建了池
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| ceph osd pool create .rgw.root 16 16
ceph osd pool create .main.rgw.root 16 16
ceph osd pool create .main.domain.rgw 16 16
ceph osd pool create .main.rgw.control 16 16
ceph osd pool create .main.rgw.gc 16 16
ceph osd pool create .main.rgw.buckets 512 512
ceph osd pool create .main.rgw.hot.buckets 512 512
ceph osd pool create .main.rgw.cold.buckets 512 512
ceph osd pool create .main.rgw.buckets.index 32 32
ceph osd pool create .main.rgw.buckets.extra 16 16
ceph osd pool create .main.log 16 16
ceph osd pool create .main.intent-log 16 16
ceph osd pool create .main.usage 16 16
ceph osd pool create .main.users 16 16
ceph osd pool create .main.users.email 16 16
ceph osd pool create .main.users.swift 16 16
ceph osd pool create .main.users.uid 16 16
|
我配置了区域、区域并添加了系统用户
1
2
3
4
5
6
7
| radosgw-admin region set --name client.radosgw.main < region.conf.json
radosgw-admin zone set --rgw-zone=main --name client.radosgw.main < zone-main.conf.json
radosgw-admin zone set --rgw-zone=fallback --name client.radosgw.main < zone-fallback.conf.json
radosgw-admin regionmap update --name client.radosgw.main
radosgw-admin user create --uid="main" --display-name="Zone main" --name client.radosgw.main --system --access-key={MAIN_USER_ACCESS} --secret={MAIN_USER_SECRET}
radosgw-admin user create --uid="fallback" --display-name="Zone fallback" --name client.radosgw.main --system --access-key={FALLBACK_USER_ACESS} --secret={FALLBACK_USER_SECRET}
|
在 MAIN 集群的 ceph.conf 中设置 RadosGW 配置
1
2
3
4
5
6
7
8
9
10
| [client.radosgw.main]
host = ceph-main-radosgw-01
rgw region = default
rgw region root pool = .rgw.root
rgw zone = main
rgw zone root pool = .main.rgw.root
rgw frontends = "civetweb port=80"
rgw dns name = s3.mydomain.com
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw_socket_path = /var/run/ceph/radosgw.sock
|
我需要在 /etc/ceph/ceph.client.radosgw.keyring 中为 [client.radosgw.main] 创建密钥环，请参阅文档。
然后，启动/重新启动 MAIN 集群的 radosgw。
在另一个 Ceph 集群 (FALLBACK) ¶
我创建了池
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| ceph osd pool create .rgw.root 16 16
ceph osd pool create .fallback.rgw.root 16 16
ceph osd pool create .fallback.domain.rgw 16 16
ceph osd pool create .fallback.rgw.control 16 16
ceph osd pool create .fallback.rgw.gc 16 16
ceph osd pool create .fallback.rgw.buckets 512 512
ceph osd pool create .fallback.rgw.hot.buckets 512 512
ceph osd pool create .fallback.rgw.cold.buckets 512 512
ceph osd pool create .fallback.rgw.buckets.index 32 32
ceph osd pool create .fallback.rgw.buckets.extra 16 16
ceph osd pool create .fallback.log 16 16
ceph osd pool create .fallback.intent-log 16 16
ceph osd pool create .fallback.usage 16 16
ceph osd pool create .fallback.users 16 16
ceph osd pool create .fallback.users.email 16 16
ceph osd pool create .fallback.users.swift 16 16
ceph osd pool create .fallback.users.uid 16 16
|
我配置了区域、区域并添加了系统用户
radosgw-admin region set --name client.radosgw.fallback < region.conf.json
radosgw-admin zone set --rgw-zone=fallback --name client.radosgw.fallback < zone-fallback.conf.json
radosgw-admin zone set --rgw-zone=main --name client.radosgw.fallback < zone-main.conf.json
radosgw-admin regionmap update --name client.radosgw.fallback
radosgw-admin user create --uid="fallback" --display-name="Zone fallback" --name client.radosgw.fallback --system --access-key={FALLBACK_USER_ACESS} --secret={FALLBACK_USER_SECRET}
radosgw-admin user create --uid="main" --display-name="Zone main" --name client.radosgw.fallback --system --access-key={MAIN_USER_ACCESS} --secret={MAIN_USER_SECRET}
在 FALLBACK 集群的 ceph.conf 中设置 RadosGW 配置
1
2
3
4
5
6
7
8
9
10
| [client.radosgw.fallback]
host = ceph-fallback-radosgw-01
rgw region = default
rgw region root pool = .rgw.root
rgw zone = fallback
rgw zone root pool = .fallback.rgw.root
rgw frontends = "civetweb port=80"
rgw dns name = s3-fallback.mydomain.com
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw_socket_path = /var/run/ceph/radosgw.sock
|
此外，我需要在 /etc/ceph/ceph.client.radosgw.keyring 中为 [client.radosgw.fallback] 创建密钥环并启动 FALLBACK 集群的 radosgw。
最后设置 RadosGW Agent ¶
/etc/ceph/radosgw-agent/default.conf
1
2
3
4
5
6
7
8
9
| src_zone: main
source: http://s3.mydomain.com:80
src_access_key: {MAIN_USER_ACCESS}
src_secret_key: {MAIN_USER_SECRET}
dest_zone: fallback
destination: http://s3-fallback.mydomain.com:80
dest_access_key: {FALLBACK_USER_ACESS}
dest_secret_key: {FALLBACK_USER_SECRET}
log_file: /var/log/radosgw/radosgw-sync.log
|
1
| /etc/init.d/radosgw-agent start
|
之后，还有一点悬念……然后我尝试在 s3.mydomain.com 上创建一个包含数据的 bucket，并验证它是否已正确同步。
为了调试，您可以启用每个侧面的 RadosGW 日志，并使用 radosgw-agent -v -c /etc/ceph/radosgw-agent/default.conf 启动 radosgw-agent。
这些步骤对我有效。设置有时并不明显。每当我设置同步时，它很少能第一次成功，但最终总是会运行起来。