CEPH快速部署(Centos7+Jewel)
生活随笔
收集整理的這篇文章主要介紹了
CEPH快速部署(Centos7+Jewel)
小編覺得挺不錯的,現在分享給大家,幫大家做個參考.
ceph介紹
Ceph是統一存儲系統，支持三種接口。
Object：有原生的API，而且也兼容Swift和S3的API
Block：支持精簡配置、快照、克隆
File：Posix接口，支持快照
Ceph也是分布式存儲系統，它的特點是：
高擴展性：使用普通x86服務器，支持10~1000臺服務器，支持TB到PB級的擴展。
高可靠性：沒有單點故障，多數據副本，自動管理，自動修復。
高性能：數據分布均衡，并行化度高。對于objects storage和block storage,不需要元數據服務器。
ceph架構
組件
Ceph的底層是RADOS，它的意思是“A reliable, autonomous, distributed object storage”。 RADOS由兩個組件組成：
OSD： Object Storage Device，提供存儲資源。
Monitor：維護整個Ceph集群的全局狀態。
環境：三臺裝有Centos7的 主機，每個主機有三個磁盤（虛擬機磁盤要大于100G）
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 | [root@mon-1?cluster]#?cat?/etc/redhat-release? CentOS?Linux?release?7.2.1511?(Core)? [root@mon-1?cluster]#?lsblk? NAME????????????MAJ:MIN?RM??SIZE?RO?TYPE?MOUNTPOINT fd0???????????????2:0????1????4K??0?disk? sda???????????????8:0????0???20G??0?disk? ├─sda1????????????8:1????0??500M??0?part?/boot └─sda2????????????8:2????0?19.5G??0?part? ??├─centos-root?253:0????0?18.5G??0?lvm??/ ??└─centos-swap?253:1????0????1G??0?lvm??[SWAP] sdb???????????????8:16???0??200G??0?disk? sdc???????????????8:32???0??200G??0?disk? sdd???????????????8:48???0??200G??0?disk? sr0??????????????11:0????1??603M??0?rom |
修改主機名并添加hosts
| 1 2 3 4 5 6 | vim?/etc/hostname?????#由于7和6.5修改主機名的方法不一樣，這里舉個例子 mon-1 [root@osd-1?ceph-osd-1]#?cat?/etc/hosts 192.168.50.123??mon-1 192.168.50.124??osd-1 192.168.50.125??osd-2 |
配置ssh無密碼登陸
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | [root@localhost?~]#?ssh-keygen??-t?rsa?-P?'' Generating?public/private?rsa?key?pair. Enter?file?in?which?to?save?the?key?(/root/.ssh/id_rsa):? Your?identification?has?been?saved?in?/root/.ssh/id_rsa. Your?public?key?has?been?saved?in?/root/.ssh/id_rsa.pub. The?key?fingerprint?is: 62:b0:4c:aa:e5:37:92:89:4d:db:c3:38:e2:f1:2a:d6?root@admin-node The?key's?randomart?image?is: +--[?RSA?2048]----+ |?????????????????| |?????????????????| |????o????????????| |???+?o???????????| |??+?o?o?S????????| |?B?B?.?.?????????| |+.@?*????????????| |oooE?o???????????| |oo..?????????????| +-----------------+ ssh-copy-id?mon-1 ssh-copy-id?osd-1 ssh-copy-id?osd-2 |
2.集群配置
主機IP功能
3.環境清理
如果之前部署失敗了，不必刪除ceph客戶端，或者重新搭建虛擬機，只需要在每個節點上執行如下指令即可將環境清理至剛安裝完ceph客戶端時的狀態！強烈建議在舊集群上搭建之前清理干凈環境，否則會發生各種異常情況。
| 1 2 3 4 5 6 7 8 9 10 11 12 13 | ps?aux|grep?ceph?|awk?'{print?$2}'|xargs?kill?-9 ps?-ef|grep?ceph #確保此時所有ceph進程都已經關閉！！！如果沒有關閉，多執行幾次。 umount?/var/lib/ceph/osd/* rm?-rf?/var/lib/ceph/osd/* rm?-rf?/var/lib/ceph/mon/* rm?-rf?/var/lib/ceph/mds/* rm?-rf?/var/lib/ceph/bootstrap-mds/* rm?-rf?/var/lib/ceph/bootstrap-osd/* rm?-rf?/var/lib/ceph/bootstrap-mon/* rm?-rf?/var/lib/ceph/tmp/* rm?-rf?/etc/ceph/* rm?-rf?/var/run/ceph/* |
安裝部署流程
yum源及ceph的安裝
需要在每個主機上執行以下命令
| 1 2 3 4 5 6 7 | yum?clean?all rm?-rf?/etc/yum.repos.d/*.repo wget?-O?/etc/yum.repos.d/CentOS-Base.repo?http://mirrors.aliyun.com/repo/Centos-7.repo wget?-O?/etc/yum.repos.d/epel.repo?http://mirrors.aliyun.com/repo/epel-7.repo sed?-i?'/aliyuncs/d'?/etc/yum.repos.d/CentOS-Base.repo sed?-i?'/aliyuncs/d'?/etc/yum.repos.d/epel.repo sed?-i?'s/$releasever/7.2.1511/g'?/etc/yum.repos.d/CentOS-Base.repo |
增加ceph源
| 1 2 3 4 5 6 7 8 9 | vim?/etc/yum.repos.d/ceph.repo [ceph] name=ceph baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ gpgcheck=0 [ceph-noarch] name=cephnoarch baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/noarch/ gpgcheck=0 |
安裝ceph客戶端
| 1 2 3 4 5 | yum?-y?install?ceph?ceph-radosgw?ntp?ntpdate 關閉selinux和firewalld sed?-i?'s/SELINUX=.*/SELINUX=disabled/'?/etc/selinux/config???#重啟機器 systemctl?stop?firewalld? systemctl?disable?firewalld |
同步各個節點時間：
| 1 2 3 | ntpdate??-s?time-a.nist.gov echo?ntpdate??-s?time-a.nist.gov?>>?/etc/rc.d/rc.local? echo?"01?01?*?*?*?/usr/sbin/ntpdate?-s?time-a.nist.gov????>>?/dev/null?2>&1"?>>?/etc/crontab |
開始部署
在部署節點(mon-1)安裝ceph-deploy，下文的部署節點統一指mon-1:
| 1 2 3 4 5 | [root@mon-1?~]#?yum?-y?install?ceph-deploy [root@mon-1?~]#?ceph-deploy?--version 1.5.36 [root@mon-1?~]#?ceph?-v ceph?version?10.2.3?(ecc23778eb545d8dd55e2e4735b53cc93f92e65b) |
在部署節點創建部署目錄
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | [root@mon-1?~]#?mkdir?cluster [root@mon-1?~]#?cd?cluster [root@mon-1?cluster]#?ceph-deploy?new?mon-1?osd-1?osd-2 [ceph_deploy.conf][DEBUG?]?found?configuration?file?at:?/root/.cephdeploy.conf [ceph_deploy.cli][INFO??]?Invoked?(1.5.36):?/usr/bin/ceph-deploy?new?mon-1?osd-1?osd-2 [ceph_deploy.cli][INFO??]?ceph-deploy?options: [ceph_deploy.cli][INFO??]??username??????????????????????:?None [ceph_deploy.cli][INFO??]??func??????????????????????????:?<function?new?at?0x1803230> [ceph_deploy.cli][INFO??]??verbose???????????????????????:?False [ceph_deploy.cli][INFO??]??overwrite_conf????????????????:?False [ceph_deploy.cli][INFO??]??quiet?????????????????????????:?False [ceph_deploy.cli][INFO??]??cd_conf???????????????????????:?<ceph_deploy.conf.cephdeploy.Conf?instance?at?0x186d440> [ceph_deploy.cli][INFO??]??cluster???????????????????????:?ceph [ceph_deploy.cli][INFO??]??ssh_copykey???????????????????:?True [ceph_deploy.cli][INFO??]??mon???????????????????????????:?['mon-1',?'osd-1',?'osd-2'] [ceph_deploy.cli][INFO??]??public_network????????????????:?None [ceph_deploy.cli][INFO??]??ceph_conf?????????????????????:?None [ceph_deploy.cli][INFO??]??cluster_network???????????????:?None [ceph_deploy.cli][INFO??]??default_release???????????????:?False [ceph_deploy.cli][INFO??]??fsid??????????????????????????:?None [ceph_deploy.new][DEBUG?]?Creating?new?cluster?named?ceph [ceph_deploy.new][INFO??]?making?sure?passwordless?SSH?succeeds [ceph_deploy.new][DEBUG?]?Writing?initial?config?to?ceph.conf...???#如果能準確無誤的走到這一步就算成功了 |
目錄內容如下
| 1 2 3 4 5 | [root@mon-1?cluster]#?ll total?12 -rw-r--r--?1?root?root??197?Dec??6?02:24?ceph.conf -rw-r--r--?1?root?root?2921?Dec??6?02:24?ceph-deploy-ceph.log -rw-------?1?root?root???73?Dec??6?02:24?ceph.mon.keyring |
根據自己的IP配置向ceph.conf中添加public_network，并稍微增大mon之間時差允許范圍(默認為0.05s，現改為2s)：
| 1 2 3 4 5 6 7 8 9 10 11 12 | [root@mon-1?cluster]#?echo?public_network=192.168.50.0/24?>>ceph.conf? [root@mon-1?cluster]#?echo?mon_clock_drift_allowed?=?2?>>?ceph.conf? [root@mon-1?cluster]#?cat?ceph.conf? [global] fsid?=?0865fe85-1655-4208-bed6-274cae945746 mon_initial_members?=?mon-1,?osd-1,?osd-2 mon_host?=?192.168.50.123,192.168.50.124,192.168.50.125 auth_cluster_required?=?cephx auth_service_required?=?cephx auth_client_required?=?cephx public_network=192.168.50.0/24 mon_clock_drift_allowed?=?2 |
部署監控節點
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | [root@mon-1?cluster]#?ceph-deploy??mon?create-initial [ceph_deploy.conf][DEBUG?]?found?configuration?file?at:?/root/.cephdeploy.conf [ceph_deploy.cli][INFO??]?Invoked?(1.5.36):?/usr/bin/ceph-deploy?mon?create-initial [ceph_deploy.cli][INFO??]?ceph-deploy?options: [ceph_deploy.cli][INFO??]??username??????????????????????:?None [ceph_deploy.cli][INFO??]??verbose???????????????????????:?False [ceph_deploy.cli][INFO??]??overwrite_conf????????????????:?False [ceph_deploy.cli][INFO??]??subcommand????????????????????:?create-initial [ceph_deploy.cli][INFO??]??quiet?????????????????????????:?False [ceph_deploy.cli][INFO??]??cd_conf???????????????????????:?<ceph_deploy.conf.cephdeploy.Conf?instance?at?0x2023fc8> [ceph_deploy.cli][INFO??]??cluster???????????????????????:?ceph [ceph_deploy.cli][INFO??]??func??????????????????????????:?<function?mon?at?0x201d140> [ceph_deploy.cli][INFO??]??ceph_conf?????????????????????:?None [ceph_deploy.cli][INFO??]??default_release???????????????:?False [ceph_deploy.cli][INFO??]??keyrings??????????????????????:?None [ceph_deploy.mon][DEBUG?]?Deploying?mon,?cluster?ceph?hosts?mon-1?osd-1?osd-2 ......省略 [ceph_deploy.gatherkeys][INFO??]?Storing?ceph.client.admin.keyring [ceph_deploy.gatherkeys][INFO??]?Storing?ceph.bootstrap-mds.keyring [ceph_deploy.gatherkeys][INFO??]?keyring?'ceph.mon.keyring'?already?exists [ceph_deploy.gatherkeys][INFO??]?Storing?ceph.bootstrap-osd.keyring [ceph_deploy.gatherkeys][INFO??]?Storing?ceph.bootstrap-rgw.keyring [ceph_deploy.gatherkeys][INFO??]?Destroy?temp?directory?/tmp/tmpVLiIFr |
目錄內容如下
| 1 2 3 4 5 6 7 8 9 | [root@mon-1?cluster]#?ll total?76 -rw-------?1?root?root???113?Dec??6?22:49?ceph.bootstrap-mds.keyring -rw-------?1?root?root???113?Dec??6?22:49?ceph.bootstrap-osd.keyring -rw-------?1?root?root???113?Dec??6?22:49?ceph.bootstrap-rgw.keyring -rw-------?1?root?root???129?Dec??6?22:49?ceph.client.admin.keyring -rw-r--r--?1?root?root???300?Dec??6?22:47?ceph.conf -rw-r--r--?1?root?root?50531?Dec??6?22:49?ceph-deploy-ceph.log -rw-------?1?root?root????73?Dec??6?22:46?ceph.mon.keyring |
查看集群狀態
| 1 2 3 4 5 6 7 8 9 10 11 | [root@mon-1?ceph]#?ceph?-s ????cluster?1b27aaf2-8b29-49b1-b50e-7ccb1f72d1fa ?????health?HEALTH_ERR ????????????no?osds ?????monmap?e1:?1?mons?at?{mon-1=192.168.50.123:6789/0} ????????????election?epoch?3,?quorum?0?mon-1 ?????osdmap?e1:?0?osds:?0?up,?0?in ????????????flags?sortbitwise ??????pgmap?v2:?64?pgs,?1?pools,?0?bytes?data,?0?objects ????????????0?kB?used,?0?kB?/?0?kB?avail ??????????????????64?creating |
開始部署OSD：
| 1 2 3 | [root@mon-1?cluster]#?ceph-deploy??--overwrite-conf?osd?prepare??mon-1:/dev/sdb??mon-1:/dev/sdc??mon-1:/dev/sdd?osd-1:/dev/sdb??osd-1:/dev/sdc??osd-1:/dev/sdd??osd-2:/dev/sdb??osd-2:/dev/sdc??osd-2:/dev/sdd??--zap-disk 這里如果部署osd-2有問題，就刪除程序和目錄，重新分區以后成功創建osd-2 ceph-deploy??--overwrite-conf?osd?prepare??osd-2:/dev/sdb??osd-2:/dev/sdc??osd-2:/dev/sdd |
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | [root@mon-1?~]#?lsblk? NAME????????????MAJ:MIN?RM??SIZE?RO?TYPE?MOUNTPOINT fd0???????????????2:0????1????4K??0?disk? sda???????????????8:0????0???20G??0?disk? ├─sda1????????????8:1????0??500M??0?part?/boot └─sda2????????????8:2????0?19.5G??0?part? ??├─centos-root?253:0????0?18.5G??0?lvm??/ ??└─centos-swap?253:1????0????1G??0?lvm??[SWAP] sdb???????????????8:16???0????2T??0?disk? ├─sdb1????????????8:17???0????2T??0?part?/var/lib/ceph/osd/ceph-0 └─sdb2????????????8:18???0????5G??0?part? sdc???????????????8:32???0????2T??0?disk? ├─sdc1????????????8:33???0????2T??0?part?/var/lib/ceph/osd/ceph-1 └─sdc2????????????8:34???0????5G??0?part? sdd???????????????8:48???0????2T??0?disk? ├─sdd1????????????8:49???0????2T??0?part?/var/lib/ceph/osd/ceph-2 └─sdd2????????????8:50???0????5G??0?part? sr0??????????????11:0????1??603M??0?rom?? [root@osd-1?~]#?lsblk? NAME????????????MAJ:MIN?RM??SIZE?RO?TYPE?MOUNTPOINT fd0???????????????2:0????1????4K??0?disk? sda???????????????8:0????0???20G??0?disk? ├─sda1????????????8:1????0??500M??0?part?/boot └─sda2????????????8:2????0?19.5G??0?part? ??├─centos-root?253:0????0?18.5G??0?lvm??/ ??└─centos-swap?253:1????0????1G??0?lvm??[SWAP] sdb???????????????8:16???0????2T??0?disk? ├─sdb1????????????8:17???0????2T??0?part?/var/lib/ceph/osd/ceph-3 └─sdb2????????????8:18???0????5G??0?part? sdc???????????????8:32???0????2T??0?disk? ├─sdc1????????????8:33???0????2T??0?part?/var/lib/ceph/osd/ceph-4 └─sdc2????????????8:34???0????5G??0?part? sdd???????????????8:48???0????2T??0?disk? ├─sdd1????????????8:49???0????2T??0?part?/var/lib/ceph/osd/ceph-5 └─sdd2????????????8:50???0????5G??0?part? sr0??????????????11:0????1??603M??0?rom?? [root@osd-2?~]#?lsblk? NAME????????????MAJ:MIN?RM??SIZE?RO?TYPE?MOUNTPOINT fd0???????????????2:0????1????4K??0?disk? sda???????????????8:0????0???20G??0?disk? ├─sda1????????????8:1????0??500M??0?part?/boot └─sda2????????????8:2????0?19.5G??0?part? ??├─centos-root?253:0????0?18.5G??0?lvm??/ ??└─centos-swap?253:1????0????1G??0?lvm??[SWAP] sdb???????????????8:16???0????2T??0?disk? ├─sdb1????????????8:17???0????2T??0?part?/var/lib/ceph/osd/ceph-6 └─sdb2????????????8:18???0????5G??0?part? sdc???????????????8:32???0????2T??0?disk? ├─sdc1????????????8:33???0????2T??0?part?/var/lib/ceph/osd/ceph-7 └─sdc2????????????8:34???0????5G??0?part? sdd???????????????8:48???0????2T??0?disk? ├─sdd1????????????8:49???0????2T??0?part?/var/lib/ceph/osd/ceph-8 └─sdd2????????????8:50???0????5G??0?part? sr0??????????????11:0????1??603M??0?rom |
這里有個WARN，去掉這個WARN只需要增加rbd池的PG就好
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 | [root@mon-1?cluster]#?ceph?osd?pool?set?rbd?pg_num?128 set?pool?0?pg_num?to?128 [root@mon-1?cluster]#?ceph?osd?pool?set?rbd?pgp_num?128 set?pool?0?pgp_num?to?128?? [root@mon-1?cluster]#?ceph?-s ????cluster?0865fe85-1655-4208-bed6-274cae945746 ?????health?HEALTH_OK ?????monmap?e3:?2?mons?at?{mon-1=192.168.50.123:6789/0,osd-1=192.168.50.124:6789/0} ????????????election?epoch?30,?quorum?0,1?mon-1,osd-1 ?????osdmap?e58:?9?osds:?9?up,?9?in ????????????flags?sortbitwise ??????pgmap?v161:?128?pgs,?1?pools,?0?bytes?data,?0?objects ????????????310?MB?used,?18377?GB?/?18378?GB?avail ?????????????????128?active+clean |
給各個節點推送config配置
請不要直接修改某個節點的/etc/ceph/ceph.conf文件的，而是要去部署節點(此處為ceph-1:/root/cluster/ceph.conf)目錄下修改。因為節點到幾十個的時候，不可能一個個去修改的，采用推送的方式快捷安全！
修改完畢后，執行如下指令，將conf文件推送至各個節點：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | [root@mon-1?cluster]#?ceph-deploy?--overwrite-conf?config?push?mon-1?osd-1?osd-2 [ceph_deploy.conf][DEBUG?]?found?configuration?file?at:?/root/.cephdeploy.conf [ceph_deploy.cli][INFO??]?Invoked?(1.5.36):?/usr/bin/ceph-deploy?admin?mon-1?osd-1?osd-2 [ceph_deploy.cli][INFO??]?ceph-deploy?options: [ceph_deploy.cli][INFO??]??username??????????????????????:?None [ceph_deploy.cli][INFO??]??verbose???????????????????????:?False [ceph_deploy.cli][INFO??]??overwrite_conf????????????????:?False [ceph_deploy.cli][INFO??]??quiet?????????????????????????:?False [ceph_deploy.cli][INFO??]??cd_conf???????????????????????:?<ceph_deploy.conf.cephdeploy.Conf?instance?at?0x1a34e18> [ceph_deploy.cli][INFO??]??cluster???????????????????????:?ceph [ceph_deploy.cli][INFO??]??client????????????????????????:?['mon-1',?'osd-1',?'osd-2'] [ceph_deploy.cli][INFO??]??func??????????????????????????:?<function?admin?at?0x1964f50> [ceph_deploy.cli][INFO??]??ceph_conf?????????????????????:?None [ceph_deploy.cli][INFO??]??default_release???????????????:?False [ceph_deploy.admin][DEBUG?]?Pushing?admin?keys?and?conf?to?mon-1 [mon-1][DEBUG?]?connected?to?host:?mon-1? [mon-1][DEBUG?]?detect?platform?information?from?remote?host [mon-1][DEBUG?]?detect?machine?type [mon-1][DEBUG?]?write?cluster?configuration?to?/etc/ceph/{cluster}.conf [root@mon-1?cluster]#?ceph-deploy?--overwrite-conf?config?push?ceph-1?ceph-2?ceph-3 [ceph_deploy.admin][DEBUG?]?Pushing?admin?keys?and?conf?to?osd-1 [osd-1][DEBUG?]?connected?to?host:?osd-1? [osd-1][DEBUG?]?detect?platform?information?from?remote?host [osd-1][DEBUG?]?detect?machine?type [osd-1][DEBUG?]?write?cluster?configuration?to?/etc/ceph/{cluster}.conf [ceph_deploy.admin][DEBUG?]?Pushing?admin?keys?and?conf?to?osd-2 [osd-2][DEBUG?]?connected?to?host:?osd-2? [osd-2][DEBUG?]?detect?platform?information?from?remote?host [osd-2][DEBUG?]?detect?machine?type [osd-2][DEBUG?]?write?cluster?configuration?to?/etc/ceph/{cluster}.conf |
此時，需要重啟各個節點的monitor服務
mon和osd的啟動方式
| 1 2 3 4 5 6 | #mon-1為各個monitor所在節點的主機名。 systemctl?start?ceph-mon@mon-1.service? systemctl?restart?ceph-mon@mon-1.service systemctl?stop?ceph-mon@mon-1.service #0為該節點的OSD的id，可以通過`ceph?osd?tree`查看 systemctl?start/stop/restart?ceph-osd@0.service |
遇到的報錯?
1.?Monitor clock skew detected
http://www.iyunv.com/thread-157755-1-1.html???
1. Monitor clock skew detected?
| 1 2 3 4 5 6 7 8 | [root@mon-1?cluster]#?ceph?-s ????cluster?f25ad2c5-fd2a-4fcc-a522-344eb498fee5 ?????health?HEALTH_ERR ????????????clock?skew?detected?on?mon.osd-2 ????????????64?pgs?are?stuck?inactive?for?more?than?300?seconds ????????????64?pgs?stuck?inactive ????????????no?osds ????????????Monitor?clock?skew?detected |
添加配置參數： ? ? ? ? ? ? ?
| 1 2 3 | vim?/etc/ceph/ceph.conf mon?clock?drift?allowed?=?2 mon?clock?drift?warn?backoff?=?30 |
?
同步配置文件
| 1 | ceph-deploy?--overwrite-conf?admin?osd-1?osd-2 |
重啟mon服務
| 1 | systemctl?restart?ceph-mon@osd-2.service |
問題總結：
本問題主要是mon節點服務器，時間偏差比較大導致，本次遇到問題為測試環境，通過修改ceph對時間偏差閥值，規避的告警信息，線上業務環境，注意排查服務器時間同步問題。
本文轉自 shouhou2581314 51CTO博客，原文鏈接：http://blog.51cto.com/thedream/1880378，如需轉載請自行聯系原作者
總結
以上是生活随笔為你收集整理的CEPH快速部署(Centos7+Jewel)的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： ubuntu16.04安装pycharm
- 下一篇： 用js实现分页