graphite+grafana监控openstack和ceph

本文环境:CentOS 7.3.1611,openstack+ceph

架构图如下,服务端部分可以分开也可以部署在同一台主机上:

未分类

一、添加epel源和base源,安装carbon和whisper,whisper用于存储监控的数据,类似rrd文件那样固定大小:

yum install python-carbon python-whisper

编辑whisper存储规则,以下是自己写的规则:

vi /etc/carbon/storage-schemas.conf
[carbon]
pattern = ^carbon.
retentions = 60:90d

[vms]
pattern = ^instances.
retentions = 60s:30d,1h:180d,1d:2y,30d:10y

[cluster]
pattern = ^servers..*.CephCollector.ceph.mon..*.cluster.
retentions = 60s:30d,1h:180d,1d:2y,30d:10y

[CephStats]
pattern = ^servers..*.CephStatsCollector.
retentions = 60s:30d,1h:180d,1d:2y,30d:10y

[cpu]
pattern = ^servers..*.cpu.
retentions = 60s:30d,1h:180d,1d:2y,30d:10y

[diskspace]
pattern = ^servers..*.diskspace.
retentions = 60s:30d,1h:180d,1d:2y,30d:10y

[iostat]
pattern = ^servers..*.iostat.
retentions = 60s:30d,1h:180d,1d:2y,30d:10y

[loadavg]
pattern = ^servers..*.loadavg.
retentions = 60s:30d,1h:180d,1d:2y,30d:10y

[memory]
pattern = ^servers..*.memory.
retentions = 60s:30d,1h:180d,1d:2y,30d:10y

[network]
pattern = ^servers..*.network.
retentions = 60s:30d,1h:180d,1d:2y,30d:10y

[default_1min_for_1day]
pattern = .*
retentions = 60s:1d

注:监控字段可以在diamond的日志(/var/log/diamond/archive.log)或者grafana查看。

60s:30d,1h:180d 就是30天内的数据保存间隔为60秒,一个月到半年的保存间隔为1小时,规则是按顺序匹配,不是最长匹配,所以 default 规则一定要放最后,default设定为那么小只是为了不存储不需要的数据,diamond监控的字段太多,不需要全部存储。

carbon配置:

vi /etc/carbon/carbon.conf

[cache]部分,MAX_UPDATES_PER_SECOND(每秒钟最大更新条数)和MAX_CREATES_PER_MINUTE(每分钟最大创建文件数,diamon客户端首次推送数据的时候才会创建whisper数据文件)可能需要调整一下。

启动carbon-cache:

systemctl start carbon-cache

二、客户端安装diamond

(1)每个被监控的客户端都需要安装agent程序

在python官网上有,附链接:https://pypi.python.org/pypi/diamond/4.0.515

解压后 python setup.py install 即安装完成。

(2)配置

cd /etc/diamond
cp diamond.conf.example diamond.conf
vi diamond.conf
# Graphite server host
host = 127.0.0.1 #填上carbon-cache服务器IP,本机的话可以不修改。
[[default]]
hostname = test-node1
# 60秒推送一次数据
interval = 60

(3)配置监控项,diamond可以监控的项目很多,包括ceph、libvirt、mysql、haproxy、nginx等等等等

diamond-setup

基本上一直按回车就可以,由于项目太多,用setup也是繁琐,直接把需要监控的项目放到/etc/diamond/collectors/就可以,具体内容可以参照diamond-setup生成的配置文件。

(4)启动

/etc/init.d/diamond start

至此,监控的服务端和客户端已经部署好,接下来的是部署用于展示的web服务,可以使用 graphite-api + Grafana,也可以 graphite-web,graphite-web是基于django的,需要安装的东西也是一大堆,而且用户体验远远不及Grafana这个专业展示软件。

三、安装 graphite-api

graphite-api的作用只是向grafana提供访问whisper数据的接口。

(1)安装

yum install graphite-api

(2)配置

graphite-api默认启动 127.0.0.1:8888,故需要外部调用或者使用其他服务端口的话,修改/etc/sysconfig/graphite-api的GRAPHITE_API_ADDRESS=0.0.0.0或者GRAPHITE_API_PORT
配置文件 /etc/graphite-api.yaml 使用默认即可。

(3)启动

systemctl start graphite-api

四、安装grafana

grafana是个很流弊的画图软件,支持很多种数据源,展示图形也很多种。

官网下载地址:https://grafana.com/grafana/download

(1)yum本地安装,自动安装依赖包:

yum localinstall grafana-4.4.1-1.x86_64.rpm

(2)安装mysql-server并创建数据库:
由于我的grafana直接装在openstack的控制节点上,已经安装好了mysql-server,直接创建数据库:

CREATE DATABASE grafana ;
GRANT ALL PRIVILEGES ON grafana.* TO 'grafana'@'127.0.0.1' IDENTIFIED BY 'grafana';
FLUSH PRIVILEGES;

mysql在这里只是储存grafana的用户信息,dashboard信息等,不会保存监控的数据。

监控的数据是直接调用graphite-api接口读取whisper数据文件。

(3)配置文件:/etc/grafana/grafana.ini
配置url、mysql连接信息,禁止检查更新、自动发送”使用计数”给grafana,登录界面不使用用户注册功能。

[server]
root_url = http://0.0.0.0:3000
[database]
type = mysql
host = 127.0.0.1:3306
name = grafana
user = grafana
password = grafana
[analytics]
reporting_enabled = false
check_for_updates = false
[security]
[snapshots]
external_enabled = false
[users]
allow_sign_up = false

(4)启动 grafana :

/etc/init.d/grafana-server start

五、配置监控

(1)浏览器登录 grafana,默认初始用户名密码是admin/admin,在grafana.ini可配置。

(2)添加数据源:Data Sources ->Add data sources,输入自定义名称,Type选择【graphite】,url为graphite-api的地址和端口,默认本机8888端口即 【http://localhost:8888】,Access选择【proxy】,其他不填,点击保存并测试连接【Save&Test】,正常返回为绿色背景提示文字

"Success
Data source is working"

(3)至此,全部部署完成,直接在页面上添加dashboard->添加panel->编辑panel即可。

附上监控图两张:

未分类

未分类

openstack集成ceph

集成Ceph相关配置

创建Pool

# ceph osd pool create volumes 64
# ceph osd pool create images 64
# ceph osd pool create vms 64

安装Ceph Client包

配置centos7 ceph yum源

在glance-api(控制节点)节点上

yum install python-rbd -y

(计算节点)在nova-compute和cinder-volume节点上

yum install ceph-common -y

openstack安装Ceph客户端认证

集群ceph存储端操作

[root@ceph ~]# ssh controller sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf

[root@ceph ~]# ssh compute sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf

如果开启了cephx authentication,需要为Nova/Cinder and Glance创建新的用户,如下

ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images'

ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'

为client.cinder, client.glance添加keyring,如下

ceph auth get-or-create client.glance | ssh controller sudo tee /etc/ceph/ceph.client.glance.keyring

ssh controller sudo chown glance:glance /etc/ceph/ceph.client.glance.keyring



ceph auth get-or-create client.cinder | ssh compute sudo tee /etc/ceph/ceph.client.cinder.keyring

ssh compute sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring

为nova-compute节点上创建临时密钥

ceph auth get-key client.cinder | ssh {your-compute-node} tee client.cinder.key

此处为

ceph auth get-key client.cinder | ssh compute tee client.cinder.key

在所有计算节点上(本例就只有一台计算节点)执行如下操作:在计算节点上为libvert替换新的key

uuidgen
536f43c1-d367-45e0-ae64-72d987417c91

cat > secret.xml <<EOF

粘贴以下内容,注意将红色key替换为新生成的key。

<secret ephemeral='no' private='no'>
<uuid>536f43c1-d367-45e0-ae64-72d987417c91</uuid>
<usage type='ceph'>
<name>client.cinder secret</name>
</usage>
</secret>

EOF
virsh secret-define --file secret.xml

以下—base64 后的秘钥为计算节点上/root目录下的client.cinder.key。是之前为计算节点创建的临时秘钥文件

virsh secret-set-value  536f43c1-d367-45e0-ae64-72d987417c91  AQCliYVYCAzsEhAAMSeU34p3XBLVcvc4r46SyA==
[root@compute ~]#rm –f client.cinder.key secret.xml

Openstack配置

在控制节点操作

vim /etc/glance/glance-api.conf
[DEFAULT]

…

default_store = rbd
show_image_direct_url = True
show_multiple_locations = True

…

[glance_store]
stores = rbd
default_store = rbd
rbd_store_pool = images
rbd_store_user = glance
rbd_store_ceph_conf = /etc/ceph/ceph.conf
rbd_store_chunk_size = 8





取消Glance cache管理,去掉cachemanagement

[paste_deploy]
flavor = keystone

在计算节点操作

vim /etc/cinder/cinder.conf



[DEFAULT]

保留之前的

enabled_backends = ceph
#glance_api_version = 2
…

[ceph]

volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2
rbd_user = cinder
volume_backend_name = ceph
rbd_secret_uuid =536f43c1-d367-45e0-ae64-72d987417c91

请注意,每个计算节点uuid不同。按照实际情况填写。本例只有一个计算节点

注意,如果配置多个cinder后端,glance_api_version = 2必须添加到[DEFAULT]中。本例注释了

每个计算节点上,设置/etc/nova/nova.conf

vim /etc/nova/nova.conf

[libvirt]
virt_type = qemu
hw_disk_discard = unmap
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
rbd_secret_uuid = 536f43c1-d367-45e0-ae64-72d987417c91
disk_cachemodes="network=writeback"
libvirt_inject_password = false
libvirt_inject_key = false
libvirt_inject_partition = -2
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_LIVE, VIR_MIGRATE_TUNNELLED

重启OpenStack

控制节点

systemctl restart openstack-glance-api.service

计算节点

systemctl restart openstack-nova-compute.service openstack-cinder-volume.service

配置文件

1、nova

[root@controller nova]# cat nova.conf
[DEFAULT]
enabled_apis = osapi_compute,metadata
rpc_backend = rabbit
auth_strategy = keystone
my_ip = 192.168.8.100
use_neutron = True
firewall_driver = nova.virt.firewall.NoopFirewallDriver
[api_database]
connection = mysql+pymysql://nova:Changeme_123@controller/nova_api
[barbican]
[cache]
[cells]
[cinder]
os_region_name = RegionOne
[conductor]
[cors]
[cors.subdomain]
[database]
connection = mysql+pymysql://nova:Changeme_123@controller/nova
[ephemeral_storage_encryption]
[glance]
api_servers = http://controller:9292
[guestfs]
[hyperv]
[image_file_url]
[ironic]
[keymgr]
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = nova
password = Changeme_123
[libvirt]
[libvirt]
virt_type = qemu
hw_disk_discard = unmap
images_type = rbd
images_rbd_pool = nova
images_rbd_ceph_conf = /etc/cinder/ceph.conf
rbd_user = cinder
rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337
disk_cachemodes="network=writeback"
libvirt_inject_password = false
libvirt_inject_key = false
libvirt_inject_partition = -2
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_LIVE, VIR_MIGRATE_TUNNELLED
[matchmaker_redis]
[metrics]
[neutron]
url = http://controller:9696
auth_url = http://controller:35357
auth_type = password
project_domain_name = default
user_domain_name = default
region_name = RegionOne
project_name = service
username = neutron
password = Changeme_123
service_metadata_proxy = True
metadata_proxy_shared_secret = Changeme_123
[osapi_v21]
[oslo_concurrency]
lock_path = /var/lib/nova/tmp
[oslo_messaging_amqp]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = Changeme_123
[oslo_middleware]
[oslo_policy]
[rdp]
[serial_console]
[spice]
[ssl]
[trusted_computing]
[upgrade_levels]
[vmware]
[vnc]
vncserver_listen = 0.0.0.0
vncserver_proxyclient_address = 192.168.8.100
enabled = True
novncproxy_base_url = http://192.168.8.100:6080/vnc_auto.html
[workarounds]
[xenserver]

cinder

[root@controller nova]# cat /etc/cinder/cinder.conf
[DEFAULT]
rpc_backend = rabbit
auth_strategy = keystone
my_ip = 192.168.8.100
glance_host = controller
enabled_backends = lvm,ceph
glance_api_servers = http://controller:9292
[BACKEND]
[BRCD_FABRIC_EXAMPLE]
[CISCO_FABRIC_EXAMPLE]
[COORDINATION]
[FC-ZONE-MANAGER]
[KEYMGR]
[cors]
[cors.subdomain]
[database]
connection = mysql+pymysql://cinder:Changeme_123@controller/cinder
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = Changeme_123
[matchmaker_redis]
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
[oslo_messaging_amqp]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = Changeme_123
[oslo_middleware]
[oslo_policy]
[oslo_reports]
[oslo_versionedobjects]
[ssl]
[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes
iscsi_protocol = iscsi
iscsi_helper = lioadm
[ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = cinder
rbd_ceph_conf = /etc/cinder/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2
rbd_user = cinder
rbd_secret_uuid =457eb676-33da-42ec-9a8c-9293d545c337
volume_backend_name = ceph

glance

[root@controller nova]# cat /etc/glance/glance-api.conf

[DEFAULT]
#default_store = rbd
show_image_direct_url = True
#show_multiple_locations = True
[cors]
[cors.subdomain]
[database]
connection = mysql+pymysql://glance:Changeme_123@controller/glance
[glance_store]
stores = rbd
default_store = rbd
rbd_store_pool = glance
rbd_store_user = glance
rbd_store_ceph_conf = /etc/glance/ceph.conf
rbd_store_chunk_size = 8
[image_format]
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
username = glance
password = Changeme_123
project_name = service
[matchmaker_redis]
[oslo_concurrency]
[oslo_messaging_amqp]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
[oslo_policy]
[paste_deploy]
flavor = keystone
[profiler]
[store_type_location_strategy]
[task]
[taskflow_executor]

ceph

[root@controller nova]# cat /etc/cinder/cinder.conf
[DEFAULT]
rpc_backend = rabbit
auth_strategy = keystone
my_ip = 192.168.8.100
glance_host = controller
enabled_backends = lvm,ceph
glance_api_servers = http://controller:9292
[BACKEND]
[BRCD_FABRIC_EXAMPLE]
[CISCO_FABRIC_EXAMPLE]
[COORDINATION]
[FC-ZONE-MANAGER]
[KEYMGR]
[cors]
[cors.subdomain]
[database]
connection = mysql+pymysql://cinder:Changeme_123@controller/cinder
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = Changeme_123
[matchmaker_redis]
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
[oslo_messaging_amqp]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = Changeme_123
[oslo_middleware]
[oslo_policy]
[oslo_reports]
[oslo_versionedobjects]
[ssl]
[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes
iscsi_protocol = iscsi
iscsi_helper = lioadm
[ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = cinder
rbd_ceph_conf = /etc/cinder/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2
rbd_user = cinder
rbd_secret_uuid =457eb676-33da-42ec-9a8c-9293d545c337
volume_backend_name = ceph
[root@controller nova]# cat /etc/glance/glance-api.conf
[DEFAULT]
#default_store = rbd
show_image_direct_url = True
#show_multiple_locations = True
[cors]
[cors.subdomain]
[database]
connection = mysql+pymysql://glance:Changeme_123@controller/glance
[glance_store]
stores = rbd
default_store = rbd
rbd_store_pool = glance
rbd_store_user = glance
rbd_store_ceph_conf = /etc/glance/ceph.conf
rbd_store_chunk_size = 8
[image_format]
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
username = glance
password = Changeme_123
project_name = service
[matchmaker_redis]
[oslo_concurrency]
[oslo_messaging_amqp]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
[oslo_policy]
[paste_deploy]
flavor = keystone
[profiler]
[store_type_location_strategy]
[task]
[taskflow_executor]

[root@controller nova]# cat /etc/cinder/ceph.conf
[global]
heartbeat interval = 5
osd pool default size = 3
osd heartbeat grace = 10
#keyring = /etc/ceph/keyring.admin
mon osd down out interval = 90
fsid = 5e8080b0-cc54-11e6-b346-000c29976397
osd heartbeat interval = 10
max open files = 131072
auth supported = cephx

[mon]
mon osd full ratio = .90
mon data = /var/lib/ceph/mon/mon$id
mon osd nearfull ratio = .75
mon clock drift allowed = .200
mon osd allow primary affinity = true

[mon.0]
host = csm-node1
mon addr = 192.168.8.102:6789

[mon.1]
host = csm-node2
mon addr = 192.168.8.103:6789

[mon.2]
host = csm-node3
mon addr = 192.168.8.104:6789

[osd]
osd mount options xfs = rw,noatime,inode64,logbsize=256k,delaylog
osd crush update on start = false
filestore xattr use omap = true
#keyring = /etc/ceph/keyring.$name
osd mkfs type = xfs
osd data = /var/lib/ceph/osd/osd$id
osd heartbeat interval = 10
osd heartbeat grace = 10
osd mkfs options xfs = -f
osd journal size = 0

[osd.0]
osd journal = /dev/sdb1
devs = /dev/sdb2
host = csm-node1
cluster addr = 192.168.8.102
public addr = 192.168.8.102

[osd.1]
osd journal = /dev/sdb1
devs = /dev/sdb2
host = csm-node2
cluster addr = 192.168.8.103
public addr = 192.168.8.103

[osd.2]
osd journal = /dev/sdb1
devs = /dev/sdb2
host = csm-node3
cluster addr = 192.168.8.104
public addr = 192.168.8.104

[client.cinder]
keyring=/etc/ceph/ceph.client.cinder.keyring