最近需要在我们的一个Kubernetes集群上部署Redis，因此重新整理，写一下如何在Kubernetes上使用Sateful Set部署Redis。
一、需求和环境
我们的需求是需要部署三节点的Redis主从复制，并部署三个节点的Redis Sentinel实现Redis的高可用。
环境信息如下：
- Kubernetes 1.6.7集群
- Ceph 11.2.0集群
Kubernetes的官方examples中已经给出了一个在k8s集群上部署Redis的例子Reliable, Scalable Redis on Kubernetes， 就是基于Redis主从复制+Sentinel实现的，但是这个例子是以无状态服务形式部署的，如果整个k8s集群重启了，Redis的状态就会丢失，因此不能用于生产环境。 但我们可以参考这个例子，以Satefult Set的形式部署。
我们的线上环境主要使用Ceph的块存储RBD作为Kubernetes的存储卷，这里可以将Redis服务的状态保存在Ceph RBD中。
二、Storage Classes和Dynamic Storage Provision
Kubernetes 1.6开始Storage Classes和Dynamic Storage Provision已经是稳定可用的了。 StorageClass是Dynamic Storage Provision的基础，k8s的管理员可以定义底层存储平台抽象。 用户通过在PVC(Persistent Volume Claim)中通过名字引用StorageClass，PV(Persistent Volume)将使用StorageClass来动态创建，这样就节省了集群管理员手动创建PV的时间。
1、在Ceph中创建存储池Pool
我们需要先在Ceph中创建一个k8s集群专用的Ceph Pool，在创建之前我们先看一下当前Ceph集群中的存储池：
ceph osd lspools
0 rbd,1 .rgw.root,2 default.rgw.control,3 default.rgw.data.root,4 default.rgw.gc,5 default.rgw.lc,6 default.rgw.log,7 default.rgw.users.uid,8 default.rgw.users.email,9 default.rgw.users.keys,10 default.rgw.buckets.index,11 default.rgw.buckets.data,
一个Ceph集群可以有多个pool，pool是逻辑上的存储池。不同的pool可以有不一样的数据处理方式，例如replica size, placement groups, crush rules,snapshot等等。 可以看到因为我们这个环境还是用Ceph的RGW作为我们的对象存储，因此除了默认的名称为rbd的pool外，还有很多rgw的pool。
下面创建一个专门给k8s集群专用的pool kube:
ceph osd pool create kube 128
pool 'kube' created
ceph osd lspools
0 rbd,1 .rgw.root,2 default.rgw.control,3 default.rgw.data.root,4 default.rgw.gc,5 default.rgw.lc,6 default.rgw.log,7 default.rgw.users.uid,8 default.rgw.users.email,9 default.rgw.users.keys,10 default.rgw.buckets.index,11 default.rgw.buckets.data,12 kube,
- 当前这个ceph集群只有3个osd，所以设置pg_num为128，可参考PLACEMENT GROUPS
2、配置k8s Node节点访问Ceph
为了让Kubernetes的Node可以调用rbd，如果Ceph集群和Kubernetes集群不是在相同的机器上，还需要在Kubernetes的Node上安装ceph-common:
yum install -y ceph-common
接下来在Kubernetes上创建ceph-secret，这个Secret将用于Kubernetes集群的StorageClass上。
我们先查看一下ceph集群上的所有用户列表：
ceph auth list
这个命令会列出针对Ceph的每种类型的进程已经创建的不同权限的用户，同时也会列出client.admin用户，这个是Ceph集群的管理员用户。
接下来我们创建一个client.kube用户：
ceph auth get-or-create client.kube
[client.kube]
key = AQAzcYVZ6sbJLhAA7qCBywM+iPRgAG97FtoXIw==
创建好的client.kube用户用户还没有任何权限，下面给其授权：
ceph auth caps client.kube mon 'allow r' osd 'allow rwx pool=kube'
updated caps for client.kube
查看用户和权限信息：
ceph auth get client.kube
exported keyring for client.kube
[client.kube]
key = AQAzcYVZ6sbJLhAA7qCBywM+iPRgAG97FtoXIw==
caps mon = "allow r"
caps osd = "allow rwx pool=kube"
因为Kubernetes的Secret需要Base64编码，下面将这个keyring转换成Base64编码：
ceph auth get-key client.kube | base64
QVFBemNZVlo2c2JKTGhBQTdxQ0J5d00raVBSZ0FHOTdGdG9YSXc9PQ==
接下来创建Secret，ceph-secret.yaml:
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret
namespace: kube-system
type: kubernetes.io/rbd
data:
key: QVFBemNZVlo2c2JKTGhBQTdxQ0J5d00raVBSZ0FHOTdGdG9YSXc9PQ==
kubectl create -f ceph-secret.yaml
secret "ceph-secret" created
3、在k8s集群创建StorageClass
首先检查我们的集群中是否有默认的StorageClass：
kubectl get storageclass
No resources found.
我们这里使用的k8s集群是使用ansible部署的Kubernetes 1.6 高可用集群，可以看出我们部署的这个集群并没有创建默认的StorageClass。
我们现在集群中创建默认的Storage Class， storege.yaml文件如下：
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: default
annotations:
storageclass.kubernetes.io/is-default-class: "true"
labels:
kubernetes.io/cluster-service: "true"
provisioner: kubernetes.io/rbd
parameters:
monitors: 192.168.61.3:6789,192.168.61.4:6789,192.168.61.5:6789
adminId: kube
adminSecretName: ceph-secret
adminSecretNamespace: kube-system
pool: kube
userId: kube
userSecretName: ceph-secret-user
- annotations中storageclass.kubernetes.io/is-default-class: “true”表示这个StorageClass是集群默认的StorageClass
-
provisioner: kubernetes.io/rbd表示这个StorageClass的类型时Ceph RBD
-
parameters配置了这个StorageClass使用的Ceph集群以及RBD的相关参数
-
monitors是逗号分隔的Ceph Mon节点地址
-
adminId指定Ceph client 的ID需要具有能在配置的Ceph RBD Pool中创建镜像的权限。默认值为admin
-
adminSecret：adminId的Secret Name,该Secret的type必须是”kubernetes.io/rbd”，该参数是必须的
-
adminSecretNamespace： adminSecret的namespace，默认为”default”
-
pool： Ceph RBD Pool，默认为”rbd”
-
userId: Ceph client Id，用来映射RBD镜像。
-
userSecretName: userId在映射RBD镜像时所需要的Secret的名称。该Secret要求必须出现在和PVC相同的namespace内，并且type必须是”kubernetes.io/rbd”。该参数是必须的
创建这个默认的StorageClass：
kubectl create -f storage.yaml
storageclass "default" created
kubectl get storageclass
NAME TYPE
default (default) kubernetes.io/rbd
- (default)表示这个名称为default的StorageClass是k8s集群默认的StorageClass
三、构建Redis的Docker镜像
参考 Reliable, Scalable Redis on Kubernetes （https://github.com/kubernetes/kubernetes/tree/master/examples/storage/redis/image） 中的Redis镜像，我们的Redis的Dockerfile定制如下：
FROM harbor.frognew.com/rg/alpine-glibc:0.1
RUN apk add --no-cache redis sed bash
COPY redis-master.conf /redis-master/redis.conf
COPY redis-slave.conf /redis-slave/redis.conf
COPY run.sh /run.sh
RUN chmod u+x /run.sh
CMD [ "/run.sh" ]
ENTRYPOINT [ "bash", "-c" ]
- alpine-glibc:0.1是我们的基础镜像，在alpine:3.6的基础上增加了glibc，并将时区设置为Asia/Shanghai
参考 Reliable, Scalable Redis on Kubernetes (https://github.com/kubernetes/kubernetes/tree/master/examples/storage/redis/image) 中的run.sh做如下定制，原来的run.sh不支持对redis设置密码，加上从环境变量$REDIS_PASS读取redis密码：
#!/bin/bash
# Copyright 2014 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
function launchmaster() {
if [[ ! -e /redis-master-data ]]; then
echo "Redis master data doesn't exist, data won't be persistent!"
mkdir /redis-master-data
fi
sed -i "s/%redis-pass%/${REDIS_PASS}/" /redis-master/redis.conf
redis-server /redis-master/redis.conf --protected-mode no
}
function launchsentinel() {
while true; do
master=$(redis-cli -a $REDIS_PASS -h ${REDIS_SENTINEL_SERVICE_HOST} -p ${REDIS_SENTINEL_SERVICE_PORT} --csv SENTINEL get-master-addr-by-name mymaster | tr ',' ' ' | cut -d' ' -f1)
if [[ -n ${master} ]]; then
master="${master//"}"
else
master=${REDIS_MASTER_SERVICE_HOST}
fi
redis-cli -a $REDIS_PASS -h ${master} INFO
if [[ "$?" == "0" ]]; then
break
fi
echo "Connecting to master failed. Waiting..."
sleep 10
done
sentinel_conf=sentinel.conf
echo "sentinel monitor mymaster ${master} 6379 2" > ${sentinel_conf}
echo "sentinel auth-pass mymaster ${REDIS_PASS}" >> ${sentinel_conf}
echo "sentinel down-after-milliseconds mymaster 60000" >> ${sentinel_conf}
echo "sentinel failover-timeout mymaster 180000" >> ${sentinel_conf}
echo "sentinel parallel-syncs mymaster 1" >> ${sentinel_conf}
echo "bind 0.0.0.0" >> ${sentinel_conf}
redis-sentinel ${sentinel_conf} --protected-mode no
}
function launchslave() {
while true; do
master=$(redis-cli -a $REDIS_PASS -h ${REDIS_SENTINEL_SERVICE_HOST} -p ${REDIS_SENTINEL_SERVICE_PORT} --csv SENTINEL get-master-addr-by-name mymaster | tr ',' ' ' | cut -d' ' -f1)
if [[ -n ${master} ]]; then
master="${master//"}"
else
echo "Failed to find master."
sleep 60
exit 1
fi
redis-cli -a $REDIS_PASS -h ${master} INFO
if [[ "$?" == "0" ]]; then
break
fi
echo "Connecting to master failed. Waiting..."
sleep 10
done
sed -i "s/%master-ip%/${master}/" /redis-slave/redis.conf
sed -i "s/%master-port%/6379/" /redis-slave/redis.conf
sed -i "s/%redis-pass%/${REDIS_PASS}/" /redis-slave/redis.conf
redis-server /redis-slave/redis.conf --protected-mode no
}
if [[ "${MASTER}" == "true" ]]; then
launchmaster
exit 0
fi
if [[ "${SENTINEL}" == "true" ]]; then
launchsentinel
exit 0
fi
launchslave
redis-master.conf的配置文件内容如下：
daemonize no
pidfile /var/run/redis.pid
port 6379
tcp-backlog 511
bind 0.0.0.0
timeout 0
tcp-keepalive 60
loglevel notice
logfile ""
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /redis-master-data
slave-serve-stale-data yes
rename-command FLUSHALL ""
rename-command FLUSHDB ""
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
requirepass %redis-pass%
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
redis-slave.conf配置文件内容如下：
daemonize no
pidfile /var/run/redis.pid
port 6379
tcp-backlog 511
bind 0.0.0.0
timeout 0
tcp-keepalive 60
loglevel notice
logfile ""
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir "/data"
slaveof %master-ip% %master-port%
masterauth %redis-pass%
slave-serve-stale-data yes
rename-command FLUSHALL ""
rename-command FLUSHDB ""
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
requirepass %redis-pass%
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
构建redis镜像并推送到我们的私有仓库：
docker build -t harbor.frognew.com/rg/redis:1.0 .
docker push harbor.frognew.com/rg/rg/redis
四、在Kubernetes集群上部署Redis
假设我们的redis要部署在devops这个namespace下，先在这个namespace下创建ceph-secret-user这个Secret：
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret-user
namespace: devops
type: kubernetes.io/rbd
data:
key: QVFBemNZVlo2c2JKTGhBQTdxQ0J5d00raVBSZ0FHOTdGdG9YSXc9PQ==
kubectl crate -f ceph-secret-user.yaml
1、redis-master.statefulset.yaml
redis-master.statefulset.yaml是redis master的Service和StatefulSet。
apiVersion: v1
kind: Service
metadata:
name: redis-master
namespace: devops
labels:
name: redis-master
spec:
ports:
- port: 6379
selector:
redis-master: "true"
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: redis-master
namespace: devops
labels:
name: redis-master
spec:
serviceName: redis-master
replicas: 1
template:
metadata:
labels:
app: redis-master
redis-master: "true"
spec:
terminationGracePeriodSeconds: 10
containers:
- name: redis
image: harbor.frognew.com/rg/redis:1.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 6379
env:
- name: MASTER
value: "true"
- name: REDIS_PASS
valueFrom:
secretKeyRef:
name: devopssecret
key: redisAuthPass
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
volumeMounts:
- name: redis-master-volume
mountPath: /redis-master-data
imagePullSecrets:
- name: regsecret
volumeClaimTemplates:
- metadata:
name: redis-master-volume
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 5Gi
2、redis-sentinel.statefulset.yaml
redis-sentinel.statefulset.yaml定义了redis-sentinel的Service和StatefulSet：
apiVersion: v1
kind: Service
metadata:
name: redis-sentinel
namespace: devops
labels:
name: redis-sentinel
spec:
ports:
- port: 26379
targetPort: 26379
selector:
redis-sentinel: "true"
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: redis-sentinel
namespace: devops
spec:
serviceName: redis-sentinel
replicas: 3
template:
metadata:
labels:
redis-sentinel: "true"
spec:
terminationGracePeriodSeconds: 10
containers:
- name: redis-sentinel
image: harbor.frognew.com/rg/redis:1.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 26379
name: redis-sentinel
env:
- name: SENTINEL
value: "true"
- name: REDIS_PASS
valueFrom:
secretKeyRef:
name: devopssecret
key: redisAuthPass
imagePullSecrets:
- name: regsecret
- sentinel的启动逻辑可以查看Docker镜像中的run.sh中launchsentinel()的逻辑
5、redis.statefulset.yaml
redis.statefulset.yaml定义了redis slave的Service和SatefulSet：
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: devops
labels:
app: redis
spec:
ports:
- port: 6379
clusterIP: None
selector:
app: redis
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: redis
namespace: devops
labels:
name: redis
spec:
serviceName: redis
replicas: 2
template:
metadata:
labels:
app: redis
spec:
terminationGracePeriodSeconds: 10
containers:
- name: redis
image: harbor.frognew.com/rg/redis:1.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 6379
env:
- name: REDIS_PASS
valueFrom:
secretKeyRef:
name: devopssecret
key: redisAuthPass
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
volumeMounts:
- name: redis-volume
mountPath: /data
imagePullSecrets:
- name: regsecret
volumeClaimTemplates:
- metadata:
name: redis-volume
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 5Gi
volumeClaimTemplates中定义了PVC，因为没有给定storageClassName，所以将使用我们前面创建的默认的StorageClass，会根据PVC动态创建StatefulSet中Pod所需的PV。
4、以StatefulSet的形式部署Redis
下面实际操作一遍基于StatefulSet的Redis的部署。
先创建redis-master的Service和StatefulSet：
kubectl create -f redis-master.statefulset.yaml
service "redis-master" created
statefulset "redis-master" created
确保这redis master Pod处于running状态：
kubectl get pods -l redis-master="true" -n devops
NAME READY STATUS RESTARTS AGE
redis-master-0 1/1 Running 0 48s
kubectl get svc -l name="redis-master" -n devops
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis-master 10.104.132.220 <none> 6379/TCP 1m
下面创建redis-sentinel的Service和StatefulSet：
kubectl create -f redis-sentinel.statefulset.yaml
service "redis-sentinel" created
statefulset "redis-sentinel" created
kubectl get svc -l name="redis-sentinel" -n devops -o wide
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
redis-sentinel 10.97.4.9 <none> 26379/TCP 16s redis-sentinel=true
查看StatefulSet确保DESIRED和CURRENT的数量是相同的。
kubectl get statefulset -n devops
NAME DESIRED CURRENT AGE
redis-master 1 1 3m
redis-sentinel 3 3 42s
查看sentinel Pod：
kubectl get pod -l redis-sentinel="true" -n devops
NAME READY STATUS RESTARTS AGE
redis-sentinel-0 1/1 Running 0 1m
redis-sentinel-1 1/1 Running 0 1m
redis-sentinel-2 1/1 Running 0 1m
下面创建redis slave的Service和StatefulSet：
kubectl create -f redis.statefulset.yaml
service "redis" created
statefulset "redis" created
注意上面的过程中，在创建redis master和slave的stateful set时可能需要一定的时间，因为涉及到PVC, PV, rbd image的创建，耐心等待。
因为redis-master这个StatefulSet的副本数为1，redis slave这个SatefulSet中的副本数为2，所以我们可以看到集群中创建了3个PVC，并创建了3个PV:
kubectl get pvc -n devops
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
redis-master-volume-redis-master-0 Bound pvc-fd2c30e3-7b14-11e7-ad4a-1866da8c6175 5Gi RWO default 10m
redis-volume-redis-0 Bound pvc-65a2ba2c-7b17-11e7-ad4a-1866da8c6175 5Gi RWO default 7m
redis-volume-redis-1 Bound pvc-6a96951c-7b17-11e7-ad4a-1866da8c6175 5Gi RWO default 7m
kubectl get pv -n devops
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-65a2ba2c-7b17-11e7-ad4a-1866da8c6175 5Gi RWO Delete Bound devops/redis-volume-redis-0 default 10m
pvc-6a96951c-7b17-11e7-ad4a-1866da8c6175 5Gi RWO Delete Bound devops/redis-volume-redis-1 default 7m
pvc-fd2c30e3-7b14-11e7-ad4a-1866da8c6175 5Gi RWO Delete Bound devops/redis-master-volume-redis-master-0 default 7m
一定要确认STATUS的状态为Bound，如果不是可以通过kubectl describe pvc -n 查看具体的事件。实际上rbd image的创建是由controller-manager调用rbd命令完成的，所以如果有问题也可以看一下controller-manager的日志。
下面的decribe pv命令详细打印出了这个PV已经使用Ceph RBD Image:
kubectl describe pv pvc-65a2ba2c-7b17-11e7-ad4a-1866da8c6175 -n devops
Name: pvc-65a2ba2c-7b17-11e7-ad4a-1866da8c6175
Labels: <none>
Annotations: pv.kubernetes.io/bound-by-controller=yes
pv.kubernetes.io/provisioned-by=kubernetes.io/rbd
StorageClass: default
Status: Bound
Claim: devops/redis-volume-redis-0
Reclaim Policy: Delete
Access Modes: RWO
Capacity: 5Gi
Message:
Source:
Type: RBD (a Rados Block Device mount on the host that shares a pod's lifetime)
CephMonitors: [192.168.61.3:6789 192.168.61.4:6879 192.168.61.5:6789]
RBDImage: kubernetes-dynamic-pvc-856cff45-7a7b-11e7-ac3c-1866da8c2fcd
FSType:
RBDPool: kube
RadosUser: kube
Keyring: /etc/ceph/keyring
SecretRef: &{ceph-secret-user}
ReadOnly: false
Events: <none>
另外可以在Ceph集群中查看创建的rbd image：
rbd list kube
kubernetes-dynamic-pvc-856cff45-7a7b-11e7-ac3c-1866da8c2fcd
kubernetes-dynamic-pvc-8b1be6fc-7a7b-11e7-ac3c-1866da8c2fcd
kubernetes-dynamic-pvc-cce6429c-7a7a-11e7-ac3c-1866da8c2fcd
rbd info -p kube --image kubernetes-dynamic-pvc-856cff45-7a7b-11e7-ac3c-1866da8c2fcd
rbd image 'kubernetes-dynamic-pvc-856cff45-7a7b-11e7-ac3c-1866da8c2fcd':
size 5120 MB in 1280 objects
order 22 (4096 kB objects)
block_name_prefix: rb.0.3e17b.238e1f29
format: 1
kubectl get statefulset -n devops
NAME DESIRED CURRENT AGE
redis 2 2 5m
redis-master 1 1 10m
redis-sentinel 3 3 7m
我们重点来看一下redis statefulset和redis service：
kubectl get svc -l app="redis" -n devops
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis None <none> 6379/TCP 7m
kubectl get pod -l app="redis" -n devops
NAME READY STATUS RESTARTS AGE
redis-0 1/1 Running 0 7m
redis-1 1/1 Running 0 7m
注意redis service的CLUSTER-IP为None，这是由有状态服务的特征决定的。 有状态服务具有以下特征：
- 要求有稳定的网络身份，即唯一不变的hostname，并保存在DNS中。hostname是由statefulset的名字后边跟随”-序号”组成，这里是redis-1, redis-2。 同时每个Pod的网络身份也是通过Service定义被创建出来了，根据Service的定义，通过ClusterIp:None指定，该Service将在DNS生成一条没有ClusterIP的记录。
-
要求有持久稳定的存储，通过PVC和PV提供。这里使用了Kubernetes的通过Dynamic Storage Provision特性，PV使用StorageClass来动态创建。
-
redis-0,redis-1这两个是reddis的slave节点。
最后我们来看一下k8s集群中redis节点：
kubectl get statefulset -n devops
NAME DESIRED CURRENT AGE
redis 2 2 8m
redis-master 1 1 12m
redis-sentinel 3 3 10m
我们以StatefulSet的形式部署了1个master, 2个slave, 3个sentinel。当其中master节点发生故障时，sentinel会从剩余redis节点中选举新的master并切换。 3个redis节点的数据都是保存在ceph rbd中。
