分类：未分类

消息队列

某次在某乎上看到有人提到消息队列的问题，然后有人在回答里提到了Redis，接着便有人在评论里指出：Redis是缓存，不是消息队列。

但不幸的是，Redis的确提供一个简易的消息队列机制，可以用于一些要求不那么高的场合。

方法就是利用Redis的列表类型的push和pop操作。

我对前文所介绍的Redis Cache作了一点简单的扩展，增加了消息队列功能。

实现

代码基本就这么点：

class RedisMQ(RedisCache):
    def __init__(self, dbname, host='localhost', port=6379, db=0):
        super(RedisMQ, self).__init__(dbname, host, port, db)

    def push(self, channel, data):
        ch = self._getkey("channel", channel)
        self.db.lpush(ch, self.SERIALIZER.dumps(data))

    def pop(self, channel, timeout=5):
        ch = self._getkey("channel", channel)
        msg = self.db.brpop(ch, timeout)
        return self.SERIALIZER.loads(msg[1]) if msg else None


class Channel(object):
    MQ = RedisMQ("msgqueue")

    def __init__(self, channel):
        self.channel = channel

    def push(self, **kwargs):
        Channel.MQ.push(self.channel, kwargs)

    def pop(self):
        return Channel.MQ.pop(self.channel)

用法

消息生产者

ch = Channel("test")
ch.push(a=123,b="hello")

消息消费者，可能是另一个线程，甚至是另一个进程，甚至是另外一台主机——只要它们共用同一个redis即可。

ch = Channel("test")
while msg=ch.pop():
    # msg: {"a": 123, "b": "hello"}
# ch is empty

1. 准备好4台机器

192.168.42.150 redis-node1 #主
192.168.42.151 redis-node2 #从 
192.168.42.152 redis-node3 #从
192.168.42.153 redis-node4 #从

将主机解析写入hosts文件,分发至每台机器

2. 安装redis,配置好基本配置

(1) 4台机器,分别安装redis

cd /usr/local/src
wget http://192.168.42.26/install_package/down/redis-3.2.3-1.el7.x86_64.rpm
yum install redis-3.2.3-1.el7.x86_64.rpm -y

(2) 4台机分别配置好,配置文件,做好备份

cp /etc/redis.conf{,.back}
vim redis.conf
daemonize yes
bind 192.168.42.150  #改为各个节点的IP

(3) 依照上面设定的从主机,在从主机配置文件中开启从配置(需要配置3台机器)

# slaveof <masterip> <masterport>
slaveof  192.168.42.150 6379

(4) 启动redis-server(4台同时启动)

redis-server /etc/redis.conf

(5) 在主机器上登录redis

[root@redis-node1 ~]# redis-cli -h 192.168.42.150
192.168.42.150:6379> 
192.168.42.150:6379> keys *
1) "magedu"
192.168.42.150:6379> 
192.168.42.150:6379> set ok "verygood!!!"
OK
192.168.42.150:6379> get ok
"verygood!!!"
192.168.42.150:6379>

登录其他3台从服务器

redis-cli -h 192.168.42.151
redis-cli -h 192.168.42.152
redis-cli -h 192.168.42.153

拿153做示例:
[root@redis-node4 src]# redis-cli -h 192.168.42.153
192.168.42.153:6379> keys *
1) "magedu"
192.168.42.153:6379> 
192.168.42.153:6379> get ok
"verygood!!!"
192.168.42.153:6379>

至此为至redis主/从已经实现,在主节点上查看信息

192.168.42.150:6379> INFO Replication
# Replication
role:master
connected_slaves:3
slave0:ip=192.168.42.151,port=6379,state=online,offset=22806,lag=1
slave1:ip=192.168.42.152,port=6379,state=online,offset=22806,lag=1
slave2:ip=192.168.42.153,port=6379,state=online,offset=22806,lag=1
master_repl_offset:22806
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:22805

接下来在四个结点上配置sentinel，实现故障转移。此处实现累似于MariaDB的MHA 在上面我们已经配置好主从复制集群,现在我们需要添加一台机器[192.168.42.154]来做高可用 同样我们需要设置好,IP,主机名,下载安装redis

192.168.42.154 redis-sentinel  #将主机解析追加至其他的4台的hosts文件中,本机也需要一份
cd /usr/local/src
wget http://192.168.42.26/install_package/down/redis-3.2.3-1.el7.x86_64.rpm
yum install redis-3.2.3-1.el7.x86_64.rpm -y

(1)配置sentinel

# sentinel monitor     #法定人数
cp /etc/redis-sentinel.conf{,.back}
vim /etc/redis-sentinel.conf

daemonize yes
sentinel monitor mymaster 192.168.42.150 6379 1
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 18000
sentinel auth-pass mymaster centos.123 #这是认证选项,我们这里的主节点并没有开启认证

(2)启动sentinel

redis-sentinel /etc/redis-sentinel.conf

查看端口是否已经启动

[root@redis-sentinel src]# ss -tnl
State      Recv-Q Send-Q  Local Address:Port                 Peer Address:Port              
LISTEN     0      128                 *:22                              *:*                  
LISTEN     0      100         127.0.0.1:25                              *:*                  
LISTEN     0      511                 *:26379                           *:*                  
LISTEN     0      128                :::22                             :::*                  
LISTEN     0      100               ::1:25                             :::*                  
LISTEN     0      511                :::26379                          :::*

(3)模拟故障：

192.168.42.150:6379
pkill redis
#查看
[root@redis-node1 ~]# ss -tnl
State      Recv-Q Send-Q                             Local Address:Port                                            Peer Address:Port              
LISTEN     0      128                                            *:22                                                         *:*                  
LISTEN     0      100                                    127.0.0.1:25                                                         *:*                  
LISTEN     0      128                                           :::22                                                        :::*                  
LISTEN     0      100                                          ::1:25

(4)查看故障是否转移

登录192.168.42.151:6379 #巧了,恰好151变成主了
192.168.42.151:6379> 
192.168.42.151:6379> INFO Replication
# Replication
role:master
connected_slaves:2
slave0:ip=192.168.42.153,port=6379,state=online,offset=5413,lag=1
slave1:ip=192.168.42.152,port=6379,state=online,offset=5413,lag=2
master_repl_offset:5413
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:5412
192.168.42.151:6379> 

登录192.168.42.152:6379查看,的确是151变成主了
192.168.42.152:6379> INFO Replication
# Replication
role:slave
master_host:192.168.42.151
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:11402
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
192.168.42.152:6379>

(4)我们再把151的进程杀了,再来看一次,可以看到只有一主一从了

192.168.42.152:6379> INFO Replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.42.153,port=6379,state=online,offset=1625,lag=1
master_repl_offset:1768
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:1767
192.168.42.152:6379>

(5)最后我们将杀死的两台redis恢复,再来查看

192.168.42.152:6379> INFO Replication
# Replication
role:master
connected_slaves:3
slave0:ip=192.168.42.153,port=6379,state=online,offset=12763,lag=1
slave1:ip=192.168.42.151,port=6379,state=online,offset=12763,lag=1
slave2:ip=192.168.42.150,port=6379,state=online,offset=12763,lag=0
master_repl_offset:12763
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:12762
192.168.42.152:6379> 

可以看出此时的1主3从又回来了,不过此时主节点,是152

回到150,和151的节点查看效果
192.168.42.150:6379> INFO Replication
# Replication
role:slave
master_host:192.168.42.152
master_port:6379
master_link_status:up
master_last_io_seconds_ago:2
master_sync_in_progress:0
slave_repl_offset:23555
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
192.168.42.150:6379> 



192.168.42.151:6379> INFO Replication
# Replication
role:slave
master_host:192.168.42.152
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:20782
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

接下来我们配置有密码认证的主从和高可用

我们之前在配置sentinel,时还记得 #法定人数吗,判断为失效至少需要2个 Sentinel进程的同意，只要同意Sentinel的数量不达标，自动failover就不会执行

    sentinel monitor #法定人数

我们现在将154这台干脆也做成redis的从服务器,而sentinel,是这5台的集合(150,151,152,153,154)

(1).将153的配置文件推到154一份

scp /etc/redis.conf{,.back} [email protected]:/etc/
 到154这边稍微修改一下
 vim /etc/redis.conf
 bind 192.168.42.154

因此这次我们是要做密码认证的,因此5台机器都需要加上密码

(2).我们现在查看之前做了 sentinel后,配置文件系统自动帮我们改了,现在我们要恢复到初始状态,重新来过

所有的配置文件还需要设置我们的密码(为了方便管理,我们这里统一设置成magedu) requirepass magedu

所有从节点加上

#masterauth 
masterauth magedu
slaveof 192.168.42.150 6379

(3).密码设置好之后,启动所有服务

redis-server /etc/redis.conf
ss -tnl

登录150,需要认证了

[root@redis-node1 ~]# redis-cli -h 192.168.42.150
192.168.42.150:6379> keys *
(error) NOAUTH Authentication required.

[root@redis-node1 ~]# redis-cli -h 192.168.42.150
192.168.42.150:6379> keys *
(error) NOAUTH Authentication required.
192.168.42.150:6379> keys *
(error) NOAUTH Authentication required.
192.168.42.150:6379> AUTH magedu
OK

同样的,我们需要登录认证其他机器

再150机器上设置key
192.168.42.150:6379> set zlyc "zai lai yi ci"
OK
192.168.42.150:6379> get zlyc
"zai lai yi ci"
192.168.42.150:6379> 

其他机器读取OK
192.168.42.151:6379> get zlyc
"zai lai yi ci"
192.168.42.151:6379>