redis实现sentinel哨兵架构的方法

Redis/缓存系统
235
0
0
2023-06-16
标签   Redis
目录
  • 1、redis哨兵(Sentinel)
  • 1.1、redis集群介绍
  • 1.2、redis哨兵(Sentinel)的工作原理
  • 1.2.1sentinel中的三个定时任务:
  • 1.3、实现哨兵
  • 1.3.1、实现哨兵需要先实现一下主从复制的架构
  • 1.3.2、编辑哨兵的配置文件
  • 1.3.2、启动哨兵
  • 1.3.3、验证哨兵端口
  • 1.3.4、查看哨兵日志
  • 1.3.5、当前sentinel状态
  • 1.3.6、验证数据是否同步
  • 1.3.7、制造Redis Master节点故障故障转移并查看其过程
  • 1.3.8、查看选举新主的过程日志监测
  • 1.3.9、恢复故障的原master重新加入redis集群

1、redis哨兵(Sentinel)

1.1、redis集群介绍

前面文章讲的主从复制集群是无法实现master和slave角色的自动切换的,如果master节点出现现redis服务异常、主机断电、磁盘损坏等问题导致master无法使用,而redis主从复制无法实现自动的故障转移(将slave 自动提升为新master),需要手动修改环境配置,才能切换到slave redis服务器,另外当单台Redis服务器性能无法满足业务写入需求的时候,也无法横向扩展Redis服务的并行写入性能。

需要解决以上的两个核心问题:

  • master和slave角色的无缝切换,让业务无感知从而不影响业务使用;
  • 可横向动态扩展Redis服务器,从而实现多台服务器并行写入以实现更高并发的目的。

Redis集群实现的方式:

  • 客户端分片: 由应用决定将不同的KEY发送到不同的Redis服务器
  • 代理分片: 由代理决定将不同的KEY发送到不同的Redis服务器,代理程序如:codis,twemproxy等
  • Redis Cluster

1.2、redis哨兵(Sentinel)的工作原理

Sentinel可以管理多个redis主从集群

Sentinel 进程是用于监控redis集群中Master主服务器工作的状态,在Master主服务器发生故障的时候,可以实现Master和Slave服务器的切换,保证系统的高可用,此功能在redis2.6+的版本已引用,Redis的哨兵模式到了2.8版本之后就稳定了下来。一般在生产环境也建议使用Redis的2.8版本的以后版本

哨兵(Sentinel) 是一个分布式系统,可以在一个架构中运行多个哨兵(sentinel) 进程,这些进程使用流言协议(gossip protocols)来接收关于Master主服务器是否下线的信息,并使用投票协议(Agreement Protocols)来决定是否执行自动故障迁移,以及选择哪个Slave作为新的Master

每个哨兵(Sentinel)进程会向其它哨兵(Sentinel)、Master、Slave定时发送消息,以确认对方是否”活”着,如果发现对方在指定配置时间(此项可配置)内未得到回应,则暂时认为对方已离线,也就是所谓的”主观认为宕机” (主观:是每个成员都具有的独自的而且可能相同也可能不同的意识),英文名称:Subjective Down,简称SDOWN

有主观宕机,对应的有客观宕机。当“哨兵群”中的多数Sentinel进程在对Master主服务器做出SDOWN 的判断,并且通过 SENTINEL is-master-down-by-addr 命令互相交流之后,得出的Master Server下线判断,这种方式就是“客观宕机”(客观:是不依赖于某种意识而已经实际存在的一切事物),英文名称是:Objectively Down, 简称 ODOWN

通过一定的vote算法,从剩下的slave从服务器节点中,选一台提升为Master服务器节点,然后自动修改相关配置,并开启故障转移(failover)

Sentinel 机制可以解决master和slave角色的自动切换问题,但单个 Master 的性能瓶颈问题无法解决,类似于MySQL中的MHA功能

Redis Sentinel中的Sentinel节点个数应该为大于等于3且最好为奇数

客户端初始化时连接的是Sentinel节点集合,不再是具体的Redis节点,但Sentinel只是配置中心不是代理。

Redis Sentinel节点与普通redis没有区别,要实现读写分离依赖于客户端程序

redis 3.0之前版本中,生产环境一般使用哨兵模式,3.0后推出redis cluster功能,可以支持更大规模的生产环境sentinel中的三个定时任务:

1.2.1sentinel中的三个定时任务:

  • 每10秒每个sentinel对master和slave执行info,发现slave节点,确认主从关系。
  • 每2秒每个sentinel通过master节点的channel交换信息(pub/sub)通过sentinel__:hello频道交互,交互对节点的“看法”和自身信息
  • 每1秒每个sentinel对其他sentinel和redis执行ping

1.3、实现哨兵

环境准备:

准备三台主机搭建主从集群,再在每个机器上搭建sentinel

IP

redis版本

主机名

10.0.0.101

redis-5.0.14

node1

10.0.0.102

redis-5.0.14

node2

10.0.0.103

redis-5.0.14

node3

1.3.1、实现哨兵需要先实现一下主从复制的架构

哨兵的前提是已经实现了一个redis的主从复制的运行环境,从而实现一个一主两从基于哨兵的高可用redis架构

注意: master的配置文件中masterauth和slave都必须相同

主从复制上篇文章已经实现了,这里继续用上一篇的环境

#主节点
[root@master etc]#redis-cli -a
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe..0.0.1:6379> info replication
# Replication
role:master
connected_slaves:
slave:ip=10.0.0.102,port=6379,state=online,offset=14,lag=1
slave:ip=10.0.0.103,port=6379,state=online,offset=14,lag=0
master_replid:efdb8dd9f91d265d4c9a8621a6879e0262ff
master_replid:0000000000000000000000000000000000000000
master_repl_offset:
second_repl_offset:-
repl_backlog_active:
repl_backlog_size:
repl_backlog_first_byte_offset:
repl_backlog_histlen:
.0.0.1:6379>
#slave节点
[root@slave ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe..0.0.1:6379> info replication
# Replication
role:slave
master_host:.0.0.101
master_port:
master_link_status:up
master_last_io_seconds_ago:
master_sync_in_progress:
slave_repl_offset:
slave_priority:
slave_read_only:
connected_slaves:
master_replid:efdb8dd9f91d265d4c9a8621a6879e0262ff
master_replid:0000000000000000000000000000000000000000
master_repl_offset:
second_repl_offset:-
repl_backlog_active:
repl_backlog_size:
repl_backlog_first_byte_offset:
repl_backlog_histlen:
#slave节点
[root@slave ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe..0.0.1:6379> info replication
# Replication
role:slave
master_host:.0.0.101
master_port:
master_link_status:up
master_last_io_seconds_ago:
master_sync_in_progress:
slave_repl_offset:
slave_priority:
slave_read_only:
connected_slaves:
master_replid:eed24063f9d79d698e875634517923d6a9c2a10
master_replid:0000000000000000000000000000000000000000
master_repl_offset:
second_repl_offset:-
repl_backlog_active:
repl_backlog_size:
repl_backlog_first_byte_offset:
repl_backlog_histlen:

1.3.2、编辑哨兵的配置文件

Sentinel实际上是一个特殊的redis服务器,有些redis指令支持,但很多指令并不支持.默认监听在26379/tcp端口.所有redis节点使用相同的以下的配置文件.

#这里因为我是源码编译安装的,所以要人源码的路径下找到sentinel.conf将其复制到redis安装的目录下
[root@master]# cd /usr/local/src/redis-.0.14[root@master /usr/local/src/redis-5.0.14# ls
-RELEASENOTES  CONTRIBUTING  INSTALL    README.md   runtest-cluster    sentinel.conf  TLS.md
BUGS             COPYING       Makefile   redis.conf  runtest-moduleapi  src            utils
CONDUCT          deps          MANIFESTO  runtest     runtest-sentinel   tests
 [root@master etc]#sed -i "/^#/d" redis-sentinel.conf
 [root@master etc]#sed -i "/^$/d" redis-sentinel.conf
 [root@master etc]#vim redis-sentinel.con

bind.0.0.0
port
daemonize no
pidfile /apps/redis/run/redis-sentinel.pid
logfile /apps/redis/log/sentinel.log
dir /tmp
sentinel monitor mymaster.0.0.100 6379 2    #mymaster是集群的名称,此行指定当前mymaster集群中master服务器的地址和端口
#为法定人数限制(quorum),即有几个sentinel认为master down了就进行故障转移,一般此值是所有sentinel节点(一般总数是>=3的 奇数,如:3,5,7等)的一半以上的整数值,比如,总数是3,即3/2=1.5,取整为2,是master的ODOWN客观下线的依据
sentinel auth-pass mymaster wm    #mymaster集群中master的密码,注意此行要在上面行的下面
sentinel down-after-milliseconds mymaster    #(SDOWN)判断mymaster集群中所有节点的主观下线的时间,单位:毫秒,建议3000
sentinel parallel-syncs mymaster    #发生故障转移后,可以同时向新master同步数据的slave的数量,数字越小总同步时间越长,但可以减轻新master的负载压力
sentinel failover-timeout mymaster    #所有slaves指向新的master所需的超时时间,单位:毫秒
sentinel deny-scripts-reconfig yes    #禁止修改脚本

[root@master etc]# scp /apps/redis/etc/redis-sentinel.conf.0.0.102:/apps/redis/etc/[root@master etc]# scp /apps/redis/etc/redis-sentinel.conf 10.0.0.103:/apps/redis/etc/

三个哨兵服务器的配置都如下:

[root@slave etc]#cat redis-sentinel.conf
port
daemonize no
pidfile /apps/redis/run/redis-sentinel.pid
logfile /apps/redis/log/sentinel.log
dir /tmp
sentinel monitor mymaster.0.0.101 6379 2
sentinel down-after-milliseconds mymaster
sentinel parallel-syncs mymaster
sentinel failover-timeout mymaster
sentinel deny-scripts-reconfig yes

1.3.2、启动哨兵

#我这里是编译安装,所以在所有节点生成新的service文件
[root@master ~]#vim /lib/systemd/system/redis-sentinel.service
[Unit]
Description=Redis Sentinel
After=network.target

[Service]
ExecStart=/apps/redis/bin/redis-sentinel /apps/redis/etc/redis-sentinel.conf --supervised systemd
ExecStop=/bin/kill -s QUIT $MAINPID
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=

[Install]
WantedBy=multi-user.target
[root@master ~]#scp /lib/systemd/system/redis-sentinel.service.0.0.102:/lib/systemd/system/
[root@master ~]#scp /lib/systemd/system/redis-sentinel.service.0.0.103:/lib/systemd/system/
#在所有节点修改一下文件的所有者所属组,不然无法启动
chown -R redis.redis /apps/redis/
#在所有节点启动哨兵
  [root@master ~]#systemctl daemon-reload
  [root@master ~]#systemctl enable --now redis-sentinel.service

1.3.3、验证哨兵端口

[root@master ~]#ss -nlt
State     Recv-Q     Send-Q         Local Address:Port          Peer Address:Port    Process
LISTEN          511                  0.0.0.0:26379              0.0.0.0:*                 #已经启动,另外两个节点同样查看
LISTEN          511                  0.0.0.0:6379               0.0.0.0:*
LISTEN          1024           127.0.0.53%lo:53                 0.0.0.0:*
LISTEN          128                  0.0.0.0:22                 0.0.0.0:*
LISTEN          128                127.0.0.1:6010               0.0.0.0:*
LISTEN          511                     [::]:26379                 [::]:*
LISTEN          128                     [::]:22                    [::]:*
LISTEN          128                    [::1]:6010                  [::]:*

1.3.4、查看哨兵日志

[root@master ~]#tail -f /apps/redis/log/sentinel.log #查看主节点日志:X 04 Nov 2022 23:32:07.023 # Configuration loaded
:X 04 Nov 2022 23:32:07.023 # systemd supervision requested, but NOTIFY_SOCKET not found
:X 04 Nov 2022 23:32:07.024 * Increased maximum number of open files to 10032 (it was originally set to 1024).
:X 04 Nov 2022 23:32:07.024 * Running mode=sentinel, port=26379.
:X 04 Nov 2022 23:32:07.032 # Sentinel ID is 2f9b5e2cb38fc5b3db19417b9626f0ebcc3a7120
:X 04 Nov 2022 23:32:07.032 # +monitor master mymaster 10.0.0.101 6379 quorum 2
:X 04 Nov 2022 23:32:07.033 * +slave slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.101 6379
:X 04 Nov 2022 23:32:07.036 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379
:X 04 Nov 2022 23:32:07.595 * +sentinel sentinel f505346a3640b4942963b26829e5d99a1fa6691e 10.0.0.102 26379 @ mymaster 10.0.0.101 6379
:X 04 Nov 2022 23:32:12.779 * +sentinel sentinel b6aebc495fe5d6e81e57ffd5d4f5bef795b3819a 10.0.0.103 26379 @ mymaster 10.0.0.101 6379

#查看从节点日志

[root@slave ~]#tail -f /apps/redis/log/sentinel.log
:X 04 Nov 2022 23:31:59.446 # Configuration loaded
:X 04 Nov 2022 23:31:59.446 # systemd supervision requested, but NOTIFY_SOCKET not found
:X 04 Nov 2022 23:31:59.446 * Increased maximum number of open files to 10032 (it was originally set to 1024).
:X 04 Nov 2022 23:31:59.447 * Running mode=sentinel, port=26379.
:X 04 Nov 2022 23:31:59.455 # Sentinel ID is f505346a3640b4942963b26829e5d99a1fa6691e
:X 04 Nov 2022 23:31:59.455 # +monitor master mymaster 10.0.0.101 6379 quorum 2
:X 04 Nov 2022 23:31:59.457 * +slave slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.101 6379
:X 04 Nov 2022 23:31:59.458 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379
:X 04 Nov 2022 23:32:08.989 * +sentinel sentinel 2f9b5e2cb38fc5b3db19417b9626f0ebcc3a7120 10.0.0.101 26379 @ mymaster 10.0.0.101 6379
:X 04 Nov 2022 23:32:12.716 * +sentinel sentinel b6aebc495fe5d6e81e57ffd5d4f5bef795b3819a 10.0.0.103 26379 @ mymaster 10.0.0.101 6379

#查看从节点日志

[root@slave ~]#tail -f /apps/redis/log/sentinel.log
:X 04 Nov 2022 23:31:59.446 # Configuration loaded
:X 04 Nov 2022 23:31:59.446 # systemd supervision requested, but NOTIFY_SOCKET not found
:X 04 Nov 2022 23:31:59.446 * Increased maximum number of open files to 10032 (it was originally set to 1024).
:X 04 Nov 2022 23:31:59.447 * Running mode=sentinel, port=26379.
:X 04 Nov 2022 23:31:59.455 # Sentinel ID is f505346a3640b4942963b26829e5d99a1fa6691e
:X 04 Nov 2022 23:31:59.455 # +monitor master mymaster 10.0.0.101 6379 quorum 2
:X 04 Nov 2022 23:31:59.457 * +slave slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.101 6379
:X 04 Nov 2022 23:31:59.458 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379
:X 04 Nov 2022 23:32:08.989 * +sentinel sentinel 2f9b5e2cb38fc5b3db19417b9626f0ebcc3a7120 10.0.0.101 26379 @ mymaster 10.0.0.101 6379
:X 04 Nov 2022 23:32:12.716 * +sentinel sentinel b6aebc495fe5d6e81e57ffd5d4f5bef795b3819a 10.0.0.103 26379 @ mymaster 10.0.0.101 6379

1.3.5、当前sentinel状态

[root@slave ~]#redis-cli -p 26379
.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:
sentinel_tilt:
sentinel_running_scripts:
sentinel_scripts_queue_length:
sentinel_simulate_failure_flags:
master:name=mymaster,status=ok,address=10.0.0.101:6379,slaves=2,sentinels=3

1.3.6、验证数据是否同步

[root@master ~]#redis-cli -a
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe..0.0.1:6379> set k1 v1
OK.0.0.1:6379> set k2 v2
OK
 
[root@slave ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe..0.0.1:6379> get k1
"v"

[root@slave ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe..0.0.1:6379> get k1
"v"

1.3.7、制造Redis Master节点故障故障转移并查看其过程

#停掉master节点的redis服务来模拟节点故障
[root@master ~]#killall redis-server
#这时候查看日志发现主节点已经变成了.0.0.102机器
[root@master ~]#tail -f /apps/redis/log/sentinel.log:X 06 Nov 2022 14:36:38.502 # -sdown sentinel b6aebc495fe5d6e81e57ffd5d4f5bef795b3819a 10.0.0.103 26379 @ mymaster 10.0.0.101 6379
:X 06 Nov 2022 14:36:38.604 # -sdown slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379
:X 06 Nov 2022 15:21:26.435 # +sdown master mymaster 10.0.0.101 6379
:X 06 Nov 2022 15:21:26.500 # +new-epoch 1
:X 06 Nov 2022 15:21:26.501 # +vote-for-leader f505346a3640b4942963b26829e5d99a1fa6691e 1
:X 06 Nov 2022 15:21:27.019 # +config-update-from sentinel f505346a3640b4942963b26829e5d99a1fa6691e 10.0.0.102 26379 @ mymaster 10.0.0.101 6379
:X 06 Nov 2022 15:21:27.019 # +switch-master mymaster 10.0.0.101 6379 10.0.0.102 6379
:X 06 Nov 2022 15:21:27.019 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.102 6379
:X 06 Nov 2022 15:21:27.019 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379
:X 06 Nov 2022 15:21:30.032 # +sdown slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379
#查看各节点sentinel信息
[root@master ~]#redis-cli -p
.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:
sentinel_tilt:
sentinel_running_scripts:
sentinel_scripts_queue_length:
sentinel_simulate_failure_flags:
master:name=mymaster,status=ok,address=10.0.0.102:6379,slaves=2,sentinels=3

#故障转移后的sentinel配置文件会被自动修改

[root@slave ~]#grep "^[a-Z]" /apps/redis/etc/redis-sentinel.conf
bind.0.0.0
port
daemonize no
pidfile "/apps/redis/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log"
dir "/tmp"
sentinel myid fa3640b4942963b26829e5d99a1fa6691e
sentinel deny-scripts-reconfig yes
sentinel monitor mymaster.0.0.102 6379 2         #这里发现主节点已经变成10.0.0.102  
sentinel down-after-milliseconds mymaster
sentinel auth-pass mymaster
sentinel config-epoch mymaster
protected-mode no
supervised systemd
sentinel leader-epoch mymaster
sentinel known-replica mymaster.0.0.101 6379
sentinel known-replica mymaster.0.0.103 6379
sentinel known-sentinel mymaster.0.0.103 26379 b6aebc495fe5d6e81e57ffd5d4f5bef795b3819a
sentinel known-sentinel mymaster.0.0.101 26379 2f9b5e2cb38fc5b3db19417b9626f0ebcc3a7120
sentinel current-epoch

#进入redis查看现在info replication信息

[root@slave ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe..0.0.1:6379> info replication
# Replication
role:master
connected_slaves:
slave:ip=10.0.0.103,port=6379,state=online,offset=905513,lag=0
master_replid:cc99085826d751e848cd7b04c9f4abc398cb
master_replid:488fa34f2c093be4886741a38323dd1a0ecf9e03
master_repl_offset:
second_repl_offset:
repl_backlog_active:
repl_backlog_size:
repl_backlog_first_byte_offset:
repl_backlog_histlen:

[root@slave ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe ..0.0.1:6379> INFO replication
# Replication
role:slave
master_host:.0.0.102
master_port:
master_link_status:up
master_last_io_seconds_ago:
master_sync_in_progress:
slave_repl_offset:
slave_priority:
slave_read_only:
connected_slaves:
master_replid:cc99085826d751e848cd7b04c9f4abc398cb
master_replid:488fa34f2c093be4886741a38323dd1a0ecf9e03
master_repl_offset:
second_repl_offset:
repl_backlog_active:
repl_backlog_size:
repl_backlog_first_byte_offset:
repl_backlog_histlen:

1.3.8、查看选举新主的过程日志监测

[root@slave ~]#tail -f /apps/redis/log/sentinel.log
:X 06 Nov 2022 15:21:26.509 # +sdown master mymaster 10.0.0.101 6379
:X 06 Nov 2022 15:21:26.534 # +new-epoch 1
:X 06 Nov 2022 15:21:26.535 # +vote-for-leader f505346a3640b4942963b26829e5d99a1fa6691e 1
:X 06 Nov 2022 15:21:26.609 # +odown master mymaster 10.0.0.101 6379 #quorum 3/2
:X 06 Nov 2022 15:21:26.609 # Next failover delay: I will not start a failover before Sun Nov  6 15:27:27 2022
:X 06 Nov 2022 15:21:27.053 # +config-update-from sentinel f505346a3640b4942963b26829e5d99a1fa6691e 10.0.0.102 26379 @ mymaster 10.0.0.101 6379
:X 06 Nov 2022 15:21:27.053 # +switch-master mymaster 10.0.0.101 6379 10.0.0.102 6379
:X 06 Nov 2022 15:21:27.053 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.102 6379
:X 06 Nov 2022 15:21:27.053 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379
:X 06 Nov 2022 15:21:30.094 # +sdown slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379

1.3.9、恢复故障的原master重新加入redis集群

#重新其他原master的redis服务,会发现会自动指向新的master
[root@master ~]#systemctl restart redis
#在原master上观察状态
[root@master ~]#redis-cli  -a
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe..0.0.1:6379> info replication
# Replication
role:slave
master_host:.0.0.102
master_port:
master_link_status:up
master_last_io_seconds_ago:
master_sync_in_progress:
slave_repl_offset:
slave_priority:
slave_read_only:
connected_slaves:
master_replid:cc99085826d751e848cd7b04c9f4abc398cb
master_replid:0000000000000000000000000000000000000000
master_repl_offset:
second_repl_offset:-
repl_backlog_active:
repl_backlog_size:
repl_backlog_first_byte_offset:
repl_backlog_histlen:

[root@master ~]#redis-cli -p
.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:
sentinel_tilt:
sentinel_running_scripts:
sentinel_scripts_queue_length:
sentinel_simulate_failure_flags:
master:name=mymaster,status=ok,address=10.0.0.102:6379,slaves=2,sentinels=3

#再回到新master节点观察状态
[root@slave ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe..0.0.1:6379> info replication
# Replication
role:master
connected_slaves:
slave:ip=10.0.0.103,port=6379,state=online,offset=976915,lag=0
slave:ip=10.0.0.101,port=6379,state=online,offset=977185,lag=0
master_replid:cc99085826d751e848cd7b04c9f4abc398cb
master_replid:488fa34f2c093be4886741a38323dd1a0ecf9e03
master_repl_offset:
second_repl_offset:
repl_backlog_active:
repl_backlog_size:
repl_backlog_first_byte_offset:
repl_backlog_histlen:

#在新主查看一下日志
[root@slave ~]#tail -f /apps/redis/log/sentinel.log
:X 06 Nov 2022 15:21:27.030 * +slave-reconf-sent slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379
:X 06 Nov 2022 15:21:27.580 * +slave-reconf-inprog slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379
:X 06 Nov 2022 15:21:27.639 # -odown master mymaster 10.0.0.101 6379
:X 06 Nov 2022 15:21:28.621 * +slave-reconf-done slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379
:X 06 Nov 2022 15:21:28.672 # +failover-end master mymaster 10.0.0.101 6379
:X 06 Nov 2022 15:21:28.672 # +switch-master mymaster 10.0.0.101 6379 10.0.0.102 6379
:X 06 Nov 2022 15:21:28.672 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.102 6379
:X 06 Nov 2022 15:21:28.672 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379
:X 06 Nov 2022 15:21:31.738 # +sdown slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379
:X 06 Nov 2022 15:54:28.253 # -sdown slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379