├── README.md ├── haproxy-server001 └── haproxy.cfg ├── redis-server001 ├── README ├── redis-sentinel.conf └── redis.conf ├── redis-server002 ├── README ├── redis-sentinel.conf └── redis.conf └── redis-server003 ├── README ├── redis-sentinel.conf └── redis.conf /README.md: -------------------------------------------------------------------------------- 1 | Redis Sentinel High Availability Setup 2 | =================== 3 | 4 | 5 | Follow this procedure to setup a High Available Redis Caching Layer. 6 | 7 | ### Environment 8 | 9 | 1. haproxy 10 | 2. redis 11 | 3. redis-sentinel 12 | 13 | Haproxy 1.5+ do have a TCP health check feature for redis. Haproxy will be configured in such a way that proxy connections will be forwarded to master instance only. Sentinel will constantly monitor the redis master instance and will promote slave node with lowest priority as next redis master in case of failure. Redis will be configured in such a way that one node will be master and other two nodes will be configured as slaveof the master instance. 14 | 15 | Sentinel will be running as a separate service. we'll need minimum 3 sentinel instances to monitor redis master instance. Our cluster will have quorum of 2 ie; two sentinel instances should agree / vote for a slave to be promoted as master incase of master failure. 16 | 17 | ### Architecture Diagram 18 | 19 | [![redis-sentinel.jpg](https://s21.postimg.org/ccw0qpnh3/redis_sentinel.jpg)](https://postimg.org/image/4wwr4wzrn/) 20 | 21 | 22 | ### Server setup 23 | 24 | **Redis, Redis-sentinel instances - redis-server-001,002,003** 25 | 26 | Install system updates 27 | ```sh 28 | yum update -y 29 | ``` 30 | Install remi repository 31 | 32 | ```sh 33 | centos 6 34 | 35 | rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm 36 | rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-6.rpm 37 | 38 | centos 7 39 | rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-7.rpm 40 | ``` 41 | 42 | Install redis 43 | 44 | ```sh 45 | yum --enablerepo=remi install redis 46 | ``` 47 | Configure redis and redis-sentinel 48 | 49 | Create redis-sentinel working directory 50 | 51 | ```sh 52 | mkdir /var/lib/redis-sentinel 53 | ``` 54 | 55 | Download and install redis and redis-sentinel configuration from redis-server-001,002 and 003 directories and Update file ownership, `redis` user should have write privileges. 56 | 57 | ``` sh 58 | chown redis /etc/redis.conf 59 | chown redis /etc/redis-sentinel.conf 60 | ``` 61 | 62 | Bootstrap Redis HA enviorment - Start Services and add services into system startup 63 | 64 | ``` sh 65 | chkconfig redis on ; chkconfig redis-sentinel on 66 | service redis start ; service redis-sentinel start 67 | ``` 68 | 69 | --- 70 | 71 | **Haproxy Instance - haproxy-server-001** 72 | 73 | Install system updates 74 | 75 | ```sh 76 | yum update -y 77 | ``` 78 | Install epel repository 79 | 80 | ```sh 81 | yum -y install epel-release 82 | ``` 83 | Install haproxy 84 | 85 | ``` sh 86 | yum -y install haproxy 87 | ``` 88 | Configure haproxy 89 | 90 | ```sh 91 | cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg_bak 92 | 93 | Download and install haproxy configuration from haproxy-server-001 directory 94 | ``` 95 | Start Haproxy service and service into system startup 96 | 97 | ```sh 98 | service haproxy start 99 | chkconfig haproxy on 100 | ``` 101 | --- 102 | At this point, you'll have 103 | 104 | 1. Redis master running on redis-server-001 instance 105 | 106 | 2. Redis slaves running on redis-server-002 and redis-server-003 instances 107 | 108 | 3. Redis Sentinel services running on all 3 redis nodes 109 | 110 | 4. Haproxy Server listerning on port 6379 and forwarding connections to master redis instance. 111 | 112 | 5. Upon master failure, redis-sentinel service will promote a slave into the master and haproxy's tcp health-check will detect next master and forward connections to the new master instance. 113 | 114 | 115 | We setup multiple HAProxy instances and manage them using cluster management services. That's how we ensure redundancy and HA for haproxy instances. 116 | -------------------------------------------------------------------------------- /haproxy-server001/haproxy.cfg: -------------------------------------------------------------------------------- 1 | defaults REDIS 2 | mode tcp 3 | timeout connect 4s 4 | timeout server 30s 5 | timeout client 30s 6 | 7 | frontend ft_redis 8 | bind HAPROXY_IPADDR:6379 name redis 9 | default_backend bk_redis 10 | 11 | backend bk_redis 12 | option tcp-check 13 | tcp-check send PING\r\n 14 | tcp-check expect string +PONG 15 | tcp-check send info\ replication\r\n 16 | tcp-check expect string role:master 17 | tcp-check send QUIT\r\n 18 | tcp-check expect string +OK 19 | 20 | server R1 REDIS_001_IPADDR:6379 check inter 1s 21 | server R2 REDIS_002_IPADDR:6379 check inter 1s 22 | server R2 REDIS_003_IPADDR:6379 check inter 1s 23 | -------------------------------------------------------------------------------- /redis-server001/README: -------------------------------------------------------------------------------- 1 | Replace : REDIS_001_IPADDR with Node1's IP Address 2 | Replace : REDIS_002_IPADDR with Node2's IP Address 3 | Replace : REDIS_003_IPADDR with Node3's IP Address 4 | Replace : CLUSTER_NAME with your identity 5 | -------------------------------------------------------------------------------- /redis-server001/redis-sentinel.conf: -------------------------------------------------------------------------------- 1 | bind REDIS_001_IPADDR 2 | port 16379 3 | 4 | dir /var/lib/redis-sentinel 5 | logfile /var/log/redis/redis-sentinel.log 6 | sentinel monitor CLUSTER_NAME REDIS_001_IPADDR 6379 2 7 | sentinel down-after-milliseconds CLUSTER_NAME 10000 8 | sentinel parallel-syncs CLUSTER_NAME 1 9 | sentinel failover-timeout CLUSTER_NAME 20000 10 | -------------------------------------------------------------------------------- /redis-server001/redis.conf: -------------------------------------------------------------------------------- 1 | ################################ GENERAL CONFIGURATION ############################### 2 | 3 | bind REDIS_001_IPADDR 4 | port 6379 5 | dir /var/lib/redis 6 | protected-mode yes 7 | tcp-backlog 511 8 | timeout 0 9 | tcp-keepalive 300 10 | daemonize no 11 | supervised no 12 | pidfile /var/run/redis_6379.pid 13 | loglevel notice 14 | logfile /var/log/redis/redis.log 15 | databases 16 16 | 17 | ################################ SNAPSHOTTING ################################ 18 | 19 | # Snapshot settings 20 | 21 | save 900 1 22 | save 300 10 23 | save 60 10000 24 | 25 | stop-writes-on-bgsave-error yes 26 | 27 | rdbcompression yes 28 | rdbchecksum yes 29 | dbfilename dump.rdb 30 | 31 | dir /var/lib/redis/ 32 | 33 | ################################# REPLICATION ################################# 34 | 35 | slave-serve-stale-data yes 36 | slave-read-only yes 37 | 38 | repl-diskless-sync no 39 | repl-diskless-sync-delay 5 40 | 41 | 42 | repl-disable-tcp-nodelay no 43 | 44 | 45 | # repl-backlog-size 1mb 46 | # repl-backlog-ttl 3600 47 | 48 | slave-priority 100 49 | 50 | ############################## APPEND ONLY MODE ############################### 51 | 52 | 53 | appendonly no 54 | appendfilename "appendonly.aof" 55 | 56 | # appendfsync always 57 | appendfsync everysec 58 | # appendfsync no 59 | 60 | no-appendfsync-on-rewrite no 61 | 62 | auto-aof-rewrite-percentage 100 63 | auto-aof-rewrite-min-size 64mb 64 | aof-load-truncated yes 65 | 66 | ################################ LUA SCRIPTING ############################### 67 | 68 | lua-time-limit 5000 69 | 70 | ################################## SLOW LOG ################################### 71 | 72 | slowlog-log-slower-than 10000 73 | slowlog-max-len 128 74 | 75 | ################################ LATENCY MONITOR ############################## 76 | 77 | latency-monitor-threshold 0 78 | ############################# EVENT NOTIFICATION ############################## 79 | 80 | 81 | notify-keyspace-events "" 82 | 83 | ############################### ADVANCED CONFIG ############################### 84 | 85 | hash-max-ziplist-entries 512 86 | hash-max-ziplist-value 64 87 | list-max-ziplist-size -2 88 | list-compress-depth 0 89 | set-max-intset-entries 512 90 | zset-max-ziplist-entries 128 91 | zset-max-ziplist-value 64 92 | hll-sparse-max-bytes 3000 93 | activerehashing yes 94 | client-output-buffer-limit normal 0 0 0 95 | client-output-buffer-limit slave 256mb 64mb 60 96 | client-output-buffer-limit pubsub 32mb 8mb 60 97 | hz 10 98 | aof-rewrite-incremental-fsync yes 99 | -------------------------------------------------------------------------------- /redis-server002/README: -------------------------------------------------------------------------------- 1 | Replace : REDIS_001_IPADDR with Node1's IP Address 2 | Replace : REDIS_002_IPADDR with Node2's IP Address 3 | Replace : REDIS_003_IPADDR with Node3's IP Address 4 | Replace : CLUSTER_NAME with your identity 5 | -------------------------------------------------------------------------------- /redis-server002/redis-sentinel.conf: -------------------------------------------------------------------------------- 1 | bind REDIS_002_IPADDR 2 | port 16379 3 | 4 | dir /var/lib/redis-sentinel 5 | logfile /var/log/redis/redis-sentinel.log 6 | sentinel monitor CLUSTER_NAME REDIS_001_IPADDR 6379 2 7 | sentinel down-after-milliseconds CLUSTER_NAME 10000 8 | sentinel parallel-syncs CLUSTER_NAME 1 9 | sentinel failover-timeout CLUSTER_NAME 20000 10 | -------------------------------------------------------------------------------- /redis-server002/redis.conf: -------------------------------------------------------------------------------- 1 | ################################ GENERAL CONFIGURATION ############################### 2 | 3 | bind REDIS_002_IPADDR 4 | port 6379 5 | dir /var/lib/redis 6 | protected-mode yes 7 | tcp-backlog 511 8 | timeout 0 9 | tcp-keepalive 300 10 | daemonize no 11 | supervised no 12 | pidfile /var/run/redis_6379.pid 13 | loglevel notice 14 | logfile /var/log/redis/redis.log 15 | databases 16 16 | 17 | ################################ SNAPSHOTTING ################################ 18 | 19 | # Snapshot settings 20 | 21 | save 900 1 22 | save 300 10 23 | save 60 10000 24 | 25 | stop-writes-on-bgsave-error yes 26 | 27 | rdbcompression yes 28 | rdbchecksum yes 29 | dbfilename dump.rdb 30 | 31 | dir /var/lib/redis/ 32 | 33 | ################################# REPLICATION ################################# 34 | 35 | slave-serve-stale-data yes 36 | slave-read-only yes 37 | 38 | repl-diskless-sync no 39 | repl-diskless-sync-delay 5 40 | 41 | 42 | repl-disable-tcp-nodelay no 43 | 44 | 45 | # repl-backlog-size 1mb 46 | # repl-backlog-ttl 3600 47 | 48 | slaveof REDIS_001_IPADDR 6379 49 | 50 | slave-priority 150 51 | 52 | ############################## APPEND ONLY MODE ############################### 53 | 54 | 55 | appendonly no 56 | appendfilename "appendonly.aof" 57 | 58 | # appendfsync always 59 | appendfsync everysec 60 | # appendfsync no 61 | 62 | no-appendfsync-on-rewrite no 63 | 64 | auto-aof-rewrite-percentage 100 65 | auto-aof-rewrite-min-size 64mb 66 | aof-load-truncated yes 67 | 68 | ################################ LUA SCRIPTING ############################### 69 | 70 | lua-time-limit 5000 71 | 72 | ################################## SLOW LOG ################################### 73 | 74 | slowlog-log-slower-than 10000 75 | slowlog-max-len 128 76 | 77 | ################################ LATENCY MONITOR ############################## 78 | 79 | latency-monitor-threshold 0 80 | ############################# EVENT NOTIFICATION ############################## 81 | 82 | 83 | notify-keyspace-events "" 84 | 85 | ############################### ADVANCED CONFIG ############################### 86 | 87 | hash-max-ziplist-entries 512 88 | hash-max-ziplist-value 64 89 | list-max-ziplist-size -2 90 | list-compress-depth 0 91 | set-max-intset-entries 512 92 | zset-max-ziplist-entries 128 93 | zset-max-ziplist-value 64 94 | hll-sparse-max-bytes 3000 95 | activerehashing yes 96 | client-output-buffer-limit normal 0 0 0 97 | client-output-buffer-limit slave 256mb 64mb 60 98 | client-output-buffer-limit pubsub 32mb 8mb 60 99 | hz 10 100 | aof-rewrite-incremental-fsync yes 101 | -------------------------------------------------------------------------------- /redis-server003/README: -------------------------------------------------------------------------------- 1 | Replace : REDIS_001_IPADDR with Node1's IP Address 2 | Replace : REDIS_002_IPADDR with Node2's IP Address 3 | Replace : REDIS_003_IPADDR with Node3's IP Address 4 | Replace : CLUSTER_NAME with your identity 5 | -------------------------------------------------------------------------------- /redis-server003/redis-sentinel.conf: -------------------------------------------------------------------------------- 1 | bind REDIS_003_IPADDR 2 | port 16379 3 | 4 | dir /var/lib/redis-sentinel 5 | logfile /var/log/redis/redis-sentinel.log 6 | sentinel monitor CLUSTER_NAME REDIS_001_IPADDR 6379 2 7 | sentinel down-after-milliseconds CLUSTER_NAME 10000 8 | sentinel parallel-syncs CLUSTER_NAME 1 9 | sentinel failover-timeout CLUSTER_NAME 20000 10 | -------------------------------------------------------------------------------- /redis-server003/redis.conf: -------------------------------------------------------------------------------- 1 | ################################ GENERAL CONFIGURATION ############################### 2 | 3 | bind REDIS_003_IPADDR 4 | port 6379 5 | dir /var/lib/redis 6 | protected-mode yes 7 | tcp-backlog 511 8 | timeout 0 9 | tcp-keepalive 300 10 | daemonize no 11 | supervised no 12 | pidfile /var/run/redis_6379.pid 13 | loglevel notice 14 | logfile /var/log/redis/redis.log 15 | databases 16 16 | 17 | ################################ SNAPSHOTTING ################################ 18 | 19 | # Snapshot settings 20 | 21 | save 900 1 22 | save 300 10 23 | save 60 10000 24 | 25 | stop-writes-on-bgsave-error yes 26 | 27 | rdbcompression yes 28 | rdbchecksum yes 29 | dbfilename dump.rdb 30 | 31 | dir /var/lib/redis/ 32 | 33 | ################################# REPLICATION ################################# 34 | 35 | slave-serve-stale-data yes 36 | slave-read-only yes 37 | 38 | repl-diskless-sync no 39 | repl-diskless-sync-delay 5 40 | 41 | 42 | repl-disable-tcp-nodelay no 43 | 44 | 45 | # repl-backlog-size 1mb 46 | # repl-backlog-ttl 3600 47 | 48 | slaveof REDIS_001_IPADDR 6379 49 | 50 | slave-priority 200 51 | 52 | ############################## APPEND ONLY MODE ############################### 53 | 54 | 55 | appendonly no 56 | appendfilename "appendonly.aof" 57 | 58 | # appendfsync always 59 | appendfsync everysec 60 | # appendfsync no 61 | 62 | no-appendfsync-on-rewrite no 63 | 64 | auto-aof-rewrite-percentage 100 65 | auto-aof-rewrite-min-size 64mb 66 | aof-load-truncated yes 67 | 68 | ################################ LUA SCRIPTING ############################### 69 | 70 | lua-time-limit 5000 71 | 72 | ################################## SLOW LOG ################################### 73 | 74 | slowlog-log-slower-than 10000 75 | slowlog-max-len 128 76 | 77 | ################################ LATENCY MONITOR ############################## 78 | 79 | latency-monitor-threshold 0 80 | ############################# EVENT NOTIFICATION ############################## 81 | 82 | 83 | notify-keyspace-events "" 84 | 85 | ############################### ADVANCED CONFIG ############################### 86 | 87 | hash-max-ziplist-entries 512 88 | hash-max-ziplist-value 64 89 | list-max-ziplist-size -2 90 | list-compress-depth 0 91 | set-max-intset-entries 512 92 | zset-max-ziplist-entries 128 93 | zset-max-ziplist-value 64 94 | hll-sparse-max-bytes 3000 95 | activerehashing yes 96 | client-output-buffer-limit normal 0 0 0 97 | client-output-buffer-limit slave 256mb 64mb 60 98 | client-output-buffer-limit pubsub 32mb 8mb 60 99 | hz 10 100 | aof-rewrite-incremental-fsync yes 101 | --------------------------------------------------------------------------------