├── README.md ├── etc ├── bootstrap-pod.sh └── redis.conf ├── notes └── gettingstarted-dashboard.yml └── redis-cluster.yml /README.md: -------------------------------------------------------------------------------- 1 | # Redis Cluster on Kubernetes 2 | 3 | This module is intended to simplify the creation and operation of a Redis Cluster deployment in Kubernetes. 4 | I don't recommend that you run this in production - it's just meant to be an illustrative example of a nontrivial Stateful Set deployment. 5 | 6 | ## Requirements 7 | 8 | - Kubernetes 1.17.0+ 9 | - Minikube to run the module locally 10 | 11 | ## How it works 12 | 13 | These directions assume some familiarity with [Redis Cluster](http://redis.io/topics/cluster-tutorial). 14 | 15 | When you create the resources in Kubernetes, it will create a 6-member (the minimum recommended size) [Stateful Set](https://kubernetes.io/docs/concepts/workloads/controllers/statefulsets/) cluster where members 0-2 are master nodes and all other members are replicas. 16 | 17 | ## Testing it out 18 | 19 | To launch the cluster, have Kubernetes create all the resources in redis-cluster.yml: 20 | 21 | ``` 22 | $ kubectl create -f redis-cluster.yml 23 | service/redis-cluster created 24 | configmap "redis-cluster-config" configured 25 | poddisruptionbudget.policy/redis-cluster-pdb created 26 | statefulset.apps/redis-cluster created 27 | ``` 28 | 29 | Wait a bit for the service to initialize. 30 | 31 | Once all the pods are initialized, you can see that Pod "redis-cluster-0" became the cluster master with the other nodes as slaves. 32 | 33 | ``` 34 | $ kubectl exec redis-cluster-0 -- redis-cli cluster nodes 35 | Defaulted container "redis-cluster" out of: redis-cluster, init-redis-cluster (init) 36 | 532505ce41c64ddf4aff143406eb90424c29c138 10.1.0.136:6379@16379 myself,master - 0 1642662784000 1 connected 0-5461 37 | 1f188afd5f2a228320cc43753e4b6a1c01c32445 10.1.0.139:6379@16379 slave 532505ce41c64ddf4aff143406eb90424c29c138 0 1642662785392 1 connected 38 | 07939f1e633cd3538f8730017aa5ce1d0f8ba680 10.1.0.140:6379@16379 slave de3b3a6d02c81f4104566828e8a523b46e31cd02 0 1642662785000 0 connected 39 | de3b3a6d02c81f4104566828e8a523b46e31cd02 10.1.0.137:6379@16379 master - 0 1642662784386 0 connected 5462-10922 40 | c53e989fa503b04ecc7c651f448a7fc07ac3c975 10.1.0.138:6379@16379 master - 0 1642662786399 2 connected 10923-16383 41 | a43a4a9d81ae095bfdd373ed2c9aebbe958d086a 10.1.0.141:6379@16379 slave c53e989fa503b04ecc7c651f448a7fc07ac3c975 0 1642662784000 2 connected 42 | ``` 43 | 44 | Also, you should be able to use redis-cli to connect to a cluster node we just created 45 | ``` 46 | $ kubectl exec -it redis-cluster-0 -- redis-cli 47 | ``` 48 | 49 | You can also check the slot configuration here: 50 | ``` 51 | $ kubectl exec redis-cluster-0 -- redis-cli --cluster check localhost 6379 52 | Defaulted container "redis-cluster" out of: redis-cluster, init-redis-cluster (init) 53 | localhost:6379 (55a74f86...) -> 0 keys | 5462 slots | 1 slaves. 54 | 10.1.0.126:6379 (f5b39569...) -> 0 keys | 5461 slots | 1 slaves. 55 | 10.1.0.125:6379 (742cf86e...) -> 0 keys | 5461 slots | 1 slaves. 56 | [OK] 0 keys in 3 masters. 57 | 0.00 keys per slot on average. 58 | >>> Performing Cluster Check (using node localhost:6379) 59 | M: 55a74f86cf241a90f66a94a6c1789e031adbcc0c localhost:6379 60 | slots:[0-5461] (5462 slots) master 61 | 1 additional replica(s) 62 | M: f5b39569a75fe72cb16e207f2947d22c625a39ab 10.1.0.126:6379 63 | slots:[10923-16383] (5461 slots) master 64 | 1 additional replica(s) 65 | S: ddea6fbe8baa7938504f9f1ff503f0f190b49bc3 10.1.0.127:6379 66 | slots: (0 slots) slave 67 | replicates 55a74f86cf241a90f66a94a6c1789e031adbcc0c 68 | M: 742cf86e53a93473d17d352d2100b7db9dc61b72 10.1.0.125:6379 69 | slots:[5462-10922] (5461 slots) master 70 | 1 additional replica(s) 71 | S: 054f92cfb064e5cd762e466799311fbc21228049 10.1.0.128:6379 72 | slots: (0 slots) slave 73 | replicates 742cf86e53a93473d17d352d2100b7db9dc61b72 74 | S: ccc463f1aa52c9afd12aa945d7869c2a44f81c9d 10.1.0.129:6379 75 | slots: (0 slots) slave 76 | replicates f5b39569a75fe72cb16e207f2947d22c625a39ab 77 | [OK] All nodes agree about slots configuration. 78 | >>> Check for open slots... 79 | >>> Check slots coverage... 80 | [OK] All 16384 slots covered. 81 | ``` 82 | 83 | To add more nodes to the cluster, you can simply use normal stateful set scaling: 84 | ``` 85 | kubectl scale -n default statefulset redis-cluster --replicas=12 86 | ``` 87 | Newly-created nodes will join the cluster as replicas. 88 | 89 | 90 | To clean this mess off your Minikube VM: 91 | ``` 92 | $ kubectl delete service,statefulsets redis-cluster 93 | $ kubectl delete configmaps redis-cluster-config 94 | $ kubectl delete poddisruptionbudgets.policy redis-cluster-pd 95 | 96 | # To prevent potential data loss, deleting a statefulset doesn't delete the pods. Gotta do that manually. 97 | $ kubectl delete pod redis-cluster-0 redis-cluster-1 redis-cluster-2 redis-cluster-3 redis-cluster-4 redis-cluster-5 98 | ``` 99 | 100 | ## TODO 101 | - Add documentation for common Redis Cluster operations: adding nodes, resharding, deleting nodes 102 | - Test some failure scenarios 103 | - Use a persistentvolume to store backup files 104 | - Create a ScheduledJob to do automated backups once [this feature](https://github.com/antirez/redis/issues/2463) is finished. 105 | - Make it easier to assign new masters 106 | - Cluster members should check whether nodes.conf exists and if so, skip pod initialization. 107 | -------------------------------------------------------------------------------- /etc/bootstrap-pod.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -ex 3 | 4 | # Find which member of the Stateful Set this pod is running 5 | # e.g. "redis-cluster-0" -> "0" 6 | PET_ORDINAL=$(cat /etc/podinfo/pod_name | rev | cut -d- -f1 | rev) 7 | MY_SHARD=$(($PET_ORDINAL % $NUM_SHARDS)) 8 | 9 | redis-server /conf/redis.conf & 10 | 11 | # TODO: Wait until redis-server process is ready 12 | sleep 1 13 | 14 | if [ $PET_ORDINAL -lt $NUM_SHARDS ]; then 15 | # Set up primary nodes. Divide slots into equal(ish) contiguous blocks 16 | NUM_SLOTS=$(( 16384 / $NUM_SHARDS )) 17 | REMAINDER=$(( 16384 % $NUM_SHARDS )) 18 | START_SLOT=$(( $NUM_SLOTS * $MY_SHARD + ($MY_SHARD < $REMAINDER ? $MY_SHARD : $REMAINDER) )) 19 | END_SLOT=$(( $NUM_SLOTS * ($MY_SHARD+1) + ($MY_SHARD+1 < $REMAINDER ? $MY_SHARD+1 : $REMAINDER) - 1 )) 20 | 21 | PEER_IP=$(perl -MSocket -e "print inet_ntoa(scalar(gethostbyname(\"redis-cluster-0.redis-cluster.$POD_NAMESPACE.svc.cluster.local\")))") 22 | redis-cli cluster meet $PEER_IP 6379 23 | redis-cli cluster addslots $(seq $START_SLOT $END_SLOT) 24 | else 25 | # Set up a replica 26 | PEER_IP=$(perl -MSocket -e "print inet_ntoa(scalar(gethostbyname(\"redis-cluster-$MY_SHARD.redis-cluster.$POD_NAMESPACE.svc.cluster.local\")))") 27 | redis-cli --cluster add-node localhost:6379 $PEER_IP:6379 --cluster-slave 28 | fi 29 | 30 | wait -------------------------------------------------------------------------------- /etc/redis.conf: -------------------------------------------------------------------------------- 1 | cluster-enabled yes 2 | cluster-require-full-coverage no 3 | cluster-node-timeout 1000 4 | cluster-migration-barrier 1 5 | cluster-config-file nodes.conf 6 | appendonly yes 7 | # Other cluster members need to be able to connect 8 | protected-mode no 9 | 10 | -------------------------------------------------------------------------------- /notes/gettingstarted-dashboard.yml: -------------------------------------------------------------------------------- 1 | Initializing the Kubernetes Dashboard 2 | === 3 | 4 | Deploy the Dashboard app using Kubectl: 5 | kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended/kubernetes-dashboard.yaml 6 | 7 | Spin up proxy between our workstation and the Kubernetes control plane: 8 | Kubectl proxy 9 | 10 | Now, we should be able to browse to http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/workloads?namespace=default 11 | 12 | Create a dashboard service account: 13 | kubectl create serviceaccount dashboard-admin-sa 14 | 15 | Next bind the dashboard-admin-service-account service account to the cluster-admin role 16 | kubectl create clusterrolebinding dashboard-admin-sa --clusterrole=cluster-admin --serviceaccount=default:dashboard-admin-sa 17 | 18 | 19 | List secrets: 20 | kubectl get secrets 21 | 22 | 23 | Use kubectl to get the access token. You can copy from here and paste it into the dashboard UI: 24 | kubectl describe secret dashboard-admin-sa-token-abcde 25 | -------------------------------------------------------------------------------- /redis-cluster.yml: -------------------------------------------------------------------------------- 1 | # 2 | # Redis Cluster service 3 | # 4 | apiVersion: v1 5 | kind: Service 6 | metadata: 7 | name: redis-cluster 8 | labels: 9 | app: redis-cluster 10 | environment: dev 11 | spec: 12 | publishNotReadyAddresses: true 13 | ports: 14 | - port: 6379 15 | targetPort: 6379 16 | name: client 17 | - port: 16379 18 | targetPort: 16379 19 | name: gossip 20 | clusterIP: None 21 | #type: ClusterIP 22 | selector: 23 | app: redis-cluster 24 | --- 25 | apiVersion: policy/v1 26 | kind: PodDisruptionBudget 27 | metadata: 28 | name: redis-cluster-pdb 29 | spec: 30 | selector: 31 | matchLabels: 32 | app: redis-cluster 33 | maxUnavailable: 0 34 | --- 35 | # 36 | # Redis configuration file for clustered mode 37 | # 38 | apiVersion: v1 39 | kind: ConfigMap 40 | metadata: 41 | name: redis-cluster-config 42 | labels: 43 | app: redis-cluster 44 | data: 45 | redis.conf: |+ 46 | cluster-enabled yes 47 | cluster-require-full-coverage no 48 | cluster-node-timeout 15000 49 | cluster-config-file nodes.conf 50 | cluster-migration-barrier 1 51 | appendonly yes 52 | # Other cluster members need to be able to connect 53 | protected-mode no 54 | # 55 | # A script to bootstrap Stateful Set members as they initialize 56 | # 57 | bootstrap-pod.sh: |+ 58 | #!/bin/sh 59 | set -ex 60 | 61 | # Find which member of the Stateful Set this pod is running 62 | # e.g. "redis-cluster-0" -> "0" 63 | PET_ORDINAL=$(cat /etc/podinfo/pod_name | rev | cut -d- -f1 | rev) 64 | MY_SHARD=$(($PET_ORDINAL % $NUM_SHARDS)) 65 | 66 | redis-server /conf/redis.conf & 67 | 68 | # TODO: Wait until redis-server process is ready 69 | sleep 1 70 | 71 | if [ $PET_ORDINAL -lt $NUM_SHARDS ]; then 72 | # Set up primary nodes. Divide slots into equal(ish) contiguous blocks 73 | NUM_SLOTS=$(( 16384 / $NUM_SHARDS )) 74 | REMAINDER=$(( 16384 % $NUM_SHARDS )) 75 | START_SLOT=$(( $NUM_SLOTS * $MY_SHARD + ($MY_SHARD < $REMAINDER ? $MY_SHARD : $REMAINDER) )) 76 | END_SLOT=$(( $NUM_SLOTS * ($MY_SHARD+1) + ($MY_SHARD+1 < $REMAINDER ? $MY_SHARD+1 : $REMAINDER) - 1 )) 77 | 78 | PEER_IP=$(perl -MSocket -e "print inet_ntoa(scalar(gethostbyname(\"redis-cluster-0.redis-cluster.$POD_NAMESPACE.svc.cluster.local\")))") 79 | redis-cli cluster meet $PEER_IP 6379 80 | redis-cli cluster addslots $(seq $START_SLOT $END_SLOT) 81 | else 82 | # Set up a replica 83 | PEER_IP=$(perl -MSocket -e "print inet_ntoa(scalar(gethostbyname(\"redis-cluster-$MY_SHARD.redis-cluster.$POD_NAMESPACE.svc.cluster.local\")))") 84 | redis-cli --cluster add-node localhost:6379 $PEER_IP:6379 --cluster-slave 85 | fi 86 | 87 | wait 88 | --- 89 | apiVersion: apps/v1 90 | kind: StatefulSet 91 | metadata: 92 | name: redis-cluster 93 | spec: 94 | podManagementPolicy: OrderedReady # default 95 | serviceName: redis-cluster 96 | replicas: 6 97 | selector: 98 | matchLabels: 99 | app: redis-cluster # has to match .spec.template.metadata.labels 100 | template: 101 | metadata: 102 | labels: 103 | app: redis-cluster 104 | name: redis-cluster 105 | spec: 106 | # affinity: # Ensure that each Redis instance is provisioned on a different k8s node 107 | # podAntiAffinity: 108 | # requiredDuringSchedulingIgnoredDuringExecution: 109 | # - labelSelector: 110 | # matchExpressions: 111 | # - key: "app" 112 | # operator: In 113 | # values: 114 | # - redis-cluster 115 | # topologyKey: "kubernetes.io/hostname" 116 | terminationGracePeriodSeconds: 10 117 | containers: 118 | - name: redis-cluster 119 | image: redis:6.2.6 120 | ports: 121 | - containerPort: 6379 122 | name: client 123 | - containerPort: 16379 124 | name: gossip 125 | command: 126 | - sh 127 | args: 128 | - /conf/bootstrap-pod.sh 129 | # Ensure that Redis is online before initializing the next node. 130 | # TODO: Test that the cluster node is init'd properly. 131 | readinessProbe: 132 | exec: 133 | command: 134 | - sh 135 | - -c 136 | - "redis-cli -h $(hostname) ping" 137 | initialDelaySeconds: 5 138 | timeoutSeconds: 5 139 | 140 | securityContext: 141 | capabilities: 142 | add: 143 | - IPC_LOCK 144 | # Mark a node as down if Redis server stops running 145 | livenessProbe: 146 | exec: 147 | command: 148 | - sh 149 | - -c 150 | - "redis-cli -h $(hostname) ping" 151 | initialDelaySeconds: 20 152 | periodSeconds: 3 153 | env: 154 | - name: POD_NAMESPACE 155 | valueFrom: 156 | fieldRef: 157 | fieldPath: metadata.namespace 158 | - name: NODE_NAME 159 | valueFrom: 160 | fieldRef: 161 | fieldPath: spec.nodeName 162 | - name: NUM_SHARDS 163 | value: "3" # If you modify this value, make sure there are at least 2 times the number of replicas 164 | volumeMounts: 165 | - name: conf 166 | mountPath: /conf 167 | readOnly: false 168 | - name: podinfo 169 | mountPath: /etc/podinfo 170 | readOnly: false 171 | initContainers: 172 | # Wait for the redis-cluster service to exist. We need it to resolve the hostnames of our nodes 173 | - name: init-redis-cluster 174 | image: busybox:1.28 175 | command: ['sh', '-c', "until nslookup redis-cluster.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for redis-cluster; sleep 2; done"] 176 | volumes: 177 | # Insert our pre-baked Redis configuration file into /conf/redis.conf 178 | - name: conf 179 | configMap: 180 | name: redis-cluster-config 181 | items: 182 | - key: redis.conf 183 | path: redis.conf 184 | - key: bootstrap-pod.sh # TODO: Move this or extract it into its own Docker image 185 | path: bootstrap-pod.sh 186 | # The init container will use this info to find cluster peers 187 | - name: podinfo 188 | downwardAPI: 189 | items: 190 | - path: "labels" 191 | fieldRef: 192 | fieldPath: metadata.labels 193 | - path: "annotations" 194 | fieldRef: 195 | fieldPath: metadata.annotations 196 | - path: "pod_name" 197 | fieldRef: 198 | fieldPath: metadata.name 199 | - path: "pod_namespace" 200 | fieldRef: 201 | fieldPath: metadata.namespace 202 | volumeClaimTemplates: 203 | - metadata: 204 | name: datadir 205 | spec: 206 | accessModes: [ "ReadWriteOnce" ] 207 | resources: 208 | requests: 209 | storage: 10Gi 210 | --------------------------------------------------------------------------------