├── LICENSE ├── README.md ├── resources └── mongodb-service.yaml └── scripts ├── configure_repset_auth.sh ├── delete_service.sh ├── generate.sh ├── recreate_service.sh └── teardown.sh /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Paul Done 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MongoDB Deployment Demo for Kubernetes on Minikube (i.e. running on local workstation) 2 | 3 | An example project demonstrating the deployment of a MongoDB Replica Set via Kubernetes on Minikube (Kubernetes running locally on a workstation). Contains example Kubernetes YAML resource files (in the 'resource' folder) and associated Kubernetes based Bash scripts (in the 'scripts' folder) to configure the environment and deploy a MongoDB Replica Set. 4 | 5 | For further background information on what these scripts and resource files do, plus general information about running MongoDB with Kubernetes, see: [http://k8smongodb.net/](http://k8smongodb.net/) 6 | 7 | 8 | ## 1 How To Run 9 | 10 | ### 1.1 Prerequisites 11 | 12 | Ensure the following dependencies are already fulfilled on your host Linux/Windows/Mac Workstation/Laptop: 13 | 14 | 1. The [VirtualBox](https://www.virtualbox.org/wiki/Downloads) hypervisor has been installed. 15 | 2. The [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) command-line tool for Kubernetes has been installed. 16 | 3. The [Minikube](https://github.com/kubernetes/minikube/releases) tool for running Kubernetes locally has been installed. 17 | 4. The Minikube cluster has been started, inside a local Virtual Machine, using the following command (also includes commands to check that kubectl is configured correctly to see the running minikube pod): 18 | 19 | ``` 20 | $ minikube start 21 | $ kubectl get nodes 22 | $ kubectl describe nodes 23 | $ kubectl get services 24 | ``` 25 | 26 | ### 1.2 Main Deployment Steps 27 | 28 | 1. To deploy the MongoDB Service (including the StatefulSet running "mongod" containers), via a command-line terminal/shell, execute the following: 29 | 30 | ``` 31 | $ cd scripts 32 | $ ./generate.sh 33 | ``` 34 | 35 | 2. Re-run the following command, until all 3 “mongod” pods (and their containers) have been successfully started (“Status=Running”; usually takes a minute or two). 36 | 37 | ``` 38 | $ kubectl get all 39 | ``` 40 | 41 | 3. Execute the following script which connects to the first Mongod instance running in a container of the Kubernetes StatefulSet, via the Mongo Shell, to (1) initialise the MongoDB Replica Set, and (2) create a MongoDB admin user (specify the password you want as the argument to the script, replacing 'abc123'). 42 | 43 | ``` 44 | $ ./configure_repset_auth.sh abc123 45 | ``` 46 | 47 | You should now have a MongoDB Replica Set initialised, secured and running in a Kubernetes StatefulSet. 48 | 49 | You can also view the the state of the deployed environment, via the Kubernetes dashboard, which can be launched in a browser with the following command: `$ minikube dashboard` 50 | 51 | 52 | ### 1.3 Example Tests To Run To Check Things Are Working 53 | 54 | Use this section to prove: 55 | 56 | 1. Data is being replicated between members of the containerised replica set. 57 | 2. Data is retained even when the MongoDB Service/StatefulSet is removed and then re-created (by virtue of re-using the same Persistent Volume Claims). 58 | 59 | #### 1.3.1 Replication Test 60 | 61 | Connect to the container running the first "mongod" replica, then use the Mongo Shell to authenticate and add some test data to a database: 62 | 63 | $ kubectl exec -it mongod-0 -c mongod-container bash 64 | $ mongo 65 | > db.getSiblingDB('admin').auth("main_admin", "abc123"); 66 | > use test; 67 | > db.testcoll.insert({a:1}); 68 | > db.testcoll.insert({b:2}); 69 | > db.testcoll.find(); 70 | 71 | Exit out of the shell and exit out of the first container (“mongod-0”). Then connect to the second container (“mongod-1”), run the Mongo Shell again and see if the previously inserted data is visible to the second "mongod" replica: 72 | 73 | $ kubectl exec -it mongod-1 -c mongod-container bash 74 | $ mongo 75 | > db.getSiblingDB('admin').auth("main_admin", "abc123"); 76 | > db.setSlaveOk(1); 77 | > use test; 78 | > db.testcoll.find(); 79 | 80 | You should see that the two records inserted via the first replica, are visible to the second replica. 81 | 82 | #### 1.3.2 Redeployment Without Data Loss Test 83 | 84 | To see if Persistent Volume Claims really are working, run a script to drop the Service & StatefulSet (thus stopping the pods and their “mongod” containers) and then a script to re-create them again: 85 | 86 | $ ./delete_service.sh 87 | $ ./recreate_service.sh 88 | $ kubectl get all 89 | 90 | As before, keep re-running the last command above, until you can see that all 3 “mongod” pods and their containers have been successfully started again. Then connect to the first container, run the Mongo Shell and query to see if the data we’d inserted into the old containerised replica-set is still present in the re-instantiated replica set: 91 | 92 | $ kubectl exec -it mongod-0 -c mongod-container bash 93 | $ mongo 94 | > db.getSiblingDB('admin').auth("main_admin", "abc123"); 95 | > use test; 96 | > db.testcoll.find(); 97 | 98 | You should see that the two records inserted earlier, are still present. 99 | 100 | ### 1.4 Undeploying & Cleaning Down the Kubernetes Environment 101 | 102 | Run the following script to undeploy the MongoDB Service & StatefulSet. 103 | 104 | $ ./teardown.sh 105 | 106 | If you want, you can shutdown the Minikube virtual machine with the following command. 107 | 108 | $ minikube stop 109 | 110 | 111 | ## 2 Project Details 112 | 113 | ### 2.1 Factors Addressed By This Project 114 | 115 | * Deployment of a MongoDB on a local Minikube Kubernetes platform 116 | * Use of Kubernetes StatefulSets and PersistentVolumeClaims to ensure data is not lost when containers are recycled 117 | * Proper configuration of a MongoDB Replica Set for full resiliency 118 | * Securing MongoDB by default for new deployments 119 | * Disabling Transparent Huge Pages to improve performance _(this is disabled by default in the Minikube host nodes)_ 120 | * Disabling NUMA to improve performance 121 | * Controlling CPU & RAM Resource Allocation 122 | * Correctly configuring WiredTiger Cache Size in containers 123 | * Controlling Anti-Affinity for Mongod Replicas to avoid a Single Point of Failure _(although in Minikube there is only one host node, so in reality all Mongod Replicas will land on the same host)_ 124 | 125 | ### 2.2 Factors To Be Potentially Addressed In The Future By This Project 126 | 127 | * Leveraging XFS filesystem for data file storage to improve performance _(not worth attempting to implement any hacks here to get this working in Minikube, as Minikube is just a demo/development environment, so raw performance gains from using XFS are not a priority)_ 128 | 129 | -------------------------------------------------------------------------------- /resources/mongodb-service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: mongodb-service 5 | labels: 6 | name: mongo 7 | spec: 8 | ports: 9 | - port: 27017 10 | targetPort: 27017 11 | clusterIP: None 12 | selector: 13 | role: mongo 14 | --- 15 | apiVersion: apps/v1beta1 16 | kind: StatefulSet 17 | metadata: 18 | name: mongod 19 | spec: 20 | serviceName: mongodb-service 21 | replicas: 3 22 | template: 23 | metadata: 24 | labels: 25 | role: mongo 26 | environment: test 27 | replicaset: MainRepSet 28 | spec: 29 | affinity: 30 | podAntiAffinity: 31 | preferredDuringSchedulingIgnoredDuringExecution: 32 | - weight: 100 33 | podAffinityTerm: 34 | labelSelector: 35 | matchExpressions: 36 | - key: replicaset 37 | operator: In 38 | values: 39 | - MainRepSet 40 | topologyKey: kubernetes.io/hostname 41 | terminationGracePeriodSeconds: 10 42 | volumes: 43 | - name: secrets-volume 44 | secret: 45 | secretName: shared-bootstrap-data 46 | defaultMode: 256 47 | containers: 48 | - name: mongod-container 49 | #image: pkdone/mongo-ent:3.4 50 | image: mongo 51 | command: 52 | - "numactl" 53 | - "--interleave=all" 54 | - "mongod" 55 | - "--wiredTigerCacheSizeGB" 56 | - "0.1" 57 | - "--bind_ip" 58 | - "0.0.0.0" 59 | - "--replSet" 60 | - "MainRepSet" 61 | - "--auth" 62 | - "--clusterAuthMode" 63 | - "keyFile" 64 | - "--keyFile" 65 | - "/etc/secrets-volume/internal-auth-mongodb-keyfile" 66 | - "--setParameter" 67 | - "authenticationMechanisms=SCRAM-SHA-1" 68 | resources: 69 | requests: 70 | cpu: 0.2 71 | memory: 200Mi 72 | ports: 73 | - containerPort: 27017 74 | volumeMounts: 75 | - name: secrets-volume 76 | readOnly: true 77 | mountPath: /etc/secrets-volume 78 | - name: mongodb-persistent-storage-claim 79 | mountPath: /data/db 80 | volumeClaimTemplates: 81 | - metadata: 82 | name: mongodb-persistent-storage-claim 83 | annotations: 84 | volume.beta.kubernetes.io/storage-class: "standard" 85 | spec: 86 | accessModes: [ "ReadWriteOnce" ] 87 | resources: 88 | requests: 89 | storage: 1Gi 90 | 91 | -------------------------------------------------------------------------------- /scripts/configure_repset_auth.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | ## 3 | # Script to connect to the first Mongod instance running in a container of the 4 | # Kubernetes StatefulSet, via the Mongo Shell, to initalise a MongoDB Replica 5 | # Set and create a MongoDB admin user. 6 | # 7 | # IMPORTANT: Only run this once 3 StatefulSet mongod pods are show with status 8 | # running (to see pod status run: $ kubectl get all) 9 | ## 10 | 11 | # Check for password argument 12 | if [[ $# -eq 0 ]] ; then 13 | echo 'You must provide one argument for the password of the "main_admin" user to be created' 14 | echo ' Usage: configure_repset_auth.sh MyPa55wd123' 15 | echo 16 | exit 1 17 | fi 18 | 19 | # Initiate replica set configuration 20 | echo "Configuring the MongoDB Replica Set" 21 | kubectl exec mongod-0 -c mongod-container -- mongo --eval 'rs.initiate({_id: "MainRepSet", version: 1, members: [ {_id: 0, host: "mongod-0.mongodb-service.default.svc.cluster.local:27017"}, {_id: 1, host: "mongod-1.mongodb-service.default.svc.cluster.local:27017"}, {_id: 2, host: "mongod-2.mongodb-service.default.svc.cluster.local:27017"} ]});' 22 | 23 | # Wait a bit until the replica set should have a primary ready 24 | echo "Waiting for the Replica Set to initialise..." 25 | sleep 30 26 | kubectl exec mongod-0 -c mongod-container -- mongo --eval 'rs.status();' 27 | 28 | # Create the admin user (this will automatically disable the localhost exception) 29 | echo "Creating user: 'main_admin'" 30 | kubectl exec mongod-0 -c mongod-container -- mongo --eval 'db.getSiblingDB("admin").createUser({user:"main_admin",pwd:"'"${1}"'",roles:[{role:"root",db:"admin"}]});' 31 | echo 32 | 33 | -------------------------------------------------------------------------------- /scripts/delete_service.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | ## 3 | # Script to just undeploy the MongoDB Service & StatefulSet but nothing else. 4 | ## 5 | 6 | # Just delete mongod stateful set + mongodb service onlys (keep rest of k8s environment in place) 7 | kubectl delete statefulsets mongod 8 | kubectl delete services mongodb-service 9 | 10 | # Show persistent volume claims are still reserved even though mongod stateful-set has been undeployed 11 | kubectl get persistentvolumes 12 | 13 | -------------------------------------------------------------------------------- /scripts/generate.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | ## 3 | # Script to deploy a Kubernetes project with a StatefulSet running a MongoDB Replica Set, to a local Minikube environment. 4 | ## 5 | 6 | # Create keyfile for the MongoD cluster as a Kubernetes shared secret 7 | TMPFILE=$(mktemp) 8 | /usr/bin/openssl rand -base64 741 > $TMPFILE 9 | kubectl create secret generic shared-bootstrap-data --from-file=internal-auth-mongodb-keyfile=$TMPFILE 10 | rm $TMPFILE 11 | 12 | # Create mongodb service with mongod stateful-set 13 | # TODO: Temporarily added no-valudate due to k8s 1.8 bug: https://github.com/kubernetes/kubernetes/issues/53309 14 | kubectl apply -f ../resources/mongodb-service.yaml --validate=false 15 | sleep 5 16 | 17 | # Print current deployment state (unlikely to be finished yet) 18 | kubectl get all 19 | kubectl get persistentvolumes 20 | echo 21 | echo "Keep running the following command until all 'mongod-n' pods are shown as running: kubectl get all" 22 | echo 23 | 24 | -------------------------------------------------------------------------------- /scripts/recreate_service.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | ## 3 | # Script to just deploy the MongoDB Service & StatefulSet back onto the exising Kubernetes cluster. 4 | ## 5 | 6 | # Show persistent volume claims are still reserved even though mongod stateful-set not deployed 7 | kubectl get persistentvolumes 8 | 9 | # Deploy just the mongodb service with mongod stateful-set only 10 | kubectl apply -f ../resources/mongodb-service.yaml 11 | sleep 5 12 | 13 | # Print current deployment state (unlikely to be finished yet) 14 | kubectl get all 15 | kubectl get persistentvolumes 16 | echo 17 | echo "Keep running the following command until all 'mongod-n' pods are shown as running: kubectl get all" 18 | echo 19 | 20 | -------------------------------------------------------------------------------- /scripts/teardown.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | ## 3 | # Script to remove/undepoy all project resources from the local Minikube environment. 4 | ## 5 | 6 | # Delete mongod stateful set + mongodb service + secrets + host vm configuer daemonset 7 | kubectl delete statefulsets mongod 8 | kubectl delete services mongodb-service 9 | kubectl delete secret shared-bootstrap-data 10 | sleep 3 11 | 12 | # Delete persistent volume claims 13 | kubectl delete persistentvolumeclaims -l role=mongo 14 | 15 | --------------------------------------------------------------------------------