├── resources ├── gce-ssd-storageclass.yaml ├── xfs-gce-ssd-persistentvolume.yaml ├── hostvm-node-configurer-daemonset.yaml └── mongodb-service.yaml ├── scripts ├── delete_service.sh ├── recreate_service.sh ├── teardown.sh ├── configure_repset_auth.sh └── generate.sh ├── LICENSE └── README.md /resources/gce-ssd-storageclass.yaml: -------------------------------------------------------------------------------- 1 | kind: StorageClass 2 | apiVersion: storage.k8s.io/v1beta1 3 | metadata: 4 | name: fast 5 | provisioner: kubernetes.io/gce-pd 6 | parameters: 7 | type: pd-ssd 8 | 9 | -------------------------------------------------------------------------------- /resources/xfs-gce-ssd-persistentvolume.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: "v1" 2 | kind: "PersistentVolume" 3 | metadata: 4 | name: data-volume-INST 5 | spec: 6 | capacity: 7 | storage: 10Gi 8 | accessModes: 9 | - ReadWriteOnce 10 | persistentVolumeReclaimPolicy: Retain 11 | storageClassName: fast 12 | gcePersistentDisk: 13 | fsType: xfs 14 | pdName: pd-ssd-disk-INST 15 | 16 | -------------------------------------------------------------------------------- /scripts/delete_service.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | ## 3 | # Script to just undeploy the MongoDB Service & StatefulSet but nothing else. 4 | ## 5 | 6 | # Just delete mongod stateful set + mongodb service onlys (keep rest of k8s environment in place) 7 | kubectl delete statefulsets mongod 8 | kubectl delete services mongodb-service 9 | 10 | # Show persistent volume claims are still reserved even though mongod stateful-set has been undeployed 11 | kubectl get persistentvolumes 12 | 13 | -------------------------------------------------------------------------------- /scripts/recreate_service.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | ## 3 | # Script to just deploy the MongoDB Service & StatefulSet back onto the exising Kubernetes cluster. 4 | ## 5 | 6 | # Show persistent volume claims are still reserved even though mongod stateful-set not deployed 7 | kubectl get persistentvolumes 8 | 9 | # Deploy just the mongodb service with mongod stateful-set only 10 | kubectl apply -f ../resources/mongodb-service.yaml 11 | sleep 5 12 | 13 | # Print current deployment state (unlikely to be finished yet) 14 | kubectl get all 15 | kubectl get persistentvolumes 16 | echo 17 | echo "Keep running the following command until all 'mongod-n' pods are shown as running: kubectl get all" 18 | echo 19 | 20 | -------------------------------------------------------------------------------- /resources/hostvm-node-configurer-daemonset.yaml: -------------------------------------------------------------------------------- 1 | kind: DaemonSet 2 | apiVersion: extensions/v1beta1 3 | metadata: 4 | name: hostvm-configurer 5 | labels: 6 | app: startup-script 7 | spec: 8 | template: 9 | metadata: 10 | labels: 11 | app: startup-script 12 | spec: 13 | hostPID: true 14 | containers: 15 | - name: hostvm-configurer-container 16 | image: gcr.io/google-containers/startup-script:v1 17 | securityContext: 18 | privileged: true 19 | env: 20 | - name: STARTUP_SCRIPT 21 | value: | 22 | #! /bin/bash 23 | set -o errexit 24 | set -o pipefail 25 | set -o nounset 26 | 27 | # Disable hugepages 28 | echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled 29 | echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag 30 | 31 | -------------------------------------------------------------------------------- /scripts/teardown.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | ## 3 | # Script to remove/undepoy all project resources from GKE & GCE. 4 | ## 5 | 6 | # Delete mongod stateful set + mongodb service + secrets + host vm configuer daemonset 7 | kubectl delete statefulsets mongod 8 | kubectl delete services mongodb-service 9 | kubectl delete secret shared-bootstrap-data 10 | kubectl delete daemonset hostvm-configurer 11 | sleep 3 12 | 13 | # Delete persistent volume claims 14 | kubectl delete persistentvolumeclaims -l role=mongo 15 | sleep 3 16 | 17 | # Delete persistent volumes 18 | for i in 1 2 3 19 | do 20 | kubectl delete persistentvolumes data-volume-$i 21 | done 22 | sleep 20 23 | 24 | # Delete GCE disks 25 | for i in 1 2 3 26 | do 27 | gcloud -q compute disks delete pd-ssd-disk-$i 28 | done 29 | 30 | # Delete whole Kubernetes cluster (including its VM instances) 31 | gcloud -q container clusters delete "gke-mongodb-demo-cluster" 32 | 33 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Paul Done 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /scripts/configure_repset_auth.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | ## 3 | # Script to connect to the first Mongod instance running in a container of the 4 | # Kubernetes StatefulSet, via the Mongo Shell, to initalise a MongoDB Replica 5 | # Set and create a MongoDB admin user. 6 | # 7 | # IMPORTANT: Only run this once all 3 StatefulSet mongod pods are shown with 8 | # status running (to see pod status run: $ kubectl get all) 9 | ## 10 | 11 | # Check for password argument 12 | if [[ $# -eq 0 ]] ; then 13 | echo 'You must provide one argument for the password of the "main_admin" user to be created' 14 | echo ' Usage: configure_repset_auth.sh MyPa55wd123' 15 | echo 16 | exit 1 17 | fi 18 | 19 | # Initiate MongoDB Replica Set configuration 20 | echo "Configuring the MongoDB Replica Set" 21 | kubectl exec mongod-0 -c mongod-container -- mongo --eval 'rs.initiate({_id: "MainRepSet", version: 1, members: [ {_id: 0, host: "mongod-0.mongodb-service.default.svc.cluster.local:27017"}, {_id: 1, host: "mongod-1.mongodb-service.default.svc.cluster.local:27017"}, {_id: 2, host: "mongod-2.mongodb-service.default.svc.cluster.local:27017"} ]});' 22 | echo 23 | 24 | # Wait for the MongoDB Replica Set to have a primary ready 25 | echo "Waiting for the MongoDB Replica Set to initialise..." 26 | kubectl exec mongod-0 -c mongod-container -- mongo --eval 'while (rs.status().hasOwnProperty("myState") && rs.status().myState != 1) { print("."); sleep(1000); };' 27 | #sleep 2 # Just a little more sleep to ensure everything is ready! 28 | sleep 20 # More sleep to ensure everything is ready! (3.6.0 workaround for https://jira.mongodb.org/browse/SERVER-31916 ) 29 | echo "...initialisation of MongoDB Replica Set completed" 30 | echo 31 | 32 | # Create the admin user (this will automatically disable the localhost exception) 33 | echo "Creating user: 'main_admin'" 34 | kubectl exec mongod-0 -c mongod-container -- mongo --eval 'db.getSiblingDB("admin").createUser({user:"main_admin",pwd:"'"${1}"'",roles:[{role:"root",db:"admin"}]});' 35 | echo 36 | 37 | -------------------------------------------------------------------------------- /scripts/generate.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | ## 3 | # Script to deploy a Kubernetes project with a StatefulSet running a MongoDB Replica Set, to GKE. 4 | ## 5 | 6 | # Create new GKE Kubernetes cluster (using host node VM images based on Ubuntu 7 | # rather than ChromiumOS default & also use slightly larger VMs than default) 8 | gcloud container clusters create "gke-mongodb-demo-cluster" --image-type=UBUNTU --machine-type=n1-standard-2 9 | 10 | # Configure host VM using daemonset to disable hugepages 11 | kubectl apply -f ../resources/hostvm-node-configurer-daemonset.yaml 12 | 13 | # Define storage class for dynamically generated persistent volumes 14 | # NOT USED IN THIS EXAMPLE AS EXPLICITLY CREATING DISKS FOR USE BY PERSISTENT 15 | # VOLUMES, HENCE COMMENTED OUT BELOW 16 | #kubectl apply -f ../resources/gce-ssd-storageclass.yaml 17 | 18 | # Register GCE Fast SSD persistent disks and then create the persistent disks 19 | echo "Creating GCE disks" 20 | for i in 1 2 3 21 | do 22 | gcloud compute disks create --size 10GB --type pd-ssd pd-ssd-disk-$i 23 | done 24 | sleep 3 25 | 26 | # Create persistent volumes using disks created above 27 | echo "Creating GKE Persistent Volumes" 28 | for i in 1 2 3 29 | do 30 | sed -e "s/INST/${i}/g" ../resources/xfs-gce-ssd-persistentvolume.yaml > /tmp/xfs-gce-ssd-persistentvolume.yaml 31 | kubectl apply -f /tmp/xfs-gce-ssd-persistentvolume.yaml 32 | done 33 | rm /tmp/xfs-gce-ssd-persistentvolume.yaml 34 | sleep 3 35 | 36 | # Create keyfile for the MongoD cluster as a Kubernetes shared secret 37 | TMPFILE=$(mktemp) 38 | /usr/bin/openssl rand -base64 741 > $TMPFILE 39 | kubectl create secret generic shared-bootstrap-data --from-file=internal-auth-mongodb-keyfile=$TMPFILE 40 | rm $TMPFILE 41 | 42 | # Create mongodb service with mongod stateful-set 43 | kubectl apply -f ../resources/mongodb-service.yaml 44 | echo 45 | 46 | # Wait until the final (3rd) mongod has started properly 47 | echo "Waiting for the 3 containers to come up (`date`)..." 48 | echo " (IGNORE any reported not found & connection errors)" 49 | sleep 30 50 | echo -n " " 51 | until kubectl --v=0 exec mongod-2 -c mongod-container -- mongo --quiet --eval 'db.getMongo()'; do 52 | sleep 5 53 | echo -n " " 54 | done 55 | echo "...mongod containers are now running (`date`)" 56 | echo 57 | 58 | # Print current deployment state 59 | kubectl get persistentvolumes 60 | echo 61 | kubectl get all 62 | 63 | -------------------------------------------------------------------------------- /resources/mongodb-service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: mongodb-service 5 | labels: 6 | name: mongo 7 | spec: 8 | ports: 9 | - port: 27017 10 | targetPort: 27017 11 | clusterIP: None 12 | selector: 13 | role: mongo 14 | --- 15 | apiVersion: apps/v1beta1 16 | kind: StatefulSet 17 | metadata: 18 | name: mongod 19 | spec: 20 | serviceName: mongodb-service 21 | replicas: 3 22 | template: 23 | metadata: 24 | labels: 25 | role: mongo 26 | environment: test 27 | replicaset: MainRepSet 28 | spec: 29 | affinity: 30 | podAntiAffinity: 31 | preferredDuringSchedulingIgnoredDuringExecution: 32 | - weight: 100 33 | podAffinityTerm: 34 | labelSelector: 35 | matchExpressions: 36 | - key: replicaset 37 | operator: In 38 | values: 39 | - MainRepSet 40 | topologyKey: kubernetes.io/hostname 41 | terminationGracePeriodSeconds: 10 42 | volumes: 43 | - name: secrets-volume 44 | secret: 45 | secretName: shared-bootstrap-data 46 | defaultMode: 256 47 | containers: 48 | - name: mongod-container 49 | #image: pkdone/mongo-ent:3.4 50 | image: mongo 51 | command: 52 | - "numactl" 53 | - "--interleave=all" 54 | - "mongod" 55 | - "--wiredTigerCacheSizeGB" 56 | - "0.25" 57 | - "--bind_ip" 58 | - "0.0.0.0" 59 | - "--replSet" 60 | - "MainRepSet" 61 | - "--auth" 62 | - "--clusterAuthMode" 63 | - "keyFile" 64 | - "--keyFile" 65 | - "/etc/secrets-volume/internal-auth-mongodb-keyfile" 66 | - "--setParameter" 67 | - "authenticationMechanisms=SCRAM-SHA-1" 68 | resources: 69 | requests: 70 | cpu: 1 71 | memory: 2Gi 72 | ports: 73 | - containerPort: 27017 74 | volumeMounts: 75 | - name: secrets-volume 76 | readOnly: true 77 | mountPath: /etc/secrets-volume 78 | - name: mongodb-persistent-storage-claim 79 | mountPath: /data/db 80 | volumeClaimTemplates: 81 | - metadata: 82 | name: mongodb-persistent-storage-claim 83 | annotations: 84 | volume.beta.kubernetes.io/storage-class: "fast" 85 | spec: 86 | accessModes: [ "ReadWriteOnce" ] 87 | resources: 88 | requests: 89 | storage: 10Gi 90 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MongoDB Deployment Demo for Kubernetes on GKE 2 | 3 | An example project demonstrating the deployment of a MongoDB Replica Set via Kubernetes on the Google Kubernetes Engine (GKE). Contains example Kubernetes YAML resource files (in the 'resource' folder) and associated Kubernetes based Bash scripts (in the 'scripts' folder) to configure the environment and deploy a MongoDB Replica Set. 4 | 5 | For further background information on what these scripts and resource files do, plus general information about running MongoDB with Kubernetes, see: [http://k8smongodb.net/](http://k8smongodb.net/) 6 | 7 | 8 | ## 1 How To Run 9 | 10 | ### 1.1 Prerequisites 11 | 12 | Ensure the following dependencies are already fulfilled on your host Linux/Windows/Mac Workstation/Laptop: 13 | 14 | 1. An account has been registered with the Google Compute Platform (GCP). You can sign up to a [free trial](https://cloud.google.com/free/) for GCP. Note: The free trial places some restrictions on account resource quotas, in particular restricting storage to a maximum of 100GB. 15 | 2. GCP’s client command line tool [gcloud](https://cloud.google.com/sdk/docs/quickstarts) has been installed. 16 | 3. Your local workstation has been initialised to: (1) use your GCP account, (2) install the Kubernetes command tool (“kubectl”), (3) configure authentication credentials, and (4) set the default GCP zone to be deployed to: 17 | 18 | ``` 19 | $ gcloud init 20 | $ gcloud components install kubectl 21 | $ gcloud auth application-default login 22 | $ gcloud config set compute/zone europe-west1-b 23 | ``` 24 | 25 | **Note:** To specify an alternative zone to deploy to, in the above command, you can first view the list of available zones by running the command: `$ gcloud compute zones list` 26 | 27 | ### 1.2 Main Deployment Steps 28 | 29 | 1. To create a Kubernetes cluster, create the required disk storage (and associated PersistentVolumes), and deploy the MongoDB Service (including the StatefulSet running "mongod" containers), via a command-line terminal/shell (ensure the script files are set to be executable): 30 | 31 | ``` 32 | $ cd scripts 33 | $ ./generate.sh 34 | ``` 35 | 36 | 2. Execute the following script which connects to the first Mongod instance running in a container of the Kubernetes StatefulSet, via the Mongo Shell, to (1) initialise the MongoDB Replica Set, and (2) create a MongoDB admin user (specify the password you want as the argument to the script, replacing 'abc123'). 37 | 38 | ``` 39 | $ ./configure_repset_auth.sh abc123 40 | ``` 41 | 42 | You should now have a MongoDB Replica Set initialised, secured and running in a Kubernetes Stateful Set. You can view the list of Pods that contain these MongoDB resources, by running the following: 43 | 44 | $ kubectl get pods 45 | 46 | You can also view the the state of the deployed environment via the [Google Cloud Platform Console](https://console.cloud.google.com) (look at both the “Kubernetes Engine” and the “Compute Engine” sections of the Console). 47 | 48 | The running replica set members will be accessible to any "app tier" containers, that are running in the same Kubernetes cluster, via the following hostnames and ports (remember to also specify the username and password, when connecting to the database): 49 | 50 | mongod-0.mongodb-service.default.svc.cluster.local:27017 51 | mongod-1.mongodb-service.default.svc.cluster.local:27017 52 | mongod-2.mongodb-service.default.svc.cluster.local:27017 53 | 54 | ### 1.3 Example Tests To Run To Check Things Are Working 55 | 56 | Use this section to prove: 57 | 58 | 1. Data is being replicated between members of the containerised replica set. 59 | 2. Data is retained even when the MongoDB Service/StatefulSet is removed and then re-created (by virtue of re-using the same Persistent Volume Claims). 60 | 61 | #### 1.3.1 Replication Test 62 | 63 | Connect to the container running the first "mongod" replica, then use the Mongo Shell to authenticate and add some test data to a database: 64 | 65 | $ kubectl exec -it mongod-0 -c mongod-container bash 66 | $ mongo 67 | > db.getSiblingDB('admin').auth("main_admin", "abc123"); 68 | > use test; 69 | > db.testcoll.insert({a:1}); 70 | > db.testcoll.insert({b:2}); 71 | > db.testcoll.find(); 72 | 73 | Exit out of the shell and exit out of the first container (“mongod-0”). Then connect to the second container (“mongod-1”), run the Mongo Shell again and see if the previously inserted data is visible to the second "mongod" replica: 74 | 75 | $ kubectl exec -it mongod-1 -c mongod-container bash 76 | $ mongo 77 | > db.getSiblingDB('admin').auth("main_admin", "abc123"); 78 | > use test; 79 | > db.setSlaveOk(1); 80 | > db.testcoll.find(); 81 | 82 | You should see that the two records inserted via the first replica, are visible to the second replica. 83 | 84 | #### 1.3.2 Redeployment Without Data Loss Test 85 | 86 | To see if Persistent Volume Claims really are working, run a script to drop the Service & StatefulSet (thus stopping the pods and their “mongod” containers) and then a script to re-create them again: 87 | 88 | $ ./delete_service.sh 89 | $ ./recreate_service.sh 90 | $ kubectl get all 91 | 92 | Keep re-running the final command above, until you can see that all 3 “mongod” pods and their containers have been successfully started again. Then connect to the first container, run the Mongo Shell and query to see if the data we’d inserted into the old containerised replica-set is still present in the re-instantiated replica set: 93 | 94 | $ kubectl exec -it mongod-0 -c mongod-container bash 95 | $ mongo 96 | > db.getSiblingDB('admin').auth("main_admin", "abc123"); 97 | > use test; 98 | > db.testcoll.find(); 99 | 100 | You should see that the two records inserted earlier, are still present. 101 | 102 | ### 1.4 Undeploying & Cleaning Down the Kubernetes Environment 103 | 104 | **Important:** This step is required to ensure you aren't continuously charged by Google Cloud for an environment you no longer need. 105 | 106 | Run the following script to undeploy the MongoDB Service & StatefulSet plus related Kubernetes resources, followed by the removal of the GCE disks before finally deleting the GKE Kubernetes cluster. 107 | 108 | $ ./teardown.sh 109 | 110 | It is also worth checking in the [Google Cloud Platform Console](https://console.cloud.google.com), to ensure all resources have been removed correctly. 111 | 112 | 113 | ## 2 Factors Addressed By This Project 114 | 115 | * Deployment of a MongoDB on the Google Kubernetes Engine 116 | * Use of Kubernetes StatefulSets and PersistentVolumeClaims to ensure data is not lost when containers are recycled 117 | * Proper configuration of a MongoDB Replica Set for full resiliency 118 | * Securing MongoDB by default for new deployments 119 | * Leveraging XFS filesystem for data file storage to improve performance 120 | * Disabling Transparent Huge Pages to improve performance 121 | * Disabling NUMA to improve performance 122 | * Controlling CPU & RAM Resource Allocation 123 | * Correctly configuring WiredTiger Cache Size in containers 124 | * Controlling Anti-Affinity for Mongod Replicas to avoid a Single Point of Failure 125 | 126 | 127 | --------------------------------------------------------------------------------