├── .gitignore ├── README.md ├── ca-config.json ├── ca-csr.json ├── cleanup.sh ├── vault-csr.json ├── vault.env └── vault.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | *.pem 2 | *.csr 3 | vault.hcl 4 | *.swp 5 | service-account.json 6 | services 7 | vault-load-balancer.yaml 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Vault on Google Kubernetes Engine 2 | 3 | This tutorial walks you through provisioning a multi-node [HashiCorp Vault](https://www.vaultproject.io) cluster on [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine). 4 | 5 | ## Cluster Features 6 | 7 | * High Availability - The Vault cluster will be provisioned in [multi-server mode](https://www.vaultproject.io/docs/concepts/ha.html) for high availability. 8 | * Google Cloud Storage Storage Backend - Vault's data is persisted in [Google Cloud Storage](https://cloud.google.com/storage). 9 | * Production Hardening - Vault is configured and deployed based on the guidance found in the [production hardening](https://www.vaultproject.io/guides/operations/production.html) guide. 10 | * Auto Initialization and Unsealing - Vault is automatically initialized and unsealed at runtime. Keys are encrypted using [Cloud KMS](https://cloud.google.com/kms) and stored on [Google Cloud Storage](https://cloud.google.com/storage). 11 | 12 | ## Tutorial 13 | 14 | ### Create a New Project 15 | 16 | In this section you will create a new GCP project and enable the APIs required by this tutorial. 17 | 18 | Generate a project ID: 19 | 20 | ``` 21 | PROJECT_ID="vault-$(($(date +%s%d)/1000000))" 22 | ``` 23 | 24 | Create a new GCP project: 25 | 26 | ``` 27 | gcloud projects create ${PROJECT_ID} \ 28 | --name "${PROJECT_ID}" 29 | ``` 30 | 31 | [Enable billing](https://cloud.google.com/billing/docs/how-to/modify-project#enable_billing_for_a_new_project) on the new project before moving on to the next step. 32 | 33 | Enable the GCP APIs required by this tutorial: 34 | 35 | ``` 36 | gcloud services enable \ 37 | cloudapis.googleapis.com \ 38 | cloudkms.googleapis.com \ 39 | container.googleapis.com \ 40 | containerregistry.googleapis.com \ 41 | iam.googleapis.com \ 42 | --project ${PROJECT_ID} 43 | ``` 44 | 45 | ### Set Configuration 46 | 47 | ``` 48 | COMPUTE_ZONE="us-west1-c" 49 | ``` 50 | 51 | ``` 52 | COMPUTE_REGION="us-west1" 53 | ``` 54 | 55 | ``` 56 | GCS_BUCKET_NAME="${PROJECT_ID}-vault-storage" 57 | ``` 58 | 59 | ``` 60 | KMS_KEY_ID="projects/${PROJECT_ID}/locations/global/keyRings/vault/cryptoKeys/vault-init" 61 | ``` 62 | 63 | ### Create KMS Keyring and Crypto Key 64 | 65 | In this section you will create a Cloud KMS [keyring](https://cloud.google.com/kms/docs/object-hierarchy#key_ring) and [cryptographic key](https://cloud.google.com/kms/docs/object-hierarchy#key) suitable for encrypting and decrypting Vault [master keys](https://www.vaultproject.io/docs/concepts/seal.html) and [root tokens](https://www.vaultproject.io/docs/concepts/tokens.html#root-tokens). 66 | 67 | Create the `vault` kms keyring: 68 | 69 | ``` 70 | gcloud kms keyrings create vault \ 71 | --location global \ 72 | --project ${PROJECT_ID} 73 | ``` 74 | 75 | Create the `vault-init` encryption key: 76 | 77 | ``` 78 | gcloud kms keys create vault-init \ 79 | --location global \ 80 | --keyring vault \ 81 | --purpose encryption \ 82 | --project ${PROJECT_ID} 83 | ``` 84 | 85 | ### Create a Google Cloud Storage Bucket 86 | 87 | Google Cloud Storage is used to [persist Vault's data](https://www.vaultproject.io/docs/configuration/storage/google-cloud-storage.html) and hold encrypted Vault master keys and root tokens. 88 | 89 | Create a GCS bucket: 90 | 91 | ``` 92 | gsutil mb -p ${PROJECT_ID} gs://${GCS_BUCKET_NAME} 93 | ``` 94 | 95 | ### Create the Vault IAM Service Account 96 | 97 | An [IAM service account](https://cloud.google.com/iam/docs/service-accounts) is used by Vault to access the GCS bucket and KMS encryption key created in the previous sections. 98 | 99 | Create the `vault` service account: 100 | 101 | ``` 102 | gcloud iam service-accounts create vault-server \ 103 | --display-name "vault service account" \ 104 | --project ${PROJECT_ID} 105 | ``` 106 | 107 | Grant access to the vault storage bucket: 108 | 109 | ``` 110 | gsutil iam ch \ 111 | serviceAccount:vault-server@${PROJECT_ID}.iam.gserviceaccount.com:objectAdmin \ 112 | gs://${GCS_BUCKET_NAME} 113 | ``` 114 | 115 | ``` 116 | gsutil iam ch \ 117 | serviceAccount:vault-server@${PROJECT_ID}.iam.gserviceaccount.com:legacyBucketReader \ 118 | gs://${GCS_BUCKET_NAME} 119 | ``` 120 | 121 | Grant access to the `vault-init` KMS encryption key: 122 | 123 | ``` 124 | gcloud kms keys add-iam-policy-binding \ 125 | vault-init \ 126 | --location global \ 127 | --keyring vault \ 128 | --member serviceAccount:vault-server@${PROJECT_ID}.iam.gserviceaccount.com \ 129 | --role roles/cloudkms.cryptoKeyEncrypterDecrypter \ 130 | --project ${PROJECT_ID} 131 | ``` 132 | 133 | ### Provision a Kubernetes Cluster 134 | 135 | In this section you will provision a three node Kubernetes cluster using [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine) with access to the `vault-server` service account across the entire cluster. 136 | 137 | Create the `vault` Kubernetes cluster: 138 | 139 | ``` 140 | gcloud container clusters create vault \ 141 | --enable-autorepair \ 142 | --machine-type e2-standard-2 \ 143 | --service-account vault-server@${PROJECT_ID}.iam.gserviceaccount.com \ 144 | --num-nodes 3 \ 145 | --zone ${COMPUTE_ZONE} \ 146 | --project ${PROJECT_ID} 147 | ``` 148 | 149 | > Warning: Each node in the `vault` Kubernetes cluster has access to the `vault-server` service account. The `vault` cluster should only be used for running Vault. Other workloads should run on a different cluster and access Vault through an internal or external load balancer. 150 | 151 | 152 | ### Provision IP Address 153 | 154 | In this section you will create a public IP address that will be used to expose the Vault server to external clients. 155 | 156 | Create the `vault` compute address: 157 | 158 | ``` 159 | gcloud compute addresses create vault \ 160 | --region ${COMPUTE_REGION} \ 161 | --project ${PROJECT_ID} 162 | ``` 163 | 164 | Store the `vault` compute address in an environment variable: 165 | 166 | ``` 167 | VAULT_LOAD_BALANCER_IP=$(gcloud compute addresses describe vault \ 168 | --region ${COMPUTE_REGION} \ 169 | --project ${PROJECT_ID} \ 170 | --format='value(address)') 171 | ``` 172 | 173 | ### Generate TLS Certificates 174 | 175 | In this section you will generate the self-signed TLS certificates used to secure communication between Vault clients and servers. 176 | 177 | Create a Certificate Authority: 178 | 179 | ``` 180 | cfssl gencert -initca ca-csr.json | cfssljson -bare ca 181 | ``` 182 | 183 | Generate the Vault TLS certificates: 184 | 185 | ``` 186 | cfssl gencert \ 187 | -ca=ca.pem \ 188 | -ca-key=ca-key.pem \ 189 | -config=ca-config.json \ 190 | -hostname="vault,vault.default.svc.cluster.local,localhost,127.0.0.1,${VAULT_LOAD_BALANCER_IP}" \ 191 | -profile=default \ 192 | vault-csr.json | cfssljson -bare vault 193 | ``` 194 | 195 | ### Deploy Vault 196 | 197 | In this section you will deploy the multi-node Vault cluster using a collection of Kubernetes and application configuration files. 198 | 199 | Create the `vault` secret to hold the Vault TLS certificates: 200 | 201 | ``` 202 | cat vault.pem ca.pem > vault-combined.pem 203 | ``` 204 | 205 | ``` 206 | kubectl create secret generic vault \ 207 | --from-file=ca.pem \ 208 | --from-file=vault.pem=vault-combined.pem \ 209 | --from-file=vault-key.pem 210 | ``` 211 | 212 | The `vault` configmap holds the Google Cloud Platform settings required bootstrap the Vault cluster. 213 | 214 | Create the `vault` configmap: 215 | 216 | ``` 217 | kubectl create configmap vault \ 218 | --from-literal api-addr=https://${VAULT_LOAD_BALANCER_IP}:8200 \ 219 | --from-literal gcs-bucket-name=${GCS_BUCKET_NAME} \ 220 | --from-literal kms-key-id=${KMS_KEY_ID} 221 | ``` 222 | 223 | #### Create the Vault StatefulSet 224 | 225 | In this section you will create the `vault` statefulset used to provision and manage two Vault server instances. 226 | 227 | Create the `vault` statefulset: 228 | 229 | ``` 230 | kubectl apply -f vault.yaml 231 | ``` 232 | ``` 233 | service "vault" created 234 | statefulset "vault" created 235 | ``` 236 | 237 | At this point the multi-node cluster is up and running: 238 | 239 | ``` 240 | kubectl get pods 241 | ``` 242 | ``` 243 | NAME READY STATUS RESTARTS AGE 244 | vault-0 2/2 Running 0 1m 245 | vault-1 2/2 Running 0 1m 246 | ``` 247 | 248 | ### Automatic Initialization and Unsealing 249 | 250 | In a typical deployment Vault must be initialized and unsealed before it can be used. In our deployment we are using the [vault-init](https://github.com/kelseyhightower/vault-init) container to automate the initialization and unseal steps. 251 | 252 | ``` 253 | kubectl logs vault-0 -c vault-init 254 | ``` 255 | ``` 256 | 2018/11/03 22:37:35 Starting the vault-init service... 257 | 2018/11/03 22:37:35 Get https://127.0.0.1:8200/v1/sys/health: dial tcp 127.0.0.1:8200: connect: connection refused 258 | 2018/11/03 22:37:45 Vault is not initialized. Initializing and unsealing... 259 | 2018/11/03 22:37:53 Encrypting unseal keys and the root token... 260 | 2018/11/03 22:37:53 Unseal keys written to gs://vault-1541283682815-vault-storage/unseal-keys.json.enc 261 | 2018/11/03 22:37:53 Root token written to gs://vault-1541283682815-vault-storage/root-token.enc 262 | 2018/11/03 22:37:53 Initialization complete. 263 | 2018/11/03 22:37:55 Unseal complete. 264 | 2018/11/03 22:37:55 Next check in 10s 265 | 2018/11/03 22:38:05 Vault is initialized and unsealed. 266 | 2018/11/03 22:38:05 Next check in 10s 267 | ``` 268 | 269 | The `vault-init` container runs every 10 seconds and ensures each vault instance is automatically unsealed. 270 | 271 | #### Health Checks 272 | 273 | A [readiness probe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes) is used to ensure Vault instances are not routed traffic when they are [sealed](https://www.vaultproject.io/docs/concepts/seal.html). 274 | 275 | > Sealed Vault instances do not forward or redirect clients even in HA setups. 276 | 277 | ### Expose the Vault Cluster 278 | 279 | In this section you will expose the Vault cluster using an external network load balancer. 280 | 281 | Generate the `vault` service configuration: 282 | 283 | ``` 284 | cat > vault-load-balancer.yaml < 8200:31805/TCP,8201:32754/TCP 316 | ``` 317 | 318 | ### Smoke Tests 319 | 320 | Source the `vault.env` script to configure the vault CLI to use the Vault cluster via the external load balancer: 321 | 322 | ``` 323 | source vault.env 324 | ``` 325 | 326 | Get the status of the Vault cluster: 327 | 328 | ``` 329 | vault status 330 | ``` 331 | ``` 332 | Key Value 333 | --- ----- 334 | Seal Type shamir 335 | Initialized true 336 | Sealed false 337 | Total Shares 5 338 | Threshold 3 339 | Version 0.11.4 340 | Cluster Name vault-cluster-46821b83 341 | Cluster ID dcd56552-27d0-fa18-4ccc-25b252464971 342 | HA Enabled true 343 | HA Cluster https://XX.XX.X.X:8201 344 | HA Mode standby 345 | Active Node Address https://XX.XXX.XXX.XX:8200 346 | ``` 347 | 348 | #### Logging in 349 | 350 | Download and decrypt the root token: 351 | 352 | ``` 353 | export VAULT_TOKEN=$(gsutil cat gs://${GCS_BUCKET_NAME}/root-token.enc | \ 354 | base64 --decode | \ 355 | gcloud kms decrypt \ 356 | --project ${PROJECT_ID} \ 357 | --location global \ 358 | --keyring vault \ 359 | --key vault-init \ 360 | --ciphertext-file - \ 361 | --plaintext-file - 362 | ) 363 | ``` 364 | 365 | #### Working with Secrets 366 | 367 | The following examples assume Vault 0.11 or later. 368 | 369 | ``` 370 | vault secrets enable -version=2 kv 371 | ``` 372 | 373 | ``` 374 | vault kv enable-versioning secret/ 375 | ``` 376 | 377 | ``` 378 | vault kv put secret/my-secret my-value=s3cr3t 379 | ``` 380 | 381 | ``` 382 | vault kv get secret/my-secret 383 | ``` 384 | 385 | ### Clean Up 386 | 387 | Ensure you are working with the right project ID: 388 | 389 | ``` 390 | echo $PROJECT_ID 391 | ``` 392 | 393 | Delete the project: 394 | 395 | ``` 396 | gcloud projects delete ${PROJECT_ID} 397 | ``` 398 | -------------------------------------------------------------------------------- /ca-config.json: -------------------------------------------------------------------------------- 1 | { 2 | "signing": { 3 | "default": { 4 | "expiry": "8760h" 5 | }, 6 | "profiles": { 7 | "default": { 8 | "usages": ["signing", "key encipherment", "server auth", "client auth"], 9 | "expiry": "8760h" 10 | } 11 | } 12 | } 13 | } 14 | -------------------------------------------------------------------------------- /ca-csr.json: -------------------------------------------------------------------------------- 1 | { 2 | "hosts": [ 3 | "cluster.local" 4 | ], 5 | "key": { 6 | "algo": "rsa", 7 | "size": 2048 8 | }, 9 | "names": [ 10 | { 11 | "C": "US", 12 | "L": "Portland", 13 | "O": "Kubernetes", 14 | "OU": "CA", 15 | "ST": "Oregon" 16 | } 17 | ] 18 | } 19 | -------------------------------------------------------------------------------- /cleanup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | rm *.pem *.csr 4 | rm vault.hcl vault-load-balancer.yaml 5 | -------------------------------------------------------------------------------- /vault-csr.json: -------------------------------------------------------------------------------- 1 | { 2 | "CN": "vault", 3 | "key": { 4 | "algo": "rsa", 5 | "size": 2048 6 | }, 7 | "names": [ 8 | { 9 | "C": "US", 10 | "L": "Portland", 11 | "O": "Kubernetes", 12 | "OU": "Vault", 13 | "ST": "Oregon" 14 | } 15 | ] 16 | } 17 | -------------------------------------------------------------------------------- /vault.env: -------------------------------------------------------------------------------- 1 | LOAD_BALANCER_IP=$(kubectl get svc \ 2 | vault-load-balancer \ 3 | -o jsonpath={.status.loadBalancer.ingress[0].ip}) 4 | export VAULT_ADDR="https://${LOAD_BALANCER_IP}:8200" 5 | export VAULT_CACERT="ca.pem" 6 | -------------------------------------------------------------------------------- /vault.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: vault 5 | spec: 6 | clusterIP: None 7 | ports: 8 | - name: http 9 | port: 8200 10 | - name: server 11 | port: 8201 12 | selector: 13 | app: vault 14 | --- 15 | apiVersion: apps/v1 16 | kind: StatefulSet 17 | metadata: 18 | name: vault 19 | labels: 20 | app: vault 21 | spec: 22 | serviceName: "vault" 23 | selector: 24 | matchLabels: 25 | app: vault 26 | replicas: 2 27 | template: 28 | metadata: 29 | labels: 30 | app: vault 31 | spec: 32 | affinity: 33 | podAntiAffinity: 34 | requiredDuringSchedulingIgnoredDuringExecution: 35 | - labelSelector: 36 | matchExpressions: 37 | - key: app 38 | operator: In 39 | values: 40 | - vault 41 | topologyKey: kubernetes.io/hostname 42 | initContainers: 43 | - name: config 44 | image: busybox 45 | env: 46 | - name: GCS_BUCKET_NAME 47 | valueFrom: 48 | configMapKeyRef: 49 | name: vault 50 | key: gcs-bucket-name 51 | command: ["/bin/sh", "-c"] 52 | args: 53 | - | 54 | cat > /etc/vault/config/vault.hcl <