├── .gitignore ├── .prettierrc ├── LICENSE ├── README.md ├── docs └── gcp.md ├── example1-single-docker-gcloud └── README.md ├── example2-kubernetes-terraform ├── README.md ├── kubernetes │ ├── deployment.yaml │ ├── ingress.yaml │ └── service.yaml └── terraform │ ├── main.tf │ ├── modules │ ├── cluster │ │ ├── main.tf │ │ ├── outputs.tf │ │ └── variables.tf │ └── network │ │ ├── main.tf │ │ ├── outputs.tf │ │ └── variables.tf │ ├── outputs.tf │ ├── secrets.auto.tfvars.example │ └── variables.tf ├── example3-autoscaling-services └── README.md └── images └── create_service_account.png /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | /config/account.json 3 | **/.terraform 4 | secrets.auto.tfvars 5 | terraform.tfstate 6 | terraform.tfstate.backup 7 | 8 | -------------------------------------------------------------------------------- /.prettierrc: -------------------------------------------------------------------------------- 1 | singleQuote: true 2 | trailingComma: all 3 | proseWrap: always 4 | overrides: 5 | - files: "*.md" 6 | options: 7 | parser: markdown 8 | - files: "*.json" 9 | options: 10 | parser: json 11 | 12 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Daniel Demmel 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Running Kubernetes on Google Cloud Platform using Terraform 2 | 3 | ## Terraform 4 | 5 | Infrastructure is managed using [Terraform][1] via the [Google Cloud 6 | provider][2], which handles resources such as: 7 | 8 | * CloudDNS and compute addresses 9 | * Container clusters 10 | * Networking 11 | 12 | ## Kubernetes 13 | 14 | Cluster components and some cluster-specific infrastructure resources are 15 | managed through Kubernetes: 16 | 17 | * App deployments, services and ingress 18 | * GCP load balancers (via ingress) 19 | * Cache stateful sets 20 | * Secrets for the app and SSL certificates 21 | * Autoscaling 22 | 23 | ## Prerequisites 24 | 25 | ### Google Cloud Platform project 26 | 27 | You need to have a service account with Project / Editor role and the GKE API 28 | enabled in the project. 29 | 30 | Create a new service account key and download as `account.json` to `config`. 31 | 32 | [More info and step by step guide with screenshots here.][3] 33 | 34 | ### Google Cloud command-line tool, `gcloud` 35 | 36 | You can either install it from the [official source][6] or using Homebrew Cask 37 | if you’re on a Mac: 38 | 39 | ```sh 40 | brew tap caskroom/cask 41 | brew cask install google-cloud-sdk 42 | ``` 43 | 44 | Once it’s installed, log in and set it up for the project: 45 | 46 | ```sh 47 | gcloud components update 48 | gcloud auth application-default login 49 | gcloud config set compute/zone europe-west2-b 50 | ``` 51 | 52 | ### Terraform 53 | 54 | * Terraform CLI 0.11+ 55 | * A GCP service account key 56 | 57 | Download the Terraform CLI [from their site][4], or install it using 58 | [Homebrew][5] is you’re on a Mac: 59 | 60 | ```sh 61 | brew install terraform 62 | ``` 63 | 64 | ### Kubernetes command-line tool, `kubectl` 65 | 66 | You can either install it from the [official source][7] or using Homebrew if 67 | you’re on a Mac: 68 | 69 | ```sh 70 | brew install kubernetes-cli 71 | ``` 72 | 73 | ## Exercise 1 - Deploying a single Docker image to GCP with `gcloud` 74 | 75 | This exercise gets us started with deploying a single Docker image to GCP using 76 | the `gcloud` CLI. 77 | 78 | [Let’s do it, onwards to Exercise 1!][8] 79 | 80 | ## Exercise 2 - Deploying a single Docker image to GCP with Terraform 81 | 82 | We’re ready to take on Kubernetes with Terraform. 83 | 84 | [Let’s do it, onwards to Exercise 2!][9] 85 | 86 | ## Exercise 3 - Deploying an autoscaling Kubernetes services to GCP with Terraform 87 | 88 | Autoscaling 89 | 90 | [Let’s do it, onwards to Exercise 3!][10] 91 | 92 | [1]: https://www.terraform.io/ 93 | [2]: https://www.terraform.io/docs/providers/google/ 94 | [3]: ./docs/gcp.md 95 | [4]: https://www.terraform.io/downloads.html 96 | [5]: https://brew.sh/ 97 | [6]: https://cloud.google.com/sdk/ 98 | [7]: https://kubernetes.io/docs/tasks/tools/install-kubectl/ 99 | [8]: ./example1-single-docker-gcloud 100 | [9]: ./example2-kubernetes-terraform 101 | [10]: ./example3-autoscaling-services 102 | -------------------------------------------------------------------------------- /docs/gcp.md: -------------------------------------------------------------------------------- 1 | Create a new service account key from the GCP [API Manager Credentials page][1], 2 | selecting “Compute Engine default service account” as the Service account and 3 | JSON as the key type. Make sure it has the Role set as Project / Editor. 4 | 5 | In a default project setup currently you need to explicitly [enable Google 6 | Kubernetes Engine API][2]. 7 | 8 | ![create_service_account.png][3] 9 | 10 | Move and rename the JSON key file into the project `config` directory, eg: 11 | 12 | ```sh 13 | mv ~/Downloads/xyz-f310a0a6a0bf.json ~/workspace/terraform-kubernetes-on-gcp/config/account.json 14 | ``` 15 | 16 | Make sure the file is named `account.json` as the Terraform config we are 17 | creating later will assume that file to exist. 18 | 19 | > DO NOT check the `account.json` file into source-control (it’s already in the 20 | > project's `.gitignore` file to help). 21 | 22 | [1]: https://console.cloud.google.com/apis/credentials/serviceaccountkey 23 | [2]: https://console.developers.google.com/apis/library/container.googleapis.com/ 24 | [3]: ../images/create_service_account.png 25 | -------------------------------------------------------------------------------- /example1-single-docker-gcloud/README.md: -------------------------------------------------------------------------------- 1 | ## Prepare 2 | 3 | First of all make sure you're in the right directory: 4 | 5 | ```sh 6 | cd example1-single-docker-gcloud 7 | ``` 8 | 9 | ## Setup 10 | 11 | Switch to your GCP project (using its ID) and pick a zone: 12 | 13 | ```sh 14 | gcloud config set project playground-195011 15 | gcloud config set compute/zone us-west1-b 16 | ``` 17 | 18 | ## Hello world deployment 19 | 20 | ```sh 21 | gcloud compute firewall-rules create allow-http --allow tcp:80 --target-tags http-server 22 | gcloud beta compute instances create-with-container nginx-docker --machine-type g1-small --tags http-server --container-image docker.io/nginxdemos/hello 23 | ``` 24 | 25 | You should see some output similar to this: 26 | 27 | ``` 28 | Creating firewall.../Created [https://www.googleapis.com/compute/v1/projects/playground-195011/global/firewalls/allow-http]. 29 | Creating firewall...done. 30 | NAME NETWORK DIRECTION PRIORITY ALLOW DENY 31 | allow-http default INGRESS 1000 tcp:80 32 | 33 | ... 34 | 35 | API [compute.googleapis.com] not enabled on project [123412341234]. 36 | Would you like to enable and retry? (Y/n)? y 37 | 38 | Enabling service compute.googleapis.com on project 123412341234... 39 | Waiting for async operation operations/tmo-acf.fdd413b0-83f5-4f5b-afc4-b0161672829d to complete... 40 | Operation finished successfully. The following command can describe the Operation details: 41 | gcloud service-management operations describe operations/tmo-acf.fdd413b0-83f5-4f5b-afc4-b0161672829d 42 | Created [https://www.googleapis.com/compute/beta/projects/playground-195011/zones/us-west1-b/instances/nginx-docker]. 43 | NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS 44 | nginx-docker us-west1-b g1-small 10.138.0.2 35.230.5.84 RUNNING 45 | ``` 46 | 47 | Now you can open the `EXTERNAL_IP` from your output in a browser window and you 48 | should see the NGINX "Hello World" page. 49 | 50 | Once you're finished basking in the glory of your first deployment, let's delete 51 | it and move on 😺 52 | 53 | ```sh 54 | gcloud compute instances delete nginx-docker 55 | ``` 56 | 57 | You should see some output similar to this: 58 | 59 | ``` 60 | The following instances will be deleted. Any attached disks configured 61 | to be auto-deleted will be deleted unless they are attached to any 62 | other instances or the `--keep-disks` flag is given and specifies them 63 | for keeping. Deleting a disk is irreversible and any data on the disk 64 | will be lost. 65 | - [nginx-vm] in [us-west1-b] 66 | 67 | Do you want to continue (Y/n)? y 68 | 69 | Deleted [https://www.googleapis.com/compute/v1/projects/playground-195011/zones/us-west1-b/instances/nginx-docker]. 70 | ``` 71 | 72 | ## App deployment 73 | 74 | First make sure you have a fresh build (see above). 75 | 76 | Then tag it and push it to GCP Container Registry: 77 | 78 | TODO 79 | 80 | Then you're ready to update the container in the VM: 81 | 82 | TODO 83 | 84 | ## Further reading 85 | 86 | For more info on deploying containers on GCP see: 87 | https://cloud.google.com/compute/docs/containers/deploying-containers 88 | 89 | Reference for the `create-with-container` command: 90 | https://cloud.google.com/sdk/gcloud/reference/beta/compute/instances/create-with-container 91 | -------------------------------------------------------------------------------- /example2-kubernetes-terraform/README.md: -------------------------------------------------------------------------------- 1 | Unfortunately we can't easily use the equivalent of `create-with-container` 2 | using Terraform (see [Github issue][1]) so have to go the Kubernetes route right 3 | away to prevent having a lot of manual deployment steps. 4 | 5 | Create a local secrets file using the example provided (and customise the values): 6 | 7 | ```sh 8 | cp ./secrets.auto.tfvars.example ./secrets.auto.tfvars 9 | ``` 10 | 11 | Let's get Terraform initialised: 12 | 13 | ```sh 14 | cd example2-kubernetes-terraform/terraform 15 | terraform init -get=true -get-plugins=true 16 | ``` 17 | 18 | Now we can do a dry run with `plan`: 19 | 20 | ```sh 21 | terraform plan 22 | ``` 23 | 24 | You should see a big block of output similar to this: 25 | 26 | ``` 27 | Refreshing Terraform state in-memory prior to plan... 28 | The refreshed state will be used to calculate this plan, but will not be 29 | persisted to local or remote state storage. 30 | 31 | 32 | ------------------------------------------------------------------------ 33 | 34 | An execution plan has been generated and is shown below. 35 | Resource actions are indicated with the following symbols: 36 | + create 37 | 38 | Terraform will perform the following actions: 39 | 40 | + google_compute_firewall.http 41 | 42 | ... 43 | 44 | Plan: 5 to add, 0 to change, 0 to destroy. 45 | 46 | ------------------------------------------------------------------------ 47 | 48 | Note: You didn't specify an "-out" parameter to save this plan, so Terraform 49 | can't guarantee that exactly these actions will be performed if 50 | "terraform apply" is subsequently run. 51 | ``` 52 | 53 | If it's all green, we can finally `apply`: 54 | 55 | ```sh 56 | terraform apply 57 | ``` 58 | 59 | Type `yes` when prompted. 60 | 61 | If all goes well, after a few minutes you should see some nice green output 62 | like: 63 | 64 | ``` 65 | Apply complete! Resources: 5 added, 0 changed, 0 destroyed. 66 | 67 | Outputs: 68 | 69 | client_certificate = LS0t... 70 | client_key = LS0t... 71 | cluster_ca_certificate = LS0t... 72 | public_ip_address = 35.186.248.173 73 | ``` 74 | 75 | Now you should have a cluster ready for your Kubernetes deployment! Let's have a 76 | look and authenticate: 77 | 78 | ``` 79 | gcloud container clusters list 80 | gcloud container clusters get-credentials terraform-playground-cluster 81 | ``` 82 | 83 | Let's finish with simple nginx test deployment: 84 | 85 | ```sh 86 | cd ../kubernetes/ 87 | kubectl apply -f deployment.yaml 88 | kubectl apply -f service.yaml 89 | kubectl apply -f ingress.yaml 90 | ``` 91 | 92 | It might take a few minutes for everything to get set up, but eventually you should see the Nginx welcome page if you open the public IP in your browser 🎉 93 | 94 | As usual, once you get everything working, make sure to clean up so you don't get stung with ongoing charges for keeping your infrastructure running! 95 | 96 | ```sh 97 | cd ../terraform/ 98 | terraform destroy 99 | ``` 100 | 101 | [1]: https://github.com/terraform-providers/terraform-provider-google/issues/1022 102 | -------------------------------------------------------------------------------- /example2-kubernetes-terraform/kubernetes/deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1beta2 # for versions starting 1.9.0 use apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: nginx-deployment 5 | spec: 6 | selector: 7 | matchLabels: 8 | app: nginx 9 | replicas: 2 # tells deployment to run 2 pods matching the template 10 | template: # create pods using pod definition in this template 11 | metadata: 12 | # unlike pod-nginx.yaml, the name is not included in the meta data as a unique name is 13 | # generated from the deployment name 14 | labels: 15 | app: nginx 16 | spec: 17 | containers: 18 | - name: nginx 19 | image: nginx:1.7.9 20 | ports: 21 | - name: nginx-http 22 | containerPort: 80 -------------------------------------------------------------------------------- /example2-kubernetes-terraform/kubernetes/ingress.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Ingress 3 | metadata: 4 | name: nginx-ingress 5 | annotations: 6 | kubernetes.io/ingress.global-static-ip-name: terraform-playground-address 7 | kubernetes.io/ingress.allow-http: "true" 8 | spec: 9 | backend: 10 | serviceName: nginx-service 11 | servicePort: nginx-http -------------------------------------------------------------------------------- /example2-kubernetes-terraform/kubernetes/service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: nginx-service 5 | labels: 6 | app: nginx 7 | spec: 8 | type: NodePort 9 | ports: 10 | - name: nginx-http 11 | protocol: TCP 12 | port: 80 13 | selector: 14 | app: nginx -------------------------------------------------------------------------------- /example2-kubernetes-terraform/terraform/main.tf: -------------------------------------------------------------------------------- 1 | provider "google" { 2 | credentials = "${file("../../config/account.json")}" 3 | project = "${var.gcloud_project}" 4 | region = "${var.gcloud_region}" 5 | } 6 | 7 | module "network" { 8 | source = "./modules/network" 9 | 10 | gcloud_region = "${var.gcloud_region}" 11 | 12 | platform_name = "${var.platform_name}" 13 | } 14 | 15 | module "cluster" { 16 | source = "./modules/cluster" 17 | 18 | gcloud_region = "${var.gcloud_region}" 19 | gcloud_zone = "${var.gcloud_zone}" 20 | 21 | platform_name = "${var.platform_name}" 22 | 23 | network_name = "${module.network.network_name}" 24 | subnetwork_name = "${module.network.subnetwork_name}" 25 | 26 | cluster_node_machine_type = "${var.cluster_node_machine_type}" 27 | cluster_node_initial_count = "${var.cluster_node_initial_count}" 28 | 29 | cluster_master_auth_username = "${var.cluster_master_auth_username}" 30 | cluster_master_auth_password = "${var.cluster_master_auth_password}" 31 | } 32 | -------------------------------------------------------------------------------- /example2-kubernetes-terraform/terraform/modules/cluster/main.tf: -------------------------------------------------------------------------------- 1 | resource "google_container_cluster" "platform_name" { 2 | name = "${var.platform_name}-cluster" 3 | network = "${var.network_name}" 4 | subnetwork = "${var.subnetwork_name}" 5 | zone = "${var.gcloud_zone}" 6 | 7 | initial_node_count = "${var.cluster_node_initial_count}" 8 | 9 | master_auth { 10 | username = "${var.cluster_master_auth_username}" 11 | password = "${var.cluster_master_auth_password}" 12 | } 13 | 14 | node_config { 15 | machine_type = "${var.cluster_node_machine_type}" 16 | 17 | oauth_scopes = [ 18 | "https://www.googleapis.com/auth/projecthosting", 19 | "https://www.googleapis.com/auth/devstorage.full_control", 20 | "https://www.googleapis.com/auth/monitoring", 21 | "https://www.googleapis.com/auth/logging.write", 22 | "https://www.googleapis.com/auth/compute", 23 | "https://www.googleapis.com/auth/cloud-platform" 24 | ] 25 | } 26 | } 27 | -------------------------------------------------------------------------------- /example2-kubernetes-terraform/terraform/modules/cluster/outputs.tf: -------------------------------------------------------------------------------- 1 | output "cluster_name" { 2 | value = "${google_container_cluster.platform_name.name}" 3 | } 4 | 5 | output "client_certificate" { 6 | value = "${google_container_cluster.platform_name.master_auth.0.client_certificate}" 7 | } 8 | 9 | output "client_key" { 10 | value = "${google_container_cluster.platform_name.master_auth.0.client_key}" 11 | } 12 | 13 | output "cluster_ca_certificate" { 14 | value = "${google_container_cluster.platform_name.master_auth.0.cluster_ca_certificate}" 15 | } 16 | -------------------------------------------------------------------------------- /example2-kubernetes-terraform/terraform/modules/cluster/variables.tf: -------------------------------------------------------------------------------- 1 | variable "gcloud_zone" {} 2 | variable "gcloud_region" {} 3 | 4 | variable "platform_name" {} 5 | 6 | variable "network_name" {} 7 | variable "subnetwork_name" {} 8 | 9 | variable "cluster_node_machine_type" {} 10 | variable "cluster_node_initial_count" {} 11 | 12 | variable "cluster_master_auth_username" {} 13 | variable "cluster_master_auth_password" {} -------------------------------------------------------------------------------- /example2-kubernetes-terraform/terraform/modules/network/main.tf: -------------------------------------------------------------------------------- 1 | resource "google_compute_network" "platform_name" { 2 | name = "${var.platform_name}-net" 3 | } 4 | 5 | resource "google_compute_firewall" "http" { 6 | name = "${var.platform_name}-firewall-http" 7 | network = "${google_compute_network.platform_name.name}" 8 | 9 | allow { 10 | protocol = "icmp" 11 | } 12 | 13 | allow { 14 | protocol = "tcp" 15 | ports = ["80", "443"] 16 | } 17 | 18 | source_ranges = ["0.0.0.0/0"] 19 | } 20 | 21 | resource "google_compute_subnetwork" "platform_name" { 22 | name = "${var.platform_name}-${var.gcloud_region}-subnet" 23 | ip_cidr_range = "10.1.2.0/24" 24 | network = "${google_compute_network.platform_name.self_link}" 25 | region = "${var.gcloud_region}" 26 | } 27 | 28 | resource "google_compute_global_address" "platform_name" { 29 | name = "${var.platform_name}-address" 30 | } -------------------------------------------------------------------------------- /example2-kubernetes-terraform/terraform/modules/network/outputs.tf: -------------------------------------------------------------------------------- 1 | output "network_name" { 2 | value = "${google_compute_network.platform_name.name}" 3 | } 4 | 5 | output "subnetwork_name" { 6 | value = "${google_compute_subnetwork.platform_name.name}" 7 | } 8 | 9 | output "public_ip_address" { 10 | value = "${google_compute_global_address.platform_name.address}" 11 | } -------------------------------------------------------------------------------- /example2-kubernetes-terraform/terraform/modules/network/variables.tf: -------------------------------------------------------------------------------- 1 | variable "platform_name" {} 2 | variable "gcloud_region" {} 3 | -------------------------------------------------------------------------------- /example2-kubernetes-terraform/terraform/outputs.tf: -------------------------------------------------------------------------------- 1 | output "client_certificate" { 2 | value = "${module.cluster.client_certificate}" 3 | } 4 | 5 | output "client_key" { 6 | value = "${module.cluster.client_key}" 7 | } 8 | 9 | output "cluster_ca_certificate" { 10 | value = "${module.cluster.cluster_ca_certificate}" 11 | } 12 | 13 | output "public_loadbalancer_ip_address_value" { 14 | value = "${module.network.public_ip_address}" 15 | } 16 | -------------------------------------------------------------------------------- /example2-kubernetes-terraform/terraform/secrets.auto.tfvars.example: -------------------------------------------------------------------------------- 1 | cluster_master_auth_username = "username1234username" 2 | cluster_master_auth_password = "password1234password" -------------------------------------------------------------------------------- /example2-kubernetes-terraform/terraform/variables.tf: -------------------------------------------------------------------------------- 1 | variable "gcloud_region" { default = "europe-west2" } 2 | variable "gcloud_zone" { default = "europe-west2-b" } 3 | variable "gcloud_project" { default = "playground-195011" } 4 | 5 | variable "platform_name" { default = "terraform-playground" } 6 | 7 | variable "cluster_master_auth_username" {} 8 | variable "cluster_master_auth_password" {} 9 | 10 | variable "cluster_node_machine_type" { default = "g1-small" } 11 | variable "cluster_node_initial_count" { default = 2 } 12 | -------------------------------------------------------------------------------- /example3-autoscaling-services/README.md: -------------------------------------------------------------------------------- 1 | TODO 2 | 3 | ```sh 4 | cd example3-autoscaling-services 5 | terraform init -get=true -get-plugins=true 6 | ``` 7 | -------------------------------------------------------------------------------- /images/create_service_account.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/daaain/terraform-kubernetes-on-gcp/bb4bba375b465cdca736370b2a4fdac454f10779/images/create_service_account.png --------------------------------------------------------------------------------