├── .gitignore ├── images ├── gocd.png ├── agent_dind.png └── kube_dind.png ├── terraform ├── .gitattributes ├── dev │ ├── variables.tf │ └── main.tf └── modules │ ├── cluster-network │ ├── outputs.tf │ ├── variables.tf │ └── main.tf │ └── container │ ├── variables.tf │ └── main.tf ├── master-cron ├── crontab ├── Dockerfile └── scripts │ └── cleanup-agents.sh ├── master ├── Dockerfile └── custom-boot.sh ├── docker-compose.override.yml ├── agent └── Dockerfile ├── kubernetes-remove.sh ├── go.env ├── kubernetes-deploy.sh ├── docker-compose.yml ├── kubernetes ├── agent.pod.yml.tmp.yml ├── agent.pod.yml ├── master.pod.yml.tmp.yml └── master.pod.yml ├── common.sh └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | *.swp 2 | .terraform 3 | -------------------------------------------------------------------------------- /images/gocd.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Stono/kube-gocd/HEAD/images/gocd.png -------------------------------------------------------------------------------- /images/agent_dind.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Stono/kube-gocd/HEAD/images/agent_dind.png -------------------------------------------------------------------------------- /images/kube_dind.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Stono/kube-gocd/HEAD/images/kube_dind.png -------------------------------------------------------------------------------- /terraform/.gitattributes: -------------------------------------------------------------------------------- 1 | * filter=git-crypt diff=git-crypt 2 | .gitattributes !filter !diff 3 | -------------------------------------------------------------------------------- /master-cron/crontab: -------------------------------------------------------------------------------- 1 | SHELL=/bin/bash 2 | PATH=/sbin:/bin:/usr/sbin:/usr/bin 3 | MAILTO=root 4 | 5 | * * * * * root /usr/local/bin/cleanup-agents.sh 6 | -------------------------------------------------------------------------------- /master/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM gocd/gocd-server:v17.6.0 2 | 3 | RUN apk add apache2-utils 4 | 5 | ENV TINI_SUBREAPER=true 6 | ADD custom-boot.sh /usr/local/bin/ 7 | CMD ["custom-boot.sh"] 8 | -------------------------------------------------------------------------------- /master-cron/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM centos:7 2 | RUN yum -y -q install jq cronie && \ 3 | yum -y -q clean all 4 | 5 | COPY crontab /etc/crontab 6 | COPY scripts/* /usr/local/bin/ 7 | CMD ["crond", "-s"] 8 | -------------------------------------------------------------------------------- /docker-compose.override.yml: -------------------------------------------------------------------------------- 1 | version: '2' 2 | 3 | services: 4 | master: 5 | build: ./master 6 | 7 | agent: 8 | build: ./agent 9 | 10 | master-cron: 11 | build: ./master-cron 12 | -------------------------------------------------------------------------------- /terraform/dev/variables.tf: -------------------------------------------------------------------------------- 1 | variable "stack" { 2 | description = "The name for the stack, used for prefixing" 3 | } 4 | 5 | variable "cluster_password" { 6 | description = "The password for logging into kubernetes ui" 7 | } 8 | -------------------------------------------------------------------------------- /terraform/modules/cluster-network/outputs.tf: -------------------------------------------------------------------------------- 1 | output "vpc_name" { 2 | value = "${google_compute_network.vpc.name}" 3 | } 4 | 5 | output "subnet_name" { 6 | value = "${google_compute_subnetwork.vpc_regional_subnet.name}" 7 | } 8 | -------------------------------------------------------------------------------- /agent/Dockerfile: -------------------------------------------------------------------------------- 1 | # Take the mtaintained dind 2 | FROM gocd/gocd-agent-centos-7:v17.6.0 3 | 4 | # Add docker to the agent 5 | ENV DOCKER_VERSION=17.03.1 6 | RUN curl --silent -O https://get.docker.com/builds/Linux/x86_64/docker-$DOCKER_VERSION-ce.tgz && \ 7 | tar xzf docker-*.tgz && \ 8 | mv docker/docker /usr/local/bin/docker && \ 9 | rm -rf docker 10 | -------------------------------------------------------------------------------- /kubernetes-remove.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source common.sh 3 | set +e 4 | 5 | echo "This action will remove all GoCD deployments!" 6 | confirm 7 | k delete -f kubernetes/master.pod.yml 8 | k delete -f kubernetes/agent.pod.yml 9 | k delete secret kube-gocd 10 | 11 | echo "Would you also like to remove all the data volumes too?" 12 | echo "WARNING: This includes your go config!" 13 | if yesno; then 14 | k delete pvc -l tier=gocd 15 | fi 16 | -------------------------------------------------------------------------------- /terraform/modules/cluster-network/variables.tf: -------------------------------------------------------------------------------- 1 | variable "stack" { 2 | description = "The name for the stack, used for prefixing" 3 | } 4 | 5 | variable "env" { 6 | description = "The environment you are creating the cluster for" 7 | } 8 | 9 | variable "subnet_range" { 10 | description = "The ip range for the subnet which kubernetes machines will be created in" 11 | } 12 | 13 | variable "subnet_region" { 14 | description = "The subnet region" 15 | } 16 | -------------------------------------------------------------------------------- /master/custom-boot.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | GO_USER_FILE=/etc/go-users 4 | GO_CONFIG_DIR=/godata/config 5 | GO_CONFIG_FILE=$GO_CONFIG_DIR/cruise-config.xml 6 | 7 | if [ ! -f "$GO_USER_FILE" ]; then 8 | echo "Creating default user file" 9 | echo $(htpasswd -nb -s $GO_USERNAME $GO_PASSWORD | xargs) > $GO_USER_FILE 10 | echo "You will need to enable password auth for $GO_USER_FILE from the GoCD GUI" 11 | else 12 | echo "$GO_USER_FILE already exists, please delete it if you wish to change passwords" 13 | fi 14 | 15 | /docker-entrypoint.sh 16 | -------------------------------------------------------------------------------- /terraform/modules/container/variables.tf: -------------------------------------------------------------------------------- 1 | variable "stack" { 2 | description = "The name for the stack, used for prefixing" 3 | } 4 | 5 | variable "env" { 6 | description = "The environment you are creating the cluster for" 7 | } 8 | 9 | variable "subnet_range" { 10 | description = "The ip range for the subnet which kubernetes machines will be created in" 11 | } 12 | 13 | variable "container_cidr_range" { 14 | description = "The CIDR range the docker containers will be DHCPd from" 15 | } 16 | 17 | variable "cluster_username" { 18 | description = "The username for logging into kubernetes ui" 19 | default = "admin" 20 | } 21 | 22 | variable "cluster_password" { 23 | description = "The password for logging into kubernetes ui" 24 | } 25 | -------------------------------------------------------------------------------- /go.env: -------------------------------------------------------------------------------- 1 | # Shared variables between gocd and agents, change these 2 | GO_USERNAME=username # Username to access GoCD 3 | GO_PASSWORD=password # Password to access GoCD 4 | AGENT_AUTO_REGISTER_KEY=some-random-key # Specify some unique key for the agents 5 | 6 | # Kubernetes options, change these 7 | KUBE_NAMESPACE=gocd # Which namespace do you want gocd in? 8 | GCP_REGISTRY_HOST=eu.gcr.io # Which regional registry shall we push to? 9 | 10 | # Agent environment, you shouldn't need to change these 11 | AGENT_AUTO_REGISTER_RESOURCES=docker 12 | AGENT_AUTO_REGISTER_ENVIRONMENTS=docker 13 | GO_SERVER_URL=https://gocd-master-internal:8154/go 14 | DOCKER_HOST=tcp://127.0.0.1:2375 15 | -------------------------------------------------------------------------------- /terraform/dev/main.tf: -------------------------------------------------------------------------------- 1 | provider "google" { 2 | credentials = "" 3 | project = "peopledata-product-team" 4 | region = "europe-west1" 5 | } 6 | 7 | # Uncomment this for remote state storage 8 | # You'll need to create the bucket first 9 | #terraform { 10 | # backend "gcs" { 11 | # bucket = "your-gcp-bucket-name" 12 | # path = "terraform.tfstate" 13 | # project = "your-project-name" 14 | # } 15 | #} 16 | 17 | module "container" { 18 | source = "../modules/container" 19 | env = "dev" 20 | stack = "${var.stack}" 21 | subnet_range = "10.34.96.0/24" # The IP range for the kubernetes VMs 22 | container_cidr_range = "10.37.64.0/19" # The IP range for the containers 23 | cluster_password = "${var.cluster_password}" # The password for the kubernetes UI 24 | } 25 | -------------------------------------------------------------------------------- /kubernetes-deploy.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source common.sh 3 | 4 | if [ ! "$1" = "--deploy-only" ]; then 5 | docker-compose build 6 | tag_and_push "kube-gocd-master" 7 | tag_and_push "kube-gocd-master-cron" 8 | tag_and_push "kube-gocd-agent" 9 | fi 10 | 11 | if ! k get namespaces | grep $KUBE_NAMESPACE &>/dev/null; then 12 | kubectl create namespace $KUBE_NAMESPACE 13 | fi 14 | 15 | if k get secrets | grep $SECRET_NAME &>/dev/null; then 16 | k delete secret $SECRET_NAME 17 | fi 18 | 19 | k create secret generic $SECRET_NAME \ 20 | --from-literal=user=$GO_USERNAME \ 21 | --from-literal=pass=$GO_PASSWORD \ 22 | --from-literal=agent_key=$AGENT_AUTO_REGISTER_KEY 23 | 24 | EXISTING=0 25 | if k get pods | grep gocd &>/dev/null; then 26 | EXISTING=1 27 | fi 28 | 29 | apply kubernetes/master.pod.yml 30 | apply kubernetes/agent.pod.yml 31 | 32 | if [ "$EXISTING" = "1" ]; then 33 | k delete pod -l tier=gocd 34 | fi 35 | -------------------------------------------------------------------------------- /terraform/modules/cluster-network/main.tf: -------------------------------------------------------------------------------- 1 | resource "google_compute_network" "vpc" { 2 | name = "${var.stack}-${var.env}-vpc" 3 | } 4 | 5 | resource "google_compute_subnetwork" "vpc_regional_subnet" { 6 | name = "${var.stack}-${var.env}-${var.subnet_region}" 7 | network = "${google_compute_network.vpc.name}" 8 | region = "${var.subnet_region}" 9 | ip_cidr_range = "${var.subnet_range}" 10 | } 11 | 12 | resource "google_compute_firewall" "standard-ports" { 13 | name = "${var.stack}-${var.env}-standard-ports" 14 | network = "${google_compute_network.vpc.name}" 15 | 16 | # Im not sure if this is used for monitoring, need to check 17 | allow { 18 | protocol = "icmp" 19 | } 20 | 21 | allow { 22 | protocol = "tcp" 23 | ports = ["22"] 24 | } 25 | 26 | # Put the IP CIDR ranges that should be able to SSH 27 | # Really, you shouldn't need to SSH on GKE, ever 28 | source_ranges = [] 29 | } 30 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '2' 2 | 3 | networks: 4 | gocd: 5 | 6 | services: 7 | master: 8 | image: stono/kube-gocd-master 9 | restart: always 10 | networks: 11 | gocd: 12 | aliases: 13 | - gocd-master-internal 14 | ports: 15 | - 8153:8153 16 | env_file: 17 | - go.env 18 | 19 | master-cron: 20 | image: stono/kube-gocd-master-cron 21 | restart: always 22 | network_mode: 'service:master' 23 | env_file: 24 | - go.env 25 | 26 | agent: 27 | image: stono/kube-gocd-agent 28 | restart: always 29 | networks: 30 | gocd: 31 | depends_on: 32 | - master 33 | volumes: 34 | - build_data:/godata 35 | env_file: 36 | - go.env 37 | 38 | dind: 39 | image: docker:stable-dind 40 | restart: always 41 | network_mode: 'service:agent' 42 | volumes: 43 | - build_data:/godata 44 | privileged: true 45 | 46 | volumes: 47 | build_data: {} 48 | -------------------------------------------------------------------------------- /terraform/modules/container/main.tf: -------------------------------------------------------------------------------- 1 | module "network" { 2 | source = "../cluster-network" 3 | env = "${var.env}" 4 | stack = "${var.stack}" 5 | subnet_range = "${var.subnet_range}" 6 | subnet_region = "europe-west1" 7 | } 8 | 9 | resource "google_container_cluster" "cluster" { 10 | depends_on = ["module.network"] 11 | name = "${var.stack}-${var.env}" 12 | cluster_ipv4_cidr = "${var.container_cidr_range}" 13 | 14 | master_auth { 15 | username = "${var.cluster_username}" 16 | password = "${var.cluster_password}" 17 | } 18 | 19 | zone = "europe-west1-c" 20 | additional_zones = ["europe-west1-d"] 21 | network = "${module.network.vpc_name}" 22 | subnetwork = "${module.network.subnet_name}" 23 | monitoring_service = "monitoring.googleapis.com" 24 | logging_service = "logging.googleapis.com" 25 | initial_node_count = 2 26 | node_version = "1.6.6" 27 | node_config { 28 | machine_type = "n1-standard-1" 29 | disk_size_gb = "100" 30 | 31 | oauth_scopes = [ 32 | "https://www.googleapis.com/auth/compute", 33 | "https://www.googleapis.com/auth/devstorage.read_write", 34 | "https://www.googleapis.com/auth/datastore", 35 | "https://www.googleapis.com/auth/logging.write", 36 | "https://www.googleapis.com/auth/monitoring" 37 | ] 38 | } 39 | } 40 | -------------------------------------------------------------------------------- /master-cron/scripts/cleanup-agents.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | API_URL="http://gocd-master-internal:8153/go/api" 4 | gocd() { 5 | curl -s -X $1 "$API_URL/$2" \ 6 | -H 'Accept: application/vnd.go.cd.v4+json' \ 7 | -H 'Content-Type: application/json' \ 8 | -u "$GO_USERNAME:$GO_PASSWORD" 9 | } 10 | 11 | get() { 12 | gocd "GET" "$1" 13 | } 14 | 15 | delete() { 16 | gocd "DELETE" "$1" 17 | } 18 | 19 | disable_agent() { 20 | curl -s -X PATCH "$API_URL/agents/$1" \ 21 | -H 'Accept: application/vnd.go.cd.v4+json' \ 22 | -H 'Content-Type: application/json' \ 23 | -u "$GO_USERNAME:$GO_PASSWORD" \ 24 | -d '{ 25 | "agent_config_state": "Disabled" 26 | }' 27 | } 28 | 29 | delete_agent() { 30 | disable_agent "$1" 31 | delete "agents/$1" 32 | } 33 | 34 | get_agents() { 35 | get "agents" 36 | } 37 | 38 | export GO_USERNAME 39 | export GO_PASSWORD 40 | export API_URL 41 | export -f delete_agent 42 | export -f disable_agent 43 | export -f get_agents 44 | export -f delete 45 | export -f get 46 | export -f gocd 47 | 48 | get_agents | jq -c '._embedded.agents[] | select(.agent_state | contains("LostContact")) | .uuid' | xargs -r -n 1 bash -i -c 'delete_agent $@' _ 49 | get_agents | jq -c '._embedded.agents[] | select(.agent_state | contains("Missing")) | .uuid' | xargs -r -n 1 bash -i -c 'delete_agent $@' _ 50 | echo "Cleanup complete." 51 | exit 0 52 | -------------------------------------------------------------------------------- /kubernetes/agent.pod.yml.tmp.yml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Deployment 3 | metadata: 4 | name: gocd-agent 5 | spec: 6 | replicas: 2 7 | revisionHistoryLimit: 1 8 | strategy: 9 | rollingUpdate: 10 | maxUnavailable: 1 11 | maxSurge: 1 12 | selector: 13 | matchLabels: 14 | app: agent 15 | tier: gocd 16 | template: 17 | metadata: 18 | labels: 19 | app: agent 20 | tier: gocd 21 | spec: 22 | volumes: 23 | - name: build-data 24 | emptyDir: {} 25 | restartPolicy: Always 26 | containers: 27 | - name: dind 28 | securityContext: 29 | privileged: true 30 | image: docker:stable-dind 31 | imagePullPolicy: Always 32 | volumeMounts: 33 | - name: build-data 34 | mountPath: /godata 35 | - name: agent 36 | image: eu.gcr.io/peopledata-product-team/kube-gocd-agent:latest 37 | imagePullPolicy: Always 38 | volumeMounts: 39 | - name: build-data 40 | mountPath: /godata 41 | env: 42 | - name: DOCKER_HOST 43 | value: 'tcp://127.0.0.1:2375' 44 | - name: GO_SERVER_URL 45 | value: 'https://gocd-master-internal:8154/go' 46 | - name: AGENT_AUTO_REGISTER_KEY 47 | valueFrom: 48 | secretKeyRef: 49 | name: kube-gocd 50 | key: agent_key 51 | - name: AGENT_AUTO_REGISTER_RESOURCES 52 | value: 53 | - name: AGENT_AUTO_REGISTER_ENVIRONMENTS 54 | value: 55 | -------------------------------------------------------------------------------- /kubernetes/agent.pod.yml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1beta1 2 | kind: StatefulSet 3 | metadata: 4 | name: gocd-agent 5 | spec: 6 | serviceName: gocd-agent-headless 7 | replicas: 2 8 | selector: 9 | matchLabels: 10 | app: agent 11 | tier: gocd 12 | template: 13 | metadata: 14 | labels: 15 | app: agent 16 | tier: gocd 17 | spec: 18 | securityContext: 19 | fsGroup: 999 20 | restartPolicy: Always 21 | containers: 22 | - name: dind 23 | securityContext: 24 | privileged: true 25 | image: docker:stable-dind 26 | imagePullPolicy: Always 27 | volumeMounts: 28 | - name: gocd-agent-data 29 | mountPath: /godata 30 | - name: agent 31 | image: $GCP_REGISTRY/kube-gocd-agent:latest 32 | imagePullPolicy: Always 33 | volumeMounts: 34 | - name: gocd-agent-data 35 | mountPath: /godata 36 | env: 37 | - name: DOCKER_HOST 38 | value: 'tcp://127.0.0.1:2375' 39 | - name: GO_SERVER_URL 40 | value: 'https://gocd-master-internal:8154/go' 41 | - name: AGENT_AUTO_REGISTER_KEY 42 | valueFrom: 43 | secretKeyRef: 44 | name: $SECRET_NAME 45 | key: agent_key 46 | - name: AGENT_AUTO_REGISTER_RESOURCES 47 | value: $AGENT_AUTO_REGISTER_RESOURCES 48 | - name: AGENT_AUTO_REGISTER_ENVIRONMENTS 49 | value: $AGENT_AUTO_REGISTER_ENVIRONMENTS 50 | volumeClaimTemplates: 51 | - metadata: 52 | name: gocd-agent-data 53 | labels: 54 | app: agent 55 | tier: gocd 56 | spec: 57 | accessModes: [ "ReadWriteOnce" ] 58 | resources: 59 | requests: 60 | storage: 50Gi 61 | -------------------------------------------------------------------------------- /kubernetes/master.pod.yml.tmp.yml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Deployment 3 | metadata: 4 | name: gocd-master 5 | spec: 6 | replicas: 1 7 | revisionHistoryLimit: 1 8 | strategy: 9 | rollingUpdate: 10 | maxUnavailable: 1 11 | maxSurge: 1 12 | selector: 13 | matchLabels: 14 | app: master 15 | tier: gocd 16 | template: 17 | metadata: 18 | labels: 19 | app: master 20 | tier: gocd 21 | spec: 22 | restartPolicy: Always 23 | containers: 24 | - name: agent 25 | image: eu.gcr.io/peopledata-product-team/kube-gocd-master:latest 26 | imagePullPolicy: Always 27 | env: 28 | - name: AGENT_AUTO_REGISTER_KEY 29 | valueFrom: 30 | secretKeyRef: 31 | name: kube-gocd 32 | key: agent_key 33 | - name: GOCD_USER 34 | valueFrom: 35 | secretKeyRef: 36 | name: kube-gocd 37 | key: user 38 | - name: GOCD_PASSWORD 39 | valueFrom: 40 | secretKeyRef: 41 | name: kube-gocd 42 | key: pass 43 | ports: 44 | - containerPort: 8153 45 | - containerPort: 8154 46 | --- 47 | apiVersion: v1 48 | kind: Service 49 | metadata: 50 | name: gocd-master-internal 51 | spec: 52 | ports: 53 | - port: 8154 54 | name: agent 55 | targetPort: 8154 56 | - port: 8153 57 | name: http 58 | targetPort: 8153 59 | selector: 60 | app: master 61 | tier: gocd 62 | --- 63 | apiVersion: v1 64 | kind: Service 65 | metadata: 66 | name: gocd-master-external 67 | spec: 68 | type: LoadBalancer 69 | ports: 70 | - port: 80 71 | name: http 72 | targetPort: 8153 73 | selector: 74 | app: master 75 | tier: gocd 76 | -------------------------------------------------------------------------------- /kubernetes/master.pod.yml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1beta1 2 | kind: StatefulSet 3 | metadata: 4 | name: gocd-master 5 | spec: 6 | serviceName: gocd-master-headless 7 | replicas: 1 8 | selector: 9 | matchLabels: 10 | app: master 11 | tier: gocd 12 | template: 13 | metadata: 14 | labels: 15 | app: master 16 | tier: gocd 17 | spec: 18 | securityContext: 19 | fsGroup: 999 20 | restartPolicy: Always 21 | containers: 22 | - name: agent 23 | image: $GCP_REGISTRY/kube-gocd-master:latest 24 | imagePullPolicy: Always 25 | volumeMounts: 26 | - name: gocd-master-data 27 | mountPath: /godata 28 | env: 29 | - name: AGENT_AUTO_REGISTER_KEY 30 | valueFrom: 31 | secretKeyRef: 32 | name: $SECRET_NAME 33 | key: agent_key 34 | - name: GO_USERNAME 35 | valueFrom: 36 | secretKeyRef: 37 | name: $SECRET_NAME 38 | key: user 39 | - name: GO_PASSWORD 40 | valueFrom: 41 | secretKeyRef: 42 | name: $SECRET_NAME 43 | key: pass 44 | ports: 45 | - containerPort: 8153 46 | - containerPort: 8154 47 | volumeClaimTemplates: 48 | - metadata: 49 | name: gocd-master-data 50 | labels: 51 | app: master 52 | tier: gocd 53 | spec: 54 | accessModes: [ "ReadWriteOnce" ] 55 | resources: 56 | requests: 57 | storage: 75Gi 58 | --- 59 | apiVersion: v1 60 | kind: Service 61 | metadata: 62 | name: gocd-master-internal 63 | spec: 64 | ports: 65 | - port: 8154 66 | name: agent 67 | targetPort: 8154 68 | - port: 8153 69 | name: http 70 | targetPort: 8153 71 | selector: 72 | app: master 73 | tier: gocd 74 | --- 75 | apiVersion: v1 76 | kind: Service 77 | metadata: 78 | name: gocd-master-external 79 | spec: 80 | type: LoadBalancer 81 | ports: 82 | - port: 80 83 | name: http 84 | targetPort: 8153 85 | selector: 86 | app: master 87 | tier: gocd 88 | -------------------------------------------------------------------------------- /common.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | 4 | function export_variable { 5 | ARG_NAME=$1 6 | ARG_VALUE=$2 7 | eval "export $ARG_NAME=$ARG_VALUE" 8 | } 9 | 10 | function enforce_arg { 11 | ARG_NAME="$1" 12 | ARG_DESC="$2" 13 | ARG_VALUE="${!1}" 14 | 15 | if [ -z "$ARG_VALUE" ]; then 16 | echo " - $ARG_NAME ($ARG_DESC) is a required value. Please ensure it is set in go.env " 17 | exit 1 18 | else 19 | export_variable "$ARG_NAME" "$ARG_VALUE" 20 | return; 21 | fi 22 | } 23 | 24 | function k() { 25 | kubectl --namespace=$KUBE_NAMESPACE $* 26 | } 27 | 28 | function tag() { 29 | docker tag stono/$1:latest $GCP_REGISTRY/$1:latest 30 | } 31 | 32 | function push() { 33 | gcloud docker -- push $GCP_REGISTRY/$1:latest 34 | } 35 | 36 | function tag_and_push() { 37 | tag $1 38 | push $1 39 | } 40 | 41 | function apply() { 42 | envsubst < $1 | kubectl --namespace=$KUBE_NAMESPACE apply -f - 43 | } 44 | 45 | function yesno { 46 | read -r -p "Do you want to continue? [y/N] " response 47 | case "$response" in 48 | [yY][eE][sS]|[yY]) 49 | echo "" 50 | return 0; 51 | ;; 52 | *) 53 | echo "" 54 | return 1 55 | ;; 56 | esac 57 | } 58 | 59 | function confirm { 60 | if ! yesno; then 61 | echo "Aborting." 62 | exit 1 63 | fi 64 | } 65 | 66 | function command_check { 67 | if ! type "$1" &> /dev/null; then 68 | echo " - $1" 69 | echo "You need $1 installed, please get it and try again" 70 | exit 1 71 | else 72 | echo " + $1" 73 | fi 74 | } 75 | 76 | function validate_requirements { 77 | echo "Checking CLI requirements..." 78 | command_check "envsubst" 79 | command_check "gcloud" 80 | command_check "kubectl" 81 | command_check "docker" 82 | command_check "docker-compose" 83 | } 84 | 85 | function validate_config { 86 | source go.env 87 | enforce_arg "GO_USERNAME" "Username for GoCD master" 88 | enforce_arg "GO_PASSWORD" "Password for GoCD master" 89 | enforce_arg "AGENT_AUTO_REGISTER_KEY" "Unique key that agents use to self register" 90 | enforce_arg "KUBE_NAMESPACE" "The namespace to deploy GoCD to" 91 | 92 | export_variable "GCP_REGISTRY" "$GCP_REGISTRY_HOST/$(gcloud config get-value project 2>/dev/null | xargs)" 93 | export_variable "SECRET_NAME" "kube-gocd" 94 | 95 | echo "Checking configuration..." 96 | echo " + GoCD username: $GO_USERNAME" 97 | echo " + GoCD password: $GO_PASSWORD" 98 | echo " + Agent registration key: $AGENT_AUTO_REGISTER_KEY" 99 | echo " + GCP registry: $GCP_REGISTRY" 100 | echo " + Kubernetes namespace: $KUBE_NAMESPACE" 101 | echo "" 102 | 103 | echo "Check, double check, and triple check the above configuration." 104 | echo "If you're not happy, quit, and edit go.env" 105 | confirm 106 | } 107 | 108 | validate_requirements 109 | echo "" 110 | validate_config 111 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # GoCD on Kubernetes (GKE) with Docker (in docker) 2 | An evolution of my journey of GoCD on Kubernetes, for building and deploying container based services. 3 | 4 | ## History 5 | [ci-in-a-box](https://github.com/Stono/ci-in-a-box) was the first iteration of a "click, deploy and off you go" solution to GoCD on kubernetes. However, through using it for the past year there were some learnings, and some aspects I wanted to change. Along with the massive change to GoCDs base images with go 17.3.0, I decided to start afresh. 6 | 7 | ### The Problems 8 | I had been running GoCD agents in docker containers, and then volume mounting the docker socket from the host, so that the agents could build and deploy docker containers. 9 | 10 | This had multiple problems: 11 | 12 | - Mounting the host docker process is in no way isolated, and privilege escalation is a massive problem 13 | - Running on kubernetes, you had access to the underlying kubernetes machine and could destroy your cluster quite easily 14 | - Agents weren't isolated, so a build job on one agent could affect a build job on another 15 | - Things like volume mounts don't work, they'd be mounting from your agents host machine, rather than from the agents filesystem 16 | - We are limited to the version of docker on the host, which for Google Container Engine (managed kubernetes) is quite old, so we're missing cool features 17 | 18 | #### Docker in Docker 19 | So I started looking at [docker in docker](https://hub.docker.com/_/docker/) and thought it would be nice if my gocd agents ran their own docker daemon, totally isolated, no reason to have access to the host they're running on. 20 | 21 | The idea on kubernetes is that your kubernetes agent pod has two containers, one is the gocd-agent itself, the other is docker-in-docker, they scale linearly, so each agent gets its own unique docker daemon. The gocd-agent talks to the docker daemon via TCP. 22 | 23 | Docker in Docker carries its own problems, theres a good blog post on [here](https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/), however the dind image is now officially supported and most of the issues have been mitigated. The only downside I can see now is its a little slower (as you're doing filesystems on top of filesystems), but that's a fair trade in exchange for a more secure setup. 24 | 25 | It's probably easier to digest as a diagram. Each agent builds against it's own isolated docker, and then pushes the build artifact up to the container registry. 26 | 27 | ![docker in docker](images/kube_dind.png) 28 | 29 | #### Shared volumes 30 | If you've ever run a CI agent inside a docker container, and your build job has needed to volume mount data from the agents working directory - you'll have found that it doesn't work. This is because you're actually acting on the daemon on the host system, so you're volume mount a directory from the parent, rather than the agent. 31 | 32 | This setup addresses that by mutually mounting `/godata` between the go agent, and docker in docker containers. 33 | 34 | ## Running/Deploying 35 | The idea here is to get you up and running as quickly as possible with GoCD. 36 | 37 | Simply edit the `go.env` file to meet your requirements and then start it up using compose, or deploy to kubernetes. 38 | 39 | Both options will run/deploy: 40 | 41 | - GoCD Server (17.6.0) 42 | - GoCD Agent (with docker-in-docker) 43 | 44 | ### Authentication 45 | You've probably noticed `GO_USERNAME` and `GO_PASSWORD` in `go.env`. These values will be written to `/etc/go-users` on the GoCD server using `htpasswd`. You will need to enable the authentication the first time you open GoCD by going to `/go/admin/security/auth_configs`. 46 | 47 | ### Customising your agent 48 | In both situations, we build custom agent and master images - inheriting from the official GoCD images. If you want to make changes to your agent, simple edit `agent/Dockerfile` 49 | 50 | ### docker-compose 51 | To run using docker-compose, do: 52 | 53 | - docker-compose build 54 | - docker-compose up -d 55 | 56 | Wait for GoCD to boot and then go to `http://127.0.0.1:8153` 57 | 58 | ### Kubernetes (on Google Container Engine) 59 | I am presuming you have already deployed a kubernetes cluster on Google Container Engine. If you haven't, head on over to the `terraform/dev` folder, and do `terraform apply`, that'll build you a shiny new cluster. Make sure you then update your kubectl command line with the credentials for you new cluster with `gcloud container clusters get-credentials [cluster name]-dev` 60 | 61 | To deploy to that cluster, do: 62 | 63 | - Make sure your gcloud cli locally is logged in, and targetting your gcloud project 64 | - Make sure you kubectl is configured to point at the cluster you wish to target 65 | - ./kubernetes-deploy.sh 66 | 67 | Wait for GoCD to boot, and then access it on the service ip. You can find the IP with `$(kubectl get services | grep master | awk '{print $2}')`. 68 | 69 | You can remove GoCD from kubernetes by running `./kubernetes-remove.sh` 70 | 71 | #### Persistence 72 | Kubernetes makes use of StatefulSets to persist your agent, and server configuration. 73 | 74 | **WARNING**: The PersistentVolumeClaims only live as long as your kubernetes cluster. Should you blow away your kubernetes cluster you **will** destroy your gocd config and history too. Make sure you have a backup strategy in place. 75 | 76 | ## The result 77 | No more `-v /var/run/docker.sock:/var/run/docker.sock` on your agents! 78 | 79 | ![result](images/gocd.png) 80 | --------------------------------------------------------------------------------