├── .gitignore ├── Dockerfile ├── README.md ├── angular ├── docker │ ├── Dockerfile │ ├── default.conf │ └── nginx.conf └── kubernetes │ └── web-deployment-and-service.yaml ├── aqueduct ├── docker │ └── Dockerfile └── kubernetes │ ├── api-deployment-and-service.yaml │ ├── config │ ├── configmap.yaml │ └── secrets.yaml │ ├── db-deployment-and-service.yaml │ ├── ingress │ ├── gke-http.yaml │ ├── gke-https.yaml │ ├── nginx-http.yaml │ └── nginx-https.yaml │ └── tasks │ ├── add-auth-client-bare-pod.yaml │ └── migration-upgrade-bare-pod.yaml ├── kube-lego ├── configmap.yaml └── deployment.yaml └── nginx_deploy.sh /.gitignore: -------------------------------------------------------------------------------- 1 | .idea/ 2 | .DS_Store -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM google/dart 2 | 3 | WORKDIR /app 4 | ADD pubspec.* /app/ 5 | RUN pub get --no-precompile 6 | ADD . /app/ 7 | RUN pub get --offline --no-precompile 8 | 9 | WORKDIR /app 10 | EXPOSE 80 11 | 12 | ENTRYPOINT ["pub", "run", "aqueduct:aqueduct", "serve", "--port", "80"] 13 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # k8s bootstrapping 2 | 3 | Directories in this repository contain Kubernetes configuration files for provisioning a cluster and deploying Aqueduct applications. 4 | 5 | ## Prerequisites 6 | 7 | The following assumes that you have a Google Cloud account with billing enabled and that tools such as `docker`, `gcloud` and `kubectl` have been installed and configured. If not, see the following references: 8 | 9 | - [Install Docker](https://www.docker.com/community-edition) 10 | - [Install gcloud](https://cloud.google.com/sdk/downloads) 11 | - Install `kubectl` by running `gcloud components install kubectl`. 12 | 13 | ## Cluster Provisioning 14 | 15 | *Tasks in this section need only be applied when a cluster is first created, not for each application deployed.* 16 | 17 | The objects in `kube-lego` and `nginx-ingress-controller` provide a cluster with low cost alternatives to load-balancing with SSL termination. They live in the `kube-system` namespace. 18 | 19 | ### SSL Certificates via Let's Encrypt 20 | 21 | Modify `kube-lego/config-map.yaml` by replacing `` with an administrator e-mail address. Then, apply the objects in the `kube-lego` directory to your cluster: 22 | 23 | ```bash 24 | kubectl apply -f kube-lego/ 25 | ``` 26 | 27 | `kube-lego` will monitor your cluster to request trusted SSL certificates for deployed applications. 28 | 29 | ### Nginx Ingress Controller 30 | 31 | *The nginx ingress controller is only necessary if you prefer not to use Google Cloud Load Balancer (GCLB costs $200+/yr).* 32 | 33 | Run the script `nginx_deploy.sh`: 34 | 35 | ```bash 36 | sh nginx_deploy.sh 37 | ``` 38 | 39 | It will take a moment for `nginx-ingress-lb` to acquire an IP address. During that time, running the command `kubectl get services -n kube-system` will show something like the following: 40 | 41 | ``` 42 | NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE 43 | default-http-backend 10.55.241.222 80:32516/TCP 11d 44 | heapster 10.55.255.209 80/TCP 11d 45 | kube-dns 10.55.240.10 53/UDP,53/TCP 11d 46 | kubernetes-dashboard 10.55.240.49 80/TCP 11d 47 | nginx-ingress-lb 10.55.249.185 80:32005/TCP,443:31623/TCP 6s 48 | ``` 49 | 50 | Where `nginx-ingress-lb`'s `EXTERNAL-IP` is ``. Once that `` flips to an IP address, note the IP address and navigate to `VPC Network->External IP adresses` in the Google Cloud console. Locate the IP address in that list and change it's type from `Ephemeral` to `Static`. (You'll be prompted for a name which can be whatever you like.) 51 | 52 | *Important: if you delete the `nginx-ingress-lb` service after reserving a static IP for it, simply re-creating the service won't work. You'll need to release the reserved address before the service will start again and then reserve the IP address again.* 53 | 54 | ## Application Deployment 55 | 56 | The `aqueduct` directory contains templates for Kubernetes objects that deploy an Aqueduct application and its PostgreSQL database. For each Aqueduct application deployed into the cluster, follow the following steps. 57 | 58 | *Note: Kubernetes object names may not contain underscores. If your application name contains underscores, substitute a dash (`-`) when replacing the `` template variable. For example, if your application name is `my_app`, use `my-app`.* 59 | 60 | 0. Copy the *contents* of `aqueduct/kubernetes` into a directory named `k8s` in your project directory. 61 | 62 | 1. Create a new namespace with the name of your application. 63 | 64 | ``` 65 | kubectl create namespace 66 | ``` 67 | 68 | 2. Modify `config/configmap.yaml` and `config/secrets.yaml` by replacing all occurrences of `` with the name of your application (remembering to use dashes and not underscores), and replacing `` with a database password. Apply these files: 69 | 70 | ``` 71 | kubectl apply -f k8s/config/ 72 | ``` 73 | 74 | Change your Aqueduct application's config.yaml to use the values from the configmap, secrets and db-service: 75 | 76 | ``` 77 | database: 78 | username: $POSTGRES_USER 79 | password: $POSTGRES_PASSWORD 80 | host: db-service 81 | port: 5432 82 | databaseName: $POSTGRES_DB 83 | ``` 84 | 85 | 3. Add the `Dockerfile` in this repository to your Aqueduct project directory. Make sure that migration files are up to date by running `aqueduct db validate` (and running `aqueduct db generate` if they are not). Migration files must be a part of the docker image. 86 | 87 | Run `docker build` in the project directory. The name of the image must have the format `gcr.io//` where `PROJECT_ID` is the name of the Google Cloud project that the target cluster is in and `APP_NAME` is your application's name. 88 | 89 | ``` 90 | docker build -t gcr.io//:latest . 91 | ``` 92 | 93 | Once built, push it to your project's private registry: 94 | 95 | ``` 96 | gcloud docker -- push gcr.io//:latest 97 | ``` 98 | 99 | 4. In both `api-deployment-and-service.yaml` and `db-deployment-and-service.yaml`, replace template variables with their appropriate values. Ensure that `` is replaced with the full name of the image built with `docker`. Apply these files. 100 | 101 | ``` 102 | # Note: this will (intentionally) only apply top-level files in k8s and will not recursively apply files in subdirectories. 103 | kubectl apply -f k8s/ 104 | ``` 105 | 106 | 5. Update the database schema to current version by replacing the template variables in `tasks/migration-upgrade-bare-pod.yaml` and then applying it: 107 | 108 | ``` 109 | kubectl apply -f k8s/tasks/migration-upgrade-bare-pod.yaml 110 | ``` 111 | 112 | Ensure this task completed by running `kubectl get pod -n db-upgrade-job`. Once it has completed, delete it: 113 | 114 | ``` 115 | kubectl delete pod -n db-upgrade-job 116 | ``` 117 | 118 | If using `ManagedAuth`, replace the template variables in `tasks/add-auth-client-bare-pod.yaml`. Apply this file, check it for completion, and delete it. Repeat for each client ID. 119 | 120 | 6. Expose your application to the world by replacing the template variables in `ingress/nginx-https.yaml` and applying it. 121 | 122 | ### Files to check in to version control 123 | 124 | Obfuscate any secret values - those in `config/secrets.yaml` and `tasks/add-auth-client-bare-pod.yaml` and check in all files in `k8s`. 125 | 126 | ### Updating the Application 127 | 128 | For application updates that do not require database schema changes, build the Docker image and push it to the registry with `gcloud`. If you are tagging images correctly, just set the image of your deployment: 129 | 130 | ``` 131 | kubectl set image deployment/api-deployment =gcr.io//: -n 132 | ``` 133 | 134 | If you are reusing the latest tag, delete all *pods* in the `api-deployment`. 135 | 136 | ``` 137 | kubectl delete pod api-deployment-xxxxxxx-xxxxx -n 138 | ``` 139 | 140 | The pods will automatically be recreated and will pull the most recent image. 141 | 142 | If a database update is required, make sure to run `aqueduct db generate` before building and pushing the Docker image. Then follow the instructions in step 5 before deleting pods. 143 | 144 | *Note: This scheme is useful for development. When deploying to production, a unique tag should be used for each image and that image name should be added to the `api-deployment-and-service.yaml`. Instead of deleting pods, re-apply this configuration file: 145 | 146 | ``` 147 | kubectl apply -f k8s/api-deployment-and-service.yaml 148 | ``` 149 | -------------------------------------------------------------------------------- /angular/docker/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM google/dart AS build-env 2 | WORKDIR /app 3 | 4 | ADD pubspec.* /app/ 5 | RUN pub get --no-precompile 6 | ADD . /app/ 7 | RUN pub get --offline --no-precompile 8 | RUN pub build 9 | 10 | FROM nginx:stable 11 | 12 | COPY --from=build-env /app/build/web /usr/share/nginx/html 13 | COPY default.conf /etc/nginx/conf.d/default.conf 14 | COPY nginx.conf /etc/nginx/nginx.conf 15 | 16 | RUN groupadd -r angular 17 | RUN useradd -m -r -g angular angular 18 | 19 | RUN touch /var/run/nginx.pid && \ 20 | chown angular:angular /var/run/nginx.pid && \ 21 | chown -R angular:angular /var/cache/nginx 22 | 23 | USER angular 24 | 25 | EXPOSE 8080 26 | -------------------------------------------------------------------------------- /angular/docker/default.conf: -------------------------------------------------------------------------------- 1 | server { 2 | listen 8080; 3 | server_name localhost; 4 | 5 | location / { 6 | root /usr/share/nginx/html; 7 | index index.html; 8 | try_files $uri $uri/ /index.html; 9 | } 10 | } 11 | -------------------------------------------------------------------------------- /angular/docker/nginx.conf: -------------------------------------------------------------------------------- 1 | worker_processes 1; 2 | 3 | error_log /var/log/nginx/error.log warn; 4 | pid /var/run/nginx.pid; 5 | 6 | 7 | events { 8 | worker_connections 1024; 9 | } 10 | 11 | 12 | http { 13 | include /etc/nginx/mime.types; 14 | default_type application/octet-stream; 15 | 16 | log_format main '$remote_addr - $remote_user [$time_local] "$request" ' 17 | '$status $body_bytes_sent "$http_referer" ' 18 | '"$http_user_agent" "$http_x_forwarded_for"'; 19 | 20 | access_log /var/log/nginx/access.log main; 21 | 22 | sendfile on; 23 | 24 | keepalive_timeout 65; 25 | 26 | include /etc/nginx/conf.d/*.conf; 27 | } 28 | -------------------------------------------------------------------------------- /angular/kubernetes/web-deployment-and-service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: web-service 5 | namespace: 6 | spec: 7 | selector: 8 | app: 9 | role: frontend 10 | type: web 11 | ports: 12 | - port: 80 13 | targetPort: 8080 14 | --- 15 | apiVersion: apps/v1beta1 16 | kind: Deployment 17 | metadata: 18 | name: web-deployment 19 | namespace: 20 | spec: 21 | replicas: 1 22 | template: 23 | metadata: 24 | labels: 25 | app: 26 | role: frontend 27 | type: web 28 | spec: 29 | containers: 30 | - name: 31 | imagePullPolicy: Always 32 | image: 33 | ports: 34 | - containerPort: 8080 35 | securityContext: 36 | runAsNonRoot: true 37 | -------------------------------------------------------------------------------- /aqueduct/docker/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM google/dart 2 | 3 | WORKDIR /app 4 | ADD pubspec.* /app/ 5 | 6 | RUN groupadd -r aqueduct 7 | RUN useradd -m -r -g aqueduct aqueduct 8 | RUN chown -R aqueduct:aqueduct /app 9 | 10 | USER aqueduct 11 | RUN pub get --no-precompile 12 | 13 | USER root 14 | ADD . /app 15 | RUN chown -R aqueduct:aqueduct /app 16 | 17 | USER aqueduct 18 | RUN pub get --offline --no-precompile 19 | 20 | EXPOSE 8082 21 | 22 | ENTRYPOINT ["pub", "run", "aqueduct:aqueduct", "serve", "--port", "8082"] -------------------------------------------------------------------------------- /aqueduct/kubernetes/api-deployment-and-service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: api-service 5 | namespace: 6 | spec: 7 | selector: 8 | app: 9 | role: backend 10 | type: api 11 | ports: 12 | - port: 80 13 | targetPort: 8082 14 | --- 15 | apiVersion: apps/v1beta1 16 | kind: Deployment 17 | metadata: 18 | name: api-deployment 19 | namespace: 20 | spec: 21 | replicas: 2 22 | template: 23 | metadata: 24 | labels: 25 | app: 26 | role: backend 27 | type: api 28 | spec: 29 | containers: 30 | - name: 31 | # In development, setting `imagePullPolicy: Always` and using :latest tag is useful. 32 | imagePullPolicy: Always 33 | image: 34 | envFrom: 35 | - secretRef: 36 | name: secrets 37 | - configMapRef: 38 | name: config 39 | ports: 40 | - containerPort: 8082 41 | securityContext: 42 | runAsNonRoot: true 43 | -------------------------------------------------------------------------------- /aqueduct/kubernetes/config/configmap.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: config 5 | namespace: 6 | data: 7 | PGDATA: /var/lib/postgresql/data/ 8 | POSTGRES_DB: 9 | POSTGRES_USER: 10 | -------------------------------------------------------------------------------- /aqueduct/kubernetes/config/secrets.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Secret 3 | metadata: 4 | name: secrets 5 | namespace: 6 | data: 7 | POSTGRES_PASSWORD: 8 | type: Opaque 9 | -------------------------------------------------------------------------------- /aqueduct/kubernetes/db-deployment-and-service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1beta1 2 | kind: Deployment 3 | metadata: 4 | name: db-deployment 5 | namespace: 6 | spec: 7 | replicas: 1 8 | revisionHistoryLimit: 0 9 | template: 10 | metadata: 11 | labels: 12 | app: 13 | role: backend 14 | type: database 15 | spec: 16 | containers: 17 | - name: postgresql 18 | image: postgres:9.6 19 | envFrom: 20 | - secretRef: 21 | name: secrets 22 | - configMapRef: 23 | name: config 24 | ports: 25 | - containerPort: 5432 26 | volumeMounts: 27 | - name: postgres-data 28 | mountPath: /var/lib/postgresql/data 29 | volumes: 30 | - name: postgres-data 31 | persistentVolumeClaim: 32 | claimName: volume-claim 33 | --- 34 | apiVersion: v1 35 | kind: Service 36 | metadata: 37 | name: db-service 38 | namespace: 39 | spec: 40 | selector: 41 | app: 42 | role: backend 43 | type: database 44 | ports: 45 | - protocol: TCP 46 | port: 5432 47 | targetPort: 5432 48 | --- 49 | apiVersion: v1 50 | kind: PersistentVolumeClaim 51 | metadata: 52 | name: volume-claim 53 | namespace: 54 | annotations: 55 | volume.beta.kubernetes.io/storage-class: "standard" 56 | spec: 57 | accessModes: 58 | - ReadWriteOnce 59 | resources: 60 | requests: 61 | storage: 40Gi 62 | -------------------------------------------------------------------------------- /aqueduct/kubernetes/ingress/gke-http.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Ingress 3 | metadata: 4 | name: 5 | namespace: 6 | annotations: 7 | kubernetes.io/ingress.global-static-ip-name: "static-ip-name" 8 | spec: 9 | rules: 10 | - host: 11 | http: 12 | paths: 13 | - path: / 14 | backend: 15 | serviceName: api-service 16 | servicePort: 80 17 | -------------------------------------------------------------------------------- /aqueduct/kubernetes/ingress/gke-https.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Ingress 3 | metadata: 4 | name: 5 | namespace: 6 | annotations: 7 | kubernetes.io/ingress.global-static-ip-name: "static-ip-name" 8 | spec: 9 | tls: 10 | - hosts: 11 | - 12 | secretName: -tls 13 | rules: 14 | - host: 15 | http: 16 | paths: 17 | - path: / 18 | backend: 19 | serviceName: api-service 20 | servicePort: 80 21 | -------------------------------------------------------------------------------- /aqueduct/kubernetes/ingress/nginx-http.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Ingress 3 | metadata: 4 | name: 5 | namespace: 6 | annotations: 7 | kubernetes.io/ingress.class: "nginx" 8 | spec: 9 | rules: 10 | - host: 11 | http: 12 | paths: 13 | - path: / 14 | backend: 15 | serviceName: api-service 16 | servicePort: 80 17 | -------------------------------------------------------------------------------- /aqueduct/kubernetes/ingress/nginx-https.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Ingress 3 | metadata: 4 | name: 5 | namespace: 6 | annotations: 7 | kubernetes.io/tls-acme: "true" 8 | kubernetes.io/ingress.class: "nginx" 9 | spec: 10 | tls: 11 | - hosts: 12 | - 13 | secretName: -tls 14 | rules: 15 | - host: 16 | http: 17 | paths: 18 | - path: / 19 | backend: 20 | serviceName: api-service 21 | servicePort: 80 22 | -------------------------------------------------------------------------------- /aqueduct/kubernetes/tasks/add-auth-client-bare-pod.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Pod 3 | metadata: 4 | name: auth-add-client-job 5 | namespace: 6 | spec: 7 | restartPolicy: Never 8 | containers: 9 | - name: add-auth-client 10 | image: 11 | envFrom: 12 | - secretRef: 13 | name: secrets 14 | - configMapRef: 15 | name: config 16 | command: ["pub"] 17 | args: ["run", "aqueduct:aqueduct", "auth", "add-client", 18 | "--id", , 19 | "--secret", , 20 | "--connect", "postgresql://$(POSTGRES_USER):$(POSTGRES_PASSWORD)@db-service:5432/$(POSTGRES_DB)"] 21 | -------------------------------------------------------------------------------- /aqueduct/kubernetes/tasks/migration-upgrade-bare-pod.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Pod 3 | metadata: 4 | name: db-upgrade-job 5 | namespace: 6 | spec: 7 | restartPolicy: Never 8 | containers: 9 | - name: migration-upgrade 10 | image: 11 | envFrom: 12 | - secretRef: 13 | name: secrets 14 | - configMapRef: 15 | name: config 16 | command: ["pub"] 17 | args: ["run", "aqueduct:aqueduct", "db", "upgrade", "--connect", "postgresql://$(POSTGRES_USER):$(POSTGRES_PASSWORD)@db-service:5432/$(POSTGRES_DB)"] 18 | -------------------------------------------------------------------------------- /kube-lego/configmap.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | metadata: 3 | name: kube-lego 4 | namespace: kube-system 5 | data: 6 | # Set this to a valid e-mail 7 | lego.email: 8 | lego.url: "https://acme-v01.api.letsencrypt.org/directory" 9 | kind: ConfigMap -------------------------------------------------------------------------------- /kube-lego/deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Deployment 3 | metadata: 4 | name: kube-lego 5 | namespace: kube-system 6 | spec: 7 | replicas: 1 8 | template: 9 | metadata: 10 | labels: 11 | app: kube-lego 12 | spec: 13 | containers: 14 | - name: kube-lego 15 | image: jetstack/kube-lego:0.1.5 16 | imagePullPolicy: Always 17 | ports: 18 | - containerPort: 8080 19 | env: 20 | - name: LEGO_EMAIL 21 | valueFrom: 22 | configMapKeyRef: 23 | name: kube-lego 24 | key: lego.email 25 | - name: LEGO_URL 26 | valueFrom: 27 | configMapKeyRef: 28 | name: kube-lego 29 | key: lego.url 30 | - name: LEGO_NAMESPACE 31 | valueFrom: 32 | fieldRef: 33 | fieldPath: metadata.namespace 34 | - name: LEGO_POD_IP 35 | valueFrom: 36 | fieldRef: 37 | fieldPath: status.podIP 38 | readinessProbe: 39 | httpGet: 40 | path: /healthz 41 | port: 8080 42 | initialDelaySeconds: 5 43 | timeoutSeconds: 1 -------------------------------------------------------------------------------- /nginx_deploy.sh: -------------------------------------------------------------------------------- 1 | kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml 2 | kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml 3 | kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml 4 | kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml 5 | kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/udp-services-configmap.yaml 6 | kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/without-rbac.yaml 7 | kubectl patch deployment -n ingress-nginx nginx-ingress-controller --type='json' --patch="$(curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/publish-service-patch.yaml)" 8 | kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/gce-gke/service.yaml 9 | kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/patch-service-without-rbac.yaml 10 | --------------------------------------------------------------------------------