├── .envrc ├── .gitignore ├── .tool-versions ├── LICENCE ├── Makefile ├── README.MD ├── ansible-playbook ├── .gitignore ├── LICENSE ├── README.md ├── ansible.cfg ├── group_vars │ ├── all.yml │ └── kube-cluster.yml ├── hosts.ini ├── reset-site.yaml ├── roles │ ├── cni │ │ ├── defaults │ │ │ └── main.yml │ │ ├── tasks │ │ │ └── main.yml │ │ └── templates │ │ │ ├── calico.yml.j2 │ │ │ ├── canal.yml.j2 │ │ │ └── flannel.yml.j2 │ ├── commons │ │ ├── os-checker │ │ │ ├── defaults │ │ │ │ └── main.yml │ │ │ └── tasks │ │ │ │ └── main.yml │ │ └── pre-install │ │ │ ├── meta │ │ │ └── main.yml │ │ │ ├── tasks │ │ │ ├── main.yml │ │ │ └── pkg.yml │ │ │ └── templates │ │ │ └── 20-extra-args.conf.j2 │ ├── docker │ │ ├── defaults │ │ │ └── main.yml │ │ ├── meta │ │ │ └── main.yml │ │ ├── tasks │ │ │ ├── main.yml │ │ │ └── pkg.yml │ │ └── templates │ │ │ ├── docker.j2 │ │ │ └── docker.service.j2 │ ├── healthcheck │ │ ├── tasks │ │ │ └── main.yml │ │ └── vars │ │ │ └── main.yml │ ├── helm │ │ ├── files │ │ │ └── rbac-config.yml │ │ └── tasks │ │ │ └── main.yml │ ├── kubernetes │ │ ├── master │ │ │ ├── handlers │ │ │ │ └── main.yml │ │ │ ├── meta │ │ │ │ └── main.yml │ │ │ └── tasks │ │ │ │ ├── init.yml │ │ │ │ └── main.yml │ │ └── node │ │ │ ├── handlers │ │ │ └── main.yml │ │ │ ├── meta │ │ │ └── main.yml │ │ │ └── tasks │ │ │ ├── join.yml │ │ │ └── main.yml │ └── metallb │ │ ├── tasks │ │ └── main.yml │ │ ├── templates │ │ └── metallb-layer-2-config.yml.j2 │ │ └── vars │ │ └── main.yml └── site.yaml ├── cloud_init.cfg ├── config ├── goldpinger.yaml └── metallb-config.yaml ├── docs ├── README.MD ├── bookmarks.html └── cka-prep-list.html ├── envvars.tmpl ├── init.sh ├── main.tf ├── output.tf ├── tasks ├── 1-deploy-kubernetes.md ├── README.MD ├── app-lifecycle-management.md ├── cluster-maintenance.md ├── logging-monitoring.md ├── networking.md ├── scheduling.md ├── security.md ├── storage.md └── troubleshooting.md └── versions.tf /.envrc: -------------------------------------------------------------------------------- 1 | if command -v asdf > /dev/null; then 2 | if ! asdf plugin-list | grep python > /dev/null; then 3 | echo "Installing asdf plugin python" 4 | asdf plugin-add python 5 | fi 6 | asdf install python $(cat ./.tool-versions | grep python | cut -f 2 -d " ") 7 | fi 8 | 9 | layout python $(asdf which python) 10 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .local 2 | .terraform 3 | .terraform.tfstate* 4 | terraform.d 5 | terraform.tfstate* 6 | volume_pool 7 | envvars.env 8 | .direnv/ -------------------------------------------------------------------------------- /.tool-versions: -------------------------------------------------------------------------------- 1 | terraform 0.12.23 2 | python 3.6.10 3 | direnv 2.21.2 -------------------------------------------------------------------------------- /LICENCE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Zachary Loeber 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | SHELL := /bin/bash 2 | .DEFAULT_GOAL := help 3 | 4 | ROOT_PATH := $(abspath $(patsubst %/,%,$(dir $(abspath $(lastword $(MAKEFILE_LIST)))))) 5 | BIN_PATH := $(ROOT_PATH)/.local/bin 6 | KUBECONFIG_PATH ?= $(ROOT_PATH)/.local/kubeconfig 7 | TASK_PATH := $(ROOT_PATH)/tasks 8 | CONFIG_PATH := $(ROOT_PATH)/config 9 | KEY_PATH := $(ROOT_PATH)/.local/.ssh 10 | KEY_NAME := $(KEY_PATH)/id_rsa 11 | 12 | POOL_NAME ?= ubuntu 13 | POOL_PATH ?= $(shell pwd)/volume_pool 14 | 15 | terraform := $(BIN_PATH)/terraform 16 | gh := $(BIN_PATH)/gh 17 | xpanes := $(BIN_PATH)/xpanes 18 | kubectl := $(BIN_PATH)/kubectl --kubeconfig $(KUBECONFIG_PATH)/config 19 | helm := $(BIN_PATH)/helm 20 | terraform-inventory := $(BIN_PATH)/terraform-inventory 21 | 22 | ENV_VARS ?= $(ROOT_PATH)/envvars.env 23 | ifneq (,$(wildcard $(ENV_VARS))) 24 | include $(ENV_VARS) 25 | export $(shell sed 's/=.*//' $(ENV_VARS)) 26 | endif 27 | 28 | KUBE_VERSION ?= 1.18.0 29 | 30 | # Generic shared variables 31 | ifeq ($(shell uname -m),x86_64) 32 | ARCH ?= "amd64" 33 | endif 34 | ifeq ($(shell uname -m),i686) 35 | ARCH ?= "386" 36 | endif 37 | ifeq ($(shell uname -m),aarch64) 38 | ARCH ?= "arm" 39 | endif 40 | ifeq ($(OS),Windows_NT) 41 | OS := Windows 42 | else 43 | OS := $(shell sh -c 'uname -s 2>/dev/null || echo not' | tr '[:upper:]' '[:lower:]') 44 | endif 45 | 46 | TF_PROVIDER_PATH := $(ROOT_PATH)/terraform.d/plugins/$(OS)_$(ARCH)/terraform-provider-libvirt 47 | TF_VERSION ?= 0.12.23 48 | 49 | .PHONY: help 50 | help: ## Help 51 | @grep --no-filename -E '^[a-zA-Z1-9_/-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' 52 | 53 | .PHONY: deps 54 | deps: .dep/terraform .dep/keygen .dep/xpanes .dep/libvirt/provider ## Install Dependencies 55 | 56 | .PHONY: .dep/terraform 57 | .dep/terraform: ## Install local terraform binary 58 | ifeq (,$(wildcard $(terraform))) 59 | @echo "Attempting to install terraform - $(TF_VERSION)" 60 | @mkdir -p $(BIN_PATH) 61 | @wget -O /tmp/terraform.zip https://releases.hashicorp.com/terraform/$(TF_VERSION)/terraform_$(TF_VERSION)_$(OS)_$(ARCH).zip 62 | @unzip -d $(BIN_PATH) /tmp/terraform.zip && rm /tmp/terraform.zip 63 | endif 64 | 65 | .PHONY: .dep/libvirt/provider 66 | .dep/libvirt/provider: ## Grab the libvirt terraform provider 67 | #ifeq "$(wildcard $(TF_PROVIDER_PATH))" "" 68 | ifeq (,$(wildcard $(TF_PROVIDER_PATH))) 69 | @echo "Attempting to grab the libvirt provider" 70 | @mkdir -p $(ROOT_PATH)/terraform.d/plugins/$(OS)_$(ARCH) 71 | $(ROOT_PATH)/init.sh 72 | endif 73 | 74 | .PHONY: .dep/keygen 75 | .dep/keygen: ## Generate an ssh key for this deployment 76 | ifeq (,$(wildcard $(KEY_NAME))) 77 | @mkdir -p $(KEY_PATH) 78 | ssh-keygen -t rsa -b 4096 -N '' -f $(KEY_NAME) -q 79 | endif 80 | 81 | .PHONY: .dep/helm 82 | .dep/helm: ## Downloads helm 3 83 | ifeq (,$(wildcard $(helm))) 84 | @echo "Attempting to install helm 3" 85 | @mkdir -p /tmp/helm3 86 | @curl --retry 3 --retry-delay 5 --fail -sSL -o - https://get.helm.sh/helm-v3.1.2-linux-amd64.tar.gz | tar -C /tmp/helm3 -zx linux-amd64/helm 87 | @mv /tmp/helm3/linux-amd64/helm $(helm) 88 | @rm -rf /tmp/helm3 89 | @chmod +x $(helm) 90 | endif 91 | 92 | .PHONY: .dep/terraform-inventory 93 | .dep/terraform-inventory: ## Downloads terraform-inventory 94 | ifeq (,$(wildcard $(terraform-inventory))) 95 | @echo "Attempting to install terraform-inventory" 96 | @mkdir -p /tmp/terraform-inventory 97 | @curl --retry 3 --retry-delay 5 --fail -sSL -L -o /tmp/terraform-inventory/terraform-inventory.zip https://github.com/adammck/terraform-inventory/releases/download/v0.9/terraform-inventory_0.9_linux_amd64.zip 98 | @unzip /tmp/terraform-inventory/terraform-inventory.zip -d /tmp/terraform-inventory 99 | @find /tmp/terraform-inventory -type f -name terraform-inventory | xargs -I {} cp -f {} $(terraform-inventory) 100 | @chmod +x $(terraform-inventory) 101 | @[ -n "/tmp" ] && [ -n "terraform-inventory" ] && rm -rf "/tmp/terraform-inventory" 102 | @echo "Deployed to: $(terraform-inventory)" 103 | endif 104 | 105 | .PHONY: .dep/xpanes 106 | .dep/xpanes: ## xpanes for tmux 107 | ifeq (,$(wildcard $(BIN_PATH)/xpanes)) 108 | wget https://raw.githubusercontent.com/greymd/tmux-xpanes/v4.1.1/bin/xpanes -O $(BIN_PATH)/xpanes 109 | chmod +x $(BIN_PATH)/xpanes 110 | endif 111 | 112 | .PHONY: libvirt/domain/remove 113 | libvirt/clean: ## Removes any dangling libvirt domains and subnets 114 | virsh destroy k8s-master || true 115 | virsh undefine k8s-master || true 116 | virsh destroy k8s-worker-1 || true 117 | virsh undefine k8s-worker-1 || true 118 | virsh destroy k8s-worker-2 || true 119 | virsh undefine k8s-worker-2 || true 120 | virsh net-destroy kube_ext || true 121 | virsh net-undefine kube_ext || true 122 | virsh net-destroy kube_node || true 123 | virsh net-undefine kube_node || true 124 | virsh pool-destroy ${POOL_NAME} || true 125 | virsh pool-undefine ${POOL_NAME} || true 126 | 127 | .PHONY: clean 128 | clean: ## Clean local cached terreform elements 129 | rm -rf ./.terraform 130 | rm terraform.tfstate* 131 | 132 | .PHONY: init 133 | init: ## Initialize terraform 134 | $(terraform) init 135 | 136 | .PHONY: plan 137 | plan: ## Plan deployment 138 | $(terraform) plan 139 | 140 | .PHONY: apply 141 | apply: ## Apply deployment 142 | $(terraform) apply 143 | 144 | .PHONY: destroy 145 | destroy: kube/clean ## Destroy the lab 146 | $(terraform) destroy 147 | 148 | .PHONY: ssh/master 149 | ssh/master: ## connect to the master node 150 | IP=$(shell $(terraform) output master_ip); ssh -o StrictHostKeyChecking=no -i $(KEY_NAME) ubuntu@$${IP} 151 | 152 | .PHONY: ssh/worker1 153 | ssh/worker1: ## connect to worker node 1 154 | IP=$(shell $(terraform) output worker_1_ip); ssh -o StrictHostKeyChecking=no -i $(KEY_NAME) ubuntu@$${IP} 155 | 156 | .PHONY: ssh/worker2 157 | ssh/worker2: ## connect to worker node 2 158 | IP=$(shell $(terraform) output worker_2_ip); ssh -o StrictHostKeyChecking=no -i $(KEY_NAME) ubuntu@$${IP} 159 | 160 | .PHONY: ssh/all 161 | ssh/all: ## Use xpanes/tmux to connect to all nodes at once (synced input) 162 | $(xpanes) -c "ssh -i $(KEY_NAME) -o StrictHostKeyChecking=no {}" \ 163 | ubuntu@$(shell $(terraform) output master_ip) \ 164 | ubuntu@$(shell $(terraform) output worker_1_ip) \ 165 | ubuntu@$(shell $(terraform) output worker_2_ip) 166 | 167 | .PHONY: ssh/all/desync 168 | ssh/all/desync: ## Use xpanes/tmux to connect to all nodes at once 169 | $(xpanes) -d -c "ssh -i $(KEY_NAME) -o StrictHostKeyChecking=no {}" \ 170 | ubuntu@$(shell $(terraform) output master_ip) \ 171 | ubuntu@$(shell $(terraform) output worker_1_ip) \ 172 | ubuntu@$(shell $(terraform) output worker_2_ip) 173 | 174 | .PHONY: show 175 | show: ## Show deployment information 176 | @echo "OS: $(OS)" 177 | @echo "ARCH: $(ARCH)" 178 | @echo "POOL_NAME: $(POOL_NAME)" 179 | @echo "POOL_PATH: $(POOL_PATH)" 180 | @echo "TF_PROVIDER_PATH: $(TF_PROVIDER_PATH)" 181 | @echo "MASTER NODE IP: $(shell $(terraform) output master_ip)" 182 | @echo "WORKER 1 NODE IP: $(shell $(terraform) output worker_1_ip)" 183 | @echo "WORKER 2 NODE IP: $(shell $(terraform) output worker_2_ip)" 184 | 185 | .PHONY: .kube/get/configfile 186 | .kube/get/configfile: ## Pull deployed kube config file from master 187 | ifeq (,$(wildcard $(KUBECONFIG_PATH)/config)) 188 | @rm -rf $(KUBECONFIG_PATH) 189 | @mkdir -p $(KUBECONFIG_PATH) 190 | @IP=$(shell $(terraform) output master_ip); \ 191 | scp -o StrictHostKeyChecking=no -i $(KEY_NAME) ubuntu@$${IP}:.kube/config $(KUBECONFIG_PATH)/config 192 | endif 193 | 194 | .PHONY: .dep/kubectl 195 | .dep/kubectl: ## install kubectl for this project 196 | ifeq (,$(wildcard $(BIN_PATH)/kubectl)) 197 | @mkdir -p $(BIN_PATH) 198 | @curl --retry 3 --retry-delay 5 --fail -sSL -o $(BIN_PATH)/kubectl https://storage.googleapis.com/kubernetes-release/release/v$(KUBE_VERSION)/bin/$(OS)/$(ARCH)/kubectl 199 | @chmod +x $(BIN_PATH)/kubectl 200 | @echo "Installed: $(BIN_PATH)/kubectl" 201 | endif 202 | 203 | # .PHONY: kube/deploy/metricsserver 204 | # kube/deploy/metricsserver: .dep/kubectl .kube/get/configfile## Deploy metrics server 205 | # @$(kubectl) apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml 206 | 207 | # .PHONY: kube/deploy/localstorage 208 | # kube/deploy/localstorage: .dep/kubectl .kube/get/configfile ## Deploy metrics server 209 | # @$(kubectl) apply -f $(TASK_PATH)/local-storageclass.yaml 210 | 211 | .PHONY: kube/deploy/nfs 212 | kube/deploy/nfs: .dep/kubectl .kube/get/configfile .dep/helm ## Deploy nfs dynamic pvc provisioning 213 | $(helm) install nfsstorage \ 214 | stable/nfs-client-provisioner \ 215 | --set nfs.server=$(shell $(terraform) output master_ip) \ 216 | --set nfs.path=/opt/nfs \ 217 | --kubeconfig $(KUBECONFIG_PATH)/config 218 | 219 | .PHONY: kube/deploy/metallb 220 | kube/deploy/metallb: .dep/kubectl .kube/get/configfile ## Deploy metallb on the cluster 221 | @$(kubectl) apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml 222 | @$(kubectl) apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml 223 | @$(kubectl) apply -f $(CONFIG_PATH)/metallb-config.yaml 224 | @$(kubectl) create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)" || true 225 | 226 | .PHONY: kube/delete/metallb 227 | kube/delete/metallb: ## Delete metallb deployment 228 | @$(kubectl) delete -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml 229 | @$(kubectl) delete -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml 230 | 231 | .PHONY: kube/export/config 232 | kube/export/config: kube/clean .kube/get/configfile ## Displays the correct export command to point kubeconfig to this cluster 233 | @echo 'export KUBECONFIG=$$(pwd)/.local/kubeconfig/config' 234 | 235 | .PHONY: kube/clean 236 | kube/clean: ## Remove old kube config and kubectl files 237 | @rm -rf $(KUBECONFIG_PATH)/config 238 | 239 | .PHONY: .dep/ansible 240 | .dep/ansible: ## configure ansible in python virtual environment 241 | ifeq (,$(wildcard $(ROOT_PATH)/.direnv/python-3.6.10/bin/ansible)) 242 | $(ROOT_PATH)/.direnv/python-3.6.10/bin/pip3 install ansible 243 | endif 244 | 245 | .PHONY: cluster/deploy 246 | cluster/deploy: ## Deploy the base cluster with ansible 247 | cd $(ROOT_PATH)/ansible-playbook && \ 248 | $(ROOT_PATH)/.direnv/python-3.6.10/bin/ansible-playbook \ 249 | $(ROOT_PATH)/ansible-playbook/site.yaml \ 250 | --private-key $(KEY_NAME) -u ubuntu 251 | 252 | # .PHONY: ansible/inventory 253 | # ansible/inventory: .dep/terraform-inventory ## Attempt to create inventory from terraform state. 254 | # TF_STATE=. $(terraform-inventory) -list 255 | 256 | -------------------------------------------------------------------------------- /README.MD: -------------------------------------------------------------------------------- 1 | # Libvirt/Terraform K8s Lab Environment 2 | 3 | ## Purpose 4 | 5 | This is meant to be a one stop location for studying for the CKA exam where you can bring up a multi-node lab locally with minimal effort and start practicing your Kubernetes kung-fu. This lab will kickstart your nodes up to the point where you need to get them clustered via kubeadm. 6 | 7 | ## Requirements 8 | 9 | This uses the libvirt provider for terraform which, as far as I can see, only creates prebuilt binaries for Linux. As such, Linux as your base host is a requirement. 10 | 11 | I got this all running using Qemu as the backing provider for libvirt. I also used the Ubuntu release version of the libvirt terraform provider (this can be changed in the init.sh script to some other supported Distro though). As such an Ubuntu based host should be used. You will also want to ensure that the folder you cloned this repo to is on a disk with plenty of space and a host that can accommodate the three VMs that will get spun up. 12 | 13 | We need a few base apps and utilities (probably more than this needs to be installed, sorry if the base list of packages isn't 100% correct) 14 | 15 | ```bash 16 | sudo apt install libvirt-clients libvirt-daemon qemu tmux genisoimage 17 | ``` 18 | 19 | Your account must have access to manage libvirt resources as well this should get you started. 20 | 21 | ```bash 22 | sudo usermod -a -G libvirt $(whoami) 23 | newgrp libvirt 24 | ``` 25 | We need to manually pull in the most recent libvirt terraform provider plugin and create an ssh key for use in the deployment. You can do all of this easily enough with a quick make task. 26 | 27 | ```bash 28 | make deps 29 | ``` 30 | 31 | ## The Lab 32 | 33 | This lab consists of 3 virtual machines with the following names and specifications: 34 | 35 | | Node | vCPUS | RAM | Disk | IP | 36 | |---|---|---|---|---| 37 | | k8s-master-1 | 2 | 4GB | 40GB | 172.16.1.11/24 | 38 | | k8s-worker-1 | 2 | 2GB | 40GB | 172.16.1.21/24 | 39 | | k8s-worker-2 | 2 | 2GB | 40GB | 172.16.1.22/24 | 40 | 41 | ```bash 42 | # If all the dependencies are in place then initialize, plan, and apply the terraform manifest to bring things up. 43 | make init plan create 44 | 45 | # Destroy just as easily. 46 | make destroy 47 | 48 | # Access the master node 49 | make ssh/master 50 | 51 | # or the worker nodes 52 | make ssh/worker1 53 | make ssh/worker2 54 | 55 | # or all the nodes at once with synchronized input 56 | make ssh/all 57 | ``` 58 | 59 | You can increase the number of master/worker nodes by updating the variables at the top of the main.tf file. The Makefile was only setup to work with the three nodes though as that's all you really need for most basic lab environments. 60 | 61 | If you need to update/change the Kubernetes component versions please update the main.tf variables. 62 | 63 | **NOTE:** When you bring the lab up for the first time it may take a few minutes to install docker and become fully ready for a Kubernetes installation. 64 | 65 | ## Cloud Init 66 | 67 | The cloud_init.cfg template is used to provide an initial configuration for the instances. This includes; 68 | 69 | - Setting the local hostname (important for the cluster to work properly) 70 | - Adding the docker repo and gpg key 71 | - Adding the kubernetes repo and gpg key 72 | - Installing some initial required packages common to all nodes 73 | - Configure the default user (ubuntu) with passwordless sudo rights. 74 | - Install containerd.io, docker-ce, docker-ce-cli, kubelet, kubeadm, and kubectl (and put kube* marked for hold) 75 | - Deploy docker service as a daemon 76 | 77 | See the cloud_init.cfg for further information or to customize the deployment further. 78 | 79 | ## Extras 80 | 81 | There are some additional configuration tasks that you can use to shortcut a cluster setup found in the `tasks/1-deploy-kubernetes.md` documentation. 82 | 83 | If you are brave and are using asdf-vm along with direnv you may also be able to use the included ansible playbook to deploy your cluster even more quickly. This assumes that the direnv Python virtual environment got setup correctly and that asdf is also working. The original playbook (see link in references) was modified to not install docker and does not do any additional deployment beyond the calico CNI setup and node joins via kubeadm. Nor has any other CNI been tested. 84 | 85 | ```bash 86 | make cluster/deploy 87 | make kube/export/config 88 | ``` 89 | 90 | It wouldn't be too hard to dissect the makefile task to run this playbook manually of course. 91 | 92 | ## Issues 93 | 94 | On Ubuntu distros SELinux is enforced by qemu even if it is disabled globally, this might cause unexpected Could not open '/var/lib/libvirt/images/': Permission denied errors. Double check that security_driver = "none" is uncommented in /etc/libvirt/qemu.conf and issue sudo systemctl restart libvirt-bin to restart the daemon. [source](https://github.com/dmacvicar/terraform-provider-libvirt/issues/546) 95 | 96 | I also had to do some things found [here](https://github.com/jedi4ever/veewee/issues/996) to get this resolved, 97 | 98 | ```bash 99 | sudo groupadd --system libvirt 100 | sudo usermod -a -G libvirt $(whoami) 101 | newgrp libvirt 102 | id $(whoami) 103 | sudo systemctl stop libvirtd.service 104 | sudo nano /etc/libvirt/qemu.conf 105 | sudo echo "user = `whoami`" >> /etc/libvirt/qemu.conf 106 | sudo echo "group = `whoami`" >> /etc/libvirt/qemu.conf 107 | sudo systemctl start libvirtd.service 108 | ``` 109 | 110 | ## Exam Tips 111 | 112 | Here are a few random tips which may or may not help you on exam day. 113 | 114 | **kubectl explain** 115 | 116 | Using `kubectl explain` will show more information on just about everything you may be authoring in YAML manfiests. The format is: `..` for individual properties. Or you can get all available properties by omitting the `.`. With some creativity and the `--recursive=true` flag you can get somthing that looks almost like a YAML snippet. 117 | 118 | ```bash 119 | # Show all available properties for pod.spec 120 | kubectl explain pod.spec | grep '<' 121 | 122 | # Show all available properties for deployment.spec.template.spec.containers 123 | kubectl explain deployment.spec.template.spec.containers --recursive=true | grep '<' 124 | ``` 125 | 126 | ## Links 127 | 128 | [CKA Practice Excercises](https://github.com/alijahnas/CKA-practice-exercises) (initial source of the terraform manifest and a great study guide!) 129 | 130 | [Additional CKA Study Repo 1](https://github.com/walidshaari/Kubernetes-Certified-Administrator) 131 | 132 | [Additional CKA Study Repo 2](https://github.com/dgkanatsios/CKAD-exercises) 133 | 134 | [A Great CKA Study Guide](http://www.kubernet.io/) 135 | 136 | [Very Good Kube Troubleshooting Handbook](https://github.com/feiskyer/kubernetes-handbook/blob/master/en/troubleshooting/index.md) 137 | 138 | [Libvirt Terraform Provider](https://github.com/dmacvicar/terraform-provider-libvirt) 139 | 140 | [xpanes for tmux](https://github.com/greymd/tmux-xpanes) 141 | 142 | [Cloud Init Examples](https://cloudinit.readthedocs.io/en/latest/topics/examples.html) 143 | 144 | [kubeadm via Ansible](https://github.com/kairen/kubeadm-ansible) 145 | -------------------------------------------------------------------------------- /ansible-playbook/.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | .vagrant/ 3 | *.retry 4 | -------------------------------------------------------------------------------- /ansible-playbook/LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright 2017 Kyle Bai(k2r2.bai@gmail.com) 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /ansible-playbook/README.md: -------------------------------------------------------------------------------- 1 | # Kubeadm Ansible Playbook 2 | 3 | Build a Kubernetes cluster using Ansible with kubeadm. The goal is easily install a Kubernetes cluster on machines running: 4 | 5 | - Ubuntu 16.04 6 | - CentOS 7 7 | - Debian 9 8 | 9 | System requirements: 10 | 11 | - Deployment environment must have Ansible `2.4.0+` 12 | - Master and nodes must have passwordless SSH access 13 | 14 | # Usage 15 | 16 | Add the system information gathered above into a file called `hosts.ini`. For example: 17 | ``` 18 | [master] 19 | 192.16.35.12 20 | 21 | [node] 22 | 192.16.35.[10:11] 23 | 24 | [kube-cluster:children] 25 | master 26 | node 27 | ``` 28 | 29 | If you're working with ubuntu, add the following properties to each host `ansible_python_interpreter='python3'`: 30 | ``` 31 | [master] 32 | 192.16.35.12 ansible_python_interpreter='python3' 33 | 34 | [node] 35 | 192.16.35.[10:11] ansible_python_interpreter='python3' 36 | 37 | [kube-cluster:children] 38 | master 39 | node 40 | 41 | ``` 42 | 43 | Before continuing, edit `group_vars/all.yml` to your specified configuration. 44 | 45 | For example, I choose to run `flannel` instead of calico, and thus: 46 | 47 | ```yaml 48 | # Network implementation('flannel', 'calico') 49 | network: flannel 50 | ``` 51 | 52 | **Note:** Depending on your setup, you may need to modify `cni_opts` to an available network interface. By default, `kubeadm-ansible` uses `eth1`. Your default interface may be `eth0`. 53 | 54 | After going through the setup, run the `site.yaml` playbook: 55 | 56 | ```sh 57 | $ ansible-playbook site.yaml 58 | ... 59 | ==> master1: TASK [addon : Create Kubernetes dashboard deployment] ************************** 60 | ==> master1: changed: [192.16.35.12 -> 192.16.35.12] 61 | ==> master1: 62 | ==> master1: PLAY RECAP ********************************************************************* 63 | ==> master1: 192.16.35.10 : ok=18 changed=14 unreachable=0 failed=0 64 | ==> master1: 192.16.35.11 : ok=18 changed=14 unreachable=0 failed=0 65 | ==> master1: 192.16.35.12 : ok=34 changed=29 unreachable=0 failed=0 66 | ``` 67 | 68 | The playbook will download `/etc/kubernetes/admin.conf` file to `$HOME/admin.conf`. 69 | 70 | If it doesn't work download the `admin.conf` from the master node: 71 | 72 | ```sh 73 | $ scp k8s@k8s-master:/etc/kubernetes/admin.conf . 74 | ``` 75 | 76 | Verify cluster is fully running using kubectl: 77 | 78 | ```sh 79 | 80 | $ export KUBECONFIG=~/admin.conf 81 | $ kubectl get node 82 | NAME STATUS AGE VERSION 83 | master1 Ready 22m v1.6.3 84 | node1 Ready 20m v1.6.3 85 | node2 Ready 20m v1.6.3 86 | 87 | $ kubectl get po -n kube-system 88 | NAME READY STATUS RESTARTS AGE 89 | etcd-master1 1/1 Running 0 23m 90 | ... 91 | ``` 92 | 93 | # Resetting the environment 94 | 95 | Finally, reset all kubeadm installed state using `reset-site.yaml` playbook: 96 | 97 | ```sh 98 | $ ansible-playbook reset-site.yaml 99 | ``` 100 | 101 | # Additional features 102 | These are features that you could want to install to make your life easier. 103 | 104 | Enable/disable these features in `group_vars/all.yml` (all disabled by default): 105 | ``` 106 | # Additional feature to install 107 | additional_features: 108 | helm: false 109 | metallb: false 110 | healthcheck: false 111 | ``` 112 | 113 | ## Helm 114 | This will install helm in your cluster (https://helm.sh/) so you can deploy charts. 115 | 116 | ## MetalLB 117 | This will install MetalLB (https://metallb.universe.tf/), very useful if you deploy the cluster locally and you need a load balancer to access the services. 118 | 119 | ## Healthcheck 120 | This will install k8s-healthcheck (https://github.com/emrekenci/k8s-healthcheck), a small application to report cluster status. 121 | 122 | # Utils 123 | Collection of scripts/utilities 124 | 125 | ## Vagrantfile 126 | This Vagrantfile is taken from https://github.com/ecomm-integration-ballerina/kubernetes-cluster and slightly modified to copy ssh keys inside the cluster (install https://github.com/dotless-de/vagrant-vbguest is highly recommended) 127 | 128 | # Tips & Tricks 129 | ## Specify user for Ansible 130 | If you use vagrant or your remote user is root, add this to `hosts.ini` 131 | ``` 132 | [master] 133 | 192.16.35.12 ansible_user='root' 134 | 135 | [node] 136 | 192.16.35.[10:11] ansible_user='root' 137 | ``` 138 | 139 | ## Access Kubernetes Dashboard 140 | As of release 1.7 Dashboard no longer has full admin privileges granted by default, so you need to create a token to access the resources: 141 | ```sh 142 | $ kubectl -n kube-system create sa dashboard 143 | $ kubectl create clusterrolebinding dashboard --clusterrole cluster-admin --serviceaccount=kube-system:dashboard 144 | $ kubectl -n kube-system get sa dashboard -o yaml 145 | apiVersion: v1 146 | kind: ServiceAccount 147 | metadata: 148 | creationTimestamp: 2017-11-27T17:06:41Z 149 | name: dashboard 150 | namespace: kube-system 151 | resourceVersion: "69076" 152 | selfLink: /api/v1/namespaces/kube-system/serviceaccounts/dashboard 153 | uid: 56b880bf-d395-11e7-9528-448a5ba4bd34 154 | secrets: 155 | - name: dashboard-token-vg52j 156 | 157 | $ kubectl -n kube-system describe secrets dashboard-token-vg52j 158 | ... 159 | token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtdG9rZW4tdmc1MmoiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNTZiODgwYmYtZDM5NS0xMWU3LTk1MjgtNDQ4YTViYTRiZDM0Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZCJ9.bVRECfNS4NDmWAFWxGbAi1n9SfQ-TMNafPtF70pbp9Kun9RbC3BNR5NjTEuKjwt8nqZ6k3r09UKJ4dpo2lHtr2RTNAfEsoEGtoMlW8X9lg70ccPB0M1KJiz3c7-gpDUaQRIMNwz42db7Q1dN7HLieD6I4lFsHgk9NPUIVKqJ0p6PNTp99pBwvpvnKX72NIiIvgRwC2cnFr3R6WdUEsuVfuWGdF-jXyc6lS7_kOiXp2yh6Ym_YYIr3SsjYK7XUIPHrBqWjF-KXO_AL3J8J_UebtWSGomYvuXXbbAUefbOK4qopqQ6FzRXQs00KrKa8sfqrKMm_x71Kyqq6RbFECsHPA 160 | 161 | $ kubectl proxy 162 | ``` 163 | > Copy and paste the `token` from above to dashboard. 164 | 165 | Login the dashboard: 166 | - Dashboard: [https://API_SERVER:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/](https://API_SERVER:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/) 167 | - Logging: [https://API_SERVER:8001/api/v1/namespaces/kube-system/services/kibana-logging/proxy/](https://API_SERVER:8001/api/v1/namespaces/kube-system/services/kibana-logging/proxy/) 168 | 169 | -------------------------------------------------------------------------------- /ansible-playbook/ansible.cfg: -------------------------------------------------------------------------------- 1 | [defaults] 2 | roles_path = ./roles 3 | inventory = ./hosts.ini 4 | 5 | remote_tmp = $HOME/.ansible/tmp 6 | local_tmp = $HOME/.ansible/tmp 7 | pipelining = True 8 | become = True 9 | host_key_checking = False 10 | deprecation_warnings = False 11 | callback_whitelist = profile_tasks 12 | -------------------------------------------------------------------------------- /ansible-playbook/group_vars/all.yml: -------------------------------------------------------------------------------- 1 | # Ansible 2 | # ansible_user: root 3 | 4 | # Kubernetes 5 | kube_version: v1.18.0 6 | token: b0f7b8.8d1767876297d85c 7 | 8 | # 1.8.x feature: --feature-gates SelfHosting=true 9 | init_opts: "" 10 | 11 | # Any other additional opts you want to add.. 12 | kubeadm_opts: "" 13 | # For example: 14 | # kubeadm_opts: '--apiserver-cert-extra-sans "k8s.domain.com,kubernetes.domain.com"' 15 | 16 | service_cidr: "10.96.0.0/12" 17 | pod_network_cidr: "192.168.0.0/16" 18 | 19 | # Network implementation('flannel', 'calico', 'canal') 20 | network: calico 21 | 22 | # Change this to an appropriate interface, preferably a private network. 23 | # For example, on DigitalOcean, you would use eth1 as that is the default private network interface. 24 | network_interface: "ens3" 25 | 26 | enable_dashboard: no 27 | 28 | # A list of insecure registries you might need to define 29 | # insecure_registries: [] 30 | insecure_registries: ['gcr.io'] 31 | 32 | systemd_dir: /lib/systemd/system 33 | system_env_dir: /etc/sysconfig 34 | network_dir: /etc/kubernetes/network 35 | kubeadmin_config: /etc/kubernetes/admin.conf 36 | kube_addon_dir: /etc/kubernetes/addon 37 | 38 | # Additional feature to install 39 | additional_features: 40 | helm: false 41 | metallb: false 42 | healthcheck: false 43 | 44 | # temporary directory used by additional features 45 | tmp_dir: /tmp/kubeadm-ansible-files 46 | 47 | -------------------------------------------------------------------------------- /ansible-playbook/group_vars/kube-cluster.yml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | master_ip: "{{ hostvars[groups['master'][0]]['ansible_default_ipv4'].address | default(groups['master'][0]) }}" 4 | -------------------------------------------------------------------------------- /ansible-playbook/hosts.ini: -------------------------------------------------------------------------------- 1 | [master] 2 | 172.16.1.11 ansible_python_interpreter='python3' 3 | 4 | [node] 5 | 172.16.1.[21:22] ansible_python_interpreter='python3' 6 | 7 | [kube-cluster:children] 8 | master 9 | node 10 | 11 | -------------------------------------------------------------------------------- /ansible-playbook/reset-site.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | - hosts: kube-cluster 4 | gather_facts: no 5 | become: yes 6 | tasks: 7 | - name: Reset Kubernetes component 8 | shell: "kubeadm reset --force" 9 | ignore_errors: True 10 | 11 | - name: Delete flannel.1 interface 12 | command: ip link delete flannel.1 13 | when: network == "flannel" or network == "canal" 14 | ignore_errors: True 15 | 16 | - name: Delete cni0 interface 17 | command: ip link delete cni0 18 | when: network == "flannel" 19 | ignore_errors: True 20 | -------------------------------------------------------------------------------- /ansible-playbook/roles/cni/defaults/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | calico_cni_opts: "interface={{ network_interface }}" 4 | flannel_cni_opts: "--iface={{ network_interface }}" 5 | -------------------------------------------------------------------------------- /ansible-playbook/roles/cni/tasks/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | - name: Create Kubernetes addon directory 4 | file: 5 | path: "{{ network_dir }}" 6 | state: directory 7 | 8 | - name: "Copy {{ network }} YAML files" 9 | template: 10 | src: "{{ item }}" 11 | dest: "{{ network_dir }}/{{ item | basename | regex_replace('\\.j2','') }}" 12 | with_fileglob: 13 | - ../templates/{{ network }}*.j2 14 | 15 | - name: "Check {{ network }} daemonset is working" 16 | shell: kubectl --kubeconfig={{ kubeadmin_config }} get ds --all-namespaces | grep {{ network }} 17 | delegate_to: "{{ groups['master'][0] }}" 18 | run_once: true 19 | register: check_net 20 | ignore_errors: true 21 | changed_when: false 22 | 23 | - name: "Create {{ network }} network daemonset" 24 | when: check_net is failed 25 | command: kubectl apply --kubeconfig={{ kubeadmin_config }} -f {{ network_dir }}/ 26 | delegate_to: "{{ groups['master'][0] }}" 27 | run_once: true 28 | -------------------------------------------------------------------------------- /ansible-playbook/roles/cni/templates/calico.yml.j2: -------------------------------------------------------------------------------- 1 | --- 2 | # Source: calico/templates/calico-config.yaml 3 | # This ConfigMap is used to configure a self-hosted Calico installation. 4 | kind: ConfigMap 5 | apiVersion: v1 6 | metadata: 7 | name: calico-config 8 | namespace: kube-system 9 | data: 10 | # Typha is disabled. 11 | typha_service_name: "none" 12 | # Configure the backend to use. 13 | calico_backend: "bird" 14 | 15 | # Configure the MTU to use 16 | veth_mtu: "1440" 17 | 18 | # The CNI network configuration to install on each node. The special 19 | # values in this config will be automatically populated. 20 | cni_network_config: |- 21 | { 22 | "name": "k8s-pod-network", 23 | "cniVersion": "0.3.1", 24 | "plugins": [ 25 | { 26 | "type": "calico", 27 | "log_level": "info", 28 | "datastore_type": "kubernetes", 29 | "nodename": "__KUBERNETES_NODE_NAME__", 30 | "mtu": __CNI_MTU__, 31 | "ipam": { 32 | "type": "calico-ipam" 33 | }, 34 | "policy": { 35 | "type": "k8s" 36 | }, 37 | "kubernetes": { 38 | "kubeconfig": "__KUBECONFIG_FILEPATH__" 39 | } 40 | }, 41 | { 42 | "type": "portmap", 43 | "snat": true, 44 | "capabilities": {"portMappings": true} 45 | } 46 | ] 47 | } 48 | 49 | --- 50 | # Source: calico/templates/kdd-crds.yaml 51 | apiVersion: apiextensions.k8s.io/v1beta1 52 | kind: CustomResourceDefinition 53 | metadata: 54 | name: felixconfigurations.crd.projectcalico.org 55 | spec: 56 | scope: Cluster 57 | group: crd.projectcalico.org 58 | version: v1 59 | names: 60 | kind: FelixConfiguration 61 | plural: felixconfigurations 62 | singular: felixconfiguration 63 | --- 64 | 65 | apiVersion: apiextensions.k8s.io/v1beta1 66 | kind: CustomResourceDefinition 67 | metadata: 68 | name: ipamblocks.crd.projectcalico.org 69 | spec: 70 | scope: Cluster 71 | group: crd.projectcalico.org 72 | version: v1 73 | names: 74 | kind: IPAMBlock 75 | plural: ipamblocks 76 | singular: ipamblock 77 | 78 | --- 79 | 80 | apiVersion: apiextensions.k8s.io/v1beta1 81 | kind: CustomResourceDefinition 82 | metadata: 83 | name: blockaffinities.crd.projectcalico.org 84 | spec: 85 | scope: Cluster 86 | group: crd.projectcalico.org 87 | version: v1 88 | names: 89 | kind: BlockAffinity 90 | plural: blockaffinities 91 | singular: blockaffinity 92 | 93 | --- 94 | 95 | apiVersion: apiextensions.k8s.io/v1beta1 96 | kind: CustomResourceDefinition 97 | metadata: 98 | name: ipamhandles.crd.projectcalico.org 99 | spec: 100 | scope: Cluster 101 | group: crd.projectcalico.org 102 | version: v1 103 | names: 104 | kind: IPAMHandle 105 | plural: ipamhandles 106 | singular: ipamhandle 107 | 108 | --- 109 | 110 | apiVersion: apiextensions.k8s.io/v1beta1 111 | kind: CustomResourceDefinition 112 | metadata: 113 | name: ipamconfigs.crd.projectcalico.org 114 | spec: 115 | scope: Cluster 116 | group: crd.projectcalico.org 117 | version: v1 118 | names: 119 | kind: IPAMConfig 120 | plural: ipamconfigs 121 | singular: ipamconfig 122 | 123 | --- 124 | 125 | apiVersion: apiextensions.k8s.io/v1beta1 126 | kind: CustomResourceDefinition 127 | metadata: 128 | name: bgppeers.crd.projectcalico.org 129 | spec: 130 | scope: Cluster 131 | group: crd.projectcalico.org 132 | version: v1 133 | names: 134 | kind: BGPPeer 135 | plural: bgppeers 136 | singular: bgppeer 137 | 138 | --- 139 | 140 | apiVersion: apiextensions.k8s.io/v1beta1 141 | kind: CustomResourceDefinition 142 | metadata: 143 | name: bgpconfigurations.crd.projectcalico.org 144 | spec: 145 | scope: Cluster 146 | group: crd.projectcalico.org 147 | version: v1 148 | names: 149 | kind: BGPConfiguration 150 | plural: bgpconfigurations 151 | singular: bgpconfiguration 152 | 153 | --- 154 | 155 | apiVersion: apiextensions.k8s.io/v1beta1 156 | kind: CustomResourceDefinition 157 | metadata: 158 | name: ippools.crd.projectcalico.org 159 | spec: 160 | scope: Cluster 161 | group: crd.projectcalico.org 162 | version: v1 163 | names: 164 | kind: IPPool 165 | plural: ippools 166 | singular: ippool 167 | 168 | --- 169 | 170 | apiVersion: apiextensions.k8s.io/v1beta1 171 | kind: CustomResourceDefinition 172 | metadata: 173 | name: hostendpoints.crd.projectcalico.org 174 | spec: 175 | scope: Cluster 176 | group: crd.projectcalico.org 177 | version: v1 178 | names: 179 | kind: HostEndpoint 180 | plural: hostendpoints 181 | singular: hostendpoint 182 | 183 | --- 184 | 185 | apiVersion: apiextensions.k8s.io/v1beta1 186 | kind: CustomResourceDefinition 187 | metadata: 188 | name: clusterinformations.crd.projectcalico.org 189 | spec: 190 | scope: Cluster 191 | group: crd.projectcalico.org 192 | version: v1 193 | names: 194 | kind: ClusterInformation 195 | plural: clusterinformations 196 | singular: clusterinformation 197 | 198 | --- 199 | 200 | apiVersion: apiextensions.k8s.io/v1beta1 201 | kind: CustomResourceDefinition 202 | metadata: 203 | name: globalnetworkpolicies.crd.projectcalico.org 204 | spec: 205 | scope: Cluster 206 | group: crd.projectcalico.org 207 | version: v1 208 | names: 209 | kind: GlobalNetworkPolicy 210 | plural: globalnetworkpolicies 211 | singular: globalnetworkpolicy 212 | 213 | --- 214 | 215 | apiVersion: apiextensions.k8s.io/v1beta1 216 | kind: CustomResourceDefinition 217 | metadata: 218 | name: globalnetworksets.crd.projectcalico.org 219 | spec: 220 | scope: Cluster 221 | group: crd.projectcalico.org 222 | version: v1 223 | names: 224 | kind: GlobalNetworkSet 225 | plural: globalnetworksets 226 | singular: globalnetworkset 227 | 228 | --- 229 | 230 | apiVersion: apiextensions.k8s.io/v1beta1 231 | kind: CustomResourceDefinition 232 | metadata: 233 | name: networkpolicies.crd.projectcalico.org 234 | spec: 235 | scope: Namespaced 236 | group: crd.projectcalico.org 237 | version: v1 238 | names: 239 | kind: NetworkPolicy 240 | plural: networkpolicies 241 | singular: networkpolicy 242 | 243 | --- 244 | 245 | apiVersion: apiextensions.k8s.io/v1beta1 246 | kind: CustomResourceDefinition 247 | metadata: 248 | name: networksets.crd.projectcalico.org 249 | spec: 250 | scope: Namespaced 251 | group: crd.projectcalico.org 252 | version: v1 253 | names: 254 | kind: NetworkSet 255 | plural: networksets 256 | singular: networkset 257 | --- 258 | # Source: calico/templates/rbac.yaml 259 | 260 | # Include a clusterrole for the kube-controllers component, 261 | # and bind it to the calico-kube-controllers serviceaccount. 262 | kind: ClusterRole 263 | apiVersion: rbac.authorization.k8s.io/v1 264 | metadata: 265 | name: calico-kube-controllers 266 | rules: 267 | # Nodes are watched to monitor for deletions. 268 | - apiGroups: [""] 269 | resources: 270 | - nodes 271 | verbs: 272 | - watch 273 | - list 274 | - get 275 | # Pods are queried to check for existence. 276 | - apiGroups: [""] 277 | resources: 278 | - pods 279 | verbs: 280 | - get 281 | # IPAM resources are manipulated when nodes are deleted. 282 | - apiGroups: ["crd.projectcalico.org"] 283 | resources: 284 | - ippools 285 | verbs: 286 | - list 287 | - apiGroups: ["crd.projectcalico.org"] 288 | resources: 289 | - blockaffinities 290 | - ipamblocks 291 | - ipamhandles 292 | verbs: 293 | - get 294 | - list 295 | - create 296 | - update 297 | - delete 298 | # Needs access to update clusterinformations. 299 | - apiGroups: ["crd.projectcalico.org"] 300 | resources: 301 | - clusterinformations 302 | verbs: 303 | - get 304 | - create 305 | - update 306 | --- 307 | kind: ClusterRoleBinding 308 | apiVersion: rbac.authorization.k8s.io/v1 309 | metadata: 310 | name: calico-kube-controllers 311 | roleRef: 312 | apiGroup: rbac.authorization.k8s.io 313 | kind: ClusterRole 314 | name: calico-kube-controllers 315 | subjects: 316 | - kind: ServiceAccount 317 | name: calico-kube-controllers 318 | namespace: kube-system 319 | --- 320 | # Include a clusterrole for the calico-node DaemonSet, 321 | # and bind it to the calico-node serviceaccount. 322 | kind: ClusterRole 323 | apiVersion: rbac.authorization.k8s.io/v1 324 | metadata: 325 | name: calico-node 326 | rules: 327 | # The CNI plugin needs to get pods, nodes, and namespaces. 328 | - apiGroups: [""] 329 | resources: 330 | - pods 331 | - nodes 332 | - namespaces 333 | verbs: 334 | - get 335 | - apiGroups: [""] 336 | resources: 337 | - endpoints 338 | - services 339 | verbs: 340 | # Used to discover service IPs for advertisement. 341 | - watch 342 | - list 343 | # Used to discover Typhas. 344 | - get 345 | - apiGroups: [""] 346 | resources: 347 | - nodes/status 348 | verbs: 349 | # Needed for clearing NodeNetworkUnavailable flag. 350 | - patch 351 | # Calico stores some configuration information in node annotations. 352 | - update 353 | # Watch for changes to Kubernetes NetworkPolicies. 354 | - apiGroups: ["networking.k8s.io"] 355 | resources: 356 | - networkpolicies 357 | verbs: 358 | - watch 359 | - list 360 | # Used by Calico for policy information. 361 | - apiGroups: [""] 362 | resources: 363 | - pods 364 | - namespaces 365 | - serviceaccounts 366 | verbs: 367 | - list 368 | - watch 369 | # The CNI plugin patches pods/status. 370 | - apiGroups: [""] 371 | resources: 372 | - pods/status 373 | verbs: 374 | - patch 375 | # Calico monitors various CRDs for config. 376 | - apiGroups: ["crd.projectcalico.org"] 377 | resources: 378 | - globalfelixconfigs 379 | - felixconfigurations 380 | - bgppeers 381 | - globalbgpconfigs 382 | - bgpconfigurations 383 | - ippools 384 | - ipamblocks 385 | - globalnetworkpolicies 386 | - globalnetworksets 387 | - networkpolicies 388 | - networksets 389 | - clusterinformations 390 | - hostendpoints 391 | verbs: 392 | - get 393 | - list 394 | - watch 395 | # Calico must create and update some CRDs on startup. 396 | - apiGroups: ["crd.projectcalico.org"] 397 | resources: 398 | - ippools 399 | - felixconfigurations 400 | - clusterinformations 401 | verbs: 402 | - create 403 | - update 404 | # Calico stores some configuration information on the node. 405 | - apiGroups: [""] 406 | resources: 407 | - nodes 408 | verbs: 409 | - get 410 | - list 411 | - watch 412 | # These permissions are only requried for upgrade from v2.6, and can 413 | # be removed after upgrade or on fresh installations. 414 | - apiGroups: ["crd.projectcalico.org"] 415 | resources: 416 | - bgpconfigurations 417 | - bgppeers 418 | verbs: 419 | - create 420 | - update 421 | # These permissions are required for Calico CNI to perform IPAM allocations. 422 | - apiGroups: ["crd.projectcalico.org"] 423 | resources: 424 | - blockaffinities 425 | - ipamblocks 426 | - ipamhandles 427 | verbs: 428 | - get 429 | - list 430 | - create 431 | - update 432 | - delete 433 | - apiGroups: ["crd.projectcalico.org"] 434 | resources: 435 | - ipamconfigs 436 | verbs: 437 | - get 438 | # Block affinities must also be watchable by confd for route aggregation. 439 | - apiGroups: ["crd.projectcalico.org"] 440 | resources: 441 | - blockaffinities 442 | verbs: 443 | - watch 444 | # The Calico IPAM migration needs to get daemonsets. These permissions can be 445 | # removed if not upgrading from an installation using host-local IPAM. 446 | - apiGroups: ["apps"] 447 | resources: 448 | - daemonsets 449 | verbs: 450 | - get 451 | --- 452 | apiVersion: rbac.authorization.k8s.io/v1 453 | kind: ClusterRoleBinding 454 | metadata: 455 | name: calico-node 456 | roleRef: 457 | apiGroup: rbac.authorization.k8s.io 458 | kind: ClusterRole 459 | name: calico-node 460 | subjects: 461 | - kind: ServiceAccount 462 | name: calico-node 463 | namespace: kube-system 464 | 465 | --- 466 | # Source: calico/templates/calico-node.yaml 467 | # This manifest installs the calico-node container, as well 468 | # as the CNI plugins and network config on 469 | # each master and worker node in a Kubernetes cluster. 470 | kind: DaemonSet 471 | apiVersion: apps/v1 472 | metadata: 473 | name: calico-node 474 | namespace: kube-system 475 | labels: 476 | k8s-app: calico-node 477 | spec: 478 | selector: 479 | matchLabels: 480 | k8s-app: calico-node 481 | updateStrategy: 482 | type: RollingUpdate 483 | rollingUpdate: 484 | maxUnavailable: 1 485 | template: 486 | metadata: 487 | labels: 488 | k8s-app: calico-node 489 | annotations: 490 | # This, along with the CriticalAddonsOnly toleration below, 491 | # marks the pod as a critical add-on, ensuring it gets 492 | # priority scheduling and that its resources are reserved 493 | # if it ever gets evicted. 494 | scheduler.alpha.kubernetes.io/critical-pod: '' 495 | spec: 496 | nodeSelector: 497 | beta.kubernetes.io/os: linux 498 | hostNetwork: true 499 | tolerations: 500 | # Make sure calico-node gets scheduled on all nodes. 501 | - effect: NoSchedule 502 | operator: Exists 503 | # Mark the pod as a critical add-on for rescheduling. 504 | - key: CriticalAddonsOnly 505 | operator: Exists 506 | - effect: NoExecute 507 | operator: Exists 508 | serviceAccountName: calico-node 509 | # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force 510 | # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods. 511 | terminationGracePeriodSeconds: 0 512 | priorityClassName: system-node-critical 513 | initContainers: 514 | # This container performs upgrade from host-local IPAM to calico-ipam. 515 | # It can be deleted if this is a fresh installation, or if you have already 516 | # upgraded to use calico-ipam. 517 | - name: upgrade-ipam 518 | image: calico/cni:v3.8.4 519 | command: ["/opt/cni/bin/calico-ipam", "-upgrade"] 520 | env: 521 | - name: KUBERNETES_NODE_NAME 522 | valueFrom: 523 | fieldRef: 524 | fieldPath: spec.nodeName 525 | - name: CALICO_NETWORKING_BACKEND 526 | valueFrom: 527 | configMapKeyRef: 528 | name: calico-config 529 | key: calico_backend 530 | volumeMounts: 531 | - mountPath: /var/lib/cni/networks 532 | name: host-local-net-dir 533 | - mountPath: /host/opt/cni/bin 534 | name: cni-bin-dir 535 | securityContext: 536 | privileged: true 537 | # This container installs the CNI binaries 538 | # and CNI network config file on each node. 539 | - name: install-cni 540 | image: calico/cni:v3.8.4 541 | command: ["/install-cni.sh"] 542 | env: 543 | # Name of the CNI config file to create. 544 | - name: CNI_CONF_NAME 545 | value: "10-calico.conflist" 546 | # The CNI network config to install on each node. 547 | - name: CNI_NETWORK_CONFIG 548 | valueFrom: 549 | configMapKeyRef: 550 | name: calico-config 551 | key: cni_network_config 552 | # Set the hostname based on the k8s node name. 553 | - name: KUBERNETES_NODE_NAME 554 | valueFrom: 555 | fieldRef: 556 | fieldPath: spec.nodeName 557 | # CNI MTU Config variable 558 | - name: CNI_MTU 559 | valueFrom: 560 | configMapKeyRef: 561 | name: calico-config 562 | key: veth_mtu 563 | # Prevents the container from sleeping forever. 564 | - name: SLEEP 565 | value: "false" 566 | volumeMounts: 567 | - mountPath: /host/opt/cni/bin 568 | name: cni-bin-dir 569 | - mountPath: /host/etc/cni/net.d 570 | name: cni-net-dir 571 | securityContext: 572 | privileged: true 573 | # Adds a Flex Volume Driver that creates a per-pod Unix Domain Socket to allow Dikastes 574 | # to communicate with Felix over the Policy Sync API. 575 | - name: flexvol-driver 576 | image: calico/pod2daemon-flexvol:v3.8.4 577 | volumeMounts: 578 | - name: flexvol-driver-host 579 | mountPath: /host/driver 580 | securityContext: 581 | privileged: true 582 | containers: 583 | # Runs calico-node container on each Kubernetes node. This 584 | # container programs network policy and routes on each 585 | # host. 586 | - name: calico-node 587 | image: calico/node:v3.8.4 588 | env: 589 | # Use Kubernetes API as the backing datastore. 590 | - name: DATASTORE_TYPE 591 | value: "kubernetes" 592 | # Wait for the datastore. 593 | - name: WAIT_FOR_DATASTORE 594 | value: "true" 595 | # Set based on the k8s node name. 596 | - name: NODENAME 597 | valueFrom: 598 | fieldRef: 599 | fieldPath: spec.nodeName 600 | # Choose the backend to use. 601 | - name: CALICO_NETWORKING_BACKEND 602 | valueFrom: 603 | configMapKeyRef: 604 | name: calico-config 605 | key: calico_backend 606 | # Cluster type to identify the deployment type 607 | - name: CLUSTER_TYPE 608 | value: "k8s,bgp" 609 | # Auto-detect the BGP IP address. 610 | - name: IP 611 | value: "autodetect" 612 | # Enable IPIP 613 | - name: CALICO_IPV4POOL_IPIP 614 | value: "Always" 615 | # Set MTU for tunnel device used if ipip is enabled 616 | - name: FELIX_IPINIPMTU 617 | valueFrom: 618 | configMapKeyRef: 619 | name: calico-config 620 | key: veth_mtu 621 | # The default IPv4 pool to create on startup if none exists. Pod IPs will be 622 | # chosen from this range. Changing this value after installation will have 623 | # no effect. This should fall within `--cluster-cidr`. 624 | - name: CALICO_IPV4POOL_CIDR 625 | value: "{{ pod_network_cidr }}" 626 | # Disable file logging so `kubectl logs` works. 627 | - name: CALICO_DISABLE_FILE_LOGGING 628 | value: "true" 629 | # Set Felix endpoint to host default action to ACCEPT. 630 | - name: FELIX_DEFAULTENDPOINTTOHOSTACTION 631 | value: "ACCEPT" 632 | # Disable IPv6 on Kubernetes. 633 | - name: FELIX_IPV6SUPPORT 634 | value: "false" 635 | # Set Felix logging to "info" 636 | - name: FELIX_LOGSEVERITYSCREEN 637 | value: "info" 638 | - name: FELIX_HEALTHENABLED 639 | value: "true" 640 | securityContext: 641 | privileged: true 642 | resources: 643 | requests: 644 | cpu: 250m 645 | livenessProbe: 646 | httpGet: 647 | path: /liveness 648 | port: 9099 649 | host: localhost 650 | periodSeconds: 10 651 | initialDelaySeconds: 10 652 | failureThreshold: 6 653 | readinessProbe: 654 | exec: 655 | command: 656 | - /bin/calico-node 657 | - -bird-ready 658 | - -felix-ready 659 | periodSeconds: 10 660 | volumeMounts: 661 | - mountPath: /lib/modules 662 | name: lib-modules 663 | readOnly: true 664 | - mountPath: /run/xtables.lock 665 | name: xtables-lock 666 | readOnly: false 667 | - mountPath: /var/run/calico 668 | name: var-run-calico 669 | readOnly: false 670 | - mountPath: /var/lib/calico 671 | name: var-lib-calico 672 | readOnly: false 673 | - name: policysync 674 | mountPath: /var/run/nodeagent 675 | volumes: 676 | # Used by calico-node. 677 | - name: lib-modules 678 | hostPath: 679 | path: /lib/modules 680 | - name: var-run-calico 681 | hostPath: 682 | path: /var/run/calico 683 | - name: var-lib-calico 684 | hostPath: 685 | path: /var/lib/calico 686 | - name: xtables-lock 687 | hostPath: 688 | path: /run/xtables.lock 689 | type: FileOrCreate 690 | # Used to install CNI. 691 | - name: cni-bin-dir 692 | hostPath: 693 | path: /opt/cni/bin 694 | - name: cni-net-dir 695 | hostPath: 696 | path: /etc/cni/net.d 697 | # Mount in the directory for host-local IPAM allocations. This is 698 | # used when upgrading from host-local to calico-ipam, and can be removed 699 | # if not using the upgrade-ipam init container. 700 | - name: host-local-net-dir 701 | hostPath: 702 | path: /var/lib/cni/networks 703 | # Used to create per-pod Unix Domain Sockets 704 | - name: policysync 705 | hostPath: 706 | type: DirectoryOrCreate 707 | path: /var/run/nodeagent 708 | # Used to install Flex Volume Driver 709 | - name: flexvol-driver-host 710 | hostPath: 711 | type: DirectoryOrCreate 712 | path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds 713 | --- 714 | 715 | apiVersion: v1 716 | kind: ServiceAccount 717 | metadata: 718 | name: calico-node 719 | namespace: kube-system 720 | 721 | --- 722 | # Source: calico/templates/calico-kube-controllers.yaml 723 | 724 | # See https://github.com/projectcalico/kube-controllers 725 | apiVersion: apps/v1 726 | kind: Deployment 727 | metadata: 728 | name: calico-kube-controllers 729 | namespace: kube-system 730 | labels: 731 | k8s-app: calico-kube-controllers 732 | spec: 733 | # The controllers can only have a single active instance. 734 | replicas: 1 735 | selector: 736 | matchLabels: 737 | k8s-app: calico-kube-controllers 738 | strategy: 739 | type: Recreate 740 | template: 741 | metadata: 742 | name: calico-kube-controllers 743 | namespace: kube-system 744 | labels: 745 | k8s-app: calico-kube-controllers 746 | annotations: 747 | scheduler.alpha.kubernetes.io/critical-pod: '' 748 | spec: 749 | nodeSelector: 750 | beta.kubernetes.io/os: linux 751 | tolerations: 752 | # Mark the pod as a critical add-on for rescheduling. 753 | - key: CriticalAddonsOnly 754 | operator: Exists 755 | - key: node-role.kubernetes.io/master 756 | effect: NoSchedule 757 | serviceAccountName: calico-kube-controllers 758 | priorityClassName: system-cluster-critical 759 | containers: 760 | - name: calico-kube-controllers 761 | image: calico/kube-controllers:v3.8.4 762 | env: 763 | # Choose which controllers to run. 764 | - name: ENABLED_CONTROLLERS 765 | value: node 766 | - name: DATASTORE_TYPE 767 | value: kubernetes 768 | readinessProbe: 769 | exec: 770 | command: 771 | - /usr/bin/check-status 772 | - -r 773 | 774 | --- 775 | 776 | apiVersion: v1 777 | kind: ServiceAccount 778 | metadata: 779 | name: calico-kube-controllers 780 | namespace: kube-system 781 | --- 782 | # Source: calico/templates/calico-etcd-secrets.yaml 783 | 784 | --- 785 | # Source: calico/templates/calico-typha.yaml 786 | 787 | --- 788 | # Source: calico/templates/configure-canal.yaml 789 | 790 | 791 | -------------------------------------------------------------------------------- /ansible-playbook/roles/cni/templates/canal.yml.j2: -------------------------------------------------------------------------------- 1 | --- 2 | # Source: calico/templates/calico-config.yaml 3 | # This ConfigMap is used to configure a self-hosted Canal installation. 4 | kind: ConfigMap 5 | apiVersion: v1 6 | metadata: 7 | name: canal-config 8 | namespace: kube-system 9 | data: 10 | # Typha is disabled. 11 | typha_service_name: "none" 12 | # The interface used by canal for host <-> host communication. 13 | # If left blank, then the interface is chosen using the node's 14 | # default route. 15 | canal_iface: "" 16 | 17 | # Whether or not to masquerade traffic to destinations not within 18 | # the pod network. 19 | masquerade: "true" 20 | 21 | # The CNI network configuration to install on each node. The special 22 | # values in this config will be automatically populated. 23 | cni_network_config: |- 24 | { 25 | "name": "k8s-pod-network", 26 | "cniVersion": "0.3.1", 27 | "plugins": [ 28 | { 29 | "type": "calico", 30 | "log_level": "info", 31 | "datastore_type": "kubernetes", 32 | "nodename": "__KUBERNETES_NODE_NAME__", 33 | "ipam": { 34 | "type": "host-local", 35 | "subnet": "usePodCidr" 36 | }, 37 | "policy": { 38 | "type": "k8s" 39 | }, 40 | "kubernetes": { 41 | "kubeconfig": "__KUBECONFIG_FILEPATH__" 42 | } 43 | }, 44 | { 45 | "type": "portmap", 46 | "snat": true, 47 | "capabilities": {"portMappings": true} 48 | } 49 | ] 50 | } 51 | 52 | # Flannel network configuration. Mounted into the flannel container. 53 | net-conf.json: | 54 | { 55 | "Network": "{{ pod_network_cidr }}", 56 | "Backend": { 57 | "Type": "vxlan" 58 | } 59 | } 60 | 61 | --- 62 | # Source: calico/templates/kdd-crds.yaml 63 | apiVersion: apiextensions.k8s.io/v1beta1 64 | kind: CustomResourceDefinition 65 | metadata: 66 | name: felixconfigurations.crd.projectcalico.org 67 | spec: 68 | scope: Cluster 69 | group: crd.projectcalico.org 70 | version: v1 71 | names: 72 | kind: FelixConfiguration 73 | plural: felixconfigurations 74 | singular: felixconfiguration 75 | --- 76 | 77 | apiVersion: apiextensions.k8s.io/v1beta1 78 | kind: CustomResourceDefinition 79 | metadata: 80 | name: bgpconfigurations.crd.projectcalico.org 81 | spec: 82 | scope: Cluster 83 | group: crd.projectcalico.org 84 | version: v1 85 | names: 86 | kind: BGPConfiguration 87 | plural: bgpconfigurations 88 | singular: bgpconfiguration 89 | 90 | --- 91 | 92 | apiVersion: apiextensions.k8s.io/v1beta1 93 | kind: CustomResourceDefinition 94 | metadata: 95 | name: ippools.crd.projectcalico.org 96 | spec: 97 | scope: Cluster 98 | group: crd.projectcalico.org 99 | version: v1 100 | names: 101 | kind: IPPool 102 | plural: ippools 103 | singular: ippool 104 | 105 | --- 106 | 107 | apiVersion: apiextensions.k8s.io/v1beta1 108 | kind: CustomResourceDefinition 109 | metadata: 110 | name: hostendpoints.crd.projectcalico.org 111 | spec: 112 | scope: Cluster 113 | group: crd.projectcalico.org 114 | version: v1 115 | names: 116 | kind: HostEndpoint 117 | plural: hostendpoints 118 | singular: hostendpoint 119 | 120 | --- 121 | 122 | apiVersion: apiextensions.k8s.io/v1beta1 123 | kind: CustomResourceDefinition 124 | metadata: 125 | name: clusterinformations.crd.projectcalico.org 126 | spec: 127 | scope: Cluster 128 | group: crd.projectcalico.org 129 | version: v1 130 | names: 131 | kind: ClusterInformation 132 | plural: clusterinformations 133 | singular: clusterinformation 134 | 135 | --- 136 | 137 | apiVersion: apiextensions.k8s.io/v1beta1 138 | kind: CustomResourceDefinition 139 | metadata: 140 | name: globalnetworkpolicies.crd.projectcalico.org 141 | spec: 142 | scope: Cluster 143 | group: crd.projectcalico.org 144 | version: v1 145 | names: 146 | kind: GlobalNetworkPolicy 147 | plural: globalnetworkpolicies 148 | singular: globalnetworkpolicy 149 | 150 | --- 151 | 152 | apiVersion: apiextensions.k8s.io/v1beta1 153 | kind: CustomResourceDefinition 154 | metadata: 155 | name: globalnetworksets.crd.projectcalico.org 156 | spec: 157 | scope: Cluster 158 | group: crd.projectcalico.org 159 | version: v1 160 | names: 161 | kind: GlobalNetworkSet 162 | plural: globalnetworksets 163 | singular: globalnetworkset 164 | 165 | --- 166 | 167 | apiVersion: apiextensions.k8s.io/v1beta1 168 | kind: CustomResourceDefinition 169 | metadata: 170 | name: networkpolicies.crd.projectcalico.org 171 | spec: 172 | scope: Namespaced 173 | group: crd.projectcalico.org 174 | version: v1 175 | names: 176 | kind: NetworkPolicy 177 | plural: networkpolicies 178 | singular: networkpolicy 179 | 180 | --- 181 | 182 | apiVersion: apiextensions.k8s.io/v1beta1 183 | kind: CustomResourceDefinition 184 | metadata: 185 | name: networksets.crd.projectcalico.org 186 | spec: 187 | scope: Namespaced 188 | group: crd.projectcalico.org 189 | version: v1 190 | names: 191 | kind: NetworkSet 192 | plural: networksets 193 | singular: networkset 194 | --- 195 | # Source: calico/templates/rbac.yaml 196 | 197 | # Include a clusterrole for the calico-node DaemonSet, 198 | # and bind it to the calico-node serviceaccount. 199 | kind: ClusterRole 200 | apiVersion: rbac.authorization.k8s.io/v1 201 | metadata: 202 | name: calico-node 203 | rules: 204 | # The CNI plugin needs to get pods, nodes, and namespaces. 205 | - apiGroups: [""] 206 | resources: 207 | - pods 208 | - nodes 209 | - namespaces 210 | verbs: 211 | - get 212 | - apiGroups: [""] 213 | resources: 214 | - endpoints 215 | - services 216 | verbs: 217 | # Used to discover service IPs for advertisement. 218 | - watch 219 | - list 220 | # Used to discover Typhas. 221 | - get 222 | - apiGroups: [""] 223 | resources: 224 | - nodes/status 225 | verbs: 226 | # Needed for clearing NodeNetworkUnavailable flag. 227 | - patch 228 | # Calico stores some configuration information in node annotations. 229 | - update 230 | # Watch for changes to Kubernetes NetworkPolicies. 231 | - apiGroups: ["networking.k8s.io"] 232 | resources: 233 | - networkpolicies 234 | verbs: 235 | - watch 236 | - list 237 | # Used by Calico for policy information. 238 | - apiGroups: [""] 239 | resources: 240 | - pods 241 | - namespaces 242 | - serviceaccounts 243 | verbs: 244 | - list 245 | - watch 246 | # The CNI plugin patches pods/status. 247 | - apiGroups: [""] 248 | resources: 249 | - pods/status 250 | verbs: 251 | - patch 252 | # Calico monitors various CRDs for config. 253 | - apiGroups: ["crd.projectcalico.org"] 254 | resources: 255 | - globalfelixconfigs 256 | - felixconfigurations 257 | - bgppeers 258 | - globalbgpconfigs 259 | - bgpconfigurations 260 | - ippools 261 | - ipamblocks 262 | - globalnetworkpolicies 263 | - globalnetworksets 264 | - networkpolicies 265 | - networksets 266 | - clusterinformations 267 | - hostendpoints 268 | verbs: 269 | - get 270 | - list 271 | - watch 272 | # Calico must create and update some CRDs on startup. 273 | - apiGroups: ["crd.projectcalico.org"] 274 | resources: 275 | - ippools 276 | - felixconfigurations 277 | - clusterinformations 278 | verbs: 279 | - create 280 | - update 281 | # Calico stores some configuration information on the node. 282 | - apiGroups: [""] 283 | resources: 284 | - nodes 285 | verbs: 286 | - get 287 | - list 288 | - watch 289 | # These permissions are only requried for upgrade from v2.6, and can 290 | # be removed after upgrade or on fresh installations. 291 | - apiGroups: ["crd.projectcalico.org"] 292 | resources: 293 | - bgpconfigurations 294 | - bgppeers 295 | verbs: 296 | - create 297 | - update 298 | --- 299 | # Flannel ClusterRole 300 | # Pulled from https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel-rbac.yml 301 | kind: ClusterRole 302 | apiVersion: rbac.authorization.k8s.io/v1 303 | metadata: 304 | name: flannel 305 | rules: 306 | - apiGroups: [""] 307 | resources: 308 | - pods 309 | verbs: 310 | - get 311 | - apiGroups: [""] 312 | resources: 313 | - nodes 314 | verbs: 315 | - list 316 | - watch 317 | - apiGroups: [""] 318 | resources: 319 | - nodes/status 320 | verbs: 321 | - patch 322 | --- 323 | # Bind the flannel ClusterRole to the canal ServiceAccount. 324 | kind: ClusterRoleBinding 325 | apiVersion: rbac.authorization.k8s.io/v1 326 | metadata: 327 | name: canal-flannel 328 | roleRef: 329 | apiGroup: rbac.authorization.k8s.io 330 | kind: ClusterRole 331 | name: flannel 332 | subjects: 333 | - kind: ServiceAccount 334 | name: canal 335 | namespace: kube-system 336 | --- 337 | apiVersion: rbac.authorization.k8s.io/v1 338 | kind: ClusterRoleBinding 339 | metadata: 340 | name: canal-calico 341 | roleRef: 342 | apiGroup: rbac.authorization.k8s.io 343 | kind: ClusterRole 344 | name: calico-node 345 | subjects: 346 | - kind: ServiceAccount 347 | name: canal 348 | namespace: kube-system 349 | 350 | --- 351 | # Source: calico/templates/calico-node.yaml 352 | # This manifest installs the canal container, as well 353 | # as the CNI plugins and network config on 354 | # each master and worker node in a Kubernetes cluster. 355 | kind: DaemonSet 356 | apiVersion: apps/v1 357 | metadata: 358 | name: canal 359 | namespace: kube-system 360 | labels: 361 | k8s-app: canal 362 | spec: 363 | selector: 364 | matchLabels: 365 | k8s-app: canal 366 | updateStrategy: 367 | type: RollingUpdate 368 | rollingUpdate: 369 | maxUnavailable: 1 370 | template: 371 | metadata: 372 | labels: 373 | k8s-app: canal 374 | annotations: 375 | # This, along with the CriticalAddonsOnly toleration below, 376 | # marks the pod as a critical add-on, ensuring it gets 377 | # priority scheduling and that its resources are reserved 378 | # if it ever gets evicted. 379 | scheduler.alpha.kubernetes.io/critical-pod: '' 380 | spec: 381 | nodeSelector: 382 | beta.kubernetes.io/os: linux 383 | hostNetwork: true 384 | tolerations: 385 | # Make sure canal gets scheduled on all nodes. 386 | - effect: NoSchedule 387 | operator: Exists 388 | # Mark the pod as a critical add-on for rescheduling. 389 | - key: CriticalAddonsOnly 390 | operator: Exists 391 | - effect: NoExecute 392 | operator: Exists 393 | serviceAccountName: canal 394 | # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force 395 | # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods. 396 | terminationGracePeriodSeconds: 0 397 | priorityClassName: system-node-critical 398 | initContainers: 399 | # This container installs the CNI binaries 400 | # and CNI network config file on each node. 401 | - name: install-cni 402 | image: calico/cni:v3.8.2 403 | command: ["/install-cni.sh"] 404 | env: 405 | # Name of the CNI config file to create. 406 | - name: CNI_CONF_NAME 407 | value: "10-canal.conflist" 408 | # The CNI network config to install on each node. 409 | - name: CNI_NETWORK_CONFIG 410 | valueFrom: 411 | configMapKeyRef: 412 | name: canal-config 413 | key: cni_network_config 414 | # Set the hostname based on the k8s node name. 415 | - name: KUBERNETES_NODE_NAME 416 | valueFrom: 417 | fieldRef: 418 | fieldPath: spec.nodeName 419 | # Prevents the container from sleeping forever. 420 | - name: SLEEP 421 | value: "false" 422 | volumeMounts: 423 | - mountPath: /host/opt/cni/bin 424 | name: cni-bin-dir 425 | - mountPath: /host/etc/cni/net.d 426 | name: cni-net-dir 427 | # Adds a Flex Volume Driver that creates a per-pod Unix Domain Socket to allow Dikastes 428 | # to communicate with Felix over the Policy Sync API. 429 | - name: flexvol-driver 430 | image: calico/pod2daemon-flexvol:v3.8.2 431 | volumeMounts: 432 | - name: flexvol-driver-host 433 | mountPath: /host/driver 434 | containers: 435 | # Runs canal container on each Kubernetes node. This 436 | # container programs network policy and routes on each 437 | # host. 438 | - name: calico-node 439 | image: calico/node:v3.8.2 440 | env: 441 | # Use Kubernetes API as the backing datastore. 442 | - name: DATASTORE_TYPE 443 | value: "kubernetes" 444 | # Configure route aggregation based on pod CIDR. 445 | - name: USE_POD_CIDR 446 | value: "true" 447 | # Wait for the datastore. 448 | - name: WAIT_FOR_DATASTORE 449 | value: "true" 450 | # Set based on the k8s node name. 451 | - name: NODENAME 452 | valueFrom: 453 | fieldRef: 454 | fieldPath: spec.nodeName 455 | # Don't enable BGP. 456 | - name: CALICO_NETWORKING_BACKEND 457 | value: "none" 458 | # Cluster type to identify the deployment type 459 | - name: CLUSTER_TYPE 460 | value: "k8s,canal" 461 | # Period, in seconds, at which felix re-applies all iptables state 462 | - name: FELIX_IPTABLESREFRESHINTERVAL 463 | value: "60" 464 | # No IP address needed. 465 | - name: IP 466 | value: "" 467 | # The default IPv4 pool to create on startup if none exists. Pod IPs will be 468 | # chosen from this range. Changing this value after installation will have 469 | # no effect. This should fall within `--cluster-cidr`. 470 | - name: CALICO_IPV4POOL_CIDR 471 | value: "192.168.0.0/16" 472 | # Disable file logging so `kubectl logs` works. 473 | - name: CALICO_DISABLE_FILE_LOGGING 474 | value: "true" 475 | # Set Felix endpoint to host default action to ACCEPT. 476 | - name: FELIX_DEFAULTENDPOINTTOHOSTACTION 477 | value: "ACCEPT" 478 | # Disable IPv6 on Kubernetes. 479 | - name: FELIX_IPV6SUPPORT 480 | value: "false" 481 | # Set Felix logging to "info" 482 | - name: FELIX_LOGSEVERITYSCREEN 483 | value: "info" 484 | - name: FELIX_HEALTHENABLED 485 | value: "true" 486 | securityContext: 487 | privileged: true 488 | resources: 489 | requests: 490 | cpu: 250m 491 | livenessProbe: 492 | httpGet: 493 | path: /liveness 494 | port: 9099 495 | host: localhost 496 | periodSeconds: 10 497 | initialDelaySeconds: 10 498 | failureThreshold: 6 499 | readinessProbe: 500 | httpGet: 501 | path: /readiness 502 | port: 9099 503 | host: localhost 504 | periodSeconds: 10 505 | volumeMounts: 506 | - mountPath: /lib/modules 507 | name: lib-modules 508 | readOnly: true 509 | - mountPath: /run/xtables.lock 510 | name: xtables-lock 511 | readOnly: false 512 | - mountPath: /var/run/calico 513 | name: var-run-calico 514 | readOnly: false 515 | - mountPath: /var/lib/calico 516 | name: var-lib-calico 517 | readOnly: false 518 | - name: policysync 519 | mountPath: /var/run/nodeagent 520 | # This container runs flannel using the kube-subnet-mgr backend 521 | # for allocating subnets. 522 | - name: kube-flannel 523 | image: quay.io/coreos/flannel:v0.11.0 524 | command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ] 525 | securityContext: 526 | privileged: true 527 | env: 528 | - name: POD_NAME 529 | valueFrom: 530 | fieldRef: 531 | fieldPath: metadata.name 532 | - name: POD_NAMESPACE 533 | valueFrom: 534 | fieldRef: 535 | fieldPath: metadata.namespace 536 | - name: FLANNELD_IFACE 537 | valueFrom: 538 | configMapKeyRef: 539 | name: canal-config 540 | key: canal_iface 541 | - name: FLANNELD_IP_MASQ 542 | valueFrom: 543 | configMapKeyRef: 544 | name: canal-config 545 | key: masquerade 546 | volumeMounts: 547 | - mountPath: /run/xtables.lock 548 | name: xtables-lock 549 | readOnly: false 550 | - name: flannel-cfg 551 | mountPath: /etc/kube-flannel/ 552 | volumes: 553 | # Used by canal. 554 | - name: lib-modules 555 | hostPath: 556 | path: /lib/modules 557 | - name: var-run-calico 558 | hostPath: 559 | path: /var/run/calico 560 | - name: var-lib-calico 561 | hostPath: 562 | path: /var/lib/calico 563 | - name: xtables-lock 564 | hostPath: 565 | path: /run/xtables.lock 566 | type: FileOrCreate 567 | # Used by flannel. 568 | - name: flannel-cfg 569 | configMap: 570 | name: canal-config 571 | # Used to install CNI. 572 | - name: cni-bin-dir 573 | hostPath: 574 | path: /opt/cni/bin 575 | - name: cni-net-dir 576 | hostPath: 577 | path: /etc/cni/net.d 578 | # Used to create per-pod Unix Domain Sockets 579 | - name: policysync 580 | hostPath: 581 | type: DirectoryOrCreate 582 | path: /var/run/nodeagent 583 | # Used to install Flex Volume Driver 584 | - name: flexvol-driver-host 585 | hostPath: 586 | type: DirectoryOrCreate 587 | path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds 588 | --- 589 | 590 | apiVersion: v1 591 | kind: ServiceAccount 592 | metadata: 593 | name: canal 594 | namespace: kube-system 595 | 596 | --- 597 | # Source: calico/templates/calico-etcd-secrets.yaml 598 | 599 | --- 600 | # Source: calico/templates/calico-kube-controllers.yaml 601 | 602 | --- 603 | # Source: calico/templates/calico-typha.yaml 604 | 605 | --- 606 | # Source: calico/templates/configure-canal.yaml 607 | 608 | 609 | -------------------------------------------------------------------------------- /ansible-playbook/roles/cni/templates/flannel.yml.j2: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: policy/v1beta1 3 | kind: PodSecurityPolicy 4 | metadata: 5 | name: psp.flannel.unprivileged 6 | annotations: 7 | seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default 8 | seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default 9 | apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default 10 | apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default 11 | spec: 12 | privileged: false 13 | volumes: 14 | - configMap 15 | - secret 16 | - emptyDir 17 | - hostPath 18 | allowedHostPaths: 19 | - pathPrefix: "/etc/cni/net.d" 20 | - pathPrefix: "/etc/kube-flannel" 21 | - pathPrefix: "/run/flannel" 22 | readOnlyRootFilesystem: false 23 | # Users and groups 24 | runAsUser: 25 | rule: RunAsAny 26 | supplementalGroups: 27 | rule: RunAsAny 28 | fsGroup: 29 | rule: RunAsAny 30 | # Privilege Escalation 31 | allowPrivilegeEscalation: false 32 | defaultAllowPrivilegeEscalation: false 33 | # Capabilities 34 | allowedCapabilities: ['NET_ADMIN'] 35 | defaultAddCapabilities: [] 36 | requiredDropCapabilities: [] 37 | # Host namespaces 38 | hostPID: false 39 | hostIPC: false 40 | hostNetwork: true 41 | hostPorts: 42 | - min: 0 43 | max: 65535 44 | # SELinux 45 | seLinux: 46 | # SELinux is unsed in CaaSP 47 | rule: 'RunAsAny' 48 | --- 49 | kind: ClusterRole 50 | apiVersion: rbac.authorization.k8s.io/v1beta1 51 | metadata: 52 | name: flannel 53 | rules: 54 | - apiGroups: ['extensions'] 55 | resources: ['podsecuritypolicies'] 56 | verbs: ['use'] 57 | resourceNames: ['psp.flannel.unprivileged'] 58 | - apiGroups: 59 | - "" 60 | resources: 61 | - pods 62 | verbs: 63 | - get 64 | - apiGroups: 65 | - "" 66 | resources: 67 | - nodes 68 | verbs: 69 | - list 70 | - watch 71 | - apiGroups: 72 | - "" 73 | resources: 74 | - nodes/status 75 | verbs: 76 | - patch 77 | --- 78 | kind: ClusterRoleBinding 79 | apiVersion: rbac.authorization.k8s.io/v1beta1 80 | metadata: 81 | name: flannel 82 | roleRef: 83 | apiGroup: rbac.authorization.k8s.io 84 | kind: ClusterRole 85 | name: flannel 86 | subjects: 87 | - kind: ServiceAccount 88 | name: flannel 89 | namespace: kube-system 90 | --- 91 | apiVersion: v1 92 | kind: ServiceAccount 93 | metadata: 94 | name: flannel 95 | namespace: kube-system 96 | --- 97 | kind: ConfigMap 98 | apiVersion: v1 99 | metadata: 100 | name: kube-flannel-cfg 101 | namespace: kube-system 102 | labels: 103 | tier: node 104 | app: flannel 105 | data: 106 | cni-conf.json: | 107 | { 108 | "cniVersion": "0.2.0", 109 | "name": "cbr0", 110 | "plugins": [ 111 | { 112 | "type": "flannel", 113 | "delegate": { 114 | "hairpinMode": true, 115 | "isDefaultGateway": true 116 | } 117 | }, 118 | { 119 | "type": "portmap", 120 | "capabilities": { 121 | "portMappings": true 122 | } 123 | } 124 | ] 125 | } 126 | net-conf.json: | 127 | { 128 | "Network": "{{ pod_network_cidr }}", 129 | "Backend": { 130 | "Type": "vxlan" 131 | } 132 | } 133 | --- 134 | apiVersion: apps/v1 135 | kind: DaemonSet 136 | metadata: 137 | name: kube-flannel-ds-amd64 138 | namespace: kube-system 139 | labels: 140 | tier: node 141 | app: flannel 142 | spec: 143 | selector: 144 | matchLabels: 145 | app: flannel 146 | template: 147 | metadata: 148 | labels: 149 | tier: node 150 | app: flannel 151 | spec: 152 | affinity: 153 | nodeAffinity: 154 | requiredDuringSchedulingIgnoredDuringExecution: 155 | nodeSelectorTerms: 156 | - matchExpressions: 157 | - key: beta.kubernetes.io/os 158 | operator: In 159 | values: 160 | - linux 161 | - key: beta.kubernetes.io/arch 162 | operator: In 163 | values: 164 | - amd64 165 | hostNetwork: true 166 | tolerations: 167 | - operator: Exists 168 | effect: NoSchedule 169 | serviceAccountName: flannel 170 | initContainers: 171 | - name: install-cni 172 | image: quay.io/coreos/flannel:v0.11.0-amd64 173 | command: 174 | - cp 175 | args: 176 | - -f 177 | - /etc/kube-flannel/cni-conf.json 178 | - /etc/cni/net.d/10-flannel.conflist 179 | volumeMounts: 180 | - name: cni 181 | mountPath: /etc/cni/net.d 182 | - name: flannel-cfg 183 | mountPath: /etc/kube-flannel/ 184 | containers: 185 | - name: kube-flannel 186 | image: quay.io/coreos/flannel:v0.11.0-amd64 187 | command: 188 | - /opt/bin/flanneld 189 | args: 190 | - --ip-masq 191 | {% if network_interface != '' -%} 192 | - {{ flannel_cni_opts }} 193 | {% endif -%} 194 | - --kube-subnet-mgr 195 | resources: 196 | requests: 197 | cpu: "100m" 198 | memory: "50Mi" 199 | limits: 200 | cpu: "100m" 201 | memory: "50Mi" 202 | securityContext: 203 | privileged: false 204 | capabilities: 205 | add: ["NET_ADMIN"] 206 | env: 207 | - name: POD_NAME 208 | valueFrom: 209 | fieldRef: 210 | fieldPath: metadata.name 211 | - name: POD_NAMESPACE 212 | valueFrom: 213 | fieldRef: 214 | fieldPath: metadata.namespace 215 | volumeMounts: 216 | - name: run 217 | mountPath: /run/flannel 218 | - name: flannel-cfg 219 | mountPath: /etc/kube-flannel/ 220 | volumes: 221 | - name: run 222 | hostPath: 223 | path: /run/flannel 224 | - name: cni 225 | hostPath: 226 | path: /etc/cni/net.d 227 | - name: flannel-cfg 228 | configMap: 229 | name: kube-flannel-cfg 230 | --- 231 | apiVersion: apps/v1 232 | kind: DaemonSet 233 | metadata: 234 | name: kube-flannel-ds-arm64 235 | namespace: kube-system 236 | labels: 237 | tier: node 238 | app: flannel 239 | spec: 240 | selector: 241 | matchLabels: 242 | app: flannel 243 | template: 244 | metadata: 245 | labels: 246 | tier: node 247 | app: flannel 248 | spec: 249 | affinity: 250 | nodeAffinity: 251 | requiredDuringSchedulingIgnoredDuringExecution: 252 | nodeSelectorTerms: 253 | - matchExpressions: 254 | - key: beta.kubernetes.io/os 255 | operator: In 256 | values: 257 | - linux 258 | - key: beta.kubernetes.io/arch 259 | operator: In 260 | values: 261 | - arm64 262 | hostNetwork: true 263 | tolerations: 264 | - operator: Exists 265 | effect: NoSchedule 266 | serviceAccountName: flannel 267 | initContainers: 268 | - name: install-cni 269 | image: quay.io/coreos/flannel:v0.11.0-arm64 270 | command: 271 | - cp 272 | args: 273 | - -f 274 | - /etc/kube-flannel/cni-conf.json 275 | - /etc/cni/net.d/10-flannel.conflist 276 | volumeMounts: 277 | - name: cni 278 | mountPath: /etc/cni/net.d 279 | - name: flannel-cfg 280 | mountPath: /etc/kube-flannel/ 281 | containers: 282 | - name: kube-flannel 283 | image: quay.io/coreos/flannel:v0.11.0-arm64 284 | command: 285 | - /opt/bin/flanneld 286 | args: 287 | - --ip-masq 288 | {% if network_interface != '' -%} 289 | - {{ flannel_cni_opts }} 290 | {% endif -%} 291 | - --kube-subnet-mgr 292 | resources: 293 | requests: 294 | cpu: "100m" 295 | memory: "50Mi" 296 | limits: 297 | cpu: "100m" 298 | memory: "50Mi" 299 | securityContext: 300 | privileged: false 301 | capabilities: 302 | add: ["NET_ADMIN"] 303 | env: 304 | - name: POD_NAME 305 | valueFrom: 306 | fieldRef: 307 | fieldPath: metadata.name 308 | - name: POD_NAMESPACE 309 | valueFrom: 310 | fieldRef: 311 | fieldPath: metadata.namespace 312 | volumeMounts: 313 | - name: run 314 | mountPath: /run/flannel 315 | - name: flannel-cfg 316 | mountPath: /etc/kube-flannel/ 317 | volumes: 318 | - name: run 319 | hostPath: 320 | path: /run/flannel 321 | - name: cni 322 | hostPath: 323 | path: /etc/cni/net.d 324 | - name: flannel-cfg 325 | configMap: 326 | name: kube-flannel-cfg 327 | --- 328 | apiVersion: apps/v1 329 | kind: DaemonSet 330 | metadata: 331 | name: kube-flannel-ds-arm 332 | namespace: kube-system 333 | labels: 334 | tier: node 335 | app: flannel 336 | spec: 337 | selector: 338 | matchLabels: 339 | app: flannel 340 | template: 341 | metadata: 342 | labels: 343 | tier: node 344 | app: flannel 345 | spec: 346 | affinity: 347 | nodeAffinity: 348 | requiredDuringSchedulingIgnoredDuringExecution: 349 | nodeSelectorTerms: 350 | - matchExpressions: 351 | - key: beta.kubernetes.io/os 352 | operator: In 353 | values: 354 | - linux 355 | - key: beta.kubernetes.io/arch 356 | operator: In 357 | values: 358 | - arm 359 | hostNetwork: true 360 | tolerations: 361 | - operator: Exists 362 | effect: NoSchedule 363 | serviceAccountName: flannel 364 | initContainers: 365 | - name: install-cni 366 | image: quay.io/coreos/flannel:v0.11.0-arm 367 | command: 368 | - cp 369 | args: 370 | - -f 371 | - /etc/kube-flannel/cni-conf.json 372 | - /etc/cni/net.d/10-flannel.conflist 373 | volumeMounts: 374 | - name: cni 375 | mountPath: /etc/cni/net.d 376 | - name: flannel-cfg 377 | mountPath: /etc/kube-flannel/ 378 | containers: 379 | - name: kube-flannel 380 | image: quay.io/coreos/flannel:v0.11.0-arm 381 | command: 382 | - /opt/bin/flanneld 383 | args: 384 | - --ip-masq 385 | {% if network_interface != '' -%} 386 | - {{ flannel_cni_opts }} 387 | {% endif -%} 388 | - --kube-subnet-mgr 389 | resources: 390 | requests: 391 | cpu: "100m" 392 | memory: "50Mi" 393 | limits: 394 | cpu: "100m" 395 | memory: "50Mi" 396 | securityContext: 397 | privileged: false 398 | capabilities: 399 | add: ["NET_ADMIN"] 400 | env: 401 | - name: POD_NAME 402 | valueFrom: 403 | fieldRef: 404 | fieldPath: metadata.name 405 | - name: POD_NAMESPACE 406 | valueFrom: 407 | fieldRef: 408 | fieldPath: metadata.namespace 409 | volumeMounts: 410 | - name: run 411 | mountPath: /run/flannel 412 | - name: flannel-cfg 413 | mountPath: /etc/kube-flannel/ 414 | volumes: 415 | - name: run 416 | hostPath: 417 | path: /run/flannel 418 | - name: cni 419 | hostPath: 420 | path: /etc/cni/net.d 421 | - name: flannel-cfg 422 | configMap: 423 | name: kube-flannel-cfg 424 | --- 425 | apiVersion: apps/v1 426 | kind: DaemonSet 427 | metadata: 428 | name: kube-flannel-ds-ppc64le 429 | namespace: kube-system 430 | labels: 431 | tier: node 432 | app: flannel 433 | spec: 434 | selector: 435 | matchLabels: 436 | app: flannel 437 | template: 438 | metadata: 439 | labels: 440 | tier: node 441 | app: flannel 442 | spec: 443 | affinity: 444 | nodeAffinity: 445 | requiredDuringSchedulingIgnoredDuringExecution: 446 | nodeSelectorTerms: 447 | - matchExpressions: 448 | - key: beta.kubernetes.io/os 449 | operator: In 450 | values: 451 | - linux 452 | - key: beta.kubernetes.io/arch 453 | operator: In 454 | values: 455 | - ppc64le 456 | hostNetwork: true 457 | tolerations: 458 | - operator: Exists 459 | effect: NoSchedule 460 | serviceAccountName: flannel 461 | initContainers: 462 | - name: install-cni 463 | image: quay.io/coreos/flannel:v0.11.0-ppc64le 464 | command: 465 | - cp 466 | args: 467 | - -f 468 | - /etc/kube-flannel/cni-conf.json 469 | - /etc/cni/net.d/10-flannel.conflist 470 | volumeMounts: 471 | - name: cni 472 | mountPath: /etc/cni/net.d 473 | - name: flannel-cfg 474 | mountPath: /etc/kube-flannel/ 475 | containers: 476 | - name: kube-flannel 477 | image: quay.io/coreos/flannel:v0.11.0-ppc64le 478 | command: 479 | - /opt/bin/flanneld 480 | args: 481 | - --ip-masq 482 | {% if network_interface != '' -%} 483 | - {{ flannel_cni_opts }} 484 | {% endif -%} 485 | - --kube-subnet-mgr 486 | resources: 487 | requests: 488 | cpu: "100m" 489 | memory: "50Mi" 490 | limits: 491 | cpu: "100m" 492 | memory: "50Mi" 493 | securityContext: 494 | privileged: false 495 | capabilities: 496 | add: ["NET_ADMIN"] 497 | env: 498 | - name: POD_NAME 499 | valueFrom: 500 | fieldRef: 501 | fieldPath: metadata.name 502 | - name: POD_NAMESPACE 503 | valueFrom: 504 | fieldRef: 505 | fieldPath: metadata.namespace 506 | volumeMounts: 507 | - name: run 508 | mountPath: /run/flannel 509 | - name: flannel-cfg 510 | mountPath: /etc/kube-flannel/ 511 | volumes: 512 | - name: run 513 | hostPath: 514 | path: /run/flannel 515 | - name: cni 516 | hostPath: 517 | path: /etc/cni/net.d 518 | - name: flannel-cfg 519 | configMap: 520 | name: kube-flannel-cfg 521 | --- 522 | apiVersion: apps/v1 523 | kind: DaemonSet 524 | metadata: 525 | name: kube-flannel-ds-s390x 526 | namespace: kube-system 527 | labels: 528 | tier: node 529 | app: flannel 530 | spec: 531 | selector: 532 | matchLabels: 533 | app: flannel 534 | template: 535 | metadata: 536 | labels: 537 | tier: node 538 | app: flannel 539 | spec: 540 | affinity: 541 | nodeAffinity: 542 | requiredDuringSchedulingIgnoredDuringExecution: 543 | nodeSelectorTerms: 544 | - matchExpressions: 545 | - key: beta.kubernetes.io/os 546 | operator: In 547 | values: 548 | - linux 549 | - key: beta.kubernetes.io/arch 550 | operator: In 551 | values: 552 | - s390x 553 | hostNetwork: true 554 | tolerations: 555 | - operator: Exists 556 | effect: NoSchedule 557 | serviceAccountName: flannel 558 | initContainers: 559 | - name: install-cni 560 | image: quay.io/coreos/flannel:v0.11.0-s390x 561 | command: 562 | - cp 563 | args: 564 | - -f 565 | - /etc/kube-flannel/cni-conf.json 566 | - /etc/cni/net.d/10-flannel.conflist 567 | volumeMounts: 568 | - name: cni 569 | mountPath: /etc/cni/net.d 570 | - name: flannel-cfg 571 | mountPath: /etc/kube-flannel/ 572 | containers: 573 | - name: kube-flannel 574 | image: quay.io/coreos/flannel:v0.11.0-s390x 575 | command: 576 | - /opt/bin/flanneld 577 | args: 578 | - --ip-masq 579 | {% if network_interface != '' -%} 580 | - {{ flannel_cni_opts }} 581 | {% endif -%} 582 | - --kube-subnet-mgr 583 | resources: 584 | requests: 585 | cpu: "100m" 586 | memory: "50Mi" 587 | limits: 588 | cpu: "100m" 589 | memory: "50Mi" 590 | securityContext: 591 | privileged: false 592 | capabilities: 593 | add: ["NET_ADMIN"] 594 | env: 595 | - name: POD_NAME 596 | valueFrom: 597 | fieldRef: 598 | fieldPath: metadata.name 599 | - name: POD_NAMESPACE 600 | valueFrom: 601 | fieldRef: 602 | fieldPath: metadata.namespace 603 | volumeMounts: 604 | - name: run 605 | mountPath: /run/flannel 606 | - name: flannel-cfg 607 | mountPath: /etc/kube-flannel/ 608 | volumes: 609 | - name: run 610 | hostPath: 611 | path: /run/flannel 612 | - name: cni 613 | hostPath: 614 | path: /etc/cni/net.d 615 | - name: flannel-cfg 616 | configMap: 617 | name: kube-flannel-cfg 618 | -------------------------------------------------------------------------------- /ansible-playbook/roles/commons/os-checker/defaults/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | systemd_dir: /lib/systemd/system 4 | system_env_dir: /etc/sysconfig 5 | network_dir: /etc/kubernetes/network 6 | kubeadmin_config: /etc/kubernetes/admin.conf 7 | kube_addon_dir: /etc/kubernetes/addon 8 | -------------------------------------------------------------------------------- /ansible-playbook/roles/commons/os-checker/tasks/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # Fact os vars 3 | 4 | - name: Get os_version from /etc/os-release 5 | when: ansible_os_family is not defined 6 | raw: "grep '^VERSION_ID=' /etc/os-release | sed s'/VERSION_ID=//'" 7 | register: os_version 8 | changed_when: False 9 | 10 | - name: Get distro name from /etc/os-release 11 | when: ansible_os_family is not defined 12 | raw: "grep '^NAME=' /etc/os-release | sed s'/NAME=//'" 13 | register: distro 14 | changed_when: False 15 | 16 | - name: Set fact ansible_os_family var to Debian 17 | when: 18 | - ansible_os_family is not defined 19 | - "'Debian' in distro.stdout" 20 | set_fact: 21 | ansible_os_family: Debian 22 | 23 | - name: Set fact ansible_os_family var to Debian 24 | when: 25 | - ansible_os_family is not defined 26 | - "'Ubuntu' in distro.stdout" 27 | set_fact: 28 | ansible_os_family: Debian 29 | 30 | - name: Set fact ansible_os_family var to RedHat 31 | when: 32 | - ansible_os_family is not defined 33 | - "'CentOS' in distro.stdout" 34 | set_fact: 35 | ansible_os_family: RedHat 36 | 37 | - name: Override config file directory for Debian 38 | when: ansible_os_family == "Debian" 39 | set_fact: 40 | system_env_dir: "/etc/default" 41 | -------------------------------------------------------------------------------- /ansible-playbook/roles/commons/pre-install/meta/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | dependencies: 3 | - { role: commons/os-checker } 4 | -------------------------------------------------------------------------------- /ansible-playbook/roles/commons/pre-install/tasks/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | - name: Install Kubernetes packages 4 | include_tasks: pkg.yml 5 | 6 | - name: Disable system swap 7 | shell: "swapoff -a" 8 | 9 | - name: Remove current swaps from fstab 10 | lineinfile: 11 | dest: /etc/fstab 12 | regexp: '(?i)^([^#][\S]+\s+(none|swap)\s+swap.*)' 13 | line: '# \1' 14 | backrefs: yes 15 | state: present 16 | 17 | - name: Disable swappiness and pass bridged IPv4 traffic to iptable's chains 18 | sysctl: 19 | name: "{{ item.name }}" 20 | value: "{{ item.value }}" 21 | state: present 22 | with_items: 23 | - { name: 'vm.swappiness', value: '0' } 24 | - { name: 'net.bridge.bridge-nf-call-iptables', value: '1' } 25 | 26 | - name: Create service drop-in directory 27 | file: 28 | path: /etc/systemd/system/kubelet.service.d/ 29 | state: directory 30 | owner: "{{ ansible_user | default(ansible_user_id) }}" 31 | group: "{{ ansible_user | default(ansible_user_id) }}" 32 | mode: 0755 33 | 34 | - name: Copy kubeadm conf to drop-in directory 35 | template: src=20-extra-args.conf.j2 dest=/etc/systemd/system/kubelet.service.d/20-extra-args.conf 36 | 37 | - name: Reload kubelet daemon 38 | systemd: 39 | name: kubelet 40 | daemon_reload: yes 41 | enabled: yes 42 | -------------------------------------------------------------------------------- /ansible-playbook/roles/commons/pre-install/tasks/pkg.yml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | - name: Add Kubernetes APT GPG key 4 | when: ansible_os_family == "Debian" 5 | apt_key: 6 | url: https://packages.cloud.google.com/apt/doc/apt-key.gpg 7 | state: present 8 | 9 | - name: Add Kubernetes APT repository 10 | when: ansible_os_family == "Debian" 11 | apt_repository: 12 | repo: deb http://apt.kubernetes.io/ kubernetes-xenial main 13 | state: present 14 | filename: 'kubernetes' 15 | 16 | - name: Add Kubernetes yum repository 17 | when: ansible_os_family == "RedHat" 18 | yum_repository: 19 | name: Kubernetes 20 | description: Kubernetes Repository 21 | file: kubernetes 22 | baseurl: http://yum.kubernetes.io/repos/kubernetes-el7-x86_64 23 | enabled: yes 24 | gpgcheck: no 25 | 26 | - name: Install kubernetes packages (RHEL/CentOS) 27 | when: ansible_os_family == "RedHat" 28 | yum: 29 | name: "{{ pkgs }}" 30 | update_cache: yes 31 | state: installed 32 | with_items: "{{ pkgs }}" 33 | 34 | - name: Install kubernetes packages (Debian/Ubuntu) 35 | when: ansible_os_family == "Debian" 36 | apt: 37 | name: "{{ pkgs }}" 38 | update_cache: yes 39 | state: present 40 | with_items: "{{ pkgs }}" 41 | -------------------------------------------------------------------------------- /ansible-playbook/roles/commons/pre-install/templates/20-extra-args.conf.j2: -------------------------------------------------------------------------------- 1 | [Service] 2 | Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false" 3 | -------------------------------------------------------------------------------- /ansible-playbook/roles/docker/defaults/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | docker_version: 17.03 -------------------------------------------------------------------------------- /ansible-playbook/roles/docker/meta/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | dependencies: 3 | - { role: commons/os-checker } 4 | -------------------------------------------------------------------------------- /ansible-playbook/roles/docker/tasks/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - name: Install Docker container engine 3 | include_tasks: pkg.yml 4 | 5 | - name: Copy Docker engine service file 6 | register: change_docker 7 | template: 8 | src: "docker.service.j2" 9 | dest: "{{ systemd_dir }}/docker.service" 10 | owner: root 11 | group: root 12 | mode: 0755 13 | 14 | - name: Copy Docker environment config file 15 | template: src=docker.j2 dest={{ system_env_dir }}/docker 16 | 17 | - name: Add any insecure registries to Docker config 18 | when: insecure_registries is defined and insecure_registries | length > 0 19 | lineinfile: dest={{ system_env_dir }}/docker regexp=^INSECURE_REGISTRY= line=INSECURE_REGISTRY="{% for reg in insecure_registries %}--insecure-registry={{ reg }} {% endfor %}" 20 | 21 | - name: Add registry to Docker config 22 | when: add_registry is defined and add_registry > 0 23 | lineinfile: dest={{ system_env_dir }}/docker regexp=^ADD_REGISTRY= line=ADD_REGISTRY="{% for reg in add_registry %}--add-registry={{ reg }} {%endfor %}" 24 | 25 | - name: Enable and check Docker service 26 | systemd: 27 | name: docker 28 | daemon_reload: yes 29 | state: started 30 | enabled: yes 31 | register: started_docker 32 | -------------------------------------------------------------------------------- /ansible-playbook/roles/docker/tasks/pkg.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - name: Install apt-transport-https 3 | when: ansible_os_family == "Debian" 4 | apt: 5 | name: "apt-transport-https" 6 | state: present 7 | update_cache: yes 8 | 9 | - name: Add Docker APT GPG key 10 | when: ansible_os_family == "Debian" 11 | apt_key: 12 | url: https://download.docker.com/linux/ubuntu/gpg 13 | 14 | - name: Add Docker APT repository 15 | when: ansible_os_family == "Debian" 16 | apt_repository: 17 | repo: deb https://download.docker.com/linux/ubuntu xenial stable 18 | state: present 19 | filename: 'docker' 20 | 21 | - name: Add Docker yum repository 22 | when: ansible_os_family == "RedHat" 23 | yum_repository: 24 | name: Docker 25 | description: Docker Repository 26 | file: docker 27 | baseurl: https://yum.dockerproject.org/repo/main/centos/7/ 28 | enabled: yes 29 | gpgcheck: yes 30 | gpgkey: https://yum.dockerproject.org/gpg 31 | 32 | - name: Install docker engine (RHEL/CentOS) 33 | when: ansible_os_family == "RedHat" 34 | yum: 35 | name: "docker-engine-{{ docker_version }}.*" 36 | state: present 37 | 38 | - name: Install docker engine (Debian/Ubuntu) 39 | when: ansible_os_family == "Debian" 40 | apt: 41 | update_cache: yes 42 | name: "docker-ce={{ docker_version }}*" 43 | state: present 44 | 45 | - name: Hold docker version 46 | when: ansible_os_family == "Debian" 47 | dpkg_selections: 48 | name: docker-ce 49 | selection: hold 50 | -------------------------------------------------------------------------------- /ansible-playbook/roles/docker/templates/docker.j2: -------------------------------------------------------------------------------- 1 | INSECURE_REGISTRY="" 2 | DOCKER_OPTS="" 3 | {% if ansible_os_family == "RedHat" -%} 4 | DOCKER_STORAGE_OPTIONS="--storage-driver=overlay" 5 | {% endif -%} 6 | -------------------------------------------------------------------------------- /ansible-playbook/roles/docker/templates/docker.service.j2: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=Docker Engine 3 | After=network.target 4 | 5 | [Service] 6 | Type=notify 7 | EnvironmentFile=-{{ system_env_dir }}/docker 8 | ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT 9 | ExecStart=/usr/bin/dockerd {% if ansible_os_family == 'Debian' -%} -H fd:// {% endif -%} \ 10 | $OPTIONS \ 11 | $DOCKER_STORAGE_OPTIONS \ 12 | $DOCKER_OPTS \ 13 | $DOCKER_NETWORK_OPTIONS \ 14 | $ADD_REGISTRY \ 15 | $BLOCK_REGISTRY \ 16 | $INSECURE_REGISTRY 17 | 18 | ExecReload=/bin/kill -s HUP $MAINPID 19 | Restart=on-failure 20 | LimitNOFILE=1048576 21 | LimitNPROC=infinity 22 | LimitCORE=infinity 23 | TimeoutStartSec=0 24 | Delegate=yes 25 | KillMode=process 26 | 27 | [Install] 28 | WantedBy=multi-user.target 29 | -------------------------------------------------------------------------------- /ansible-playbook/roles/healthcheck/tasks/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - name: "Create tmp directory" 3 | file: 4 | path: "{{ tmp_dir }}" 5 | state: directory 6 | mode: 0755 7 | tags: healthcheck 8 | 9 | - name: "Create checkout directory" 10 | file: 11 | path: "{{ tmp_dir }}/healthcheck" 12 | state: directory 13 | mode: 0755 14 | tags: healthcheck 15 | 16 | - name: "Clone git repo" 17 | git: 18 | repo: "{{ healthcheck_git_url }}" 19 | dest: "{{ tmp_dir }}/healthcheck" 20 | tags: healthcheck 21 | 22 | - name: "Install Healthcheck" 23 | shell: "kubectl apply -f {{ tmp_dir }}/healthcheck/kubernetes/" 24 | tags: healthcheck 25 | 26 | - name: "Clean-up" 27 | file: 28 | path: "{{ tmp_dir }}" 29 | state: absent 30 | ignore_errors: yes 31 | tags: healthcheck 32 | -------------------------------------------------------------------------------- /ansible-playbook/roles/healthcheck/vars/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | healthcheck_git_url: https://github.com/emrekenci/k8s-healthcheck.git 3 | -------------------------------------------------------------------------------- /ansible-playbook/roles/helm/files/rbac-config.yml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ServiceAccount 3 | metadata: 4 | name: tiller 5 | namespace: kube-system 6 | --- 7 | apiVersion: rbac.authorization.k8s.io/v1 8 | kind: ClusterRoleBinding 9 | metadata: 10 | name: tiller 11 | roleRef: 12 | apiGroup: rbac.authorization.k8s.io 13 | kind: ClusterRole 14 | name: cluster-admin 15 | subjects: 16 | - kind: ServiceAccount 17 | name: tiller 18 | namespace: kube-system 19 | -------------------------------------------------------------------------------- /ansible-playbook/roles/helm/tasks/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - name: "Create tmp directory" 3 | file: 4 | path: "{{ tmp_dir }}" 5 | state: directory 6 | mode: 0755 7 | tags: helm 8 | 9 | - name: "Check if Helm is installed" 10 | shell: command -v helm >/dev/null 2>&1 11 | register: helm_exists 12 | ignore_errors: yes 13 | tags: helm 14 | 15 | - name: "Install Helm" 16 | block: 17 | - name: "Get Helm installer" 18 | get_url: 19 | url: https://raw.githubusercontent.com/helm/helm/master/scripts/get 20 | dest: "{{ tmp_dir }}/get_helm.sh" 21 | mode: 0755 22 | 23 | - name: "Run the installer" 24 | shell: "{{ tmp_dir }}/get_helm.sh" 25 | 26 | when: helm_exists.rc > 0 27 | tags: helm 28 | 29 | - name: "Copy yaml file" 30 | copy: 31 | src: "rbac-config.yml" 32 | dest: "{{ tmp_dir }}/rbac-config.yml" 33 | mode: 0644 34 | tags: helm 35 | 36 | - name: "RBAC configuration" 37 | shell: "kubectl apply -f {{ tmp_dir }}/rbac-config.yml" 38 | tags: helm 39 | 40 | - name: "Init Helm" 41 | shell: "helm init --service-account tiller" 42 | tags: helm 43 | 44 | - name: "Update Helm repo" 45 | shell: "helm repo update" 46 | tags: helm 47 | 48 | - name: "Clean-up" 49 | file: 50 | path: "{{ tmp_dir }}" 51 | state: absent 52 | ignore_errors: yes 53 | tags: helm 54 | -------------------------------------------------------------------------------- /ansible-playbook/roles/kubernetes/master/handlers/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | - name: Deploy kubernetes dashboard into cluster 4 | when: init_cluster and started_kubelet and enable_dashboard 5 | command: | 6 | kubectl --kubeconfig={{ kubeadmin_config }} \ 7 | apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended/kubernetes-dashboard.yaml 8 | register: create_result 9 | until: create_result.rc == 0 10 | retries: 5 11 | delay: 2 12 | ignore_errors: true 13 | -------------------------------------------------------------------------------- /ansible-playbook/roles/kubernetes/master/meta/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | dependencies: 3 | - { role: commons/os-checker } 4 | - { role: commons/pre-install, pkgs: ["kubelet", "kubeadm", "kubectl"] } 5 | -------------------------------------------------------------------------------- /ansible-playbook/roles/kubernetes/master/tasks/init.yml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | - name: Reset Kubernetes component 4 | shell: "kubeadm reset --force" 5 | register: reset_cluster 6 | 7 | - name: Init Kubernetes cluster 8 | when: reset_cluster is succeeded 9 | shell: | 10 | kubeadm init --kubernetes-version {{ kube_version }} \ 11 | --pod-network-cidr {{ pod_network_cidr }} \ 12 | --token {{ token }} \ 13 | --apiserver-advertise-address {{ master_ip }} \ 14 | {{ kubeadm_opts }} \ 15 | {{ init_opts }} 16 | register: init_cluster 17 | 18 | - name: Create Kubernetes config directory 19 | file: 20 | path: ".kube/" 21 | state: directory 22 | 23 | - name: Copy admin.conf to Home directory 24 | when: init_cluster is succeeded 25 | copy: 26 | src: "{{ kubeadmin_config }}" 27 | dest: ".kube/config" 28 | owner: "{{ ansible_user | default(ansible_user_id) }}" 29 | group: "{{ ansible_user | default(ansible_user_id) }}" 30 | mode: 0755 31 | remote_src: true 32 | 33 | - name: Deploy kubernetes dashboard into cluster 34 | when: init_cluster is succeeded and enable_dashboard 35 | command: | 36 | kubectl --kubeconfig={{ kubeadmin_config }} \ 37 | apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended.yaml 38 | register: create_result 39 | until: create_result.rc == 0 40 | retries: 5 41 | delay: 2 42 | ignore_errors: true 43 | -------------------------------------------------------------------------------- /ansible-playbook/roles/kubernetes/master/tasks/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | - name: Check if kubeadm has already run 4 | stat: 5 | path: "/etc/kubernetes/pki/ca.key" 6 | register: kubeadm_ca 7 | 8 | - name: Init cluster if needed 9 | include_tasks: init.yml 10 | when: not kubeadm_ca.stat.exists 11 | run_once: yes 12 | 13 | - name: Enable and check kubelet service 14 | systemd: 15 | name: kubelet 16 | daemon_reload: yes 17 | state: started 18 | enabled: yes 19 | register: started_kubelet 20 | 21 | - name: "Copy config file" 22 | fetch: 23 | src: /etc/kubernetes/admin.conf 24 | dest: "{{ lookup('env', 'HOME') }}/admin.conf" 25 | flat: yes 26 | run_once: yes 27 | ignore_errors: yes 28 | 29 | -------------------------------------------------------------------------------- /ansible-playbook/roles/kubernetes/node/handlers/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - name: Recreate kube-dns 3 | command: kubectl --kubeconfig={{ kubeadmin_config }} -n kube-system delete pods -l k8s-app=kube-dns 4 | delegate_to: "{{ groups['master'][0] }}" 5 | run_once: true 6 | ignore_errors: true 7 | -------------------------------------------------------------------------------- /ansible-playbook/roles/kubernetes/node/meta/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | dependencies: 3 | - { role: commons/os-checker } 4 | - { role: commons/pre-install , pkgs: ["kubelet", "kubeadm"] } 5 | -------------------------------------------------------------------------------- /ansible-playbook/roles/kubernetes/node/tasks/join.yml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | - name: Reset Kubernetes component 4 | shell: "kubeadm reset --force" 5 | register: reset_cluster 6 | 7 | - name: Join to Kubernetes cluster 8 | when: reset_cluster is succeeded 9 | shell: | 10 | kubeadm join --token {{ token }} \ 11 | --discovery-token-unsafe-skip-ca-verification \ 12 | {{ master_ip }}:6443 13 | register: join_cluster 14 | notify: 15 | - Recreate kube-dns 16 | -------------------------------------------------------------------------------- /ansible-playbook/roles/kubernetes/node/tasks/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | - name: Check if kubelet.conf exists 4 | stat: 5 | path: "/etc/kubernetes/kubelet.conf" 6 | register: kubelet_conf 7 | 8 | - name: Join to cluster if needed 9 | include_tasks: join.yml 10 | when: not kubelet_conf.stat.exists 11 | 12 | - name: Enable and check kubelet service 13 | systemd: 14 | name: kubelet 15 | daemon_reload: yes 16 | state: started 17 | enabled: yes 18 | -------------------------------------------------------------------------------- /ansible-playbook/roles/metallb/tasks/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - name: "Create tmp directory" 3 | file: 4 | path: "{{ tmp_dir }}" 5 | state: directory 6 | mode: 0755 7 | tags: metallb 8 | 9 | - name: "Install MetalLB" 10 | shell: "kubectl apply -f {{ metallb_yaml_url }}" 11 | tags: metallb 12 | 13 | - name: "Create configmap file" 14 | template: 15 | src: metallb-layer-2-config.yml.j2 16 | dest: "{{ tmp_dir }}/metallb-layer-2-config.yml" 17 | tags: metallb 18 | 19 | - name: "Create MetalLB configmap in kubernetes" 20 | shell: "kubectl apply -f {{ tmp_dir }}/metallb-layer-2-config.yml" 21 | tags: metallb 22 | 23 | - name: "Clean-up" 24 | file: 25 | path: "{{ tmp_dir }}" 26 | state: absent 27 | ignore_errors: yes 28 | tags: metallb 29 | -------------------------------------------------------------------------------- /ansible-playbook/roles/metallb/templates/metallb-layer-2-config.yml.j2: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | namespace: metallb-system 5 | name: config 6 | data: 7 | config: | 8 | address-pools: 9 | - name: metallb-ip-space 10 | protocol: layer2 11 | addresses: 12 | - {{ metallb_address_space }} 13 | -------------------------------------------------------------------------------- /ansible-playbook/roles/metallb/vars/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | metallb_version: v0.7.3 3 | metallb_yaml_url: "https://raw.githubusercontent.com/google/metallb/{{ metallb_version }}/manifests/metallb.yaml" 4 | metallb_address_space: 192.168.205.200-192.168.205.210 5 | -------------------------------------------------------------------------------- /ansible-playbook/site.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | - hosts: master 3 | gather_facts: yes 4 | become: yes 5 | roles: 6 | - { role: kubernetes/master, tags: master } 7 | - { role: cni, tags: cni } 8 | 9 | - hosts: node 10 | gather_facts: yes 11 | become: yes 12 | roles: 13 | - { role: kubernetes/node, tags: node } 14 | 15 | - hosts: master 16 | gather_facts: yes 17 | become: yes 18 | tasks: 19 | - name: "Helm role" 20 | include_role: 21 | name: helm 22 | when: "additional_features.helm" 23 | run_once: yes 24 | tags: helm 25 | 26 | - name: "MetalLB role" 27 | include_role: 28 | name: metallb 29 | when: "additional_features.metallb" 30 | run_once: yes 31 | tags: metallb 32 | 33 | - name: "Healthcheck role" 34 | include_role: 35 | name: healthcheck 36 | when: "additional_features.healthcheck" 37 | run_once: yes 38 | tags: healthcheck 39 | 40 | -------------------------------------------------------------------------------- /cloud_init.cfg: -------------------------------------------------------------------------------- 1 | #cloud-config 2 | # vim: syntax=yaml 3 | # 4 | # *********************** 5 | # ---- for more examples look at: ------ 6 | # ---> https://cloudinit.readthedocs.io/en/latest/topics/examples.html 7 | # ****************************** 8 | # 9 | # This is the configuration syntax that the write_files module 10 | # will know how to understand. encoding can be given b64 or gzip or (gz+b64). 11 | # The content will be decoded accordingly and then written to the path that is 12 | # provided. 13 | hostname: ${hostname} 14 | package_update: true 15 | package_upgrade: true 16 | packages: 17 | - apt-transport-https 18 | - ca-certificates 19 | - curl 20 | - software-properties-common 21 | - gnupg2 22 | - nfs-common 23 | - ebtables 24 | - ethtool 25 | disable_root: 0 26 | ssh_pwauth: 1 27 | users: 28 | - name: ubuntu 29 | shell: /bin/bash 30 | sudo: ALL=(ALL) NOPASSWD:ALL 31 | ssh-authorized-keys: 32 | - ${public_key} 33 | growpart: 34 | mode: auto 35 | devices: ['/'] 36 | apt: 37 | preserve_sources_list: true 38 | sources: 39 | docker.list: 40 | source: "deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable" 41 | key: | 42 | -----BEGIN PGP PUBLIC KEY BLOCK----- 43 | 44 | mQINBFit2ioBEADhWpZ8/wvZ6hUTiXOwQHXMAlaFHcPH9hAtr4F1y2+OYdbtMuth 45 | lqqwp028AqyY+PRfVMtSYMbjuQuu5byyKR01BbqYhuS3jtqQmljZ/bJvXqnmiVXh 46 | 38UuLa+z077PxyxQhu5BbqntTPQMfiyqEiU+BKbq2WmANUKQf+1AmZY/IruOXbnq 47 | L4C1+gJ8vfmXQt99npCaxEjaNRVYfOS8QcixNzHUYnb6emjlANyEVlZzeqo7XKl7 48 | UrwV5inawTSzWNvtjEjj4nJL8NsLwscpLPQUhTQ+7BbQXAwAmeHCUTQIvvWXqw0N 49 | cmhh4HgeQscQHYgOJjjDVfoY5MucvglbIgCqfzAHW9jxmRL4qbMZj+b1XoePEtht 50 | ku4bIQN1X5P07fNWzlgaRL5Z4POXDDZTlIQ/El58j9kp4bnWRCJW0lya+f8ocodo 51 | vZZ+Doi+fy4D5ZGrL4XEcIQP/Lv5uFyf+kQtl/94VFYVJOleAv8W92KdgDkhTcTD 52 | G7c0tIkVEKNUq48b3aQ64NOZQW7fVjfoKwEZdOqPE72Pa45jrZzvUFxSpdiNk2tZ 53 | XYukHjlxxEgBdC/J3cMMNRE1F4NCA3ApfV1Y7/hTeOnmDuDYwr9/obA8t016Yljj 54 | q5rdkywPf4JF8mXUW5eCN1vAFHxeg9ZWemhBtQmGxXnw9M+z6hWwc6ahmwARAQAB 55 | tCtEb2NrZXIgUmVsZWFzZSAoQ0UgZGViKSA8ZG9ja2VyQGRvY2tlci5jb20+iQI3 56 | BBMBCgAhBQJYrefAAhsvBQsJCAcDBRUKCQgLBRYCAwEAAh4BAheAAAoJEI2BgDwO 57 | v82IsskP/iQZo68flDQmNvn8X5XTd6RRaUH33kXYXquT6NkHJciS7E2gTJmqvMqd 58 | tI4mNYHCSEYxI5qrcYV5YqX9P6+Ko+vozo4nseUQLPH/ATQ4qL0Zok+1jkag3Lgk 59 | jonyUf9bwtWxFp05HC3GMHPhhcUSexCxQLQvnFWXD2sWLKivHp2fT8QbRGeZ+d3m 60 | 6fqcd5Fu7pxsqm0EUDK5NL+nPIgYhN+auTrhgzhK1CShfGccM/wfRlei9Utz6p9P 61 | XRKIlWnXtT4qNGZNTN0tR+NLG/6Bqd8OYBaFAUcue/w1VW6JQ2VGYZHnZu9S8LMc 62 | FYBa5Ig9PxwGQOgq6RDKDbV+PqTQT5EFMeR1mrjckk4DQJjbxeMZbiNMG5kGECA8 63 | g383P3elhn03WGbEEa4MNc3Z4+7c236QI3xWJfNPdUbXRaAwhy/6rTSFbzwKB0Jm 64 | ebwzQfwjQY6f55MiI/RqDCyuPj3r3jyVRkK86pQKBAJwFHyqj9KaKXMZjfVnowLh 65 | 9svIGfNbGHpucATqREvUHuQbNnqkCx8VVhtYkhDb9fEP2xBu5VvHbR+3nfVhMut5 66 | G34Ct5RS7Jt6LIfFdtcn8CaSas/l1HbiGeRgc70X/9aYx/V/CEJv0lIe8gP6uDoW 67 | FPIZ7d6vH+Vro6xuWEGiuMaiznap2KhZmpkgfupyFmplh0s6knymuQINBFit2ioB 68 | EADneL9S9m4vhU3blaRjVUUyJ7b/qTjcSylvCH5XUE6R2k+ckEZjfAMZPLpO+/tF 69 | M2JIJMD4SifKuS3xck9KtZGCufGmcwiLQRzeHF7vJUKrLD5RTkNi23ydvWZgPjtx 70 | Q+DTT1Zcn7BrQFY6FgnRoUVIxwtdw1bMY/89rsFgS5wwuMESd3Q2RYgb7EOFOpnu 71 | w6da7WakWf4IhnF5nsNYGDVaIHzpiqCl+uTbf1epCjrOlIzkZ3Z3Yk5CM/TiFzPk 72 | z2lLz89cpD8U+NtCsfagWWfjd2U3jDapgH+7nQnCEWpROtzaKHG6lA3pXdix5zG8 73 | eRc6/0IbUSWvfjKxLLPfNeCS2pCL3IeEI5nothEEYdQH6szpLog79xB9dVnJyKJb 74 | VfxXnseoYqVrRz2VVbUI5Blwm6B40E3eGVfUQWiux54DspyVMMk41Mx7QJ3iynIa 75 | 1N4ZAqVMAEruyXTRTxc9XW0tYhDMA/1GYvz0EmFpm8LzTHA6sFVtPm/ZlNCX6P1X 76 | zJwrv7DSQKD6GGlBQUX+OeEJ8tTkkf8QTJSPUdh8P8YxDFS5EOGAvhhpMBYD42kQ 77 | pqXjEC+XcycTvGI7impgv9PDY1RCC1zkBjKPa120rNhv/hkVk/YhuGoajoHyy4h7 78 | ZQopdcMtpN2dgmhEegny9JCSwxfQmQ0zK0g7m6SHiKMwjwARAQABiQQ+BBgBCAAJ 79 | BQJYrdoqAhsCAikJEI2BgDwOv82IwV0gBBkBCAAGBQJYrdoqAAoJEH6gqcPyc/zY 80 | 1WAP/2wJ+R0gE6qsce3rjaIz58PJmc8goKrir5hnElWhPgbq7cYIsW5qiFyLhkdp 81 | YcMmhD9mRiPpQn6Ya2w3e3B8zfIVKipbMBnke/ytZ9M7qHmDCcjoiSmwEXN3wKYI 82 | mD9VHONsl/CG1rU9Isw1jtB5g1YxuBA7M/m36XN6x2u+NtNMDB9P56yc4gfsZVES 83 | KA9v+yY2/l45L8d/WUkUi0YXomn6hyBGI7JrBLq0CX37GEYP6O9rrKipfz73XfO7 84 | JIGzOKZlljb/D9RX/g7nRbCn+3EtH7xnk+TK/50euEKw8SMUg147sJTcpQmv6UzZ 85 | cM4JgL0HbHVCojV4C/plELwMddALOFeYQzTif6sMRPf+3DSj8frbInjChC3yOLy0 86 | 6br92KFom17EIj2CAcoeq7UPhi2oouYBwPxh5ytdehJkoo+sN7RIWua6P2WSmon5 87 | U888cSylXC0+ADFdgLX9K2zrDVYUG1vo8CX0vzxFBaHwN6Px26fhIT1/hYUHQR1z 88 | VfNDcyQmXqkOnZvvoMfz/Q0s9BhFJ/zU6AgQbIZE/hm1spsfgvtsD1frZfygXJ9f 89 | irP+MSAI80xHSf91qSRZOj4Pl3ZJNbq4yYxv0b1pkMqeGdjdCYhLU+LZ4wbQmpCk 90 | SVe2prlLureigXtmZfkqevRz7FrIZiu9ky8wnCAPwC7/zmS18rgP/17bOtL4/iIz 91 | QhxAAoAMWVrGyJivSkjhSGx1uCojsWfsTAm11P7jsruIL61ZzMUVE2aM3Pmj5G+W 92 | 9AcZ58Em+1WsVnAXdUR//bMmhyr8wL/G1YO1V3JEJTRdxsSxdYa4deGBBY/Adpsw 93 | 24jxhOJR+lsJpqIUeb999+R8euDhRHG9eFO7DRu6weatUJ6suupoDTRWtr/4yGqe 94 | dKxV3qQhNLSnaAzqW/1nA3iUB4k7kCaKZxhdhDbClf9P37qaRW467BLCVO/coL3y 95 | Vm50dwdrNtKpMBh3ZpbB1uJvgi9mXtyBOMJ3v8RZeDzFiG8HdCtg9RvIt/AIFoHR 96 | H3S+U79NT6i0KPzLImDfs8T7RlpyuMc4Ufs8ggyg9v3Ae6cN3eQyxcK3w0cbBwsh 97 | /nQNfsA6uu+9H7NhbehBMhYnpNZyrHzCmzyXkauwRAqoCbGCNykTRwsur9gS41TQ 98 | M8ssD1jFheOJf3hODnkKU+HKjvMROl1DK7zdmLdNzA1cvtZH/nCC9KPj1z8QC47S 99 | xx+dTZSx4ONAhwbS/LN3PoKtn8LPjY9NP9uDWI+TWYquS2U+KHDrBDlsgozDbs/O 100 | jCxcpDzNmXpWQHEtHU7649OXHP7UeNST1mCUCH5qdank0V1iejF6/CfTFU4MfcrG 101 | YT90qFF93M3v01BbxP+EIY2/9tiIPbrd 102 | =0YYh 103 | -----END PGP PUBLIC KEY BLOCK----- 104 | kubernetes.list: 105 | source: "deb https://apt.kubernetes.io/ kubernetes-xenial main" 106 | keyserver: "https://packages.cloud.google.com/apt/doc/apt-key.gpg" 107 | keyid: 54A6 47F9 048D 5688 D7DA 2ABE 6A03 0B21 BA07 F4FB 108 | write_files: 109 | - owner: root:root 110 | path: /root/daemon.json 111 | permissions: '0644' 112 | content: | 113 | { 114 | "exec-opts": ["native.cgroupdriver=systemd"], 115 | "log-driver": "json-file", 116 | "log-opts": { 117 | "max-size": "100m" 118 | }, 119 | "storage-driver": "overlay2" 120 | } 121 | - owner: root:root 122 | path: /etc/sysctl.d/k8s.conf 123 | permissions: '0644' 124 | content: | 125 | net.bridge.bridge-nf-call-ip6tables = 1 126 | net.bridge.bridge-nf-call-iptables = 1 127 | - owner: root:root 128 | path: /root/install-nfs.sh 129 | permissions: '0644' 130 | content: | 131 | echo "NFS Server Install" 132 | export DEBIAN_FRONTEND=noninteractive 133 | mkdir -p /opt/nfs 134 | apt-get update 135 | apt-get install -y \ 136 | nfs-kernel-server rpcbind 137 | chown nobody:nogroup /opt/nfs 138 | echo "/opt/nfs 172.0.0.0/8(rw,sync,no_root_squash)" >> /etc/exports 139 | systemctl enable rpcbind.service 140 | systemctl enable nfs-server.service 141 | systemctl start rpcbind.service 142 | systemctl start nfs-server.service 143 | - owner: root:root 144 | path: /root/nfs-deployment.yaml 145 | permissions: '0644' 146 | content: | 147 | kind: Deployment 148 | apiVersion: extensions/v1beta1 149 | metadata: 150 | name: nfs-client-provisioner 151 | spec: 152 | replicas: 1 153 | strategy: 154 | type: Recreate 155 | template: 156 | metadata: 157 | labels: 158 | app: nfs-client-provisioner 159 | spec: 160 | serviceAccount: nfs-client-provisioner 161 | containers: 162 | - name: nfs-client-provisioner 163 | image: jicki/nfs-client-provisioner:latest 164 | volumeMounts: 165 | - name: nfs-client-root 166 | mountPath: /persistentvolumes 167 | env: 168 | - name: PROVISIONER_NAME 169 | value: fuseim.pri/ifs 170 | - name: NFS_SERVER 171 | value: 172.16.1.11 172 | - name: NFS_PATH 173 | value: /opt/nfs 174 | volumes: 175 | - name: nfs-client-root 176 | nfs: 177 | server: 172.16.1.11 178 | path: /opt/nfs 179 | - owner: root:root 180 | path: /root/nfs-rbac.yaml 181 | permissions: '0644' 182 | content: | 183 | apiVersion: v1 184 | kind: ServiceAccount 185 | metadata: 186 | name: nfs-client-provisioner 187 | --- 188 | kind: ClusterRole 189 | apiVersion: rbac.authorization.k8s.io/v1 190 | metadata: 191 | name: nfs-client-provisioner-runner 192 | rules: 193 | - apiGroups: [""] 194 | resources: ["persistentvolumes"] 195 | verbs: ["get", "list", "watch", "create", "delete"] 196 | - apiGroups: [""] 197 | resources: ["persistentvolumeclaims"] 198 | verbs: ["get", "list", "watch", "update"] 199 | - apiGroups: ["storage.k8s.io"] 200 | resources: ["storageclasses"] 201 | verbs: ["get", "list", "watch"] 202 | - apiGroups: [""] 203 | resources: ["events"] 204 | verbs: ["list", "watch", "create", "update", "patch"] 205 | --- 206 | kind: ClusterRoleBinding 207 | apiVersion: rbac.authorization.k8s.io/v1 208 | metadata: 209 | name: run-nfs-client-provisioner 210 | subjects: 211 | - kind: ServiceAccount 212 | name: nfs-client-provisioner 213 | namespace: default 214 | roleRef: 215 | kind: ClusterRole 216 | name: nfs-client-provisioner-runner 217 | apiGroup: rbac.authorization.k8s.io 218 | - owner: root:root 219 | path: /root/nfs-storageclass.yaml 220 | permissions: '0644' 221 | content: | 222 | apiVersion: storage.k8s.io/v1 223 | kind: StorageClass 224 | metadata: 225 | name: nfs-storage 226 | provisioner: fuseim.pri/ifs 227 | - owner: root:root 228 | path: /root/bootstrap-nfs-provisioner.sh 229 | permissions: '0644' 230 | content: | 231 | #!/bin/bash 232 | echo "Configuring NFS" 233 | kubectl apply -f nfs-rbac.yaml 234 | kubectl apply -f nfs-deployment.yaml 235 | kubectl apply -f nfs-storageclass.yaml 236 | echo "Making nfs-storage the default storage class" 237 | kubectl patch storageclass nfs-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' 238 | runcmd: 239 | - [ "/bin/bash", "-c", "sudo apt-get install -y containerd.io=1.2.10-3 docker-ce=5:19.03.4~3-0~ubuntu-bionic docker-ce-cli=5:19.03.4~3-0~ubuntu-bionic kubelet=${kube_version}-00 kubeadm=${kube_version}-00 kubectl=${kube_version}-00" ] 240 | - [ "/bin/bash", "-c", "sudo apt-mark hold kubelet kubeadm kubectl" ] 241 | - [ "/bin/bash", "-c", "sudo cp /root/daemon.json /etc/docker/daemon.json" ] 242 | - [ "/bin/bash", "-c", "sudo mkdir -p /etc/systemd/system/docker.service.d" ] 243 | - [ "/bin/bash", "-c", "sudo systemctl daemon-reload" ] 244 | - [ "/bin/bash", "-c", "sudo systemctl restart docker" ] 245 | -------------------------------------------------------------------------------- /config/goldpinger.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: apps/v1 3 | kind: DaemonSet 4 | metadata: 5 | name: goldpinger 6 | namespace: default 7 | labels: 8 | app: goldpinger 9 | spec: 10 | updateStrategy: 11 | type: RollingUpdate 12 | selector: 13 | matchLabels: 14 | app: goldpinger 15 | template: 16 | metadata: 17 | annotations: 18 | prometheus.io/scrape: 'true' 19 | prometheus.io/port: '8080' 20 | labels: 21 | app: goldpinger 22 | spec: 23 | serviceAccount: "goldpinger-serviceaccount" 24 | tolerations: 25 | - key: node-role.kubernetes.io/master 26 | effect: NoSchedule 27 | securityContext: 28 | runAsNonRoot: true 29 | runAsUser: 1000 30 | fsGroup: 2000 31 | containers: 32 | - name: goldpinger 33 | env: 34 | - name: HOST 35 | value: "0.0.0.0" 36 | - name: PORT 37 | value: "8080" 38 | # injecting real hostname will make for easier to understand graphs/metrics 39 | - name: HOSTNAME 40 | valueFrom: 41 | fieldRef: 42 | fieldPath: spec.nodeName 43 | # podIP is used to select a randomized subset of nodes to ping. 44 | - name: POD_IP 45 | valueFrom: 46 | fieldRef: 47 | fieldPath: status.podIP 48 | image: "docker.io/bloomberg/goldpinger:2.0.2" 49 | imagePullPolicy: Always 50 | securityContext: 51 | allowPrivilegeEscalation: false 52 | readOnlyRootFilesystem: true 53 | resources: 54 | limits: 55 | memory: 80Mi 56 | requests: 57 | cpu: 1m 58 | memory: 40Mi 59 | ports: 60 | - containerPort: 8080 61 | name: http 62 | readinessProbe: 63 | httpGet: 64 | path: /healthz 65 | port: 8080 66 | initialDelaySeconds: 20 67 | periodSeconds: 5 68 | livenessProbe: 69 | httpGet: 70 | path: /healthz 71 | port: 8080 72 | initialDelaySeconds: 20 73 | periodSeconds: 5 74 | --- 75 | apiVersion: v1 76 | kind: Service 77 | metadata: 78 | name: goldpinger 79 | namespace: default 80 | labels: 81 | app: goldpinger 82 | spec: 83 | type: NodePort 84 | ports: 85 | - port: 8080 86 | nodePort: 30080 87 | name: http 88 | selector: 89 | app: goldpinger 90 | --- 91 | apiVersion: rbac.authorization.k8s.io/v1beta1 92 | kind: ClusterRoleBinding 93 | metadata: 94 | name: default 95 | roleRef: 96 | apiGroup: rbac.authorization.k8s.io 97 | kind: ClusterRole 98 | name: view 99 | subjects: 100 | - kind: ServiceAccount 101 | name: default 102 | namespace: default 103 | -------------------------------------------------------------------------------- /config/metallb-config.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | namespace: metallb-system 5 | name: config 6 | data: 7 | config: | 8 | address-pools: 9 | - name: default 10 | protocol: layer2 11 | addresses: 12 | - 172.16.1.100-172.16.1.200 13 | -------------------------------------------------------------------------------- /docs/README.MD: -------------------------------------------------------------------------------- 1 | ## Additional Docs & Tools 2 | 3 | Some additional documents found online that maybe useful in your studies are included in this folder. 4 | 5 | Of particular value is the bookmarks.html file. This can be imported into your browser and used on the exam to quickly get to various resources. It is updated as of version 1.18 and should be sorted manually after importing to be most effective. Each link is formated to quickly find information (generally like so: `-`) 6 | 7 | The cka-prep-list.html is a copy of a great resource I found online [here](https://docs.google.com/spreadsheets/d/1mYzfkxu1Iaup3KgO7zhbz7C4nFadSEvogo-lDaFIXK0/edit#gid=254552495). The bookmarks file comes from this spreadsheet actually. 8 | -------------------------------------------------------------------------------- /docs/bookmarks.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | Bookmarks 4 |

Bookmarks

5 |

6 |

Bookmarks bar

7 |

8 |

CKA

9 |

10 |

_Docs Home 11 |
_kubectl - Commands 12 |
_kubectl - Cheat Sheet 13 |
_kubectl - Overview 14 |
_kubectl - View Find Resources 15 |
_kubectl - JSONPath Filtering Format 16 |
_kubectl - Generators 17 |
Workloads - Pod Templates 18 |
Cluster - Taints and Tolerations 19 |
Workloads - Assigning Pods to Nodes 20 |
Workloads - Labels and Selectors 21 |
Workloads - Init Containers 22 |
Workloads - Init Containers (Example initialization) 23 |
Workloads - Resource Limits 24 |
Workloads - Liveness Readiness and Startup Probes 25 |
Workloads - Environment Variables 26 |
Workloads - Command and Arguments 27 |
Workloads - Inject Data Into Applications 28 |
Workloads - Static Pods 29 |
Security - Create TLS secret 30 |
Security - Credentials Securely using Secrets 31 |
Workloads - ConfigMaps 32 |
Workloads - Private Registry 33 |
Workloads - Private Registry (Example) 34 |
Workloads - Resource Quotas 35 |
Workloads - Limit Ranges 36 |
Workloads - Default Namespace Mem/CPU Limits 37 |
Workloads - Publishing Services By Type 38 |
Workloads - Services Examples + TLS secret example 39 |
Workloads - DaemonSet 40 |
Workloads - Deployments 41 |
Workloads - Deployments (Updating) 42 |
Workloads - Jobs 43 |
Workloads - CronJobs 44 |
Workloads - CronJobs (Example) 45 |
Workloads - Service Accounts 46 |
Workloads - Multi-container (Example) 47 |
Security - Network Policies 48 |
Security - RBAC 49 |
Security - Secrets 50 |
Security - Pod/Container Security Context 51 |
Security - Certificates 52 |
Security - Certificates (with kubeadm) 53 |
Security - Pod Security Policies 54 |
Security - CSR via API 55 |
Security - CSR for User Auth Cert (openssl) 56 |
Ingress - Controller 57 |
Ingress - Networking (Examples) 58 |
Storage - hostPath 59 |
Storage - emptyDir 60 |
Storage - Persistent Volumes 61 |
Storage - PVC 62 |
Storage - PVC (Example) 63 |
Cluster - Release Binaries 64 |
Cluster - Node Maintenance 65 |
Cluster - Node Troubleshooting 66 |
Cluster - Troubleshooting (kubeadm) 67 |
Cluster - Configure Multiple Schedulers 68 |
Cluster - etcd Backup 69 |
Cluster - Required Ports 70 |
Cluster - kubadm (Single master cluster) 71 |
Cluster - kubeadm (Cluster Upgrade) 72 |
Cluster - kubeadm (setup HA) 73 |
Cluster - Debugging (DNS Resolution) 74 |
Cluster - Create Certificates (openssl) 75 |
Cluster - TLS bootstrapping 76 |
Cluster - Encryption at Rest 77 |

78 |

79 |

80 | -------------------------------------------------------------------------------- /envvars.tmpl: -------------------------------------------------------------------------------- 1 | KUBE_VERSION=${kube_version} -------------------------------------------------------------------------------- /init.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | ## Install the latest libvirt terraform provider 3 | 4 | PROVIDERREPO=dmacvicar/terraform-provider-libvirt 5 | PROVIDERNAME=terraform-provider-libvirt 6 | ARCH=amd64 7 | OS=`uname -s | tr '[:upper:]' '[:lower:']` 8 | DISTRO=Ubuntu 9 | 10 | IGNORED_EXT='(.tar.gz.asc|.txt|.tar.xz|.asc|.MD|.hsm|+ent.hsm|.rpm|.deb|.sha256|.src.tar.gz)' 11 | 12 | function get_github_latest_urls { 13 | # Description: Scrape github releases for most recent release of a project based on: 14 | # vendor, repo, os, and arch 15 | local vendorapp="${1?"Usage: $0 vendor/app"}" 16 | local OS="${OS:-"linux"}" 17 | local ARCH="${ARCH:-"amd64"}" 18 | curl -s "https://api.github.com/repos/${vendorapp}/releases/latest" | \ 19 | jq -r --arg OS ${OS} --arg ARCH ${ARCH} \ 20 | '.assets[] | .browser_download_url' 21 | } 22 | 23 | latesturl=(`get_github_latest_urls "dmacvicar/terraform-provider-libvirt" | grep -v -E "${IGNORED_EXT}" | grep terraform-provider-libvirt | grep ${DISTRO}`) 24 | applist=() 25 | cnt=${#latesturl[@]} 26 | for ((i=0;i

Solution 6 |

7 | 8 | There are two deployment paths below, one for flannel and another for calico. The only difference between them is the default pod network passed in at the kubeadm init step. Both work well and are on the exam. Calico should be used if you are going to be testing out pod security policies. 9 | 10 | ```bash 11 | ## Flannel based cluster deployment 12 | 13 | make ssh/master 14 | 15 | # Deploy initial master 16 | sudo kubeadm init --pod-network-cidr=10.244.0.0/16 17 | 18 | # Configure kubectl access 19 | mkdir -p $HOME/.kube 20 | sudo mkdir -p /root/.kube 21 | sudo cp -i /etc/kubernetes/admin.conf /root/.kube/config 22 | sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 23 | sudo chown $(id -u):$(id -g) $HOME/.kube/config 24 | 25 | # Deploy Flannel as a network plugin 26 | kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml 27 | ``` 28 | 29 | ```bash 30 | ## Alternatively, a calico based deployment 31 | 32 | make ssh/master 33 | 34 | # Deploy initial master 35 | sudo kubeadm init --pod-network-cidr=192.168.0.0/16 36 | 37 | # Configure kubectl access 38 | mkdir -p $HOME/.kube 39 | sudo mkdir /root/.kube 40 | sudo cp -i /etc/kubernetes/admin.conf /root/.kube/config 41 | sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 42 | sudo chown $(id -u):$(id -g) $HOME/.kube/config 43 | 44 | kubectl apply -f https://docs.projectcalico.org/v3.11/manifests/calico.yaml 45 | ``` 46 | Copy the output from the first step and deploy to the other nodes using something like this: 47 | 48 | ```bash 49 | make ssh/worker1 50 | sudo kubeadm join 172.16.1.11:6443 --token xqckqj.2j6umqdoe416ra9p --discovery-token-ca-cert-hash sha256:6701e97f40377b98e0ae2d35add6ada9050158ab876f9669b22ff09dedae8897 51 | exit 52 | 53 | make ssh/worker2 54 | sudo kubeadm join 172.16.1.11:6443 --token xqckqj.2j6umqdoe416ra9p --discovery-token-ca-cert-hash sha256:6701e97f40377b98e0ae2d35add6ada9050158ab876f9669b22ff09dedae8897 55 | exit 56 | ``` 57 | 58 | Then validate on the master node that all nodes are up and running 59 | 60 | ```bash 61 | make ssh/master 62 | kubectl get nodes 63 | ``` 64 | 65 |

66 | 67 | **TASKS (additional)** 68 | 69 | - Deploy metallb as a loadbalancer 70 | 71 |
Solution 72 |

73 | 74 | Since this is local we use metallb to create services of type LoadBalancer on our cluster. This is not an exam item but would be required for testing out anything relating to loadbalancers in your testing. It has been almost entirely automated and only takes a minute to get deployed anyway. 75 | 76 | ```bash 77 | make kube/clean kube/deploy/metallb 78 | ``` 79 | 80 | The kube config file will have been pulled into the ./.local/kubeconfig/ folder when we get to this point. You can use this to optionally run any kubectl commands outside of the nodes by using the following command. 81 | 82 | ```bash 83 | export KUBECONFIG=$(pwd)/.local/kubeconfig/config 84 | kubectl get nodes 85 | ``` 86 | 87 |

88 | 89 | **TASKS (additional)** 90 | 91 | - Deploy NFS as a persistent storage provider 92 | 93 |
Solution 94 |

95 | 96 | On each node within the `/root` path will be a set of files which can be used to bootstrap an nfs server. 97 | 98 | ```bash 99 | # Run only on master (though the files will be on all nodes) 100 | make ssh/master 101 | sudo su - 102 | chmod +x *.sh 103 | ./install-nfs.sh 104 | exit 105 | ``` 106 | 107 | Then provision the helm chart using this task 108 | 109 | ```bash 110 | make kube/deploy/nfs 111 | ``` 112 | 113 |

-------------------------------------------------------------------------------- /tasks/README.MD: -------------------------------------------------------------------------------- 1 | ## Tasks 2 | 3 | These are tasks you can perform to prepare for the exam. 4 | 5 | The only task that is absolutely necessary to have a working cluster is `1-deploy-kubernetes.md`. This has been modified for this lab to get you to a working state as quickly as possible. Additional tasks which are almost certainly not on the exam are included as well. These are to extend the functionality of the cluster to allow for the use of external loadbalancer services and persistent volumes. 6 | 7 | 8 | > **NOTE:** Most of these are shamelessly taken from [alijahnas/CKA-practice-exercises](https://github.com/alijahnas/CKA-practice-exercises). -------------------------------------------------------------------------------- /tasks/app-lifecycle-management.md: -------------------------------------------------------------------------------- 1 | 2 | ## Application Lifecycle Management 3 | 4 | **TASK:** Create a deployment named nginx-deploy in the ngx namespace using nginx image version 1.16 with three replicas. Check that the deployment rolled out and show running pods. 5 | 6 |
Solution 7 |

8 | ```bash 9 | # Create the template from kubectl 10 | kubectl create ns ngx -o yaml --dry-run=true > nginx-deploy.yml 11 | echo '---' >> nginx-deploy.yml 12 | kubectl create deployment nginx-deploy --image=nginx:1.16 --dry-run=true -o yaml -n ngx >> nginx-deploy.yml 13 | 14 | # Edit the template and add the namespace, and the replica number 15 | nano nginx-deploy.yml 16 | ``` 17 | 18 | ```bash 19 | # Deploy 20 | kubectl deploy -f ./nginx-deploy.yml 21 | 22 | # Validate 23 | kubectl -n ngx rollout status deployment nginx-deploy 24 | kubectl -n ngx get deployment 25 | kubectl -n ngx get pods 26 | ``` 27 |

28 |
29 | 30 | **TASKS:** 31 | 32 | - Scale the prior deployment to 5 replicas and check the status again. 33 | - Then change prior deployment's image tag of nginx container from 1.16 to 1.17. 34 | 35 |
Solution 36 |

37 | 38 | ```bash 39 | kubectl -n ngx scale deployment nginx-deploy --replicas=5 40 | kubectl -n ngx rollout status deployment nginx-deploy 41 | kubectl -n ngx get deploy 42 | kubectl -n ngx get pods 43 | ``` 44 | 45 | Change the image tag: 46 | 47 | ```bash 48 | kubectl -n ngx edit deployment/nginx-deploy 49 | ... 50 | spec: 51 | containers: 52 | - image: nginx:1.17 53 | imagePullPolicy: IfNotPresent 54 | ... 55 | ``` 56 | 57 | Check that new replicaset was created and new pods were deployed: 58 | 59 | ```bash 60 | kubectl -n ngx get replicaset 61 | kubectl -n ngx get pods 62 | ``` 63 |

64 |
65 | 66 | **TASKS:** 67 | - Check the history of the deployment and rollback to previous revision. 68 | - Then check that the nginx image was reverted to 1.16. 69 | 70 |
Solution 71 |

72 | 73 | ```bash 74 | kubectl -n ngx rollout history deployment nginx-deploy 75 | kubectl -n ngx rollout undo deployment nginx-deploy 76 | 77 | kubectl -n ngx get replicaset 78 | kubectl -n ngx get pods 79 | kubectl -n ngx get pods nginx-deploy-7ff78f74b9-72xc8 -o jsonpath='{.spec.containers[0].image}' 80 | ``` 81 |

82 |
83 | 84 | ## Environment variables 85 | 86 | Doc: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/ 87 | 88 | **TASKS:** 89 | - Create a pod with the latest busybox image running a sleep for 1 hour, and give it an environment variable named `PLANET` with the value `blue`. 90 | - Then exec a command in the container to show that it has the configured environment variable. 91 | 92 |
Solution 93 |

94 | 95 | 96 | The pod yaml `envvar.yml`: 97 | ```bash 98 | cat < 122 |

123 | 124 | ## ConfigMaps 125 | 126 | Doc: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/ 127 | 128 | **TASKS** 129 | - Create a configmap named `space` with two values `planet=blue` and `moon=white`. 130 | - Create a pod similar to the previous where you have two environment variables taken from the above configmap and show them in the container. 131 | 132 |
Solution 133 |

134 | 135 | ```bash 136 | kubectl create configmap space --from-literal=planet=blue --from-literal=moon=white 137 | 138 | cat < 169 |

170 | 171 | 172 | **TASKS** 173 | - Create a configmap named `space-system` that contains a file named `system.conf` with the values `planet=blue` and `moon=white`. 174 | - Mount the configmap to a pod and display it from the container through the path `/etc/system.conf` 175 | 176 |
Solution 177 |

178 | 179 | ```bash 180 | cat << EOF > system.conf 181 | planet=blue 182 | moon=white 183 | EOF 184 | 185 | kubectl create configmap space-system --from-file=system.conf 186 | 187 | cat << EOF | kubectl apply -f - 188 | apiVersion: v1 189 | kind: Pod 190 | metadata: 191 | labels: 192 | run: confvolume 193 | name: confvolume 194 | spec: 195 | containers: 196 | - image: busybox:latest 197 | name: confvolume 198 | args: 199 | - sleep 200 | - "3600" 201 | volumeMounts: 202 | - name: system 203 | mountPath: /etc/system.conf 204 | subPath: system.conf 205 | resources: {} 206 | volumes: 207 | - name: system 208 | configMap: 209 | name: space-system 210 | EOF 211 | 212 | kubectl exec confvolume -- cat /etc/system.conf 213 | ``` 214 |

215 |
216 | 217 | ## Secrets 218 | 219 | Doc: https://kubernetes.io/docs/concepts/configuration/secret/ 220 | 221 | **TASKS** 222 | - Create a secret from files containing a username and a password. 223 | - Use the secrets to define environment variables and display them. 224 | - Mount the secret to a pod to `admin-cred` folder and display it. 225 | 226 |
Solution 227 |

228 | 229 | Create secret. 230 | 231 | ```bash 232 | echo -n 'admin' > username 233 | echo -n 'admin-pass' > password 234 | 235 | kubectl create secret generic admin-cred --from-file=username --from-file=password 236 | ``` 237 | 238 | Use secret as environment variables. 239 | 240 | ```yaml 241 | apiVersion: v1 242 | kind: Pod 243 | metadata: 244 | labels: 245 | run: secretenv 246 | name: secretenv 247 | spec: 248 | containers: 249 | - image: busybox:latest 250 | name: secretenv 251 | args: 252 | - sleep 253 | - "3600" 254 | env: 255 | - name: USERNAME 256 | valueFrom: 257 | secretKeyRef: 258 | name: admin-cred 259 | key: username 260 | - name: PASSWORD 261 | valueFrom: 262 | secretKeyRef: 263 | name: admin-cred 264 | key: password 265 | 266 | ``` 267 | 268 | ```bash 269 | kubectl apply -f secretenv.yml 270 | 271 | kubectl exec secretenv -- env | grep -E "USERNAME|PASSWORD" 272 | USERNAME=admin 273 | PASSWORD=admin-pass 274 | ``` 275 | 276 | Mount a secret to pod: 277 | 278 | ```yaml 279 | apiVersion: v1 280 | kind: Pod 281 | metadata: 282 | labels: 283 | run: secretvolume 284 | name: secretvolume 285 | spec: 286 | containers: 287 | - image: busybox:latest 288 | name: secretvolume 289 | args: 290 | - sleep 291 | - "3600" 292 | volumeMounts: 293 | - name: admincred 294 | mountPath: /etc/admin-cred 295 | readOnly: true 296 | volumes: 297 | - name: admincred 298 | secret: 299 | secretName: admin-cred 300 | 301 | ``` 302 | 303 | ```bash 304 | kubectl apply -f secretvolume.yml 305 | 306 | kubectl exec secretvolume -- ls /etc/admin-cred 307 | password 308 | username 309 | 310 | kubectl exec secretvolume -- cat /etc/admin-cred/username 311 | admin 312 | 313 | kubectl exec secretvolume -- cat /etc/admin-cred/password 314 | admin-pass 315 | ``` 316 | 317 |

318 |
319 | 320 | 321 | ## Know how to scale applications 322 | 323 | Docs: 324 | - https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ 325 | - https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ 326 | 327 | **TASKS** 328 | 329 | - Create a deployment with the latest nginx image and scale the deployment to 4 replicas. 330 | 331 |
Solution 332 |

333 | 334 | ```bash 335 | kubectl create deployment scalable --image=nginx:latest 336 | kubectl scale deployment scalable --replicas=4 337 | kubectl get pods 338 | ``` 339 |

340 |
341 | 342 | **TASKS** 343 | 344 | - Autoscale a deployment to have a minimum of two pods and a maximum of 6 pods and that transitions when cpu usage goes above 70%. 345 | 346 |
Solution 347 |

348 | 349 | In order to use Horizontal Pod Autoscaling, you need to have the metrics server installed in you cluster. 350 | 351 | ```bash 352 | # Install metrics server 353 | git clone https://github.com/kubernetes-sigs/metrics-server 354 | kubectl apply -f metrics-server/deploy/kubernetes/ 355 | 356 | # Autoscale a deployment 357 | kubectl create deployment autoscalable --image=nginx:latest 358 | kubectl autoscale deployment autoscalable --min=2 --max=6 --cpu-percent=70 359 | kubectl get hpa 360 | kubectl get pods 361 | ``` 362 |

363 |
364 | -------------------------------------------------------------------------------- /tasks/cluster-maintenance.md: -------------------------------------------------------------------------------- 1 | # Preparation For Cluster Upgrade 2 | 3 | To setup this scenario simply change main.tf to use kubernetes 1.17, deploy the cluster, then upgrade to 1.18 using kubeadm. 4 | 5 | You can generally follow the same set of steps shown for upgrading 1.16 to 1.17. 6 | 7 | [link to upgrade directions](https://v1-18.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/) 8 | 9 | 10 | # Cluster Maintenance (11%) 11 | 12 | ## Understand Kubernetes Cluster upgrade process 13 | 14 |
Solution 15 |

16 | 17 | Doc: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ 18 | 19 | After installing Kubernetes v1.17 here: [install](https://github.com/alijahnas/CKA-practice-exercises/blob/master/installation-configuration-validation.md#install-kubernetes-masters-and-nodes) 20 | 21 | We will now upgrade the cluster to v1.18. 22 | 23 | On master node: 24 | 25 | ```bash 26 | # Upgrade kubeadm 27 | sudo apt-mark unhold kubeadm 28 | sudo apt-get update && sudo apt-get install -y kubeadm=1.18.0-00 29 | sudo apt-mark hold kubeadm 30 | 31 | # Upgrade master node 32 | kubectl drain k8s-master --ignore-daemonsets 33 | sudo kubeadm upgrade plan 34 | sudo kubeadm upgrade apply v1.18.0 35 | 36 | # Update Flannel 37 | kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml 38 | 39 | # Make master node reschedulable 40 | kubectl uncordon k8s-master 41 | 42 | # Upgrade kubelet and kubectl 43 | sudo apt-mark unhold kubelet kubectl 44 | sudo apt-get update && sudo apt-get install -y kubelet=1.18.0-00 kubectl=1.18.0-00 45 | sudo apt-mark hold kubelet kubectl 46 | sudo systemctl restart kubelet 47 | ``` 48 | 49 | On worker nodes: 50 | 51 | ```bash 52 | # Upgrade kubeadm 53 | sudo apt-mark unhold kubeadm 54 | sudo apt-get update && sudo apt-get install -y kubeadm=1.18.0-00 55 | sudo apt-mark hold kubeadm 56 | 57 | # Upgrade worker node 58 | kubectl drain k8s-worker-1 --ignore-daemonsets # On master node, or on worker node if you have proper config 59 | sudo kubeadm upgrade node 60 | 61 | # Upgrade kubelet and kubectl 62 | sudo apt-mark unhold kubelet kubectl 63 | sudo apt-get update && sudo apt-get install -y kubelet=1.18.0-00 kubectl=1.18.0-00 64 | sudo apt-mark hold kubelet kubectl 65 | sudo systemctl restart kubelet 66 | 67 | # Make worker node reschedulable 68 | kubectl uncordon k8s-worker-1 # On master node, or on worker node if you have proper config 69 | ``` 70 | 71 | Verify that the nodes are upgraded to v1.18: 72 | 73 | ```bash 74 | kubectl get nodes 75 | $NAME STATUS ROLES AGE VERSION 76 | k8s-master Ready master 172m v1.18.0 77 | k8s-worker-1 Ready 164m v1.18.0 78 | k8s-worker-2 Ready 164m v1.18.0 79 | ``` 80 | 81 |

82 |
83 | 84 | ## Facilitate operating system upgrades 85 | 86 |
Solution 87 |

88 | 89 | When having a one master node in you cluster, you cannot upgrade the OS system (with reboot) without loosing temporarily access to your cluster. 90 | 91 | Here we will upgrade our worker nodes: 92 | 93 | ```bash 94 | # Hold kubernetes from upgrading 95 | sudo apt-mark hold kubeadm kubelet kubectl 96 | 97 | # Upgrade node 98 | kubectl drain k8s-worker-1 --ignore-daemonsets # On master node, or on worker node if you have proper config 99 | sudo apt update && sudo apt upgrade -y # Be careful about container runtime (e.g., docker) upgrade. 100 | 101 | # Reboot node if necessary 102 | sudo reboot 103 | 104 | # Make worker node reschedulable 105 | kubectl uncordon k8s-worker-1 # On master node, or on worker node if you have proper config 106 | ``` 107 | 108 |

109 |
110 | 111 | ## Implement backup and restore methodologies 112 | 113 |
Solution 114 |

115 | 116 | ### Backup etcd cluster 117 | 118 |

Solution 119 |

120 | 121 | Check the version of your etcd cluster depending on how you installed it. 122 | 123 | ```bash 124 | kubectl exec -it -n kube-system etcd-k8s-master -- etcd --version 125 | etcd Version: 3.4.3 126 | Git SHA: 3cf2f69b5 127 | Go Version: go1.12.12 128 | Go OS/Arch: linux/amd64 129 | ``` 130 | 131 | ```bash 132 | # Download etcd client (should already be root for exam I think) 133 | sudo su - 134 | # Download etcd client (should already exist on the exam) 135 | apt install etcd-client 136 | # Get all key information from the deployment 137 | cat /etc/kubernetes/manifests/etcd.yaml 138 | 139 | # save etcd snapshot 140 | ETCDCTL_API=3 etcdctl snapshot save --endpoints=https://172.16.1.11:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key backupetcd1 141 | 142 | # View the snapshot 143 | ETCDCTL_API=3 etcdctl --write-out=table snapshot status backupetcd1 144 | ``` 145 | 146 |

147 |
148 | 149 | ### Restore an etcd cluster from a snapshot 150 | 151 |
Solution 152 |

153 | 154 | Doc: https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/recovery.md#restoring-a-cluster 155 | 156 |

157 |
158 | 159 |

160 |
-------------------------------------------------------------------------------- /tasks/logging-monitoring.md: -------------------------------------------------------------------------------- 1 | # Logging/Monitoring (5%) 2 | 3 | ## Understand how to monitor all cluster components 4 | 5 | Doc: https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/ 6 | 7 | Questions: 8 | - Install the metrics server and show metrics for nodes and for pods in `kube-system` namespace. 9 | 10 |
Solution 11 |

12 | 13 | ```bash 14 | git clone https://github.com/kubernetes-sigs/metrics-server 15 | # Add --kubelet-insecure-tls to metrics-server/deploy/kubernetes/metrics-server-deployment.yaml if necessary 16 | ... 17 | containers: 18 | - name: metrics-server 19 | image: k8s.gcr.io/metrics-server-amd64:v0.3.6 20 | imagePullPolicy: IfNotPresent 21 | args: 22 | - --cert-dir=/tmp 23 | - --secure-port=4443 24 | - --kubelet-insecure-tls 25 | ... 26 | 27 | # Launch the metrics server 28 | kubectl apply -f kubectl apply -f metrics-server/deploy/kubernetes/ 29 | 30 | # Wait for the server to get metrics and show them 31 | kubectl top nodes 32 | NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 33 | k8s-master 276m 13% 754Mi 19% 34 | k8s-worker-1 87m 4% 391Mi 20% 35 | k8s-worker-2 130m 6% 519Mi 27% 36 | 37 | kubectl top pods -n kube-system 38 | NAME CPU(cores) MEMORY(bytes) 39 | coredns-66bff467f8-vr7ws 5m 5Mi 40 | coredns-66bff467f8-w89dn 6m 6Mi 41 | etcd-k8s-master 44m 34Mi 42 | kube-apiserver-k8s-master 78m 239Mi 43 | kube-controller-manager-k8s-master 31m 37Mi 44 | kube-flannel-ds-amd64-jgvg6 5m 9Mi 45 | kube-flannel-ds-amd64-mdp7q 6m 9Mi 46 | kube-flannel-ds-amd64-n9bfw 6m 9Mi 47 | kube-proxy-f5c9j 3m 12Mi 48 | kube-proxy-qqhkq 1m 12Mi 49 | kube-proxy-tlmvq 2m 12Mi 50 | kube-scheduler-k8s-master 7m 10Mi 51 | metrics-server-64b57fd654-zjcj9 1m 11Mi 52 | 53 | ``` 54 | 55 |

56 |
57 | 58 | 59 | ## Understand how to monitor applications 60 | 61 | Doc: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ 62 | 63 | Questions: 64 | - Create an nginx pod with a liveness and a readiness probe for the port 80. 65 | 66 |
Solution 67 |

68 | 69 | pod-ness.yml: 70 | ```yaml 71 | apiVersion: v1 72 | kind: Pod 73 | metadata: 74 | name: nginx 75 | labels: 76 | run: nginx 77 | spec: 78 | containers: 79 | - name: nginx 80 | image: nginx:latest 81 | readinessProbe: 82 | httpGet: 83 | path: / 84 | port: 80 85 | initialDelaySeconds: 5 86 | periodSeconds: 5 87 | livenessProbe: 88 | httpGet: 89 | path: / 90 | port: 80 91 | ``` 92 | 93 | ```bash 94 | kubectl apply -f pod-ness.yml 95 | kubectl describe pods nginx 96 | ... 97 | Liveness: http-get http://:80/ delay=0s timeout=1s period=10s #success=1 #failure=3 98 | Readiness: http-get http://:80/ delay=5s timeout=1s period=5s #success=1 #failure=3 99 | ... 100 | 101 | ``` 102 | 103 |

104 |
105 | 106 | ## Manage cluster component logs 107 | 108 | Questions: 109 | - Get cluster components logs. 110 | 111 |
Solution 112 |

113 | 114 | Logs depend on how your cluster was deployed. 115 | 116 | For our deployment done in [Installation, Configuration & Validation 12%](https://github.com/alijahnas/CKA-practice-exercises/blob/master/installation-configuration-validation.md) here is how to get logs. 117 | 118 | ```bash 119 | # Kubelet 120 | sudo journalctl -u kubelet 121 | 122 | # API server 123 | kubectl -n kube-system logs kube-apiserver-k8s-master 124 | 125 | # Controller Manager 126 | kubectl -n kube-system logs kube-controller-manager-k8s-master 127 | 128 | # Scheduler 129 | kubectl -n kube-system logs kube-scheduler-k8s-master 130 | 131 | ``` 132 | 133 |

134 |
135 | 136 | 137 | ## Manage application logs 138 | 139 | Doc: https://kubernetes.io/docs/concepts/cluster-administration/logging/ 140 | 141 | Questions: 142 | - Get logs from the nginx pod deployed earlier and redirect them to a file. 143 | 144 |
Solution 145 |

146 | 147 | ```bash 148 | kubectl logs nginx > nginx.log 149 | ``` 150 | 151 |

152 |
153 | -------------------------------------------------------------------------------- /tasks/networking.md: -------------------------------------------------------------------------------- 1 | # Networking (11%) 2 | 3 | ## Understand the networking configuration of the cluster nodes 4 | 5 | Doc: https://kubernetes.io/docs/concepts/cluster-administration/networking/ 6 | 7 | ## Understand Pod networking concepts 8 | 9 | Doc: https://kubernetes.io/docs/concepts/cluster-administration/networking/ 10 | 11 | ## Understand service networking 12 | 13 | Doc: https://kubernetes.io/docs/concepts/services-networking/service/ 14 | 15 | Questions: 16 | - Create a deployment with the latest nginx image and two replicas. 17 | - Expose it's port 80 through a service of type NodePort. 18 | - Show all elements, including the endpoints. 19 | - Get the nginx index page through the NodePort. 20 | 21 |
Solution 22 |

23 | 24 | ```bash 25 | kubectl create deployment nginx --image=nginx:latest 26 | kubectl scale deployment nginx --replicas=2 27 | kubectl expose deployment nginx --port=80 --target-port=80 --type=NodePort 28 | kubectl describe svc nginx 29 | Name: nginx 30 | Namespace: default 31 | Labels: app=nginx 32 | Annotations: 33 | Selector: app=nginx 34 | Type: NodePort 35 | IP: 10.96.36.225 36 | Port: 80/TCP 37 | TargetPort: 80/TCP 38 | NodePort: 30811/TCP 39 | Endpoints: 10.244.1.25:80,10.244.1.26:80 40 | Session Affinity: None 41 | External Traffic Policy: Cluster 42 | Events: 43 | 44 | kubectl get pods -l app=nginx -o wide 45 | NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 46 | nginx-674ff86d-9s9z6 1/1 Running 0 10m 10.244.1.25 k8s-worker-1 47 | nginx-674ff86d-p52qm 1/1 Running 0 10m 10.244.1.26 k8s-worker-1 48 | 49 | # We are getting the page through IP address of the master node and the port allocated by the NodePort service 50 | curl http://172.16.1.11:30811 51 | ... 52 |

Welcome to nginx!

53 |

If you see this page, the nginx web server is successfully installed and 54 | working. Further configuration is required.

55 | ... 56 | 57 | ``` 58 | 59 |

60 |
61 | 62 | 63 | ## Deploy and configure network load balancer 64 | 65 | Doc: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/ 66 | 67 | Questions: 68 | - Do the same exercice as in the previous section but use a Load Balancer service type rather than a NodePort. 69 | 70 | Hint: If you are not running your cluster on a cloud providing a load balancer service, you can use [MetalLB](https://metallb.universe.tf/installation/) 71 | 72 |
Solution 73 |

74 | 75 | ```bash 76 | # We will deploy MetalLB first to provide Load Balancer service type 77 | mkdir metallb 78 | cd metallb 79 | wget https://raw.githubusercontent.com/google/metallb/v0.9.3/manifests/namespace.yaml 80 | wget https://raw.githubusercontent.com/google/metallb/v0.9.3/manifests/metallb.yaml 81 | 82 | # We are giving MetalLB an IP range from our cluster infra to allocate from 83 | cat << EOF > metallb-config.yaml 84 | apiVersion: v1 85 | kind: ConfigMap 86 | metadata: 87 | namespace: metallb-system 88 | name: config 89 | data: 90 | config: | 91 | address-pools: 92 | - name: default 93 | protocol: 94 | addresses: 95 | - 172.16.1.101-172.16.1.150 96 | EOF 97 | 98 | # Apply the manifests 99 | kubectl apply -f namespace.yaml 100 | kubectl apply -f metallb-config.yml 101 | kubectl apply -f metallb.yaml 102 | kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)" 103 | 104 | # Now we create the deployment with a Load Balancer service type 105 | kubectl create deployment nginx --image=nginx:latest 106 | kubectl scale deployment nginx --replicas=2 107 | kubectl expose deployment nginx --port=80 --target-port=80 --type=LoadBalancer 108 | kubectl describe svc nginx 109 | Name: nginx 110 | Namespace: default 111 | Labels: app=nginx 112 | Annotations: 113 | Selector: app=nginx 114 | Type: LoadBalancer 115 | IP: 10.99.146.85 116 | LoadBalancer Ingress: 172.16.1.101 117 | Port: 80/TCP 118 | TargetPort: 80/TCP 119 | NodePort: 32402/TCP 120 | Endpoints: 10.244.1.25:80,10.244.1.26:80 121 | Session Affinity: None 122 | External Traffic Policy: Cluster 123 | Events: 124 | Type Reason Age From Message 125 | ---- ------ ---- ---- ------- 126 | Normal IPAllocated 3s metallb-controller Assigned IP "172.16.1.101" 127 | Normal nodeAssigned 3s metallb-speaker announcing from node "k8s-worker-1" 128 | 129 | # We are getting the page through the IP address allocated by MetalLB from the pool we provided 130 | curl http://172.16.1.101:80 131 | ... 132 |

Welcome to nginx!

133 |

If you see this page, the nginx web server is successfully installed and 134 | working. Further configuration is required.

135 | ... 136 | 137 | ``` 138 | 139 |

140 |
141 | 142 | 143 | ## Know how to use Ingress rules 144 | 145 | Doc: https://kubernetes.io/docs/concepts/services-networking/ingress/ 146 | 147 | Questions: 148 | - Keep the previous deployment of nginx and add a new deployment using the image `bitnami/apache` with two replicas. 149 | - Expose its port 8080 through a service and query it. 150 | - Create an ingress service that redirects /nginx to the nginx service and /apache to the apache service. 151 | 152 |
Solution 153 |

154 | 155 | ```bash 156 | kubectl create deployment apache --image=bitnami/apache:latest 157 | kubectl scale deployment apache --replicas=2 158 | kubectl expose deployment apache --port=8080 --target-port=8080 --type=LoadBalancer # Replace by NodePort if you don't have a LoadBalancer provider 159 | kubectl describe svc apache 160 | Name: apache 161 | Namespace: default 162 | Labels: app=apache 163 | Annotations: 164 | Selector: app=apache 165 | Type: LoadBalancer 166 | IP: 10.101.123.225 167 | LoadBalancer Ingress: 172.16.1.102 168 | Port: 8080/TCP 169 | TargetPort: 8080/TCP 170 | NodePort: 31041/TCP 171 | Endpoints: 10.244.1.28:8080,10.244.2.68:8080 172 | Session Affinity: None 173 | External Traffic Policy: Cluster 174 | Events: 175 | Type Reason Age From Message 176 | ---- ------ ---- ---- ------- 177 | Normal IPAllocated 5m55s metallb-controller Assigned IP "172.16.1.102" 178 | Normal nodeAssigned 5m55s metallb-speaker announcing from node "k8s-worker-2" 179 | 180 | curl http://172.16.1.102:8080 181 |

It works!

182 | ``` 183 | 184 | web-ingress.yml: 185 | ```yaml 186 | apiVersion: networking.k8s.io/v1beta1 187 | kind: Ingress 188 | metadata: 189 | name: web-ingress 190 | annotations: 191 | kubernetes.io/ingress.class: "nginx" 192 | nginx.ingress.kubernetes.io/rewrite-target: / 193 | spec: 194 | rules: 195 | - host: nginx-or-apache.com 196 | http: 197 | paths: 198 | - path: /nginx 199 | backend: 200 | serviceName: nginx 201 | servicePort: 80 202 | - path: /apache 203 | backend: 204 | serviceName: apache 205 | servicePort: 8080 206 | 207 | ``` 208 | 209 | ```bash 210 | # Install the nginx ingress controller if necessary then create the ingress 211 | kubectl apply -f web-ingress.yml 212 | kubectl describe ingress web-ingress 213 | 214 | ``` 215 | 216 |

217 |
218 | 219 | ## Know how to configure and use the cluster DNS 220 | 221 | Doc: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ 222 | 223 | Questions: 224 | - Create a busybox pod and resolve the nginx and apache services created earlier from within the pod. 225 | 226 |
Solution 227 |

228 | 229 | ```bash 230 | kubectl run busybox --image=busybox --rm -it --restart=Never -- sh 231 | If you don't see a command prompt, try pressing enter. 232 | # nslookup apache 233 | Server: 10.96.0.10 234 | Address: 10.96.0.10:53 235 | 236 | Name: apache.default.svc.cluster.local 237 | Address: 10.105.144.161 238 | 239 | # nslookup nginx 240 | Server: 10.96.0.10 241 | Address: 10.96.0.10:53 242 | 243 | Name: nginx.default.svc.cluster.local 244 | Address: 10.99.146.85 245 | 246 | ``` 247 | 248 |

249 |
250 | 251 | 252 | ## Understand CNI 253 | 254 | Doc: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/ 255 | -------------------------------------------------------------------------------- /tasks/scheduling.md: -------------------------------------------------------------------------------- 1 | # Scheduling (5%) 2 | 3 | ## Use label selectors to schedule Pods 4 | 5 | Doc: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ 6 | 7 | Questions: 8 | - Label a node with `kind=special` and schedule a pod to that node. 9 | 10 |
Solution 11 |

12 | 13 | pod-selector.yml: 14 | ```yaml 15 | apiVersion: v1 16 | kind: Pod 17 | metadata: 18 | labels: 19 | run: podsel 20 | name: podsel 21 | spec: 22 | containers: 23 | - image: busybox:latest 24 | name: podsel 25 | args: 26 | - sleep 27 | - "3600" 28 | nodeSelector: 29 | kind: special 30 | ``` 31 | 32 | ```bash 33 | kubectl label nodes k8s-worker-1 kind=special 34 | kubectl apply -f pod-selector.yml 35 | 36 | kubectl get pods -o wide 37 | NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 38 | podsel 1/1 Running 0 14s 10.244.1.6 k8s-worker-1 39 | 40 | ``` 41 | 42 |

43 |
44 | 45 | Questions: 46 | - Use antiaffinity to launch a pod to a different node than the pod where the first one was scheduled. 47 | 48 |
Solution 49 |

50 | 51 | pod-antiaffinity.yml: 52 | ```yaml 53 | apiVersion: v1 54 | kind: Pod 55 | metadata: 56 | labels: 57 | run: podaff 58 | name: podaff 59 | spec: 60 | containers: 61 | - image: busybox:latest 62 | name: podaff 63 | args: 64 | - sleep 65 | - "3600" 66 | affinity: 67 | podAntiAffinity: 68 | requiredDuringSchedulingIgnoredDuringExecution: 69 | - labelSelector: 70 | matchExpressions: 71 | - key: run 72 | operator: In 73 | values: 74 | - podsel 75 | topologyKey: kubernetes.io/hostname 76 | ``` 77 | 78 | ```bash 79 | kubectl apply -f pod-antiaffinity.yml 80 | 81 | kubectl get pods -o wide 82 | NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 83 | podaff 1/1 Running 0 8m47s 10.244.2.57 k8s-worker-2 84 | podsel 1/1 Running 0 16m 10.244.1.8 k8s-worker-1 85 | 86 | ``` 87 | 88 |

89 |
90 | 91 | Questions: 92 | - Taint a node with `type=special:NoSchedule`, make the other node unschedulable, and create a pod to tolerate this taint. 93 | 94 |
Solution 95 |

96 | 97 | pod-toleration.yml: 98 | ```yaml 99 | apiVersion: v1 100 | kind: Pod 101 | metadata: 102 | labels: 103 | run: podtol 104 | name: podtol 105 | spec: 106 | containers: 107 | - image: busybox:latest 108 | name: podtol 109 | args: 110 | - sleep 111 | - "3600" 112 | tolerations: 113 | - key: "type" 114 | operator: "Equal" 115 | value: "special" 116 | effect: "NoSchedule" 117 | 118 | ``` 119 | 120 | ```bash 121 | kubectl taint node k8s-worker-1 type=special:NoSchedule 122 | kubectl cordon k8s-worker-2 123 | kubectl apply -f pod-toleration.yml 124 | 125 | kubectl get pods -o wide 126 | NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 127 | podtol 1/1 Running 0 16s 10.244.1.13 k8s-worker-1 128 | 129 | ``` 130 | 131 |

132 |
133 | 134 | 135 | ## Understand the role of DaemonSets 136 | 137 | Doc: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/ 138 | 139 | Questions: 140 | - Create a DaemonSet and see that it runs on all nodes. 141 | 142 |
Solution 143 |

144 | 145 | daemonset.yml 146 | ```yaml 147 | apiVersion: apps/v1 148 | kind: DaemonSet 149 | metadata: 150 | labels: 151 | type: daemon 152 | name: daemontest 153 | spec: 154 | selector: 155 | matchLabels: 156 | run: daemon 157 | template: 158 | metadata: 159 | labels: 160 | run: daemon 161 | name: daemonpod 162 | spec: 163 | containers: 164 | - image: busybox:latest 165 | name: daemonpod 166 | args: 167 | - sleep 168 | - "3600" 169 | ``` 170 | 171 | ```bash 172 | kubectl apply -f daemonset.yml 173 | 174 | kubectl get pods -o wide 175 | NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 176 | daemontest-9lwqc 1/1 Running 0 4m13s 10.244.1.14 k8s-worker-1 177 | daemontest-ch7rq 1/1 Running 0 4m13s 10.244.2.65 k8s-worker-2 178 | 179 | ``` 180 | 181 |

182 |
183 | 184 | 185 | ## Understand how resource limits can affect Pod Scheduling 186 | 187 | Doc: https://kubernetes.io/docs/concepts/policy/resource-quotas/ 188 | 189 | Questions: 190 | - Create a pod with a busybox container that requests 1G of memory and half a CPU, and has limits at 2G of memory and a whole CPU. 191 | 192 |
Solution 193 |

194 | 195 | pod-quota.yml 196 | ```yaml 197 | apiVersion: v1 198 | kind: Pod 199 | metadata: 200 | labels: 201 | run: podquota 202 | name: podquota 203 | spec: 204 | containers: 205 | - image: busybox:latest 206 | name: podquota 207 | args: 208 | - sleep 209 | - "3600" 210 | resources: 211 | requests: 212 | memory: "1Gi" 213 | cpu: "500m" 214 | limits: 215 | memory: "2Gi" 216 | cpu: "1" 217 | 218 | ``` 219 | 220 | ```bash 221 | kubectl apply -f pod-quota.yml 222 | 223 | kubectl describe pod pod-quota 224 | ... 225 | Limits: 226 | cpu: 1 227 | memory: 2Gi 228 | Requests: 229 | cpu: 500m 230 | memory: 1Gi 231 | ... 232 | ``` 233 | 234 |

235 |
236 | 237 | 238 | ## Understand how to run multiple schedulers and how to configure Pods to use them 239 | 240 | Doc: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/ 241 | 242 | ## Manually schedule a Pod without a scheduler 243 | 244 | Doc: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodename 245 | 246 | Questions: 247 | - Force a pod to be on a specific node with using the scheduler, and show that it was assigned to it. 248 | 249 |
Solution 250 |

251 | 252 | pod-node.yml: 253 | ```yaml 254 | apiVersion: v1 255 | kind: Pod 256 | metadata: 257 | labels: 258 | run: podnode 259 | name: podnode 260 | spec: 261 | containers: 262 | - image: busybox:latest 263 | name: podnode 264 | args: 265 | - sleep 266 | - "3600" 267 | nodeName: k8s-worker-2 268 | 269 | ``` 270 | 271 | ```bash 272 | kubectl apply -f pod-node.yml 273 | 274 | kubectl get pods -o wide 275 | NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 276 | podnode 1/1 Running 0 10s 10.244.2.66 k8s-worker-2 277 | 278 | ``` 279 | 280 |

281 |
282 | 283 | ## Display scheduler events 284 | 285 | Doc: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/#verifying-that-the-pods-were-scheduled-using-the-desired-schedulers 286 | 287 | Questions: 288 | - Check the scheduler events. 289 | 290 |
Solution 291 |

292 | 293 | ```bash 294 | kubectl get events 295 | kubectl get events --all-namespaces 296 | ``` 297 | 298 |

299 |
300 | 301 | ## Know how to configure the Kubernetes scheduler 302 | 303 | Doc: https://kubernetes.io/docs/concepts/scheduling/scheduler-perf-tuning/ 304 | -------------------------------------------------------------------------------- /tasks/security.md: -------------------------------------------------------------------------------- 1 | # Security (12%) 2 | 3 | ## Know how to configure authentication and authorization 4 | 5 | Docs: 6 | - https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/ 7 | - https://kubernetes.io/docs/reference/access-authn-authz/rbac/ 8 | 9 | Questions: 10 | - Create a service account name lister, give it the ability to list pods. 11 | - Launch a busybox pod with this service account and list pods from within the pod (podception) using curl. Check that other resources like services are forbidden. 12 | 13 |
Solution 14 |

15 | 16 | pod-sa.yml: 17 | ```yaml 18 | apiVersion: v1 19 | kind: Pod 20 | metadata: 21 | labels: 22 | run: podsa 23 | name: podsa 24 | spec: 25 | serviceAccountName: lister 26 | containers: 27 | - image: tutum/curl:latest 28 | name: podsa 29 | command: ["sleep","3600"] 30 | ``` 31 | 32 | ```bash 33 | # Creating the lister service account and giving it the capabilities 34 | kubectl create serviceaccount lister 35 | kubectl create role lister-role --verb=get,watch,list --resource=pods 36 | kubectl create rolebinding lister-rolebinding --role=lister-role --serviceaccount=default:lister 37 | 38 | kubectl apply -f pod-sa.yml 39 | kubectl exec -it podsa -- bash 40 | # From within the pod now 41 | API=https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT 42 | TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token) 43 | CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt 44 | NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace) 45 | # Listing pods 46 | curl -H "Authorization: Bearer $TOKEN" --cacert $CACERT $API/api/v1/namespaces/$NAMESPACE/pods 47 | "kind": "PodList", 48 | "apiVersion": "v1", 49 | "metadata": { 50 | "selfLink": "/api/v1/namespaces/default/pods", 51 | "resourceVersion": "1085753" 52 | }, 53 | "items": [ 54 | { 55 | ... 56 | 57 | # Trying to list services is forbidden 58 | curl -H "Authorization: Bearer $TOKEN" --cacert $CACERT $K8S/api/v1/amespaces/$NAMESPACE/service 59 | { 60 | "kind": "Status", 61 | "apiVersion": "v1", 62 | "metadata": { 63 | 64 | }, 65 | "status": "Failure", 66 | "message": "service is forbidden: User \"system:serviceaccount:default:lister\" cannot list resource \"service\" in API group \"\" in the namespace \"default\"", 67 | "reason": "Forbidden", 68 | "details": { 69 | "kind": "service" 70 | }, 71 | "code": 403 72 | } 73 | ``` 74 | 75 |

76 |
77 | 78 | 79 | ## Understand Kubernetes security primitives 80 | 81 | Docs: 82 | - https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/ 83 | - https://kubernetes.io/docs/concepts/configuration/secret/ 84 | - https://kubernetes.io/docs/concepts/policy/pod-security-policy/ 85 | 86 | ## Know how to configure network policies 87 | 88 | Doc: https://kubernetes.io/docs/concepts/services-networking/network-policies/ 89 | 90 | Be sure to have deployed a CNI that supports network policies like Calico. 91 | 92 | Questions: 93 | - Create one busybox pod with label `role: client` and one deployment of two nginx pods with label `role: server`. Expose the nginx port 80 with a service. 94 | - Create a network policy that denies all ingress traffic. Check that the busybox pod can't reach the nginx service. 95 | - Add one exception for port 80 from the busybox pod and check again connectivity to the nginx service. 96 | 97 |
Solution 98 |

99 | 100 | server-client.yml: 101 | ```yaml 102 | --- 103 | apiVersion: apps/v1 104 | kind: Deployment 105 | metadata: 106 | labels: 107 | app: nginx 108 | role: server 109 | name: nginx-deploy 110 | spec: 111 | replicas: 2 112 | selector: 113 | matchLabels: 114 | app: nginx 115 | role: server 116 | template: 117 | metadata: 118 | labels: 119 | app: nginx 120 | role: server 121 | spec: 122 | containers: 123 | - image: nginx:latest 124 | name: nginx 125 | 126 | --- 127 | apiVersion: v1 128 | kind: Service 129 | metadata: 130 | name: nginx 131 | spec: 132 | selector: 133 | app: nginx 134 | role: server 135 | ports: 136 | - protocol: TCP 137 | port: 80 138 | targetPort: 80 139 | 140 | --- 141 | apiVersion: v1 142 | kind: Pod 143 | metadata: 144 | labels: 145 | role: client 146 | name: busybox 147 | spec: 148 | containers: 149 | - image: busybox:latest 150 | name: busybox 151 | args: 152 | - sleep 153 | - "3600" 154 | 155 | --- 156 | apiVersion: networking.k8s.io/v1 157 | kind: NetworkPolicy 158 | metadata: 159 | name: default-deny 160 | spec: 161 | podSelector: {} 162 | policyTypes: 163 | - Ingress 164 | 165 | ``` 166 | 167 | ```bash 168 | kubectl apply -f server-client.yml 169 | kubectl exec busybox -- wget nginx 170 | ``` 171 | 172 | allow-80.yml 173 | ```yaml 174 | apiVersion: networking.k8s.io/v1 175 | kind: NetworkPolicy 176 | metadata: 177 | name: allow-80 178 | spec: 179 | podSelector: 180 | matchLabels: 181 | role: server 182 | ingress: 183 | - from: 184 | - podSelector: 185 | matchLabels: 186 | role: client 187 | ports: 188 | - port: 80 189 | 190 | ``` 191 | 192 | ```bash 193 | kubectl apply -f allow-80.yml 194 | kubectl exec busybox -- wget nginx 195 | ``` 196 | 197 |

198 |
199 | 200 | 201 | ## Create and manage TLS certificates for cluster components 202 | 203 | Docs: 204 | - https://kubernetes.io/docs/concepts/cluster-administration/certificates/ 205 | - https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/ 206 | 207 | Kelsey Hightower, in his now famous Kubernetes the hard way, has a great guide on creating and distributing certificates for cluster components here: [Certificates the hard way](https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/04-certificate-authority.md) 208 | 209 | Kubernetes also offers an [API](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/) to provision TLS certificates signed by a CA that you control. 210 | 211 | Questions: 212 | - Create a certificate signing request with `cfssl` for a user named `new-admin` and create a certificate through the API that it will use to authenticate, and give it the cluster-admin role. 213 | - Create a config with this user and list nodes with it. 214 | 215 |
Solution 216 |

217 | 218 | ```bash 219 | # Download cfssl first 220 | wget -q --show-progress --https-only --timestamping https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 221 | chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 222 | sudo mv cfssl_linux-amd64 /usr/local/bin/cfssl 223 | sudo mv cfssljson_linux-amd64 /usr/local/bin/cfssljson 224 | 225 | cat << EOF | cfssl genkey - | cfssljson -bare new-admin 226 | { 227 | "CN": "new-admin", 228 | "key": { 229 | "algo": "rsa", 230 | "size": 2048 231 | }, 232 | "names": [ 233 | { 234 | "C": "FR", 235 | "L": "Paris", 236 | "O": "system:authenticated", 237 | "OU": "CKA practice exercises", 238 | "ST": "IDF" 239 | } 240 | ] 241 | } 242 | EOF 243 | 244 | cat << EOF > new-admin-csr.yml 245 | apiVersion: certificates.k8s.io/v1beta1 246 | kind: CertificateSigningRequest 247 | metadata: 248 | name: new-admin-csr 249 | spec: 250 | request: $(cat new-admin.csr | base64 | tr -d '\n') 251 | usages: 252 | - digital signature 253 | - key encipherment 254 | - client auth 255 | EOF 256 | 257 | kubectl apply -f new-admin-csr.yml 258 | 259 | # Approve the CSR through the API 260 | kubectl certificate approve new-admin-csr 261 | 262 | # Get the signed certificate 263 | kubectl get csr new-admin-csr -o jsonpath='{.status.certificate}' | base64 --decode > new-admin.crt 264 | 265 | # Create a ClusterRoleBinding for user new-admin and give cluster-admin role 266 | kubectl create clusterrolebinding cluster-new-admin --clusterrole=cluster-admin --user=new-admin 267 | 268 | # Create config for user new-admin 269 | kubectl config set-credentials new-admin --client-certificate=new-admin.crt --client-key=new-admin-key.pem --embed-certs=true 270 | kubectl config set-context new-admin@kubernetes --cluster=kubernetes --user=new-admin 271 | kubectl config view 272 | apiVersion: v1 273 | clusters: 274 | - cluster: 275 | certificate-authority-data: DATA+OMITTED 276 | server: https://172.16.1.11:6443 277 | name: kubernetes 278 | contexts: 279 | - context: 280 | cluster: kubernetes 281 | user: kubernetes-admin 282 | name: kubernetes-admin@kubernetes 283 | - context: 284 | cluster: kubernetes 285 | user: new-admin 286 | name: new-admin@kubernetes 287 | current-context: new-admin@kubernetes 288 | kind: Config 289 | preferences: {} 290 | users: 291 | - name: kubernetes-admin 292 | user: 293 | client-certificate-data: REDACTED 294 | client-key-data: REDACTED 295 | - name: new-admin 296 | user: 297 | client-certificate-data: REDACTED 298 | client-key-data: REDACTED 299 | 300 | # Use context to list nodes 301 | kubectl config use-context new-admin@kubernetes 302 | Switched to context "new-admin@kubernetes". 303 | 304 | kubectl get nodes 305 | NAME STATUS ROLES AGE VERSION 306 | k8s-master Ready master 5d v1.18.0 307 | k8s-worker-1 Ready 5d v1.18.0 308 | k8s-worker-2 Ready 5d v1.18.0 309 | 310 | ``` 311 | 312 |

313 |
314 | 315 | ## Work with images securely 316 | 317 | Docs: 318 | - https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ 319 | - https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account 320 | 321 | ## Define security contexts 322 | 323 | Doc: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ 324 | 325 | Questions: 326 | - Create a pod that runs as user with ID 9001, group 9002, and check the ids from within the pod 327 | 328 |
Solution 329 |

330 | 331 | pod-context.yml: 332 | ```yaml 333 | apiVersion: v1 334 | kind: Pod 335 | metadata: 336 | labels: 337 | run: over9000 338 | name: over9000 339 | spec: 340 | securityContext: 341 | runAsUser: 9001 342 | runAsGroup: 9002 343 | containers: 344 | - image: busybox:latest 345 | name: over9000 346 | args: 347 | - sleep 348 | - "9001" 349 | 350 | ``` 351 | 352 | ```bash 353 | kubectl apply -f pod-context.yml 354 | kubectl exec over9000 -- id 355 | uid=9001 gid=9002 356 | ``` 357 | 358 |

359 |
360 | 361 | ## Secure persistent key value store 362 | 363 | For securing etcd check: https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#securing-etcd-clusters 364 | 365 | For encrypting data at rest check: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/ 366 | 367 | For secrets usage check [Application configuration with secrets](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/) -------------------------------------------------------------------------------- /tasks/storage.md: -------------------------------------------------------------------------------- 1 | # Storage (7%) 2 | 3 | ## Understand persistent volumes and know how to create them 4 | 5 | Doc: https://kubernetes.io/docs/concepts/storage/persistent-volumes/ 6 | 7 | Questions: 8 | - Create a pod and mount a volume with hostPath directory. 9 | - Check that the contents of the directory are accessible through the pod. 10 | 11 |
Solution 12 |

13 | 14 | pv-pod.yml: 15 | ```yaml 16 | apiVersion: v1 17 | kind: Pod 18 | metadata: 19 | labels: 20 | run: pv-pod 21 | name: pv-pod 22 | spec: 23 | containers: 24 | - image: busybox:latest 25 | name: pv-pod 26 | args: 27 | - sleep 28 | - "3600" 29 | volumeMounts: 30 | - name: data 31 | mountPath: "/data" 32 | volumes: 33 | - name: data 34 | hostPath: 35 | path: "/home/ubuntu/data/" 36 | ``` 37 | 38 | ```bash 39 | # Create directory and file inside it 40 | mkdir /home/ubuntu/data 41 | touch data/file 42 | 43 | kubectl apply -f pv-pod.yml 44 | kubectl exec pv-pod -- ls /data 45 | file 46 | ``` 47 | 48 |

49 |
50 | 51 | ## Understand access modes for volumes 52 | 53 | Doc: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes 54 | 55 | ## Understand persistent volume claims primitive 56 | 57 | Doc: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes 58 | 59 | Questions: 60 | - Create a persistent volume from hostPath and a persistent volume claim corresponding tothat PV. Create a pod that uses the PVC and check that the volume is mounted in the pod. 61 | - Create a file from the pod in the volume then delete it and create a new pod with the same volume and show the created file by the first pod. 62 | 63 |
Solution 64 |

65 | 66 | pv-data.yml: 67 | ```yaml 68 | apiVersion: v1 69 | kind: PersistentVolume 70 | metadata: 71 | name: pv-data 72 | spec: 73 | storageClassName: "local" 74 | capacity: 75 | storage: 1Gi 76 | accessModes: 77 | - ReadWriteOnce 78 | hostPath: 79 | path: "/home/ubuntu/data" 80 | 81 | ``` 82 | 83 | pvc-data.yml: 84 | ```yaml 85 | apiVersion: v1 86 | kind: PersistentVolumeClaim 87 | metadata: 88 | name: pvc-data 89 | spec: 90 | storageClassName: "local" 91 | accessModes: 92 | - ReadWriteOnce 93 | resources: 94 | requests: 95 | storage: 1Gi 96 | 97 | ``` 98 | 99 | pvc-pod.yml: 100 | ```yaml 101 | apiVersion: v1 102 | kind: Pod 103 | metadata: 104 | labels: 105 | run: pvc-pod 106 | name: pvc-pod 107 | spec: 108 | containers: 109 | - image: busybox:latest 110 | name: pvc-pod 111 | args: 112 | - sleep 113 | - "3600" 114 | volumeMounts: 115 | - name: data 116 | mountPath: "/data" 117 | volumes: 118 | - name: data 119 | persistentVolumeClaim: 120 | claimName: pvc-data 121 | 122 | ``` 123 | 124 | Create a pod with the PVC. Create a file on volume. Delete the pod and create a new one with the same volume. Check that the file has persisted. 125 | 126 | ```bash 127 | kubectl apply -f pv-data.yml 128 | kubectl apply -f pvc-data.yml 129 | kubectl apply -f pvc-pod.yml 130 | 131 | kubectl get pv 132 | NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE 133 | pv-data 1Gi RWO Retain Bound default/pvc-data local 20m 134 | 135 | kubectl get pvc 136 | NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE 137 | pvc-data Bound pv-data 1Gi RWO local 20m 138 | 139 | # Check that the volume has been mounted 140 | kubectl exec pvc-pod -- ls /data/ 141 | file 142 | 143 | # Create a new file 144 | kubectl exec pvc-pod -- touch /data/file2 145 | 146 | # Delete the pod 147 | kubectl delete -f pvc-pod.yml 148 | 149 | # Copy the pvc-pod.yml and change the name of the pod to pvc-pod-2 150 | kubectl apply -f pvc-pod-2.yml 151 | 152 | # Check that the file from previous pod has persisted on volume 153 | kubectl exec pvc-pod-2 -- ls /data/ 154 | file 155 | file2 156 | ``` 157 | 158 |

159 |
160 | 161 | 162 | ## Understand Kubernetes storage objects 163 | 164 | Docs: 165 | - Persistent Volumes: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistent-volumes 166 | - Persistent Volume Claims: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims 167 | - Storage classes: https://kubernetes.io/docs/concepts/storage/storage-classes/ 168 | 169 | ## Know how to configure applications with persistent storage 170 | 171 | Doc: https://kubernetes.io/docs/tasks/configure-pod-container/configure-volume-storage/ 172 | 173 |
Solution 174 |

175 | 176 | Check the section [Understand persistent volume claims primitive](https://github.com/alijahnas/CKA-practice-exercises/blob/master/storage.md#understand-persistent-volume-claims-primitive) above. 177 | 178 |

179 |
180 | -------------------------------------------------------------------------------- /tasks/troubleshooting.md: -------------------------------------------------------------------------------- 1 | # Troubleshooting (10%) 2 | 3 | ## Troubleshoot application failure 4 | 5 | Doc: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/ 6 | 7 | Questions: 8 | - Launch a pod with a busybox container that launches with the `sheep 3600` command (this command doesn't exist. 9 | - Get the logs from the pod, then correct the error to make it launch `sleep 3600`. 10 | 11 |
Solution 12 |

13 | 14 | podfail.yml: 15 | ```yml 16 | apiVersion: v1 17 | kind: Pod 18 | metadata: 19 | labels: 20 | run: podfail 21 | name: podfail 22 | spec: 23 | containers: 24 | - image: busybox:latest 25 | name: podfail 26 | args: 27 | - sheep 28 | - "3600" 29 | ``` 30 | 31 | ```bash 32 | kubectl apply -f podfail.yml 33 | 34 | kubectl describe pods podfail 35 | ... 36 | Warning Failed 3m14s (x4 over 4m2s) kubelet, k8s-worker-1 Error: failed to start container "podfail": Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"sheep\": executable file not found in $PATH": unknown 37 | ... 38 | 39 | kubectl delete -f podfail.yml 40 | # Change sheep to sleep 41 | kubectl apply -f podfail.yml 42 | ``` 43 | 44 |

45 |
46 | 47 | 48 | ## Troubleshoot control plane failure 49 | 50 | Doc: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/ 51 | 52 | Questions: 53 | - Get logs from the control plane in the `kube-system` namespace. 54 | 55 |
Solution 56 |

57 | 58 | Check: https://github.com/alijahnas/CKA-practice-exercises/blob/master/logging-monitoring.md#manage-cluster-component-logs 59 | 60 |

61 |
62 | 63 | 64 | ## Troubleshoot worker node failure 65 | 66 | Doc: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/ 67 | 68 | Questions: 69 | - Check the node status and the system logs for kubelet on the failing node. 70 | 71 |
Solution 72 |

73 | 74 | ```bash 75 | kubectl describe node k8s-worker-1 76 | 77 | # From k8s-worker-1 if reachable 78 | sudo journalctl -u kubelet | grep -i error 79 | ``` 80 | 81 |

82 |
83 | 84 | 85 | ## Troubleshoot networking 86 | 87 | Doc: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ 88 | 89 | Questions: 90 | - Check the `kube-dns` service running in the `kube-system` namespace and check the endpoints behind the service. Check the pods that serve the endpoints. 91 | 92 |
Solution 93 |

94 | 95 | ```bash 96 | kubectl -n kube-system describe svc kube-dns 97 | Name: kube-dns 98 | Namespace: kube-system 99 | Labels: k8s-app=kube-dns 100 | kubernetes.io/cluster-service=true 101 | kubernetes.io/name=KubeDNS 102 | Annotations: prometheus.io/port: 9153 103 | prometheus.io/scrape: true 104 | Selector: k8s-app=kube-dns 105 | Type: ClusterIP 106 | IP: 10.96.0.10 107 | Port: dns 53/UDP 108 | TargetPort: 53/UDP 109 | Endpoints: 10.244.0.9:53,10.244.2.64:53 110 | Port: dns-tcp 53/TCP 111 | TargetPort: 53/TCP 112 | Endpoints: 10.244.0.9:53,10.244.2.64:53 113 | Port: metrics 9153/TCP 114 | TargetPort: 9153/TCP 115 | Endpoints: 10.244.0.9:9153,10.244.2.64:9153 116 | Session Affinity: None 117 | Events: 118 | 119 | kubectl -n kube-system describe ep kube-dns 120 | Name: kube-dns 121 | Namespace: kube-system 122 | Labels: k8s-app=kube-dns 123 | kubernetes.io/cluster-service=true 124 | kubernetes.io/name=KubeDNS 125 | Annotations: 126 | Subsets: 127 | Addresses: 10.244.0.9,10.244.2.64 128 | NotReadyAddresses: 129 | Ports: 130 | Name Port Protocol 131 | ---- ---- -------- 132 | dns-tcp 53 TCP 133 | metrics 9153 TCP 134 | dns 53 UDP 135 | 136 | Events: 137 | 138 | kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide 139 | NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 140 | coredns-66bff467f8-vr7ws 1/1 Running 1 3d5h 10.244.0.9 k8s-master 141 | coredns-66bff467f8-w89dn 1/1 Running 1 3d5h 10.244.2.64 k8s-worker-2 142 | 143 | ``` 144 | 145 |

146 |
147 | -------------------------------------------------------------------------------- /versions.tf: -------------------------------------------------------------------------------- 1 | 2 | terraform { 3 | required_version = ">= 0.12" 4 | } 5 | --------------------------------------------------------------------------------