├── .dockerignore ├── .gitignore ├── Dockerfile ├── LICENSE ├── Makefile ├── README.md ├── bin └── kustomize ├── cmd ├── cert.go ├── main.go ├── webhook.go └── webhookconfig.go ├── deploy ├── clusterrole.yaml ├── clusterrolebinding.yaml ├── configmap.yaml ├── deployment.yaml ├── kustomization.yaml ├── namespace.yaml ├── nginx-configmap.yaml ├── service.yaml └── serviceaccount.yaml ├── go.mod ├── go.sum ├── medium-article.md ├── mutating-admission-webhook.jpg ├── mutating-admission-webhook.xml └── prestop.sh /.dockerignore: -------------------------------------------------------------------------------- 1 | # Binaries for programs and plugins 2 | *.exe 3 | *.dll 4 | *.so 5 | *.dylib 6 | 7 | # Test binary, build with `go test -c` 8 | *.test 9 | 10 | # Output of the go coverage tool, specifically when used with LiteIDE 11 | *.out 12 | 13 | vendor/* 14 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Binaries for programs and plugins 2 | *.exe 3 | *.dll 4 | *.so 5 | *.dylib 6 | 7 | # Test binary, build with `go test -c` 8 | *.test 9 | 10 | # Output of the go coverage tool, specifically when used with LiteIDE 11 | *.out 12 | 13 | vendor/* 14 | 15 | build/_output 16 | 17 | # GOPATH 18 | .go -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | # Build the sidecar-injector binary 2 | FROM golang:1.17 as builder 3 | 4 | WORKDIR /workspace 5 | # Copy the Go Modules manifests 6 | COPY go.mod go.mod 7 | COPY go.sum go.sum 8 | # cache deps before building and copying source so that we don't need to re-download as much 9 | # and so that source changes don't invalidate our downloaded layer 10 | RUN go mod download 11 | 12 | # Copy the go source 13 | COPY cmd/ cmd/ 14 | 15 | # Build 16 | RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o sidecar-injector ./cmd 17 | 18 | 19 | FROM alpine:latest 20 | 21 | # install curl for prestop script 22 | RUN apk --no-cache add curl 23 | 24 | WORKDIR / 25 | 26 | # install binary 27 | COPY --from=builder /workspace/sidecar-injector . 28 | 29 | # install the prestop script 30 | COPY ./prestop.sh . 31 | 32 | USER 65532:65532 33 | 34 | ENTRYPOINT ["/sidecar-injector"] 35 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Setting SHELL to bash allows bash commands to be executed by recipes. 2 | # This is a requirement for 'setup-envtest.sh' in the test target. 3 | # Options are set to exit when a recipe line exits non-zero or a piped command fails. 4 | SHELL = /usr/bin/env bash -o pipefail 5 | .SHELLFLAGS = -ec 6 | 7 | # Get the currently used golang install path (in GOPATH/bin, unless GOBIN is set) 8 | ifeq (,$(shell go env GOBIN)) 9 | GOBIN=$(shell go env GOPATH)/bin 10 | else 11 | GOBIN=$(shell go env GOBIN) 12 | endif 13 | 14 | # Tools for deploy 15 | KUBECTL?=kubectl 16 | PWD=$(shell pwd) 17 | KUSTOMIZE?=$(PWD)/$(PERMANENT_TMP_GOPATH)/bin/kustomize 18 | KUSTOMIZE_VERSION?=v3.5.4 19 | KUSTOMIZE_ARCHIVE_NAME?=kustomize_$(KUSTOMIZE_VERSION)_$(GOHOSTOS)_$(GOHOSTARCH).tar.gz 20 | kustomize_dir:=$(dir $(KUSTOMIZE)) 21 | 22 | IMAGE = quay.io/morvencao/sidecar-injector:latest 23 | 24 | all: build 25 | .PHONY: all 26 | 27 | ##@ General 28 | 29 | # The help target prints out all targets with their descriptions organized 30 | # beneath their categories. The categories are represented by '##@' and the 31 | # target descriptions by '##'. The awk commands is responsible for reading the 32 | # entire set of makefiles included in this invocation, looking for lines of the 33 | # file as xyz: ## something, and then pretty-format the target and help. Then, 34 | # if there's a line with ##@ something, that gets pretty-printed as a category. 35 | # More info on the usage of ANSI control characters for terminal formatting: 36 | # https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_parameters 37 | # More info on the awk command: 38 | # http://linuxcommand.org/lc3_adv_awk.php 39 | 40 | .PHONY: help 41 | help: ## Display this help. 42 | @awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf " \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST) 43 | 44 | ##@ Development 45 | 46 | .PHONY: fmt 47 | fmt: ## Run go fmt against code. 48 | go fmt ./... 49 | 50 | .PHONY: vet 51 | vet: ## Run go vet against code. 52 | go vet ./... 53 | 54 | .PHONY: test 55 | test: fmt vet ## Run tests. 56 | go test ./... -coverprofile cover.out 57 | 58 | ##@ Build 59 | 60 | .PHONY: build 61 | build: fmt vet ## Build binary. 62 | go build -o bin/sidecar-injector ./cmd/ 63 | 64 | .PHONY: docker-build 65 | docker-build: test ## Build docker image. 66 | docker build -t ${IMAGE} . 67 | 68 | .PHONY: docker-push 69 | docker-push: ## Push docker image. 70 | docker push ${IMAGE} 71 | 72 | ##@ Deployment 73 | 74 | deploy: kustomize 75 | cp deploy/kustomization.yaml deploy/kustomization.yaml.tmp 76 | cd deploy && $(KUSTOMIZE) edit set image sidecar-injector=$(IMAGE) 77 | $(KUSTOMIZE) build deploy | $(KUBECTL) apply -f - 78 | mv deploy/kustomization.yaml.tmp deploy/kustomization.yaml 79 | 80 | undeploy: kustomize 81 | $(KUSTOMIZE) build deploy | $(KUBECTL) delete --ignore-not-found -f - 82 | 83 | KUSTOMIZE = $(shell pwd)/bin/kustomize 84 | .PHONY: kustomize 85 | kustomize: ## Download kustomize locally if necessary. 86 | $(call go-get-tool,$(KUSTOMIZE),sigs.k8s.io/kustomize/kustomize/v3@v3.8.7) 87 | 88 | # go-get-tool will 'go get' any package $2 and install it to $1. 89 | PROJECT_DIR := $(shell dirname $(abspath $(lastword $(MAKEFILE_LIST)))) 90 | define go-get-tool 91 | @[ -f $(1) ] || { \ 92 | set -e ;\ 93 | TMP_DIR=$$(mktemp -d) ;\ 94 | cd $$TMP_DIR ;\ 95 | go mod init tmp ;\ 96 | echo "Downloading $(2)" ;\ 97 | GOBIN=$(PROJECT_DIR)/bin go get $(2) ;\ 98 | rm -rf $$TMP_DIR ;\ 99 | } 100 | endef 101 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # kube-sidecar-injector 2 | 3 | This repo is used for [a tutorial at Medium](https://medium.com/ibm-cloud/diving-into-kubernetes-mutatingadmissionwebhook-6ef3c5695f74) to create a Kubernetes [MutatingAdmissionWebhook](https://kubernetes.io/docs/admin/admission-controllers/#mutatingadmissionwebhook-beta-in-19) that injects a nginx sidecar container into pod prior to persistence of the object. 4 | 5 | ## Prerequisites 6 | 7 | - [git](https://git-scm.com/downloads) 8 | - [go](https://golang.org/dl/) version v1.17+ 9 | - [docker](https://docs.docker.com/install/) version 19.03+ 10 | - [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) version v1.19+ 11 | - Access to a Kubernetes v1.19+ cluster with the `admissionregistration.k8s.io/v1` API enabled. Verify that by the following command: 12 | 13 | ``` 14 | kubectl api-versions | grep admissionregistration.k8s.io 15 | ``` 16 | The result should be: 17 | ``` 18 | admissionregistration.k8s.io/v1 19 | admissionregistration.k8s.io/v1beta1 20 | ``` 21 | 22 | > Note: In addition, the `MutatingAdmissionWebhook` and `ValidatingAdmissionWebhook` admission controllers should be added and listed in the correct order in the admission-control flag of kube-apiserver. 23 | 24 | ## Build and Deploy 25 | 26 | 1. Build and push docker image: 27 | 28 | ```bash 29 | make docker-build docker-push IMAGE=quay.io//sidecar-injector:latest 30 | ``` 31 | 32 | 2. Deploy the kube-sidecar-injector to kubernetes cluster: 33 | 34 | ```bash 35 | make deploy IMAGE=quay.io//sidecar-injector:latest 36 | ``` 37 | 38 | 3. Verify the kube-sidecar-injector is up and running: 39 | 40 | ```bash 41 | # kubectl -n sidecar-injector get pod 42 | # kubectl -n sidecar-injector get pod 43 | NAME READY STATUS RESTARTS AGE 44 | sidecar-injector-7c8bc5f4c9-28c84 1/1 Running 0 30s 45 | ``` 46 | 47 | ## How to use 48 | 49 | 1. Create a new namespace `test-ns` and label it with `sidecar-injector=enabled`: 50 | 51 | ``` 52 | # kubectl create ns test-ns 53 | # kubectl label namespace test-ns sidecar-injection=enabled 54 | # kubectl get namespace -L sidecar-injection 55 | NAME STATUS AGE SIDECAR-INJECTION 56 | default Active 26m 57 | test-ns Active 13s enabled 58 | kube-public Active 26m 59 | kube-system Active 26m 60 | sidecar-injector Active 17m 61 | ``` 62 | 63 | 2. Deploy an app in Kubernetes cluster, take `alpine` app as an example 64 | 65 | ```bash 66 | kubectl -n test-ns run alpine \ 67 | --image=alpine \ 68 | --restart=Never \ 69 | --command -- sleep infinity 70 | ``` 71 | 72 | 3. Verify sidecar container is injected: 73 | 74 | ``` 75 | # kubectl -n test-ns get pod 76 | NAME READY STATUS RESTARTS AGE 77 | alpine 2/2 Running 0 10s 78 | # kubectl -n test-ns get pod alpine -o jsonpath="{.spec.containers[*].name}" 79 | alpine sidecar-nginx 80 | ``` 81 | 82 | ## Troubleshooting 83 | 84 | Sometimes you may find that pod is injected with sidecar container as expected, check the following items: 85 | 86 | 1. The sidecar-injector pod is in running state and no error logs. 87 | 2. The namespace in which application pod is deployed has the correct labels(`sidecar-injector=enabled`) as configured in `mutatingwebhookconfiguration`. 88 | 3. Check if the application pod has annotation `sidecar-injector-webhook.morven.me/inject:"yes"`. 89 | -------------------------------------------------------------------------------- /bin/kustomize: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/morvencao/kube-sidecar-injector/4e010f4cdee8baf3cd3f3f59ec9b95e5db9b9f01/bin/kustomize -------------------------------------------------------------------------------- /cmd/cert.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bytes" 5 | "crypto/rand" 6 | "crypto/rsa" 7 | "crypto/x509" 8 | "crypto/x509/pkix" 9 | "encoding/pem" 10 | "math/big" 11 | "time" 12 | ) 13 | 14 | // generateCert generate a self-signed CA for given organization 15 | // and sign certificate with the CA for given common name and dns names 16 | // it resurns the CA, certificate and private key in PEM format 17 | func generateCert(orgs, dnsNames []string, commonName string) (*bytes.Buffer, *bytes.Buffer, *bytes.Buffer, error) { 18 | // init CA config 19 | ca := &x509.Certificate{ 20 | SerialNumber: big.NewInt(2022), 21 | Subject: pkix.Name{Organization: orgs}, 22 | NotBefore: time.Now(), 23 | NotAfter: time.Now().AddDate(1, 0, 0), // expired in 1 year 24 | IsCA: true, 25 | ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth}, 26 | KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign, 27 | BasicConstraintsValid: true, 28 | } 29 | 30 | // generate private key for CA 31 | caPrivateKey, err := rsa.GenerateKey(rand.Reader, 4096) 32 | if err != nil { 33 | return nil, nil, nil, err 34 | } 35 | 36 | // create the CA certificate 37 | caBytes, err := x509.CreateCertificate(rand.Reader, ca, ca, &caPrivateKey.PublicKey, caPrivateKey) 38 | if err != nil { 39 | return nil, nil, nil, err 40 | } 41 | 42 | // CA certificate with PEM encoded 43 | caPEM := new(bytes.Buffer) 44 | _ = pem.Encode(caPEM, &pem.Block{ 45 | Type: "CERTIFICATE", 46 | Bytes: caBytes, 47 | }) 48 | 49 | // new certificate config 50 | newCert := &x509.Certificate{ 51 | DNSNames: dnsNames, 52 | SerialNumber: big.NewInt(1024), 53 | Subject: pkix.Name{ 54 | CommonName: commonName, 55 | Organization: orgs, 56 | }, 57 | NotBefore: time.Now(), 58 | NotAfter: time.Now().AddDate(1, 0, 0), // expired in 1 year 59 | ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth}, 60 | KeyUsage: x509.KeyUsageDigitalSignature, 61 | } 62 | 63 | // generate new private key 64 | newPrivateKey, err := rsa.GenerateKey(rand.Reader, 4096) 65 | if err != nil { 66 | return nil, nil, nil, err 67 | } 68 | 69 | // sign the new certificate 70 | newCertBytes, err := x509.CreateCertificate(rand.Reader, newCert, ca, &newPrivateKey.PublicKey, caPrivateKey) 71 | if err != nil { 72 | return nil, nil, nil, err 73 | } 74 | 75 | // new certificate with PEM encoded 76 | newCertPEM := new(bytes.Buffer) 77 | _ = pem.Encode(newCertPEM, &pem.Block{ 78 | Type: "CERTIFICATE", 79 | Bytes: newCertBytes, 80 | }) 81 | 82 | // new private key with PEM encoded 83 | newPrivateKeyPEM := new(bytes.Buffer) 84 | _ = pem.Encode(newPrivateKeyPEM, &pem.Block{ 85 | Type: "RSA PRIVATE KEY", 86 | Bytes: x509.MarshalPKCS1PrivateKey(newPrivateKey), 87 | }) 88 | 89 | return caPEM, newCertPEM, newPrivateKeyPEM, nil 90 | } 91 | -------------------------------------------------------------------------------- /cmd/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "context" 5 | "crypto/tls" 6 | "flag" 7 | "fmt" 8 | "log" 9 | "net/http" 10 | "os" 11 | "os/signal" 12 | "syscall" 13 | ) 14 | 15 | var ( 16 | infoLogger *log.Logger 17 | warningLogger *log.Logger 18 | errorLogger *log.Logger 19 | ) 20 | 21 | var ( 22 | port int 23 | sidecarConfigFile string 24 | webhookNamespace, webhookServiceName string 25 | ) 26 | 27 | func init() { 28 | // init loggers 29 | infoLogger = log.New(os.Stderr, "INFO: ", log.Ldate|log.Ltime|log.Lshortfile) 30 | warningLogger = log.New(os.Stderr, "WARNING: ", log.Ldate|log.Ltime|log.Lshortfile) 31 | errorLogger = log.New(os.Stderr, "ERROR: ", log.Ldate|log.Ltime|log.Lshortfile) 32 | 33 | // webhook server running namespace 34 | webhookNamespace = os.Getenv("POD_NAMESPACE") 35 | } 36 | 37 | func main() { 38 | // init command flags 39 | flag.IntVar(&port, "port", 8443, "Webhook server port.") 40 | flag.StringVar(&webhookServiceName, "service-name", "sidecar-injector", "Webhook service name.") 41 | flag.StringVar(&sidecarConfigFile, "sidecar-config-file", "/etc/webhook/config/sidecarconfig.yaml", "Sidecar injector configuration file.") 42 | // flag.StringVar(&certFile, "tlsCertFile", "/etc/webhook/certs/cert.pem", "x509 Certificate file.") 43 | // flag.StringVar(&keyFile, "tlsKeyFile", "/etc/webhook/certs/key.pem", "x509 private key file.") 44 | flag.Parse() 45 | 46 | dnsNames := []string{ 47 | webhookServiceName, 48 | webhookServiceName + "." + webhookNamespace, 49 | webhookServiceName + "." + webhookNamespace + ".svc", 50 | } 51 | commonName := webhookServiceName + "." + webhookNamespace + ".svc" 52 | 53 | org := "morven.me" 54 | caPEM, certPEM, certKeyPEM, err := generateCert([]string{org}, dnsNames, commonName) 55 | if err != nil { 56 | errorLogger.Fatalf("Failed to generate ca and certificate key pair: %v", err) 57 | } 58 | 59 | pair, err := tls.X509KeyPair(certPEM.Bytes(), certKeyPEM.Bytes()) 60 | if err != nil { 61 | errorLogger.Fatalf("Failed to load certificate key pair: %v", err) 62 | } 63 | 64 | sidecarConfig, err := loadConfig(sidecarConfigFile) 65 | if err != nil { 66 | errorLogger.Fatalf("Failed to load configuration: %v", err) 67 | } 68 | 69 | // create or update the mutatingwebhookconfiguration 70 | err = createOrUpdateMutatingWebhookConfiguration(caPEM, webhookServiceName, webhookNamespace) 71 | if err != nil { 72 | errorLogger.Fatalf("Failed to create or update the mutating webhook configuration: %v", err) 73 | } 74 | 75 | whsvr := &WebhookServer{ 76 | sidecarConfig: sidecarConfig, 77 | server: &http.Server{ 78 | Addr: fmt.Sprintf(":%v", port), 79 | TLSConfig: &tls.Config{Certificates: []tls.Certificate{pair}}, 80 | }, 81 | } 82 | 83 | // define http server and server handler 84 | mux := http.NewServeMux() 85 | mux.HandleFunc(webhookInjectPath, whsvr.serve) 86 | whsvr.server.Handler = mux 87 | 88 | // start webhook server in new rountine 89 | go func() { 90 | if err := whsvr.server.ListenAndServeTLS("", ""); err != nil { 91 | errorLogger.Fatalf("Failed to listen and serve webhook server: %v", err) 92 | } 93 | }() 94 | 95 | // listening OS shutdown singal 96 | signalChan := make(chan os.Signal, 1) 97 | signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM) 98 | <-signalChan 99 | 100 | infoLogger.Printf("Got OS shutdown signal, shutting down webhook server gracefully...") 101 | whsvr.server.Shutdown(context.Background()) 102 | } 103 | -------------------------------------------------------------------------------- /cmd/webhook.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "crypto/sha256" 5 | "encoding/json" 6 | "fmt" 7 | "io/ioutil" 8 | "net/http" 9 | "strings" 10 | 11 | "gopkg.in/yaml.v2" 12 | admissionv1 "k8s.io/api/admission/v1" 13 | corev1 "k8s.io/api/core/v1" 14 | metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" 15 | "k8s.io/apimachinery/pkg/runtime" 16 | "k8s.io/apimachinery/pkg/runtime/serializer" 17 | ) 18 | 19 | var ( 20 | runtimeScheme = runtime.NewScheme() 21 | codecs = serializer.NewCodecFactory(runtimeScheme) 22 | deserializer = codecs.UniversalDeserializer() 23 | ) 24 | 25 | var ignoredNamespaces = []string{ 26 | metav1.NamespaceSystem, 27 | metav1.NamespacePublic, 28 | } 29 | 30 | const ( 31 | admissionWebhookAnnotationInjectKey = "sidecar-injector-webhook.morven.me/inject" 32 | admissionWebhookAnnotationStatusKey = "sidecar-injector-webhook.morven.me/status" 33 | ) 34 | 35 | type WebhookServer struct { 36 | sidecarConfig *Config 37 | server *http.Server 38 | } 39 | 40 | // Webhook Server parameters 41 | type WhSvrParameters struct { 42 | port int // webhook server port 43 | certFile string // path to the x509 certificate for https 44 | keyFile string // path to the x509 private key matching `CertFile` 45 | sidecarCfgFile string // path to sidecar injector configuration file 46 | } 47 | 48 | type Config struct { 49 | Containers []corev1.Container `yaml:"containers"` 50 | Volumes []corev1.Volume `yaml:"volumes"` 51 | } 52 | 53 | type patchOperation struct { 54 | Op string `json:"op"` 55 | Path string `json:"path"` 56 | Value interface{} `json:"value,omitempty"` 57 | } 58 | 59 | func loadConfig(configFile string) (*Config, error) { 60 | data, err := ioutil.ReadFile(configFile) 61 | if err != nil { 62 | return nil, err 63 | } 64 | infoLogger.Printf("New configuration: sha256sum %x", sha256.Sum256(data)) 65 | 66 | var cfg Config 67 | if err := yaml.Unmarshal(data, &cfg); err != nil { 68 | return nil, err 69 | } 70 | 71 | return &cfg, nil 72 | } 73 | 74 | // Check whether the target resoured need to be mutated 75 | func mutationRequired(ignoredList []string, metadata *metav1.ObjectMeta) bool { 76 | // skip special kubernete system namespaces 77 | for _, namespace := range ignoredList { 78 | if metadata.Namespace == namespace { 79 | infoLogger.Printf("Skip mutation for %v for it's in special namespace:%v", metadata.Name, metadata.Namespace) 80 | return false 81 | } 82 | } 83 | 84 | annotations := metadata.GetAnnotations() 85 | if annotations == nil { 86 | annotations = map[string]string{} 87 | } 88 | 89 | status := annotations[admissionWebhookAnnotationStatusKey] 90 | 91 | // determine whether to perform mutation based on annotation for the target resource 92 | var required bool 93 | if strings.ToLower(status) == "injected" { 94 | required = false 95 | } else { 96 | switch strings.ToLower(annotations[admissionWebhookAnnotationInjectKey]) { 97 | default: 98 | required = true 99 | case "n", "not", "false", "off": 100 | required = false 101 | } 102 | } 103 | 104 | infoLogger.Printf("Mutation policy for %v/%v: status: %q required:%v", metadata.Namespace, metadata.Name, status, required) 105 | return required 106 | } 107 | 108 | func addContainer(target, added []corev1.Container, basePath string) (patch []patchOperation) { 109 | first := len(target) == 0 110 | var value interface{} 111 | for _, add := range added { 112 | value = add 113 | path := basePath 114 | if first { 115 | first = false 116 | value = []corev1.Container{add} 117 | } else { 118 | path = path + "/-" 119 | } 120 | patch = append(patch, patchOperation{ 121 | Op: "add", 122 | Path: path, 123 | Value: value, 124 | }) 125 | } 126 | return patch 127 | } 128 | 129 | func addVolume(target, added []corev1.Volume, basePath string) (patch []patchOperation) { 130 | first := len(target) == 0 131 | var value interface{} 132 | for _, add := range added { 133 | value = add 134 | path := basePath 135 | if first { 136 | first = false 137 | value = []corev1.Volume{add} 138 | } else { 139 | path = path + "/-" 140 | } 141 | patch = append(patch, patchOperation{ 142 | Op: "add", 143 | Path: path, 144 | Value: value, 145 | }) 146 | } 147 | return patch 148 | } 149 | 150 | func updateAnnotation(target map[string]string, added map[string]string) (patch []patchOperation) { 151 | for key, value := range added { 152 | if target == nil || target[key] == "" { 153 | target = map[string]string{} 154 | patch = append(patch, patchOperation{ 155 | Op: "add", 156 | Path: "/metadata/annotations", 157 | Value: map[string]string{ 158 | key: value, 159 | }, 160 | }) 161 | } else { 162 | patch = append(patch, patchOperation{ 163 | Op: "replace", 164 | Path: "/metadata/annotations/" + key, 165 | Value: value, 166 | }) 167 | } 168 | } 169 | return patch 170 | } 171 | 172 | // create mutation patch for resoures 173 | func createPatch(pod *corev1.Pod, sidecarConfig *Config, annotations map[string]string) ([]byte, error) { 174 | var patch []patchOperation 175 | 176 | patch = append(patch, addContainer(pod.Spec.Containers, sidecarConfig.Containers, "/spec/containers")...) 177 | patch = append(patch, addVolume(pod.Spec.Volumes, sidecarConfig.Volumes, "/spec/volumes")...) 178 | patch = append(patch, updateAnnotation(pod.Annotations, annotations)...) 179 | 180 | return json.Marshal(patch) 181 | } 182 | 183 | // main mutation process 184 | func (whsvr *WebhookServer) mutate(ar *admissionv1.AdmissionReview) *admissionv1.AdmissionResponse { 185 | req := ar.Request 186 | var pod corev1.Pod 187 | if err := json.Unmarshal(req.Object.Raw, &pod); err != nil { 188 | warningLogger.Printf("Could not unmarshal raw object: %v", err) 189 | return &admissionv1.AdmissionResponse{ 190 | Result: &metav1.Status{ 191 | Message: err.Error(), 192 | }, 193 | } 194 | } 195 | 196 | infoLogger.Printf("AdmissionReview for Kind=%v, Namespace=%v Name=%v (%v) UID=%v patchOperation=%v UserInfo=%v", 197 | req.Kind, req.Namespace, req.Name, pod.Name, req.UID, req.Operation, req.UserInfo) 198 | 199 | // determine whether to perform mutation 200 | if !mutationRequired(ignoredNamespaces, &pod.ObjectMeta) { 201 | infoLogger.Printf("Skipping mutation for %s/%s due to policy check", pod.Namespace, pod.Name) 202 | return &admissionv1.AdmissionResponse{ 203 | Allowed: true, 204 | } 205 | } 206 | 207 | annotations := map[string]string{admissionWebhookAnnotationStatusKey: "injected"} 208 | patchBytes, err := createPatch(&pod, whsvr.sidecarConfig, annotations) 209 | if err != nil { 210 | return &admissionv1.AdmissionResponse{ 211 | Result: &metav1.Status{ 212 | Message: err.Error(), 213 | }, 214 | } 215 | } 216 | 217 | infoLogger.Printf("AdmissionResponse: patch=%v\n", string(patchBytes)) 218 | return &admissionv1.AdmissionResponse{ 219 | Allowed: true, 220 | Patch: patchBytes, 221 | PatchType: func() *admissionv1.PatchType { 222 | pt := admissionv1.PatchTypeJSONPatch 223 | return &pt 224 | }(), 225 | } 226 | } 227 | 228 | // Serve method for webhook server 229 | func (whsvr *WebhookServer) serve(w http.ResponseWriter, r *http.Request) { 230 | var body []byte 231 | if r.Body != nil { 232 | if data, err := ioutil.ReadAll(r.Body); err == nil { 233 | body = data 234 | } 235 | } 236 | if len(body) == 0 { 237 | warningLogger.Println("empty body") 238 | http.Error(w, "empty body", http.StatusBadRequest) 239 | return 240 | } 241 | 242 | // verify the content type is accurate 243 | contentType := r.Header.Get("Content-Type") 244 | if contentType != "application/json" { 245 | warningLogger.Printf("Content-Type=%s, expect application/json", contentType) 246 | http.Error(w, "invalid Content-Type, expect `application/json`", http.StatusUnsupportedMediaType) 247 | return 248 | } 249 | 250 | var admissionResponse *admissionv1.AdmissionResponse 251 | ar := admissionv1.AdmissionReview{} 252 | if _, _, err := deserializer.Decode(body, nil, &ar); err != nil { 253 | warningLogger.Printf("Can't decode body: %v", err) 254 | admissionResponse = &admissionv1.AdmissionResponse{ 255 | Result: &metav1.Status{ 256 | Message: err.Error(), 257 | }, 258 | } 259 | } else { 260 | admissionResponse = whsvr.mutate(&ar) 261 | } 262 | 263 | admissionReview := admissionv1.AdmissionReview{ 264 | TypeMeta: metav1.TypeMeta{ 265 | APIVersion: "admission.k8s.io/v1", 266 | Kind: "AdmissionReview", 267 | }, 268 | } 269 | if admissionResponse != nil { 270 | admissionReview.Response = admissionResponse 271 | if ar.Request != nil { 272 | admissionReview.Response.UID = ar.Request.UID 273 | } 274 | } 275 | 276 | resp, err := json.Marshal(admissionReview) 277 | if err != nil { 278 | warningLogger.Printf("Can't encode response: %v", err) 279 | http.Error(w, fmt.Sprintf("could not encode response: %v", err), http.StatusInternalServerError) 280 | } 281 | infoLogger.Printf("Ready to write reponse ...") 282 | if _, err := w.Write(resp); err != nil { 283 | warningLogger.Printf("Can't write response: %v", err) 284 | http.Error(w, fmt.Sprintf("could not write response: %v", err), http.StatusInternalServerError) 285 | } 286 | } 287 | -------------------------------------------------------------------------------- /cmd/webhookconfig.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bytes" 5 | "context" 6 | "os" 7 | "reflect" 8 | 9 | admissionregistrationv1 "k8s.io/api/admissionregistration/v1" 10 | apierrors "k8s.io/apimachinery/pkg/api/errors" 11 | metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" 12 | "k8s.io/client-go/kubernetes" 13 | "k8s.io/client-go/tools/clientcmd" 14 | ) 15 | 16 | var ( 17 | webhookConfigName = "sidecar-injector-webhook" 18 | webhookInjectPath = "/inject" 19 | ) 20 | 21 | func createOrUpdateMutatingWebhookConfiguration(caPEM *bytes.Buffer, webhookService, webhookNamespace string) error { 22 | infoLogger.Println("Initializing the kube client...") 23 | 24 | kubeconfig := os.Getenv("KUBECONFIG") 25 | config, err := clientcmd.BuildConfigFromFlags("", kubeconfig) 26 | if err != nil { 27 | return err 28 | } 29 | clientset, err := kubernetes.NewForConfig(config) 30 | if err != nil { 31 | return err 32 | } 33 | mutatingWebhookConfigV1Client := clientset.AdmissionregistrationV1() 34 | 35 | infoLogger.Printf("Creating or updating the mutatingwebhookconfiguration: %s", webhookConfigName) 36 | fail := admissionregistrationv1.Fail 37 | sideEffect := admissionregistrationv1.SideEffectClassNone 38 | mutatingWebhookConfig := &admissionregistrationv1.MutatingWebhookConfiguration{ 39 | ObjectMeta: metav1.ObjectMeta{ 40 | Name: webhookConfigName, 41 | }, 42 | Webhooks: []admissionregistrationv1.MutatingWebhook{{ 43 | Name: "sidecar-injector.morven.me", 44 | AdmissionReviewVersions: []string{"v1", "v1beta1"}, 45 | SideEffects: &sideEffect, 46 | ClientConfig: admissionregistrationv1.WebhookClientConfig{ 47 | CABundle: caPEM.Bytes(), // self-generated CA for the webhook 48 | Service: &admissionregistrationv1.ServiceReference{ 49 | Name: webhookService, 50 | Namespace: webhookNamespace, 51 | Path: &webhookInjectPath, 52 | }, 53 | }, 54 | Rules: []admissionregistrationv1.RuleWithOperations{ 55 | { 56 | Operations: []admissionregistrationv1.OperationType{ 57 | admissionregistrationv1.Create, 58 | admissionregistrationv1.Update, 59 | }, 60 | Rule: admissionregistrationv1.Rule{ 61 | APIGroups: []string{""}, 62 | APIVersions: []string{"v1"}, 63 | Resources: []string{"pods"}, 64 | }, 65 | }, 66 | }, 67 | NamespaceSelector: &metav1.LabelSelector{ 68 | MatchLabels: map[string]string{ 69 | "sidecar-injection": "enabled", 70 | }, 71 | }, 72 | FailurePolicy: &fail, 73 | }}, 74 | } 75 | 76 | foundWebhookConfig, err := mutatingWebhookConfigV1Client.MutatingWebhookConfigurations().Get(context.TODO(), webhookConfigName, metav1.GetOptions{}) 77 | if err != nil && apierrors.IsNotFound(err) { 78 | if _, err := mutatingWebhookConfigV1Client.MutatingWebhookConfigurations().Create(context.TODO(), mutatingWebhookConfig, metav1.CreateOptions{}); err != nil { 79 | warningLogger.Printf("Failed to create the mutatingwebhookconfiguration: %s", webhookConfigName) 80 | return err 81 | } 82 | infoLogger.Printf("Created mutatingwebhookconfiguration: %s", webhookConfigName) 83 | } else if err != nil { 84 | warningLogger.Printf("Failed to check the mutatingwebhookconfiguration: %s", webhookConfigName) 85 | return err 86 | } else { 87 | // there is an existing mutatingWebhookConfiguration 88 | if len(foundWebhookConfig.Webhooks) != len(mutatingWebhookConfig.Webhooks) || 89 | !(foundWebhookConfig.Webhooks[0].Name == mutatingWebhookConfig.Webhooks[0].Name && 90 | reflect.DeepEqual(foundWebhookConfig.Webhooks[0].AdmissionReviewVersions, mutatingWebhookConfig.Webhooks[0].AdmissionReviewVersions) && 91 | reflect.DeepEqual(foundWebhookConfig.Webhooks[0].SideEffects, mutatingWebhookConfig.Webhooks[0].SideEffects) && 92 | reflect.DeepEqual(foundWebhookConfig.Webhooks[0].FailurePolicy, mutatingWebhookConfig.Webhooks[0].FailurePolicy) && 93 | reflect.DeepEqual(foundWebhookConfig.Webhooks[0].Rules, mutatingWebhookConfig.Webhooks[0].Rules) && 94 | reflect.DeepEqual(foundWebhookConfig.Webhooks[0].NamespaceSelector, mutatingWebhookConfig.Webhooks[0].NamespaceSelector) && 95 | reflect.DeepEqual(foundWebhookConfig.Webhooks[0].ClientConfig.CABundle, mutatingWebhookConfig.Webhooks[0].ClientConfig.CABundle) && 96 | reflect.DeepEqual(foundWebhookConfig.Webhooks[0].ClientConfig.Service, mutatingWebhookConfig.Webhooks[0].ClientConfig.Service)) { 97 | mutatingWebhookConfig.ObjectMeta.ResourceVersion = foundWebhookConfig.ObjectMeta.ResourceVersion 98 | if _, err := mutatingWebhookConfigV1Client.MutatingWebhookConfigurations().Update(context.TODO(), mutatingWebhookConfig, metav1.UpdateOptions{}); err != nil { 99 | warningLogger.Printf("Failed to update the mutatingwebhookconfiguration: %s", webhookConfigName) 100 | return err 101 | } 102 | infoLogger.Printf("Updated the mutatingwebhookconfiguration: %s", webhookConfigName) 103 | } 104 | infoLogger.Printf("The mutatingwebhookconfiguration: %s already exists and has no change", webhookConfigName) 105 | } 106 | 107 | return nil 108 | } 109 | -------------------------------------------------------------------------------- /deploy/clusterrole.yaml: -------------------------------------------------------------------------------- 1 | kind: ClusterRole 2 | apiVersion: rbac.authorization.k8s.io/v1 3 | metadata: 4 | name: sidecar-injector 5 | labels: 6 | app: sidecar-injector 7 | rules: 8 | - apiGroups: ["admissionregistration.k8s.io"] 9 | resources: ["mutatingwebhookconfigurations"] 10 | verbs: ["create", "get", "delete", "list", "patch", "update", "watch"] 11 | -------------------------------------------------------------------------------- /deploy/clusterrolebinding.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: rbac.authorization.k8s.io/v1 2 | kind: ClusterRoleBinding 3 | metadata: 4 | name: sidecar-injector 5 | labels: 6 | app: sidecar-injector 7 | roleRef: 8 | apiGroup: rbac.authorization.k8s.io 9 | kind: ClusterRole 10 | name: sidecar-injector 11 | subjects: 12 | - kind: ServiceAccount 13 | name: sidecar-injector 14 | namespace: sidecar-injector 15 | -------------------------------------------------------------------------------- /deploy/configmap.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: sidecar-injector 5 | labels: 6 | app: sidecar-injector 7 | data: 8 | sidecarconfig.yaml: | 9 | containers: 10 | - name: sidecar-nginx 11 | image: nginx:1.12.2 12 | imagePullPolicy: IfNotPresent 13 | volumeMounts: 14 | - name: nginx-conf 15 | mountPath: /etc/nginx 16 | volumes: 17 | - name: nginx-conf 18 | configMap: 19 | name: nginx-configmap 20 | -------------------------------------------------------------------------------- /deploy/deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: sidecar-injector 5 | labels: 6 | app: sidecar-injector 7 | spec: 8 | replicas: 1 9 | selector: 10 | matchLabels: 11 | app: sidecar-injector 12 | template: 13 | metadata: 14 | labels: 15 | app: sidecar-injector 16 | spec: 17 | serviceAccountName: sidecar-injector 18 | containers: 19 | - name: sidecar-injector 20 | image: sidecar-injector 21 | imagePullPolicy: Always 22 | args: 23 | - -service-name=sidecar-injector 24 | - -sidecar-config-file=/etc/webhook/config/sidecarconfig.yaml 25 | env: 26 | - name: POD_NAMESPACE 27 | valueFrom: 28 | fieldRef: 29 | fieldPath: metadata.namespace 30 | lifecycle: 31 | preStop: 32 | exec: 33 | command: ["/bin/sh", "-c", "/prestop.sh"] 34 | volumeMounts: 35 | - name: webhook-config 36 | mountPath: /etc/webhook/config 37 | volumes: 38 | - name: webhook-config 39 | configMap: 40 | name: sidecar-injector 41 | -------------------------------------------------------------------------------- /deploy/kustomization.yaml: -------------------------------------------------------------------------------- 1 | namespace: sidecar-injector 2 | 3 | resources: 4 | - namespace.yaml 5 | - clusterrole.yaml 6 | - clusterrolebinding.yaml 7 | - deployment.yaml 8 | - service.yaml 9 | - serviceaccount.yaml 10 | - configmap.yaml 11 | - nginx-configmap.yaml 12 | 13 | images: 14 | - name: sidecar-injector 15 | newName: quay.io/morvencao/sidecar-injector 16 | newTag: latest 17 | apiVersion: kustomize.config.k8s.io/v1beta1 18 | kind: Kustomization 19 | -------------------------------------------------------------------------------- /deploy/namespace.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Namespace 3 | metadata: 4 | name: sidecar-injector 5 | 6 | -------------------------------------------------------------------------------- /deploy/nginx-configmap.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: nginx-configmap 5 | data: 6 | nginx.conf: | 7 | worker_processes 1; 8 | 9 | events { 10 | worker_connections 1024; 11 | } 12 | 13 | http { 14 | default_type application/octet-stream; 15 | 16 | sendfile on; 17 | keepalive_timeout 65; 18 | 19 | server { 20 | listen 80; 21 | server_name localhost; 22 | 23 | location / { 24 | root html; 25 | index index.html index.htm; 26 | } 27 | 28 | error_page 500 502 503 504 /50x.html; 29 | location = /50x.html { 30 | root html; 31 | } 32 | } 33 | } 34 | -------------------------------------------------------------------------------- /deploy/service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: sidecar-injector 5 | labels: 6 | app: sidecar-injector 7 | spec: 8 | ports: 9 | - port: 443 10 | targetPort: 8443 11 | selector: 12 | app: sidecar-injector 13 | -------------------------------------------------------------------------------- /deploy/serviceaccount.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ServiceAccount 3 | metadata: 4 | name: sidecar-injector 5 | labels: 6 | app: sidecar-injector 7 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/morvencao/kube-sidecar-injector 2 | 3 | go 1.17 4 | 5 | require ( 6 | gopkg.in/yaml.v2 v2.4.0 7 | k8s.io/api v0.19.15 8 | k8s.io/apimachinery v0.19.15 9 | k8s.io/client-go v0.19.15 10 | ) 11 | 12 | require ( 13 | github.com/davecgh/go-spew v1.1.1 // indirect 14 | github.com/go-logr/logr v0.2.0 // indirect 15 | github.com/gogo/protobuf v1.3.2 // indirect 16 | github.com/golang/protobuf v1.4.2 // indirect 17 | github.com/google/gofuzz v1.1.0 // indirect 18 | github.com/googleapis/gnostic v0.4.1 // indirect 19 | github.com/imdario/mergo v0.3.5 // indirect 20 | github.com/json-iterator/go v1.1.10 // indirect 21 | github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect 22 | github.com/modern-go/reflect2 v1.0.1 // indirect 23 | github.com/spf13/pflag v1.0.5 // indirect 24 | golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 // indirect 25 | golang.org/x/net v0.0.0-20201110031124-69a78807bb2b // indirect 26 | golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6 // indirect 27 | golang.org/x/sys v0.0.0-20201112073958-5cba982894dd // indirect 28 | golang.org/x/text v0.3.3 // indirect 29 | golang.org/x/time v0.0.0-20191024005414-555d28b269f0 // indirect 30 | google.golang.org/appengine v1.6.5 // indirect 31 | google.golang.org/protobuf v1.24.0 // indirect 32 | gopkg.in/inf.v0 v0.9.1 // indirect 33 | k8s.io/klog/v2 v2.2.0 // indirect 34 | k8s.io/utils v0.0.0-20200729134348-d5654de09c73 // indirect 35 | sigs.k8s.io/structured-merge-diff/v4 v4.1.2 // indirect 36 | sigs.k8s.io/yaml v1.2.0 // indirect 37 | ) 38 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= 2 | cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= 3 | cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU= 4 | cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU= 5 | cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY= 6 | cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc= 7 | cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0= 8 | cloud.google.com/go v0.51.0/go.mod h1:hWtGJ6gnXH+KgDv+V0zFGDvpi07n3z8ZNj3T1RW0Gcw= 9 | cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o= 10 | cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE= 11 | cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I= 12 | cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw= 13 | dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= 14 | github.com/Azure/go-autorest/autorest v0.9.0/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI= 15 | github.com/Azure/go-autorest/autorest v0.9.6/go.mod h1:/FALq9T/kS7b5J5qsQ+RSTUdAmGFqi0vUdVNNx8q630= 16 | github.com/Azure/go-autorest/autorest/adal v0.5.0/go.mod h1:8Z9fGy2MpX0PvDjB1pEgQTmVqjGhiHBW7RJJEciWzS0= 17 | github.com/Azure/go-autorest/autorest/adal v0.8.2/go.mod h1:ZjhuQClTqx435SRJ2iMlOxPYt3d2C/T/7TiQCVZSn3Q= 18 | github.com/Azure/go-autorest/autorest/date v0.1.0/go.mod h1:plvfp3oPSKwf2DNjlBjWF/7vwR+cUD/ELuzDCXwHUVA= 19 | github.com/Azure/go-autorest/autorest/date v0.2.0/go.mod h1:vcORJHLJEh643/Ioh9+vPmf1Ij9AEBM5FuBIXLmIy0g= 20 | github.com/Azure/go-autorest/autorest/mocks v0.1.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0= 21 | github.com/Azure/go-autorest/autorest/mocks v0.2.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0= 22 | github.com/Azure/go-autorest/autorest/mocks v0.3.0/go.mod h1:a8FDP3DYzQ4RYfVAxAN3SVSiiO77gL2j2ronKKP0syM= 23 | github.com/Azure/go-autorest/logger v0.1.0/go.mod h1:oExouG+K6PryycPJfVSxi/koC6LSNgds39diKLz7Vrc= 24 | github.com/Azure/go-autorest/tracing v0.5.0/go.mod h1:r/s2XiOKccPW3HrqB+W0TQzfbtp2fGCgRFtBroKn4Dk= 25 | github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= 26 | github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= 27 | github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ= 28 | github.com/PuerkitoBio/purell v1.0.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= 29 | github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= 30 | github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= 31 | github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= 32 | github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= 33 | github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= 34 | github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= 35 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 36 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 37 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 38 | github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ= 39 | github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM= 40 | github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE= 41 | github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc= 42 | github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= 43 | github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= 44 | github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= 45 | github.com/evanphx/json-patch v4.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= 46 | github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= 47 | github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= 48 | github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= 49 | github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= 50 | github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas= 51 | github.com/go-logr/logr v0.2.0 h1:QvGt2nLcHH0WK9orKa+ppBPAxREcH364nPUedEpK0TY= 52 | github.com/go-logr/logr v0.2.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU= 53 | github.com/go-openapi/jsonpointer v0.0.0-20160704185906-46af16f9f7b1/go.mod h1:+35s3my2LFTysnkMfxsJBAMHj/DoqoB9knIWoYG/Vk0= 54 | github.com/go-openapi/jsonreference v0.0.0-20160704190145-13c6e3589ad9/go.mod h1:W3Z9FmVs9qj+KR4zFKmDPGiLdk1D9Rlm7cyMvf57TTg= 55 | github.com/go-openapi/spec v0.0.0-20160808142527-6aced65f8501/go.mod h1:J8+jY1nAiCcj+friV/PDoE1/3eeccG9LYBs0tYvLOWc= 56 | github.com/go-openapi/swag v0.0.0-20160704191624-1d0bd113de87/go.mod h1:DXUve3Dpr1UfpPtxFw+EFuQ41HhCWZfha5jSVRG7C7I= 57 | github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= 58 | github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= 59 | github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= 60 | github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= 61 | github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= 62 | github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= 63 | github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= 64 | github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y= 65 | github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 66 | github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 67 | github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 68 | github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= 69 | github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= 70 | github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= 71 | github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= 72 | github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= 73 | github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= 74 | github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= 75 | github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0= 76 | github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= 77 | github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= 78 | github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= 79 | github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= 80 | github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 81 | github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 82 | github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4= 83 | github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 84 | github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= 85 | github.com/google/gofuzz v1.1.0 h1:Hsa8mG0dQ46ij8Sl2AYJDUv1oA9/d6Vk+3LG99Oe02g= 86 | github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= 87 | github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs= 88 | github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= 89 | github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= 90 | github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= 91 | github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= 92 | github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= 93 | github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= 94 | github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= 95 | github.com/googleapis/gnostic v0.4.1 h1:DLJCy1n/vrD4HPjOvYcT8aYQXpPIzoRZONaYwyycI+I= 96 | github.com/googleapis/gnostic v0.4.1/go.mod h1:LRhVm6pbyptWbWbuZ38d1eyptfvIytN3ir6b65WBswg= 97 | github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= 98 | github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= 99 | github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= 100 | github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= 101 | github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= 102 | github.com/imdario/mergo v0.3.5 h1:JboBksRwiiAJWvIYJVo46AfV+IAIKZpfrSzVKj42R4Q= 103 | github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= 104 | github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= 105 | github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68= 106 | github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= 107 | github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= 108 | github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk= 109 | github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= 110 | github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= 111 | github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= 112 | github.com/kr/pretty v0.2.0 h1:s5hAObm+yFO5uHYt5dYjxi2rXrsnmRpJx4OYvIWUaQs= 113 | github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= 114 | github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= 115 | github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= 116 | github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= 117 | github.com/mailru/easyjson v0.0.0-20160728113105-d5b7844b561a/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= 118 | github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= 119 | github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= 120 | github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= 121 | github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= 122 | github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI= 123 | github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= 124 | github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= 125 | github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw= 126 | github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= 127 | github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= 128 | github.com/onsi/ginkgo v1.11.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= 129 | github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= 130 | github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= 131 | github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= 132 | github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 133 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 134 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 135 | github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= 136 | github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= 137 | github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk= 138 | github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= 139 | github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= 140 | github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= 141 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 142 | github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= 143 | github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk= 144 | github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= 145 | github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= 146 | github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= 147 | go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= 148 | go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8= 149 | go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= 150 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 151 | golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 152 | golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 153 | golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 154 | golang.org/x/crypto v0.0.0-20191206172530-e9b2fee46413/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= 155 | golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI= 156 | golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= 157 | golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= 158 | golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= 159 | golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= 160 | golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek= 161 | golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= 162 | golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= 163 | golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= 164 | golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= 165 | golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= 166 | golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= 167 | golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= 168 | golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= 169 | golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= 170 | golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs= 171 | golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= 172 | golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o= 173 | golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= 174 | golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY= 175 | golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= 176 | golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 177 | golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 178 | golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 179 | golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 180 | golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 181 | golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 182 | golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 183 | golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 184 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 185 | golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 186 | golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 187 | golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= 188 | golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 189 | golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 190 | golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 191 | golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= 192 | golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= 193 | golang.org/x/net v0.0.0-20201110031124-69a78807bb2b h1:uwuIcX0g4Yl1NC5XAz37xsr2lTtcqevgzYNVt49waME= 194 | golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= 195 | golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= 196 | golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= 197 | golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= 198 | golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6 h1:pE8b58s1HRDMi8RDc79m0HISf9D4TzseP40cEA6IGfs= 199 | golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= 200 | golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 201 | golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 202 | golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 203 | golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 204 | golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 205 | golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 206 | golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 207 | golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 208 | golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 209 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 210 | golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 211 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 212 | golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 213 | golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 214 | golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 215 | golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 216 | golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 217 | golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 218 | golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 219 | golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 220 | golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 221 | golang.org/x/sys v0.0.0-20201112073958-5cba982894dd h1:5CtCZbICpIOFdgO940moixOPjc0178IU44m4EjOO5IY= 222 | golang.org/x/sys v0.0.0-20201112073958-5cba982894dd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 223 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 224 | golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 225 | golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= 226 | golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k= 227 | golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= 228 | golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= 229 | golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= 230 | golang.org/x/time v0.0.0-20191024005414-555d28b269f0 h1:/5xXl8Y5W96D+TtHSlonuFqGHIWVuyCkGJLwGh9JJFs= 231 | golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= 232 | golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 233 | golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 234 | golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 235 | golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= 236 | golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 237 | golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 238 | golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 239 | golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= 240 | golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= 241 | golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= 242 | golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= 243 | golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= 244 | golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= 245 | golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 246 | golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 247 | golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 248 | golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 249 | golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 250 | golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= 251 | golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= 252 | golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= 253 | golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 254 | golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 255 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 256 | golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= 257 | golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 258 | google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE= 259 | google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M= 260 | google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg= 261 | google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg= 262 | google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= 263 | google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= 264 | google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= 265 | google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= 266 | google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0= 267 | google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM= 268 | google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= 269 | google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= 270 | google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= 271 | google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= 272 | google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= 273 | google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= 274 | google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= 275 | google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= 276 | google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8= 277 | google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= 278 | google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= 279 | google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= 280 | google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= 281 | google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM= 282 | google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= 283 | google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= 284 | google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= 285 | google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= 286 | google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= 287 | google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= 288 | google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= 289 | google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= 290 | google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= 291 | google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= 292 | google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= 293 | google.golang.org/protobuf v1.24.0 h1:UhZDfRO8JRQru4/+LlLE0BRKGF8L+PICnvYZmx/fEGA= 294 | google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4= 295 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 296 | gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 297 | gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= 298 | gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 299 | gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= 300 | gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= 301 | gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= 302 | gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= 303 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= 304 | gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 305 | gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 306 | gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 307 | gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= 308 | gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= 309 | honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 310 | honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 311 | honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 312 | honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 313 | honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= 314 | k8s.io/api v0.19.15 h1:i22aQYrQ9gaBHEAS9XvyR5ZfrTDAd+Q+JwWM+xIBv30= 315 | k8s.io/api v0.19.15/go.mod h1:rMRWjnIJQmurd/FdLobht6dCSbJQ+UDpyOwPaoFS7lI= 316 | k8s.io/apimachinery v0.19.15 h1:P37ni6/yFxRMrqgM75k/vt5xq9vnNiR3rJPTmWXrNho= 317 | k8s.io/apimachinery v0.19.15/go.mod h1:RMyblyny2ZcDQ/oVE+lC31u7XTHUaSXEK2IhgtwGxfc= 318 | k8s.io/client-go v0.19.15 h1:lDBvFBjDIExh0mFS6JbG+5B7ghuPhqXjBzlaxG81ToU= 319 | k8s.io/client-go v0.19.15/go.mod h1:OJMQWgHQJRDtO2BVtpkHUQOq/e5WHpXc02lSdPI0S/k= 320 | k8s.io/gengo v0.0.0-20200413195148-3a45101e95ac/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0= 321 | k8s.io/klog/v2 v2.0.0/go.mod h1:PBfzABfn139FHAV07az/IF9Wp1bkk3vpT2XSJ76fSDE= 322 | k8s.io/klog/v2 v2.2.0 h1:XRvcwJozkgZ1UQJmfMGpvRthQHOvihEhYtDfAaxMz/A= 323 | k8s.io/klog/v2 v2.2.0/go.mod h1:Od+F08eJP+W3HUb4pSrPpgp9DGU4GzlpG/TmITuYh/Y= 324 | k8s.io/kube-openapi v0.0.0-20200805222855-6aeccd4b50c6/go.mod h1:UuqjUnNftUyPE5H64/qeyjQoUZhGpeFDVdxjTeEVN2o= 325 | k8s.io/utils v0.0.0-20200729134348-d5654de09c73 h1:uJmqzgNWG7XyClnU/mLPBWwfKKF1K8Hf8whTseBgJcg= 326 | k8s.io/utils v0.0.0-20200729134348-d5654de09c73/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA= 327 | rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8= 328 | sigs.k8s.io/structured-merge-diff/v4 v4.0.1/go.mod h1:bJZC9H9iH24zzfZ/41RGcq60oK1F7G282QMXDPYydCw= 329 | sigs.k8s.io/structured-merge-diff/v4 v4.1.2 h1:Hr/htKFmJEbtMgS/UD0N+gtgctAqz81t3nu+sPzynno= 330 | sigs.k8s.io/structured-merge-diff/v4 v4.1.2/go.mod h1:j/nl6xW8vLS49O8YvXW1ocPhZawJtm+Yrr7PPRQ0Vg4= 331 | sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o= 332 | sigs.k8s.io/yaml v1.2.0 h1:kr/MCeFWJWTwyaHoR9c8EjH9OumOmoF9YGiZd7lFm/Q= 333 | sigs.k8s.io/yaml v1.2.0/go.mod h1:yfXDCHCao9+ENCvLSE62v9VSji2MKu5jeNfTrofGhJc= 334 | -------------------------------------------------------------------------------- /medium-article.md: -------------------------------------------------------------------------------- 1 | # Diving into Kubernetes MutatingAdmissionWebhook 2 | 3 | [Admission controllers](https://kubernetes.io/docs/admin/admission-controllers/) are powerful tools for intercepting requests to the Kubernetes API server prior to persistence of the object. However, they are not very flexible due to the requirement that they are compiled into binary into `kube-apiserver` and configured by the cluster administrator. Starting in Kubernetes 1.7, [Initializers](https://v1-8.docs.kubernetes.io/docs/admin/extensible-admission-controllers/#initializers) and [External Admission Webhooks](https://v1-8.docs.kubernetes.io/docs/admin/extensible-admission-controllers/#external-admission-webhooks) are introduced to address this limitation. In Kubernetes 1.9, `Initializers` stays in alpha phase while `External Admission Webhooks` have been promoted to beta and split into [MutatingAdmissionWebhook](https://kubernetes.io/docs/admin/admission-controllers/#mutatingadmissionwebhook-beta-in-19) and [ValidatingAdmissionWebhook](https://kubernetes.io/docs/admin/admission-controllers/#validatingadmissionwebhook-alpha-in-18-beta-in-19). 4 | 5 | `MutatingAdmissionWebhook` together with `ValidatingAdmissionWebhook` are a special kind of `admission controllers` which process mutating and validating on requests matching the rules defined in [MutatingWebhookConfiguration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#mutatingwebhookconfiguration-v1beta1-admissionregistration)(explained below). 6 | 7 | In this article, we'll dive into the details of `MutatingAdmissionWebhook` and write a working webhook admission server step by step. 8 | 9 | ## Benefit of `Webhooks` 10 | 11 | `Webhooks` allow Kubernetes cluster-admin to create additional mutating and validating admission plugins to the admission chain of `apiserver` without recompiling them. This provides end-developer with the freedom and flexibility to customize admission logic on multiple actions("CREATA", "UPDATE", "DELETE"...) on any resource. The possible applications are vast. Some common use cases includes: 12 | - Mutating resources before creating them. [Istio](https://github.com/istio), a representative example, injecting [Envoy](https://github.com/envoyproxy/envoy) sidecar container to target pods to implement traffic management and policy enforcement. 13 | - Automated provisioning of `StorageClass`. Observes creation of `PersistentVolumeClaim` objects and automatically adds storage class to them based on predefined policy. Users that do not need to care about `StorageClass` creating. 14 | - Validating complex custom resource. Make sure custom resource can only be created after its definition and all dependencies created and available. 15 | - Restricting namespace. On multi-tenant systems, avoid resources created in reserved namespaces. 16 | 17 | Besides the user-cases listed above, many more aplications can be created based on the power of `webhooks`. 18 | 19 | ## `Webhooks` vs `Initializers` 20 | 21 | Based on feedback from the community and use cases in alpha phase of both `External Admission Webhooks` and `Initializers`, the Kubernetes community decided to promote webhooks to beta and split it into `MutatingAdmissionWebhook` and `ValidatingAdmissionWebhook`. These updates make webhooks consistent with other admission controllers and enforce `mutate-before-validate`. `Initializers` can also implement dynamic admission control by modifying Kubernetes resources before they are actually created. If you're unfamiliar with `Initializers`, please refer to the acrtcle: https://medium.com/ibm-cloud/kubernetes-initializers-deep-dive-and-tutorial-3bc416e4e13e. 22 | 23 | So what's the difference between `Webhooks` and `Initializers`? 24 | 25 | - `Webhooks` can be applied on more actions, including 'mutate' or 'admit' on resoures 'CREATE' 'UPDATE' and 'DELETE', whereas `Initializers` can't 'admit' resources for 'DELETE' requests. 26 | - `Webhooks` are not allowed to query resources before created, while `Initializers` are capable of watching the uninitialized resources by the query parameter `?includeUninitialized=true`, which makes resources creating progress transparent. 27 | - Since the `Initializers` persist the 'pre-create' states to `etcd`, higher latency and increased `etcd` burden will be introduced accordingly, especially when `apiserver` upgrades or fails. `Webhooks`, however, consume less memory and computing resources. 28 | - More robustness on failures for `Webhooks` than `Initializers`. Failure policy can be configured in `Webhooks` configuraton to avoid hanging onto resources that are created. Buggy `Initializers`, on the other hand, may block all matched resources creating. 29 | 30 | Besides the difference listed above, `Initializer` is stuck in some open issues with long expected development time including quota replenishment bug. Promotion of `Webhooks` to beta may be a signal that more support for it in the future, but that depends. If stable behavior is preferred, suggest you choose `Webhooks`. 31 | 32 | ## How MutatingAdmissionWebhook works 33 | 34 | `MutatingAdmissionWebhook` intercepts requests matching the rules defined in `MutatingWebhookConfiguration` before presisting into [etcd](https://github.com/coreos/etcd). `MutatingAdmissionWebhook` executes the mutation by sending admission requests to webhook server. Webhook server is just plain http server that adhere to the [API](https://github.com/kubernetes/kubernetes/blob/v1.9.0/pkg/apis/admission/types.go). 35 | 36 | The following diagram describes how `MutatingAdmissionWebhook` works in details: 37 | 38 | ![](https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/mutating-admission-webhook.jpg) 39 | 40 | The `MutatingAdmissionWebhook` needs three objects to function: 41 | 42 | 1. **MutatingWebhookConfiguration** 43 | 44 | `MutatingAdmissionWebhook` need to be registered in the `apiserver` by providing `MutatingWebhookConfiguration`. During the registration process, MutatingAdmissionWebhook states: 45 | - How to connect to the webhook admission server 46 | - How to verify the webhook admission server 47 | - The URL path of the webhook admission server 48 | - Rules defining which resource and what action it handles 49 | - How unrecognized errors from the webhook admission server are handled 50 | 51 | 2. **MutatingAdmissionWebhook itself** 52 | 53 | `MutatingAdmissionWebhook` is a plugin-style admission controller that can be configured into the `apiserver`. The `MutatingAdmissionWebhook` plugin get the list of interested admission webhooks from `MutatingWebhookConfiguration`. Then the `MutatingAdmissionWebhook` observes the requests to apiserver and intercepts requests matching the rules in admission webhooks and calls them in parallel. 54 | 55 | 3. **Webhook Admission Server** 56 | 57 | `Webhook Admission Server` is just plain http server that adhere to Kubernetes [API](https://github.com/kubernetes/kubernetes/blob/v1.9.0/pkg/apis/admission/types.go). 58 | For each request to the `apiserver`, the `MutatingAdmissionWebhook` sends an `admissionReview`([API](https://github.com/kubernetes/kubernetes/blob/v1.9.0/pkg/apis/admission/types.go) for reference) to the relevant webhook admission server. The webhook admission server gathers information like `object`, `oldobject`, and `userInfo` from `admissionReview`, and sends back a `admissionReview` response including `AdmissionResponse` whose `Allowed` and `Result` fields are filled with the admission decision and optional `Patch` to mutate the resoures. 59 | 60 | ## Tutorial for MutatingAdmissionWebhook 61 | 62 | Write a complete Webhook Admission Server may be intimidating. To make it easier, we'll write a simple Webhook Admission Server that implements injecting nginx sidecar container and volume. The complete code can be found in [kube-mutating-webhook-tutorial](https://github.com/morvencao/kube-mutating-webhook-tutorial). The project refers to [Kunernetes webhook example](https://github.com/kubernetes/kubernetes/tree/release-1.9/test/images/webhook) and [Istio sidecar injection implementation](https://github.com/istio/istio/tree/master/pilot/pkg/kube/inject). 63 | 64 | In the following sections, I'll show you how to write a working containerized webhook admission server and deploy it to a Kubernetes cluster. 65 | 66 | #### Prerequisites 67 | 68 | `MutatingAdmissionWebhook` requires a Kubernetes 1.9.0 or above with the `admissionregistration.k8s.io/v1beta1` API enabled. Verify that by the following command: 69 | ``` 70 | kubectl api-versions | grep admissionregistration.k8s.io/v1beta1 71 | ``` 72 | The result should be: 73 | ``` 74 | admissionregistration.k8s.io/v1beta1 75 | ``` 76 | In addition, the `MutatingAdmissionWebhook` and `ValidatingAdmissionWebhook` admission controllers should be added and listed in the correct order in the `admission-control` flag of `kube-apiserver`. 77 | 78 | ### Write the Webhook Server 79 | 80 | `Webhook Admission Server` is just plain http server that adhere to Kubernetes [API](https://github.com/kubernetes/kubernetes/blob/v1.9.0/pkg/apis/admission/types.go). 81 | I'll paste some pseudo code to describe the main logic: 82 | ``` 83 | sidecarConfig, err := loadConfig(parameters.sidecarCfgFile) 84 | pair, err := tls.LoadX509KeyPair(parameters.certFile, parameters.keyFile) 85 | 86 | whsvr := &WebhookServer { 87 | sidecarConfig: sidecarConfig, 88 | server: &http.Server { 89 | Addr: fmt.Sprintf(":%v", 443), 90 | TLSConfig: &tls.Config{Certificates: []tls.Certificate{pair}}, 91 | }, 92 | } 93 | 94 | // define http server and server handler 95 | mux := http.NewServeMux() 96 | mux.HandleFunc("/mutate", whsvr.serve) 97 | whsvr.server.Handler = mux 98 | 99 | // start webhook server in new rountine 100 | go func() { 101 | if err := whsvr.server.ListenAndServeTLS("", ""); err != nil { 102 | glog.Errorf("Filed to listen and serve webhook server: %v", err) 103 | } 104 | }() 105 | ``` 106 | Explanation for the above code: 107 | 108 | - `sidecarCfgFile` contains sidecar injector template defined in `ConfigMap` below. 109 | - `certFile` and `keyFile` key pair that will be needed for TLS communication between `apiserver` and `webhook server`. 110 | - Line 19 starts https server listening on 443 on path '/mutate'. 111 | 112 | Next we'll focus on the main logic of handler function `serve`: 113 | ``` 114 | // Serve method for webhook server 115 | func (whsvr *WebhookServer) serve(w http.ResponseWriter, r *http.Request) { 116 | var body []byte 117 | if r.Body != nil { 118 | if data, err := ioutil.ReadAll(r.Body); err == nil { 119 | body = data 120 | } 121 | } 122 | 123 | var reviewResponse *v1beta1.AdmissionResponse 124 | ar := v1beta1.AdmissionReview{} 125 | deserializer := codecs.UniversalDeserializer() 126 | if _, _, err := deserializer.Decode(body, nil, &ar); err != nil { 127 | glog.Error(err) 128 | reviewResponse = toAdmissionResponse(err) 129 | } else { 130 | reviewResponse = mutate(ar) 131 | } 132 | 133 | response := v1beta1.AdmissionReview{} 134 | if reviewResponse != nil { 135 | response.Response = reviewResponse 136 | response.Response.UID = ar.Request.UID 137 | } 138 | // reset the Object and OldObject, they are not needed in a response. 139 | ar.Request.Object = runtime.RawExtension{} 140 | ar.Request.OldObject = runtime.RawExtension{} 141 | 142 | resp, err := json.Marshal(response) 143 | if err != nil { 144 | glog.Error(err) 145 | } 146 | if _, err := w.Write(resp); err != nil { 147 | glog.Error(err) 148 | } 149 | } 150 | ``` 151 | The `serve` function is plain http handler with `http request` and `response writer` parameters. 152 | - Firstly unmarshals the request to `AdmissionReview`, which contains information like `object`, `oldobject` and `userInfo`... 153 | - Then calls Webhook core function `mutate` to create `patch` that injects sidecar container and volume. 154 | - Finally, unmarshals the response with admission decision and optional patch, sends it back to `apiserver`. 155 | 156 | For the part of `mutate` function, you get the free rein to complete it in your preferred way. Let's take my implementation as an example: 157 | ``` 158 | // main mutation process 159 | func (whsvr *WebhookServer) mutate(ar *v1beta1.AdmissionReview) *v1beta1.AdmissionResponse { 160 | req := ar.Request 161 | var pod corev1.Pod 162 | if err := json.Unmarshal(req.Object.Raw, &pod); err != nil { 163 | glog.Errorf("Could not unmarshal raw object: %v", err) 164 | return &v1beta1.AdmissionResponse { 165 | Result: &metav1.Status { 166 | Message: err.Error(), 167 | }, 168 | } 169 | } 170 | 171 | // determine whether to perform mutation 172 | if !mutationRequired(ignoredNamespaces, &pod.ObjectMeta) { 173 | glog.Infof("Skipping mutation for %s/%s due to policy check", pod.Namespace, pod.Name) 174 | return &v1beta1.AdmissionResponse { 175 | Allowed: true, 176 | } 177 | } 178 | 179 | annotations := map[string]string{admissionWebhookAnnotationStatusKey: "injected"} 180 | patchBytes, err := createPatch(&pod, whsvr.sidecarConfig, annotations) 181 | 182 | return &v1beta1.AdmissionResponse { 183 | Allowed: true, 184 | Patch: patchBytes, 185 | PatchType: func() *v1beta1.PatchType { 186 | pt := v1beta1.PatchTypeJSONPatch 187 | return &pt 188 | }(), 189 | } 190 | } 191 | ``` 192 | From the code above, the `mutate` function calls [mutationRequired](https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/webhook.go#L98-L130) to detemine whether mutation is required or not. For those requiring mutation, the `mutate` function gets mutation 'patch' from another function [createPatch](https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/webhook.go#L196-L205). Pay attention to the little trick in function `mutationRequired`, we skip the `pods` without annotation `sidecar-injector-webhook.morven.me/inject: true`. That will be mentioned latter when we deployment applications. For complete code, please refer to https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/webhook.go. 193 | 194 | #### Create Dockerfile and Build the Container 195 | 196 | Create the `build` script: 197 | ``` 198 | dep ensure 199 | CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o kube-mutating-webhook-tutorial . 200 | docker build --no-cache -t morvencao/sidecar-injector:v1 . 201 | rm -rf kube-mutating-webhook-tutorial 202 | 203 | docker push morvencao/sidecar-injector:v1 204 | ``` 205 | 206 | And create `Dockerfile` as dependency of build script: 207 | ``` 208 | FROM alpine:latest 209 | 210 | ADD kube-mutating-webhook-tutorial /kube-mutating-webhook-tutorial 211 | ENTRYPOINT ["./kube-mutating-webhook-tutorial"] 212 | ``` 213 | 214 | Before actually building the container, you need a [Docker](https://hub.docker.com/) ID account and change the image name&tag(in `Dockerfile` and `deployment.yaml`) to yours, then execute: 215 | ``` 216 | [root@mstnode kube-mutating-webhook-tutorial]# ./build 217 | Sending build context to Docker daemon 44.89MB 218 | Step 1/3 : FROM alpine:latest 219 | ---> 3fd9065eaf02 220 | Step 2/3 : ADD kube-mutating-webhook-tutorial /kube-mutating-webhook-tutorial 221 | ---> 432de60c2b3f 222 | Step 3/3 : ENTRYPOINT ["./kube-mutating-webhook-tutorial"] 223 | ---> Running in da6e956d1755 224 | Removing intermediate container da6e956d1755 225 | ---> 619faa936145 226 | Successfully built 619faa936145 227 | Successfully tagged morvencao/sidecar-injector:v1 228 | The push refers to repository [docker.io/morvencao/sidecar-injector] 229 | efd05fe119bb: Pushed 230 | cd7100a72410: Layer already exists 231 | v1: digest: sha256:7a4889928ec5a8bcfb91b610dab812e5228d8dfbd2b540cd7a341c11f24729bf size: 739 232 | ``` 233 | 234 | #### Create Sidecar Injection Configuration 235 | 236 | Now let's create a Kubernetes `ConfigMap`, which includes `container` and `volume` information that will be injected into the target pod. 237 | ``` 238 | apiVersion: v1 239 | kind: ConfigMap 240 | metadata: 241 | name: sidecar-injector-webhook-configmap 242 | data: 243 | sidecarconfig.yaml: | 244 | containers: 245 | - name: sidecar-nginx 246 | image: nginx:1.12.2 247 | imagePullPolicy: IfNotPresent 248 | ports: 249 | - containerPort: 80 250 | volumeMounts: 251 | - name: nginx-conf 252 | mountPath: /etc/nginx 253 | volumes: 254 | - name: nginx-conf 255 | configMap: 256 | name: nginx-configmap 257 | ``` 258 | From the above manifest, another ConfigMap including `nginx conf` is required. Here we put it in [nginxconfigmap.yaml](https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/deploy/nginxconfigmap.yaml). 259 | 260 | Then deploy the two `ConfigMap`s to cluster: 261 | ``` 262 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deploy/nginxconfigmap.yaml 263 | configmap "nginx-configmap" created 264 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deploy/configmap.yaml 265 | configmap "sidecar-injector-webhook-configmap" created 266 | ``` 267 | 268 | #### Create Secret Including Signed key/cert Pair 269 | 270 | Supporting `TLS` for external webhook server is required, because admission is a high security operation. so we need to create TLS certificate signed by `Kubernetes CA` for to secure the communcation between webhook server and `apiserver`. For the complete creating and approving `CSR` process, please refer to https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/. 271 | 272 | For simplicity purposes, we refer to the [script](https://github.com/istio/istio/blob/master/install/kubernetes/webhook-create-signed-cert.sh) from `Istio` and write a similar script called `webhook-create-signed-cert.sh` to automatically create the cert/key pair and include it in a Kubernetes `secret`. 273 | ``` 274 | #!/bin/bash 275 | while [[ $# -gt 0 ]]; do 276 | case ${1} in 277 | --service) 278 | service="$2" 279 | shift 280 | ;; 281 | --secret) 282 | secret="$2" 283 | shift 284 | ;; 285 | --namespace) 286 | namespace="$2" 287 | shift 288 | ;; 289 | esac 290 | shift 291 | done 292 | 293 | [ -z ${service} ] && service=sidecar-injector-webhook-svc 294 | [ -z ${secret} ] && secret=sidecar-injector-webhook-certs 295 | [ -z ${namespace} ] && namespace=default 296 | 297 | csrName=${service}.${namespace} 298 | tmpdir=$(mktemp -d) 299 | echo "creating certs in tmpdir ${tmpdir} " 300 | 301 | cat <> ${tmpdir}/csr.conf 302 | [req] 303 | req_extensions = v3_req 304 | distinguished_name = req_distinguished_name 305 | [req_distinguished_name] 306 | [ v3_req ] 307 | basicConstraints = CA:FALSE 308 | keyUsage = nonRepudiation, digitalSignature, keyEncipherment 309 | extendedKeyUsage = serverAuth 310 | subjectAltName = @alt_names 311 | [alt_names] 312 | DNS.1 = ${service} 313 | DNS.2 = ${service}.${namespace} 314 | DNS.3 = ${service}.${namespace}.svc 315 | EOF 316 | 317 | openssl genrsa -out ${tmpdir}/server-key.pem 2048 318 | openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf 319 | 320 | # clean-up any previously created CSR for our service. Ignore errors if not present. 321 | kubectl delete csr ${csrName} 2>/dev/null || true 322 | 323 | # create server cert/key CSR and send to k8s API 324 | cat <&2 359 | exit 1 360 | fi 361 | echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem 362 | 363 | 364 | # create the secret with CA cert and server cert/key 365 | kubectl create secret generic ${secret} \ 366 | --from-file=key.pem=${tmpdir}/server-key.pem \ 367 | --from-file=cert.pem=${tmpdir}/server-cert.pem \ 368 | --dry-run -o yaml | 369 | kubectl -n ${namespace} apply -f - 370 | ``` 371 | 372 | Then execute it and a Kubernetes `secret` including cert/key pair is created: 373 | ``` 374 | [root@mstnode kube-mutating-webhook-tutorial]# ./deploy/webhook-create-signed-cert.sh 375 | creating certs in tmpdir /tmp/tmp.wXZywp0wAF 376 | Generating RSA private key, 2048 bit long modulus 377 | ...........................................+++ 378 | ..........+++ 379 | e is 65537 (0x10001) 380 | certificatesigningrequest "sidecar-injector-webhook-svc.default" created 381 | NAME AGE REQUESTOR CONDITION 382 | sidecar-injector-webhook-svc.default 0s https://mycluster.icp:9443/oidc/endpoint/OP#admin Pending 383 | certificatesigningrequest "sidecar-injector-webhook-svc.default" approved 384 | secret "sidecar-injector-webhook-certs" created 385 | ``` 386 | 387 | #### Create the Sidecar Injector Deployment and Service 388 | 389 | The `deployment` brings up 1 `pod` in which the `sidecar-injector` container is running. The container starts with special arguments: 390 | - `sidecarCfgFile` pointing to the sidecar injector configuration file mounted from `sidecar-injector-webhook-configmap` ConfigMap created above 391 | - `tlsCertFile` and `tlsKeyFile` are cert/key pair mounted from `sidecar-injector-webhook-certs` Secret create by script above 392 | - `alsologtostderr` `v=4` and `2>&1` are logging arguments 393 | ``` 394 | apiVersion: extensions/v1beta1 395 | kind: Deployment 396 | metadata: 397 | name: sidecar-injector-webhook-deployment 398 | labels: 399 | app: sidecar-injector 400 | spec: 401 | replicas: 1 402 | template: 403 | metadata: 404 | labels: 405 | app: sidecar-injector 406 | spec: 407 | containers: 408 | - name: sidecar-injector 409 | image: morvencao/sidecar-injector:v1 410 | imagePullPolicy: IfNotPresent 411 | args: 412 | - -sidecarCfgFile=/etc/webhook/config/sidecarconfig.yaml 413 | - -tlsCertFile=/etc/webhook/certs/cert.pem 414 | - -tlsKeyFile=/etc/webhook/certs/key.pem 415 | - -alsologtostderr 416 | - -v=4 417 | - 2>&1 418 | volumeMounts: 419 | - name: webhook-certs 420 | mountPath: /etc/webhook/certs 421 | readOnly: true 422 | - name: webhook-config 423 | mountPath: /etc/webhook/config 424 | volumes: 425 | - name: webhook-certs 426 | secret: 427 | secretName: sidecar-injector-webhook-certs 428 | - name: webhook-config 429 | configMap: 430 | name: sidecar-injector-webhook-configmap 431 | ``` 432 | 433 | The `service` exposes the `pod` defined above labeled by `app=sidecar-injector` to make it accessible in cluster. This `service` will be referred by the `MutatingWebhookConfiguration` in `clientConfig` section and by default `spec.ports.port` should be **443**(default https port). 434 | ``` 435 | apiVersion: v1 436 | kind: Service 437 | metadata: 438 | name: sidecar-injector-webhook-svc 439 | labels: 440 | app: sidecar-injector 441 | spec: 442 | ports: 443 | - port: 443 444 | targetPort: 443 445 | selector: 446 | app: sidecar-injector 447 | ``` 448 | 449 | Next we deploy the above `Deployment` and `Service` to cluster and verify the `sidecar injector` webhook server is running: 450 | ``` 451 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deploy/deployment.yaml 452 | deployment "sidecar-injector-webhook-deployment" created 453 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deploy/service.yaml 454 | service "sidecar-injector-webhook-svc" created 455 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl get deployment 456 | NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE 457 | sidecar-injector-webhook-deployment 1 1 1 1 2m 458 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl get pod 459 | NAME READY STATUS RESTARTS AGE 460 | sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 3m 461 | ``` 462 | 463 | #### Configure webhook admission controller on the fly 464 | 465 | `MutatingWebhookConfiguration` specifies which webhook admission servers are enabled and which resources are subject to the admission server. It is recommended that you firstly deploy the webhook admission server and make sure it is working properly before creating the `MutatingWebhookConfiguration`. Otherwise, requests will be unconditionally accepted or rejected based on `failurePolicy`. 466 | 467 | For now, we create the `MutatingWebhookConfiguration` manifest with the following content: 468 | ``` 469 | apiVersion: admissionregistration.k8s.io/v1beta1 470 | kind: MutatingWebhookConfiguration 471 | metadata: 472 | name: sidecar-injector-webhook-cfg 473 | labels: 474 | app: sidecar-injector 475 | webhooks: 476 | - name: sidecar-injector.morven.me 477 | clientConfig: 478 | service: 479 | name: sidecar-injector-webhook-svc 480 | namespace: default 481 | path: "/mutate" 482 | caBundle: ${CA_BUNDLE} 483 | rules: 484 | - operations: [ "CREATE" ] 485 | apiGroups: [""] 486 | apiVersions: ["v1"] 487 | resources: ["pods"] 488 | namespaceSelector: 489 | matchLabels: 490 | sidecar-injector: enabled 491 | ``` 492 | 493 | Line 8: `name` - name for the webhook, should be fully qualified. Mutiple mutating webhooks are sorted by providing order. 494 | Line 9: `clientConfig` - describes how to connect to the webhook admission server and the TLS certificate. In our case, we specify the `sidecar injector` service. 495 | Line 15: `rules` - specifies what resources and what actions the webhook server handles. In our case, only intercepts request for creating of pods. 496 | Line 20: `namespaceSelector` - `namespaceSelector` decides whether to send admission request the webhook server on an object based on whether the namespace for that object matches the selector. 497 | 498 | Before deploying the `MutatingWebhookConfiguration`, we need to replace the `${CA_BUNDLE}` with apiserver's default `caBundle`. Let's write the script `webhook-patch-ca-bundle.sh` to automate this process: 499 | ``` 500 | #!/bin/bash 501 | set -o errexit 502 | set -o nounset 503 | set -o pipefail 504 | 505 | ROOT=$(cd $(dirname $0)/../../; pwd) 506 | 507 | export CA_BUNDLE=$(kubectl get configmap -n kube-system extension-apiserver-authentication -o=jsonpath='{.data.client-ca-file}' | base64 | tr -d '\n') 508 | 509 | if command -v envsubst >/dev/null 2>&1; then 510 | envsubst 511 | else 512 | sed -e "s|\${CA_BUNDLE}|${CA_BUNDLE}|g" 513 | fi 514 | ``` 515 | 516 | Then execute: 517 | ``` 518 | [root@mstnode kube-mutating-webhook-tutorial]# cat ./deploy/mutatingwebhook.yaml |\ 519 | > ./deploy/webhook-patch-ca-bundle.sh >\ 520 | > ./deploy/mutatingwebhook-ca-bundle.yaml 521 | ``` 522 | 523 | Finally we can deploy `MutatingWebhookConfiguration`: 524 | ``` 525 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deploy/mutatingwebhook-ca-bundle.yaml 526 | mutatingwebhookconfiguration "sidecar-injector-webhook-cfg" created 527 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl get mutatingwebhookconfiguration 528 | NAME AGE 529 | sidecar-injector-webhook-cfg 11s 530 | ``` 531 | 532 | #### Verification and Troubleshooting 533 | 534 | Now it's time to verify sidecar injector works as expected and try to see how to troubleshoot if you encounter issues. 535 | Typically we create and deploy a sleep application in `default` namespace to see if the sidecar can be injected. 536 | ``` 537 | [root@mstnode kube-mutating-webhook-tutorial]# cat < apiVersion: extensions/v1beta1 539 | > kind: Deployment 540 | > metadata: 541 | > name: sleep 542 | > spec: 543 | > replicas: 1 544 | > template: 545 | > metadata: 546 | > annotations: 547 | > sidecar-injector-webhook.morven.me/inject: "true" 548 | > labels: 549 | > app: sleep 550 | > spec: 551 | > containers: 552 | > - name: sleep 553 | > image: tutum/curl 554 | > command: ["/bin/sleep","infinity"] 555 | > imagePullPolicy: IfNotPresent 556 | > EOF 557 | deployment "sleep" created 558 | ``` 559 | 560 | Pay close attention to the `spec.template.metadata.annotations` as there is a new annotation added: 561 | ``` 562 | sidecar-injector-webhook.morven.me/inject: "true" 563 | ``` 564 | The sidecar injector has some logic to check the existence of the above annotation before injecting sidecar container and volume. 565 | You're free to delete the logic or customize it before build the sidecar injector container. 566 | 567 | Check the `deployment` and `pod`: 568 | ``` 569 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl get deployment 570 | NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE 571 | sidecar-injector-webhook-deployment 1 1 1 1 18m 572 | sleep 1 1 1 1 58s 573 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl get pod 574 | NAME READY STATUS RESTARTS AGE 575 | sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 18m 576 | sleep-6d79d8dc54-r66vz 1/1 Running 0 1m 577 | ``` 578 | It's not there. What's going on? 579 | Let's check the sidecar injector logs: 580 | ``` 581 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl logs -f sidecar-injector-webhook-deployment-bbb689d69-fdbgj 582 | I0314 08:48:15.140858 1 webhook.go:88] New configuration: sha256sum 21669464280f76170b88241fd79ecbca3dcebaec5c152a4a9a3e921ff742157f 583 | 584 | ``` 585 | We can't find any logs that indicate webhook server got admission request, seems that request hadn't been sent to `sidecar injector` webhook server. 586 | So there is a possibility that the issue is caused by configuration in `MutatingWebhookConfiguration`. Do a double check of `MutatingWebhookConfiguration` and we find following content: 587 | ``` 588 | namespaceSelector: 589 | matchLabels: 590 | sidecar-injector: enabled 591 | ``` 592 | 593 | #### Control sidecar injector with `namespaceSelector` 594 | 595 | We have configured 'namespaceSelector' in `MutatingWebhookConfiguration`, which means only resources in namespace matching the selector will be sent to webhook server. So we need label the `default` namespace with `sidecar-injector=enabled`: 596 | 597 | ``` 598 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl label namespace default sidecar-injector=enabled 599 | namespace "default" labeled 600 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl get namespace -L sidecar-injector 601 | NAME STATUS AGE sidecar-injector 602 | default Active 1d enabled 603 | kube-public Active 1d 604 | kube-system Active 1d 605 | ``` 606 | 607 | We have configured the `MutatingWebhookConfiguration` resulting in the sidecar injection occuring at pod creation time. Kill the running pod and verify a new pod is created with the injected sidecar. 608 | ``` 609 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl delete pod sleep-6d79d8dc54-r66vz 610 | pod "sleep-6d79d8dc54-r66vz" deleted 611 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl get pods 612 | NAME READY STATUS RESTARTS AGE 613 | sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 29m 614 | sleep-6d79d8dc54-b8ztx 0/2 ContainerCreating 0 3s 615 | sleep-6d79d8dc54-r66vz 1/1 Terminating 0 11m 616 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl get pod sleep-6d79d8dc54-b8ztx -o yaml 617 | apiVersion: v1 618 | kind: Pod 619 | metadata: 620 | annotations: 621 | kubernetes.io/psp: default 622 | sidecar-injector-webhook.morven.me/inject: "true" 623 | sidecar-injector-webhook.morven.me/status: injected 624 | labels: 625 | app: sleep 626 | pod-template-hash: "2835848710" 627 | name: sleep-6d79d8dc54-b8ztx 628 | namespace: default 629 | spec: 630 | containers: 631 | - command: 632 | - /bin/sleep 633 | - infinity 634 | image: tutum/curl 635 | imagePullPolicy: IfNotPresent 636 | name: sleep 637 | resources: {} 638 | volumeMounts: 639 | - mountPath: /var/run/secrets/kubernetes.io/serviceaccount 640 | name: default-token-d7t2r 641 | readOnly: true 642 | - image: nginx:1.12.2 643 | imagePullPolicy: IfNotPresent 644 | name: sidecar-nginx 645 | ports: 646 | - containerPort: 80 647 | protocol: TCP 648 | resources: {} 649 | terminationMessagePath: /dev/termination-log 650 | terminationMessagePolicy: File 651 | volumeMounts: 652 | - mountPath: /etc/nginx 653 | name: nginx-conf 654 | volumes: 655 | - name: default-token-d7t2r 656 | secret: 657 | defaultMode: 420 658 | secretName: default-token-d7t2r 659 | - configMap: 660 | defaultMode: 420 661 | name: nginx-configmap 662 | name: nginx-conf 663 | ... 664 | ``` 665 | We can see that sidecar container and volume have been injected into sleep application successfully. Until now, we have working sidecar injector with `MutatingAdmissionWebhook`. With `namespaceSelector` we can easily control whether the pods in specified namespace will be injected or not. 666 | 667 | But there is a problem for this, with the above configurations, all of the pods in `default` namespace will be injected with a sidecar, this may be not expected for some cases. 668 | 669 | #### Control sidecar injector with `annotation` 670 | 671 | Thanks to flexibility of `MutatingAdmissionWebhook`, we can easily customized the mutating logic to filter resources with specified annotations. Remember the annotation `sidecar-injector-webhook.morven.me/inject: "true"` mentioned above? It can be used as an extra control on sidecar injector. I have written [some code](https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/webhook.go#L98-L130) in webhook server to skip injecting for pod without the annotation. 672 | 673 | Let's give it a try. In this case, we create another sleep application without `sidecar-injector-webhook.morven.me/inject: "true"` annotation in `podTemplateSpec`: 674 | ``` 675 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl delete deployment sleep 676 | deployment "sleep" deleted 677 | [root@mstnode kube-mutating-webhook-tutorial]# cat < kind: Deployment 680 | > metadata: 681 | > name: sleep 682 | > spec: 683 | > replicas: 1 684 | > template: 685 | > metadata: 686 | > labels: 687 | > app: sleep 688 | > spec: 689 | > containers: 690 | > - name: sleep 691 | > image: tutum/curl 692 | > command: ["/bin/sleep","infinity"] 693 | > imagePullPolicy: IfNotPresent 694 | > EOF 695 | deployment "sleep" created 696 | ``` 697 | 698 | And then verify the sidecar injector skipped the pod: 699 | ``` 700 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl get deployment 701 | NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE 702 | sidecar-injector-webhook-deployment 1 1 1 1 45m 703 | sleep 1 1 1 1 17s 704 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl get pod 705 | NAME READY STATUS RESTARTS AGE 706 | sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 45m 707 | sleep-776b7bcdcd-4bz58 1/1 Running 0 21s 708 | ``` 709 | 710 | The output shows that the sleep application contains only one container, no extra container and volume injected. 711 | Then we patch the sleep deployment to add the additional annotation and verify it will be injected after recreated: 712 | ``` 713 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl patch deployment sleep -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar-injector-webhook.morven.me/inject": "true"}}}}}' 714 | deployment "sleep" patched 715 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl delete pod sleep-776b7bcdcd-4bz58 716 | pod "sleep-776b7bcdcd-4bz58" deleted 717 | [root@mstnode kube-mutating-webhook-tutorial]# kubectl get pods 718 | NAME READY STATUS RESTARTS AGE 719 | sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 49m 720 | sleep-3e42ff9e6c-6f87b 0/2 ContainerCreating 0 18s 721 | sleep-776b7bcdcd-4bz58 1/1 Terminating 0 3m 722 | ``` 723 | As expected, the pod has been injected with extra sidecar container. 724 | Now, we got working sidecar injector with `mutatingAdmissionWebhook` and its coarse-grained control by `namespaceSelector` and fine-grained control by additional `annotation`. 725 | 726 | ## Conclusion 727 | 728 | `MutatingAdmissionWebhook` is one of easiest ways of extending Kubernetes with new policy controls, resources mutation... 729 | 730 | This feature will enable more workloads and support more ecosystem components, including [Istio](https://github.com/istio/istio) service mesh platform. Starting with Istio 0.5.0, Istio has refactored to support their auto injection code with `MutatingAdmissionWebhook` replacing `initializers`. 731 | 732 | ## Reference 733 | 734 | - http://blog.kubernetes.io/2018/01/extensible-admission-is-beta.html 735 | - https://docs.google.com/document/d/1c4kdkY3ha9rm0OIRbGleCeaHknZ-NR1nNtDp-i8eH8E/view 736 | - https://v1-8.docs.kubernetes.io/docs/admin/extensible-admission-controllers/ 737 | - https://github.com/kubernetes/kubernetes/tree/release-1.9/test/images/webhook -------------------------------------------------------------------------------- /mutating-admission-webhook.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/morvencao/kube-sidecar-injector/4e010f4cdee8baf3cd3f3f59ec9b95e5db9b9f01/mutating-admission-webhook.jpg -------------------------------------------------------------------------------- /mutating-admission-webhook.xml: -------------------------------------------------------------------------------- 1 | 7Vzbkps4EP0aV+0+2CUB4vI4M8lsHpKt2clWsvuIQbbZwUAAzyVfvwIkGwnJ2OZiT8qkamI3Qojuo9PdauGJfrd+/SN1k9WX2MfhRAP+60T/MNE0CKFN/iskb5XEhE4lWKaBX4nATvA1+InplUy6CXycUVklyuM4zIOEF3pxFGEv52RumsYvfLNFHPqcIHGXuCH46rlhU/o98PMVey4Adic+4WC5ore2ET0xd72nZRpvInq/iaYvyqM6vXZZX7R9tnL9+KUm0j9O9Ls0jvPq0/r1DoeFbnm13SvObsed4ig/5AJUXfDshhv66E+bOZ66SZDh9BmndJT5G9NMjl9Jx7erfB0SASQfszyNn/BdHMYpkURxRFreLoIwFERuGCwj8tUjQyMd67ek+zwgOr+hJ9aB7xe3uX1ZBTn+mrhecc8XgjAiK3WKi1GD4p5MbcWX0J3j8HareOG29AHJvfCrUklwq3oCaRyvcZ6+kSb0Aocai4JZZyh92UGDNVnVQKFTmUvBuNx2vDMI+UBtIreP1bBPwyK8ZmS6qxurrjjecgSqoDy2Zxj0NblBB9e7BWeoVfMatJqqN4wedO9IdG+GBfr94Jl8XOblY1Yij3CgKFvE5DnrtjJ/bGJ2YpqVrHdDGjjJ6+4c6+RPd42zwo6fgwX23rwQs27JuKue+bsRsWwQRMwNV4keOBB6oBw9xSPcu+sgLGz7CYfPuCCDMWAF2SUMVUYTVVB3JBMadQcVhEpUJWnDeL2j6nOwDvJHN1oW1H4inLhxXuHUgJMzJpy0Y+A0BsK+krgh8PCN5xF15/2A7Aq9w6BnjMpk+qVB7wGnWZDlRHnf4nCzxp8L/V8ROCoCHTQiAo2OCBwDlF82uZsH0fLGXwdZFsTRdzxfxfHTkLi8ovUwtKJRXXUz1z4rWm0ZWr+RLN2/4vVC8WqO6t/Ny8KrlF0fcRZvUg//RcTuFaIXANFRKdVWQrS7z8bTh9gvHoTxH2BLogqUjb4ud3FgMAC/UmoYehMMaKClUk29ssLAwOoMzN5Qafoou1q+i+URkNDAYJaXLYIINsGRf1MUhsg3L3RJYOPxZlCqTm2SQ5SK/UahqVWlNZXJNMZkKQ5JoPbMdy9TI73DQxyUs4CZ2BGWrSyBkCvHSq+ql5DaOnIEk+ZuusR5o6PSqtvHPszQshD6aug2QwukrJ1saJHdhzO0zLEPb2hisPTtn4KLZ8VCBhX8SwRTMAMkIKaSB5wG5GmKKubBFch2gGigUiALmi4KNH2xw3aBaADQqAt281OTDmjK4sNZebSnG3NliHCBZXTlfFmEQfKJjnIVp8FP0tANj4B+twTDEiILSQ6sSyZCHxVg1u/YNPQa5BULIQDp94KEIOEgRL+LFFSjLo64+qSnas5PuLVXjrLQmSmLsHbt4GBj9uX1zOG8ni7LYEaFWx1s5wAVkoDKPCeoNABmNrKgoVV/LZsHg45mu5Pk72kYK+6i28BBWvXX4G+ChkOcLHM6NWe+e/xwTZq7JM2mLimeDZU06+ryrWR7U5a40ckxFNhXF6Plhbs4WgTLTUpkcdQeWcnG07LTiWAmwaX9clV0VEMYv5Q7Ct7oVCvTDDGoy+NkDETqJo/IbfTObbiTIBKafUBSVs9V+795GHtPf6+CaF8gXTePUvNbD1jzf5XLK3bgEZBOqk2Q/23WCd2sXPlD/z4IWSS8c5acqyy7GSsC08+bNArosSwBFIf6QxGGjWBe4QEJMty3WrOkaJCpB2wI/AvsvcMSmmsWt7eafKjuf7IzVheIu9DvvgpHwlc4THdd0Ew0zxKeVw9i4SNTXDIzzfIQaLLYMo+Kf2dJdkdw+vye5u2rAHWnr0kmqdYHw8r2kw8LMb6SckXYGRCGmk58OISpS7JDIUzIO64QGx9i+pgkZshW5bqu82qqdd7rKu8ogDIEzrLHW+U11BsHrnWD94soxEfrCI2IKNmyWutCbj+p7J4sdJevwoZJWQoMZwAYk3oarNmsBjFU2bO1rmDLLT3SErCQjJrwxKzWbsmOe0pqdV3MasHeYYntDcRlwZ3TWkO90Hgom1oFmYISU7U3rKdeBbiiRbqc/0ameNFKIyMCwuffm1wsi0pdtuf5ET8H+GVPLFm/KsU/NjhTv+rUSuEpJo/qzssGxVNSC5PW6HaCPhxF3vvnpJI2GvysZGD6yj0d7mQb0dXndcV/SmoGM9NhOQkF3VTrZa5OdSC8MoxMvo94schw17KJcdxCZa/8znh6Ii5VCnzO/MCW0He+AJrKnSy8ixhkX4txTjJHhjOzawdfJDaQJTt7dDnPBDNLqx08Ih1rptcOYxAvgMy9rN7eXm/xGrBbe+SY/XoZ2dsK7yfk0vjpbJ054jrvjsVmxCXMkUMnoiGC1LSHCbmQPERUTgYxN6EbH3qbDOpl3mvI9YuGXIptMsOHXJDfEsPC/X4DrpP2A/cbcBGW5oMuw9ofdfFbpmYGMBsOQbmTbwziv8DoDO2Nzsx+ojNrX3RmglGiM3FTK/VwyuhMaG84LQ6mY3sTcOPp7pDUW7N/FYeUJXGUqX+j6d15JNjZIylW7Qb3SNvCE+sWWsIs7sUnIfV2jaMwLYKyAUDw+B7ARZzpfXnsQ9iQgJqCmQ41UzB9TzHOEPj5lTJWk3jj86asumLNcaSUlQQVwq5tqM0E2rm4tFXcf9wShTTao343wKFr2np+Hh83SEBnS1s1R6gUTIUXhPoh+QtIXCdHVAp03iHYsOkSRstP2cspF5SfmmBWf9tH+DUpTefeFeqQn9bvYYh+xZCNYeAEVaD69vZ2S8IpvlV3bHv6g+J9uR624tzB9dgDuJ53mgu0+ZBuLuNdpwHk6+7X3avmu5/Q1z/+Dw== -------------------------------------------------------------------------------- /prestop.sh: -------------------------------------------------------------------------------- 1 | set -exo pipefail 2 | 3 | # Point to the internal API server hostname 4 | APISERVER=https://kubernetes.default.svc 5 | 6 | # Path to ServiceAccount token 7 | SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount 8 | 9 | # Read this Pod's namespace 10 | NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace) 11 | 12 | # Read the ServiceAccount bearer token 13 | TOKEN=$(cat ${SERVICEACCOUNT}/token) 14 | 15 | # Reference the internal certificate authority (CA) 16 | CACERT=${SERVICEACCOUNT}/ca.crt 17 | 18 | MutatingWebhookConfigurationName=sidecar-injector-webhook 19 | 20 | # Delete the validatingwebhookconfiguration with TOKEN 21 | curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X DELETE ${APISERVER}/apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/${MutatingWebhookConfigurationName} 22 | 23 | --------------------------------------------------------------------------------