├── .DEREK.yml ├── .gitignore ├── LICENSE ├── README.md └── images ├── add-dns.png ├── buy-dns.png └── registry.png /.DEREK.yml: -------------------------------------------------------------------------------- 1 | curators: 2 | - alexellis 3 | 4 | features: 5 | - dco_check 6 | - comments 7 | - pr_description_required 8 | # - hacktoberfest 9 | - release_notes 10 | 11 | contributing_url: https://github.com/openfaas/faas/blob/master/CONTRIBUTING.md 12 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.sh 2 | kubeconfig 3 | k3sup* 4 | *.txt 5 | htpasswd 6 | *.yaml 7 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Alex Ellis 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Setup a private Docker registry with TLS on Kubernetes 2 | 3 | This tutorial will show you how to deploy your own registry on Kubernetes for storing Docker images. You will also learn how to setup [TLS certificates](https://en.wikipedia.org/wiki/Transport_Layer_Security) which will be issued for free from [LetsEncrypt.com](https://letsencrypt.com). 4 | 5 | ### Conceptual architecture 6 | 7 | ![Registry](/images/registry.png) 8 | 9 | You will learn how each part works together by following the tutorial. 10 | 11 | ### Do I need my own container registry? 12 | 13 | The primary purpose of a container registry is to store and host artifacts packaged in the [Docker or OCI-image format](https://blog.docker.com/2017/07/demystifying-open-container-initiative-oci-specifications/). 14 | 15 | At present there are managed registries for container images offered by almost every cloud provider. Even companies who do not offer compute are starting to offer registries such as [jFrog](https://jfrog.com), [GitLab.com](https://docs.gitlab.com/ee/user/project/container_registry.html), [Docker Inc](https://hub.docker.com), and [GitHub.com](https://github.com/features/package-registry). 16 | 17 | So why would you want to set up your own? 18 | 19 | * latency 20 | 21 | Hosting a registry inside your Kubernetes cluster is the fastest possible way to push and pull images. This matters for use-case such as auto-scaling and affects the overall speed to deploy from a CI/CD pipeline. 22 | 23 | * costs 24 | 25 | Bandwidth in and out of a datacentre is rarely free, let alone across regions. By hosting Docker images where they are produced and consumed keeps costs to the absolute minimum. 26 | 27 | * regulations 28 | 29 | Some regulations and legal restrictions such as GDPR may mean that storing artifacts with a SaaS provider is just not tenable. 30 | 31 | * security 32 | 33 | Although we don't explore it in the scope of this tutorial, additional security can be added to self-hosted registries using Open Source software like the [CNCF's Harbor](https://goharbor.io). Harbor scans Docker images for CVEs and other vulnerabilities. 34 | 35 | * automation & portability 36 | 37 | You may be able to automate a hosted registry on AWS, but completely different code is required to automate a registry on GCP. By using an Open Source registry that we can self-host, we regain the portability aspect. 38 | 39 | * ease of use 40 | 41 | It is relatively easy to integrate one or more registries into an existing Kubernetes cluster, in any availability region that you choose. 42 | 43 | ### Pre-reqs 44 | 45 | * A domain name, or sub-domain which you own. You need to be able to update DNS A records for your domain. I bought mine at [domains.google](https://domains.google). 46 | 47 | * [Docker](https://www.docker.com/) - we'll use a Docker container to generate some of our configuration 48 | 49 | * [Kubernetes](https://kubernetes.io/) - this tutorial is written with k3s in mind, but also works on Kubernetes with a few tweaks. 50 | 51 | If you're a Civo user, then we'll be hosting our registry on [Civo Cloud](https://civo.com). You can use [k3sup (ketchup)](https://www.civo.com/learn/kubernetes-on-civo-in-5-minutes-flat) to deploy Kubernetes on your own Instances within a few minutes. If you have access to the #Kube100 program, then you can use the managed k3s service. 52 | 53 | * [helm](https://helm.sh) - a packaging tool used to install cert-manager and docker-registry 54 | Some developers have concerns about using helm's server-side component called `tiller`. Rest assured, you can use the `helm template` command to avoid installing `tiller` if it bothers you. 55 | 56 | * [cert-manager](https://github.com/jetstack/cert-manager) - a tool by [JetStack](https://jetstack.io/) which provides and renews TLS certificates from [LetsEncrypt](https://letsencrypt.org). 57 | 58 | * [docker-registry](https://hub.docker.com/_/registry) - This is a helm chart for Docker's own open source registry 59 | 60 | * [nginx-ingress](https://kubernetes.github.io/ingress-nginx/) - The Nginx IngressController configures instances of [Nginx](https://nginx.com) to handle incoming HTTP/s traffic. 61 | 62 | > Note: If you are using k3s, you can skip installing Nginx IngressController 63 | 64 | ### Tutorial 65 | 66 | We'll first install helm, then tiller, then Kubernetes users can add [Nginx](https://nginx.com/) in *Host* mode and k3s users can skip this because they will be using [Traefik](https://traefik.io). After that we'll add cert-manager and an Issuer to obtain certificates, followed by the registry. After everything is installed, we can then make use of our registry using the password created during the tutorial. You'll finish off by testing everything end-to-end, and if you get stuck, there are some helpful tips on how to troubleshoot. 67 | 68 | Some components are installed in their own namespaces such as cert-manager, all others will be installed into the `default` namespace. You can control the namespace with `kubectl get --namespace/-n NAME` or `kubectl get --all-namespaces/-A`. 69 | 70 | There will also be some ways to take the tutorial further in the appendix. 71 | 72 | #### Install the helm CLI/client 73 | 74 | Instructions for latest Helm install 75 | 76 | * On MacOS and Linux: 77 | 78 | curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash 79 | 80 | * Or via Homebrew on Mac: 81 | 82 | brew install kubernetes-helm 83 | 84 | For Windows users, go to [helm.sh](https://helm.sh). 85 | 86 | #### Install tiller 87 | 88 | * Create RBAC permissions for tiller 89 | 90 | ```sh 91 | kubectl -n kube-system create sa tiller \ 92 | && kubectl create clusterrolebinding tiller \ 93 | --clusterrole cluster-admin \ 94 | --serviceaccount=kube-system:tiller 95 | ``` 96 | 97 | * Install the server-side Tiller component on your cluster 98 | 99 | ```sh 100 | helm init --skip-refresh --upgrade --service-account tiller 101 | ``` 102 | 103 | > Note: this step installs a server component in your cluster. It can take anywhere between a few seconds to a few minutes to be installed properly. You should see tiller appear on: `kubectl get pods -n kube-system`. 104 | 105 | * Now wait for tiller to become ready: 106 | 107 | ```sh 108 | kubectl rollout status -n kube-system deploy/tiller-deploy 109 | 110 | deployment "tiller-deploy" successfully rolled out 111 | ``` 112 | 113 | #### Your built-in IngressController with k3s 114 | 115 | Given that neither Civo's k3s service nor k3sup offer a cloud LoadBalancer, we need to use an IngressController. Fortunately k3s comes with one called [Traefik](https://traefik.io/). 116 | 117 | For k3s, don't install an IngressController, you already have one, skip ahead. 118 | 119 | #### Add an IngressController if not using k3s 120 | 121 | **If you're not using k3s**, then install Nginx Ingress instead: 122 | 123 | ``` 124 | helm install stable/nginx-ingress --name nginxingress --set rbac.create=true,controller.hostNetwork=true,controller.daemonset.useHostPort=true,dnsPolicy=ClusterFirstWithHostNet,controller.kind=DaemonSet 125 | ``` 126 | 127 | #### Install cert-manager 128 | 129 | You can now install cert-manager, the version used is v0.9.1. 130 | 131 | ```sh 132 | # Install the CustomResourceDefinition resources separately 133 | kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.9/deploy/manifests/00-crds.yaml 134 | 135 | # Create the namespace for cert-manager 136 | kubectl create namespace cert-manager 137 | 138 | # Label the cert-manager namespace to disable resource validation 139 | kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true 140 | 141 | # Add the Jetstack Helm repository 142 | helm repo add jetstack https://charts.jetstack.io 143 | 144 | # Update your local Helm chart repository cache 145 | helm repo update 146 | 147 | # Install the cert-manager Helm chart 148 | helm install \ 149 | --name cert-manager \ 150 | --namespace cert-manager \ 151 | --version v0.9.1 \ 152 | jetstack/cert-manager 153 | 154 | ``` 155 | 156 | See also: [cert-manager v0.9.0 docs](https://docs.cert-manager.io/en/release-0.9/) 157 | 158 | #### Create a ClusterIssuer 159 | 160 | The way that cert-manager issues certificates is through an [Issuer](https://docs.cert-manager.io/en/release-0.9/tutorials/acme/http-validation.html). The `Issuer` can issue certificates for the namespace it is created in, but a `ClusterIssuer` can create certificates for any namespace, so that's the one we will use today. 161 | 162 | Save `issuer.yaml`: 163 | 164 | ```yaml 165 | apiVersion: certmanager.k8s.io/v1alpha1 166 | kind: ClusterIssuer 167 | metadata: 168 | name: letsencrypt-prod 169 | namespace: default 170 | spec: 171 | acme: 172 | # The ACME server URL 173 | server: https://acme-v02.api.letsencrypt.org/directory 174 | # Email address used for ACME registration 175 | email: user@example.com 176 | # Name of a secret used to store the ACME account private key 177 | privateKeySecretRef: 178 | name: letsencrypt-prod 179 | # Enable the HTTP-01 challenge provider 180 | solvers: 181 | - http01: 182 | ingress: 183 | class: traefik 184 | ``` 185 | 186 | * Edit the file: 187 | 188 | Edit the line: `email: user@example.com`. 189 | 190 | If using Nginx instead of k3s and Traefik, then edit the following: 191 | 192 | ``` 193 | solvers: 194 | - http01: 195 | ingress: 196 | class: nginx 197 | ``` 198 | 199 | Then run `kubectl apply -f issuer.yaml`. 200 | 201 | > Note you may receive an error, if you do then wait 1-2 minutes and try again whilst cert-manager registers itself 202 | 203 | You can check the status of your issuer like this: 204 | 205 | ``` 206 | kubectl describe clusterissuer/letsencrypt-prod 207 | ``` 208 | 209 | Look for it to become `Ready`. 210 | 211 | #### Configure DNS 212 | 213 | For this tutorial a domain `on-k3s.dev` was purchased from Google Domains to show a full worked example. 214 | 215 | ![](/images/buy-dns.png) 216 | 217 | Once you have purchased your domain, you need to point the DNS records at the hosts in the k3s cluster where Nginx is going to be listening on port 80 (HTTP) and port 443 (HTTPS/TLS). 218 | 219 | ![](/images/add-dns.png) 220 | 221 | You can find your IP addresses with the Civo UI, or by typing in `civo instance ls` through the CLI. 222 | 223 | #### Install the registry 224 | 225 | At this stage we can install the registry, but we are going to install it without persistence. If you need persistence see the appendix for how to do this. 226 | 227 | Save the following as `install-registry.sh`: 228 | 229 | ```sh 230 | export SHA=$(head -c 16 /dev/urandom | shasum | cut -d " " -f 1) 231 | export USER=admin 232 | 233 | echo $USER > registry-creds.txt 234 | echo $SHA >> registry-creds.txt 235 | 236 | docker run --entrypoint htpasswd registry:2 -Bbn admin $SHA > ./htpasswd 237 | 238 | helm install stable/docker-registry \ 239 | --name private-registry \ 240 | --namespace default \ 241 | --set persistence.enabled=false \ 242 | --set secrets.htpasswd=$(cat ./htpasswd) 243 | ``` 244 | 245 | You will need to have `docker` installed and ready for this step. If it's not started, then start it up now. 246 | 247 | Then run the script: 248 | 249 | ``` 250 | chmod +x install-registry.sh 251 | ./install-registry.sh 252 | ``` 253 | 254 | It will install the Docker registry from [the docker-registry](https://github.com/helm/charts/tree/master/stable/docker-registry) chart. 255 | 256 | Later, when you want to use your registry you can find your username and password in the `registry-creds.txt` file. 257 | 258 | #### Get a TLS certificate for the registry 259 | 260 | Now let's get a TLS certificate for the registry. 261 | 262 | Save `ingress.yaml`, then edit it: 263 | 264 | ```yaml 265 | --- 266 | apiVersion: extensions/v1beta1 267 | kind: Ingress 268 | metadata: 269 | name: registry 270 | namespace: default 271 | annotations: 272 | certmanager.k8s.io/cluster-issuer: letsencrypt-prod 273 | kubernetes.io/ingress.class: "traefik" 274 | nginx.ingress.kubernetes.io/proxy-body-size: 50m 275 | labels: 276 | app: docker-registry 277 | spec: 278 | tls: 279 | - hosts: 280 | - registry.example.com 281 | secretName: registry.example.com-cert 282 | rules: 283 | - host: registry.example.com 284 | http: 285 | paths: 286 | - path: / 287 | backend: 288 | serviceName: private-registry-docker-registry 289 | servicePort: 5000 290 | ``` 291 | 292 | Update the file: 293 | 294 | * Everywhere that you see `registry.example.com`, replace it for your domain. 295 | * If using Nginx, then change this line: `kubernetes.io/ingress.class:` 296 | 297 | Note the special setting: `.ingress.kubernetes.io/proxy-body-size: 50m`. This value can be customized and allows large Docker images to be stored in the registry. 298 | 299 | Now run: 300 | 301 | ```sh 302 | kubectl apply -f ingress.yaml 303 | ``` 304 | 305 | ### Check the certificate 306 | 307 | Now check the certificate with the following: 308 | 309 | ``` 310 | kubectl get cert -n default 311 | 312 | NAME READY SECRET AGE 313 | registry.on-k3s.dev-cert True registry.on-k3s.dev-cert 47s 314 | ``` 315 | 316 | For any of the entries listed, you can check the status with `kubectl describe`: 317 | 318 | ``` 319 | kubectl describe cert/registry.on-k3s.dev-cert 320 | 321 | Status: 322 | Conditions: 323 | Last Transition Time: 2019-08-29T13:26:20Z 324 | Message: Certificate is up to date and has not expired 325 | Reason: Ready 326 | Status: True 327 | Type: Ready 328 | Not After: 2019-11-27T12:26:18Z 329 | Events: 330 | Type Reason Age From Message 331 | ---- ------ ---- ---- ------- 332 | Normal Generated 64s cert-manager Generated new private key 333 | Normal GenerateSelfSigned 64s cert-manager Generated temporary self signed certificate 334 | Normal OrderCreated 63s cert-manager Created Order resource "registry.on-k3s.dev-cert-3194477141" 335 | Normal OrderComplete 31s cert-manager Order "registry.on-k3s.dev-cert-3194477141" completed successfully 336 | Normal CertIssued 31s cert-manager Certificate issued successfully 337 | ``` 338 | 339 | Look for hints in the *Status* and *Events* sections. 340 | 341 | ### Now let's test the registry 342 | 343 | ``` 344 | export DOCKER_PASSWORD="" # Populate this with your password used above 345 | export DOCKER_USERNAME="admin" 346 | export SERVER="registry.example.com" 347 | 348 | echo $DOCKER_PASSWORD | docker login $SERVER --username $DOCKER_USERNAME --password-stdin 349 | ``` 350 | 351 | > Replace `example.com` with your domain 352 | 353 | Sometimes it can take a few minutes for your new domain to become available. If it's an existing domain, then the DNS record should be synchronised already. 354 | 355 | Once logged in, you can tag an image from the Docker Hub and push it into your own registry. 356 | 357 | ```sh 358 | export SERVER="registry.example.com" 359 | 360 | docker pull functions/figlet:latest 361 | 362 | docker tag functions/figlet:latest $SERVER/functions/figlet:latest 363 | 364 | docker push $SERVER/functions/figlet:latest 365 | ``` 366 | 367 | Now that we can log into our registry and push images, we need to enable the same from within our cluster. This is done by attaching an image pull secret to the namespace's service account. 368 | 369 | ```sh 370 | export DOCKER_PASSWORD="" # Populate this with your password used above 371 | export DOCKER_USERNAME="admin" 372 | 373 | kubectl create secret docker-registry my-private-repo \ 374 | --docker-username=$DOCKER_USERNAME \ 375 | --docker-password=$DOCKER_PASSWORD \ 376 | --docker-server=$SERVER \ 377 | --namespace default 378 | ``` 379 | 380 | Now edit the service account and grant it permission to access the secret: 381 | 382 | ```sh 383 | kubectl edit serviceaccount default -n default 384 | ``` 385 | 386 | Add the following and save: 387 | 388 | ```yaml 389 | imagePullSecrets: 390 | - name: my-private-repo 391 | ``` 392 | 393 | To check that it's available in Kubernetes, you can run the following [OpenFaaS function](https://github.com/openfaas/faas/), which prints an ASCII logo and then exits. 394 | 395 | ```sh 396 | export SERVER="" 397 | 398 | kubectl run --rm -t -i figlet --restart Never --image $SERVER/functions/figlet:latest -- figlet Kubernetes 399 | 400 | _ __ _ _ 401 | | |/ / _| |__ ___ _ __ _ __ ___| |_ ___ ___ 402 | | ' / | | | '_ \ / _ \ '__| '_ \ / _ \ __/ _ \/ __| 403 | | . \ |_| | |_) | __/ | | | | | __/ || __/\__ \ 404 | |_|\_\__,_|_.__/ \___|_| |_| |_|\___|\__\___||___/ 405 | 406 | pod "figlet" deleted 407 | ``` 408 | 409 | This will print out the Kubernetes logo in ASCII art and then delete the Pod used to run the code. 410 | 411 | If it didn't work, find out why with this command: 412 | 413 | ``` 414 | kubectl get events --sort-by=.metadata.creationTimestamp -n default 415 | ``` 416 | 417 | ### Take things further (appendix) 418 | 419 | You can take things further and start to explore more advanced use-cases for your registry. 420 | 421 | #### Enable persistence 422 | 423 | It is desirable, but not essential to enable persistence for a registry. When available, persistence means that if the registry crashes, then the images can be recovered. 424 | 425 | There are two routes to enable persistence. 426 | 427 | * Use S3, or S3-compatible buckets 428 | S3 is a protocol and standard for storing objects. You can use [an AWS account](https://aws.amazon.com/s3/) and S3 as a backing for your registry's storage, or you can [install Minio onto your Civo instances](https://www.civo.com/learn/back-up-your-data-using-restic-minio-and-civo) and use it as an S3 target instead. 429 | 430 | * Use PersistentVolumes in Kubernetes 431 | Storage in Kubernetes comes in the shape of Volumes. When volumes are not ephemeral, then they are called PersistentVolumes or (PVs). In order to use PVs with k3s, you'll have to install [Rancher's Longhorn project](https://www.civo.com/learn/cloud-native-stateful-storage-for-kubernetes-with-rancher-labs-longhorn). 432 | 433 | The helm chart explains the options for using PVs or S3: [docker-registry chart](https://github.com/helm/charts/tree/master/stable/docker-registry). 434 | 435 | ## Wrapping up 436 | 437 | We have now built a Kubernetes cluster using k3s and have a working registry with TLS, authentication and a public URL. 438 | 439 | * helm provided us with charts (packaged software for Kubernetes) 440 | * docker-registry gave us a registry with authentication 441 | * cert-manager provided TLS certificates from LetsEncrypt 442 | * Traefik was built into k3s, or we used Nginx on upstream Kubernetes. 443 | 444 | You can now share the registry with your team or use it in your CI/CD pipeline using a tool like Jenkins to build and ship Docker images. You may like to try installing other software to start building applications on Kubernetes such as [OpenFaaS](https://www.civo.com/learn/deploy-openfaas-with-k3s-on-civo). 445 | -------------------------------------------------------------------------------- /images/add-dns.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alexellis/k8s-tls-registry/894e1248769cb7b3d95866ff81ad3beaa97fde40/images/add-dns.png -------------------------------------------------------------------------------- /images/buy-dns.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alexellis/k8s-tls-registry/894e1248769cb7b3d95866ff81ad3beaa97fde40/images/buy-dns.png -------------------------------------------------------------------------------- /images/registry.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alexellis/k8s-tls-registry/894e1248769cb7b3d95866ff81ad3beaa97fde40/images/registry.png --------------------------------------------------------------------------------