├── .gitignore ├── kustomize ├── base │ ├── podinfo │ │ ├── namespace │ │ │ └── namespace.yaml │ │ ├── kustomization.yaml │ │ └── helmrelease.yaml │ ├── cluster │ │ ├── kustomization.yaml │ │ └── namespaces │ │ │ └── demo │ │ │ └── namespace.yaml │ └── helm-operators │ │ └── common │ │ ├── kustomization.yaml │ │ ├── demo.yaml │ │ ├── default.yaml │ │ ├── podinfo.yaml │ │ └── custom-repositories.yaml ├── .flux.yaml └── dev │ ├── cluster │ └── kustomization.yaml │ ├── helm-version-v3.yaml │ ├── helm-operators │ └── kustomization.yaml │ ├── podinfo │ ├── kustomization.yaml │ └── helmrelease.yaml │ ├── flux-patch.yaml │ └── kustomization.yaml ├── ci ├── kustomization-without-flux-patch-template.yaml └── kustomization-with-flux-patch-template.yaml ├── bin ├── deprek8s-check ├── kubeval-each-environment ├── check-duplicate-release-name └── kubeval-helmreleases-for-environment ├── docs ├── configuration.md ├── pre-reqs.md ├── automated-image-upgrades.md ├── kustomize-setup.md └── ci.md ├── .github └── workflows │ └── kustomize-diff.yml ├── Makefile ├── README.md ├── .circleci └── config.yml └── scripts └── flux-init.sh /.gitignore: -------------------------------------------------------------------------------- 1 | _test/ 2 | gitops.yaml 3 | temp/ 4 | -------------------------------------------------------------------------------- /kustomize/base/podinfo/namespace/namespace.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Namespace 3 | metadata: 4 | name: podinfo 5 | -------------------------------------------------------------------------------- /kustomize/.flux.yaml: -------------------------------------------------------------------------------- 1 | version: 1 2 | patchUpdated: 3 | generators: 4 | - command: kustomize build . 5 | patchFile: flux-patch.yaml 6 | -------------------------------------------------------------------------------- /ci/kustomization-without-flux-patch-template.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: kustomize.config.k8s.io/v1beta1 2 | kind: Kustomization 3 | resources: 4 | - kustomize/$ENV 5 | -------------------------------------------------------------------------------- /kustomize/base/cluster/kustomization.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: kustomize.config.k8s.io/v1beta1 2 | kind: Kustomization 3 | resources: 4 | - namespaces/demo/namespace.yaml 5 | -------------------------------------------------------------------------------- /kustomize/dev/cluster/kustomization.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: kustomize.config.k8s.io/v1beta1 2 | kind: Kustomization 3 | resources: 4 | - ../../../kustomize/base/cluster 5 | -------------------------------------------------------------------------------- /kustomize/dev/helm-version-v3.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: helm.fluxcd.io/v1 3 | kind: HelmRelease 4 | metadata: 5 | name: ignored-but-required 6 | spec: 7 | helmVersion: v3 -------------------------------------------------------------------------------- /kustomize/base/podinfo/kustomization.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: kustomize.config.k8s.io/v1beta1 2 | kind: Kustomization 3 | resources: 4 | - namespace/namespace.yaml 5 | - helmrelease.yaml 6 | -------------------------------------------------------------------------------- /kustomize/base/cluster/namespaces/demo/namespace.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Namespace 3 | metadata: 4 | annotations: 5 | iam.amazonaws.com/permitted: '.*' 6 | name: demo 7 | -------------------------------------------------------------------------------- /kustomize/dev/helm-operators/kustomization.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: kustomize.config.k8s.io/v1beta1 2 | kind: Kustomization 3 | resources: 4 | - ../../../kustomize/base/helm-operators/common 5 | -------------------------------------------------------------------------------- /ci/kustomization-with-flux-patch-template.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: kustomize.config.k8s.io/v1beta1 2 | kind: Kustomization 3 | resources: 4 | - kustomize/$ENV 5 | patchesStrategicMerge: 6 | - kustomize/$ENV/flux-patch.yaml 7 | -------------------------------------------------------------------------------- /kustomize/dev/podinfo/kustomization.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: kustomize.config.k8s.io/v1beta1 2 | kind: Kustomization 3 | patchesStrategicMerge: 4 | - helmrelease.yaml 5 | resources: 6 | - ../../../kustomize/base/podinfo 7 | -------------------------------------------------------------------------------- /kustomize/dev/flux-patch.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: helm.fluxcd.io/v1 3 | kind: HelmRelease 4 | metadata: 5 | name: podinfo 6 | namespace: podinfo 7 | spec: 8 | values: 9 | image: 10 | tag: 4.0.6 11 | -------------------------------------------------------------------------------- /kustomize/base/helm-operators/common/kustomization.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: kustomize.config.k8s.io/v1beta1 2 | kind: Kustomization 3 | resources: 4 | - demo.yaml 5 | - default.yaml 6 | - podinfo.yaml 7 | patches: 8 | - path: custom-repositories.yaml 9 | target: 10 | kind: HelmRelease 11 | -------------------------------------------------------------------------------- /kustomize/dev/kustomization.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: kustomize.config.k8s.io/v1beta1 2 | kind: Kustomization 3 | resources: 4 | - cluster 5 | - helm-operators 6 | - podinfo 7 | # Patch all helm release with to `spec.helmVersion: v3` 8 | patches: 9 | - path: helm-version-v3.yaml 10 | target: 11 | kind: HelmRelease 12 | -------------------------------------------------------------------------------- /bin/deprek8s-check: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | set -euo pipefail 4 | 5 | mkdir -p /tmp/kustomize 6 | 7 | for env in kustomize/*; do 8 | if [ $env == 'kustomize/base' ]; then continue ; fi 9 | 10 | printf "\nChecking %s\n" "${env#*/}" 11 | 12 | kustomize build ${env} > /tmp/${env}.yaml 13 | 14 | conftest test -p /policies/deprecations /tmp/${env}.yaml 15 | 16 | done 17 | -------------------------------------------------------------------------------- /kustomize/base/podinfo/helmrelease.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: helm.fluxcd.io/v1 3 | kind: HelmRelease 4 | metadata: 5 | annotations: 6 | fluxcd.io/automated: "false" 7 | name: podinfo 8 | namespace: podinfo 9 | spec: 10 | chart: 11 | name: podinfo 12 | repository: https://stefanprodan.github.io/podinfo 13 | version: 4.0.6 14 | releaseName: podinfo 15 | values: 16 | image: 17 | tag: 4.0.3 -------------------------------------------------------------------------------- /kustomize/dev/podinfo/helmrelease.yaml: -------------------------------------------------------------------------------- 1 | # For more on image automation see https://docs.fluxcd.io/en/stable/references/helm-operator-integration/#annotations 2 | 3 | --- 4 | apiVersion: helm.fluxcd.io/v1 5 | kind: HelmRelease 6 | metadata: 7 | annotations: 8 | fluxcd.io/automated: "true" 9 | filter.fluxcd.io/chart-image: semver:~4.0 10 | name: podinfo 11 | namespace: podinfo 12 | spec: 13 | values: 14 | image: 15 | repository: stefanprodan/podinfo 16 | tag: 4.0.3 17 | -------------------------------------------------------------------------------- /bin/kubeval-each-environment: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | set -euo pipefail 4 | 5 | mkdir -p /tmp/kustomize 6 | 7 | for env in kustomize/*; do 8 | if [ $env == 'kustomize/base' ]; then continue ; fi 9 | 10 | printf "\nChecking %s\n" "${env#*/}" 11 | 12 | kustomize build ${env} > /tmp/${env}.yaml 13 | 14 | export KUBEVAL_SCHEMA_LOCATION=file:///usr/local/kubeval/schemas 15 | 16 | kubeval --kubernetes-version 1.17.0 --strict --force-color --ignore-missing-schemas /tmp/${env}.yaml 17 | 18 | done 19 | -------------------------------------------------------------------------------- /docs/configuration.md: -------------------------------------------------------------------------------- 1 | # Configuration 2 | 3 | The configuration of Flux is handled in `scripts/flux-init.sh`. 4 | 5 | There are two variables of interest, these are: 6 | 7 | ``` 8 | REPO_URL=${1:-git@github.com:swade1987/gitops-with-kustomize} 9 | REPO_BRANCH=master 10 | ``` 11 | 12 | The above needs to be updated to point to your repository, and the branch you want Flux to reconcile against. 13 | 14 | Finally, an important flag to note within the Flux installation is the following: 15 | 16 | ``` 17 | --set manifestGeneration=true \ 18 | ``` 19 | 20 | The above is required to be used as we are using `Kustomize`. -------------------------------------------------------------------------------- /bin/check-duplicate-release-name: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | set -euo pipefail 4 | 5 | EXIT_CODE=0 6 | 7 | for env in kustomize/*; do 8 | if [ $env == 'kustomize/base' ]; then continue ; fi 9 | printf "\nChecking %s\n" "${env#*/}" 10 | dupes=$( 11 | cd ${env} ; \ 12 | kustomize build . | \ 13 | yq r - -d'*' spec.releaseName | \ 14 | grep -v ^\-\ null | \ 15 | sed 's/^-\ //g' | \ 16 | sort | \ 17 | uniq -d 18 | ) 19 | 20 | if [[ $(wc -w <<< "$dupes") == 0 ]]; then 21 | echo "OK" 22 | else 23 | echo " Duplicates found:" 24 | printf " %s" "$dupes" 25 | EXIT_CODE=1 26 | fi 27 | done 28 | 29 | exit $EXIT_CODE 30 | -------------------------------------------------------------------------------- /docs/pre-reqs.md: -------------------------------------------------------------------------------- 1 | # Pre-requisites 2 | 3 | ## k3d 4 | 5 | The demo uses [rancher/k3d](https://github.com/rancher/k3d) to spin up a cluster locally. 6 | 7 | The installation guide for `k3d` can be found [here](https://github.com/rancher/k3d#get). 8 | 9 | ## Kustomize 10 | 11 | To validate that your Kustomization's are valid it is recommended to have `Kustomize` installed. 12 | 13 | The installation guide for this can be found [here](https://github.com/kubernetes-sigs/kustomize). 14 | 15 | ## Helm 3 16 | 17 | Flux is installed using [Helm 3](https://helm.sh/blog/helm-3-released/). 18 | 19 | The installation guide for this can be found [here](https://helm.sh/docs/intro/install/). -------------------------------------------------------------------------------- /.github/workflows/kustomize-diff.yml: -------------------------------------------------------------------------------- 1 | name: kustomize-diff 2 | on: 3 | pull_request: 4 | paths: 5 | - 'kustomize/**' 6 | 7 | jobs: 8 | kustomize-diff: 9 | runs-on: ubuntu-latest 10 | steps: 11 | - id: checkout 12 | uses: actions/checkout@v2 13 | with: 14 | fetch-depth: 0 15 | - id: kustomize-diff 16 | uses: eeveebank/github-action-kustomize-diff@master 17 | - id: comment 18 | uses: unsplash/comment-on-pr@master 19 | env: 20 | GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} 21 | with: 22 | msg: ${{ steps.kustomize-diff.outputs.diff }} 23 | check_for_duplicate_msg: false 24 | -------------------------------------------------------------------------------- /kustomize/base/helm-operators/common/demo.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: helm.fluxcd.io/v1 3 | kind: HelmRelease 4 | metadata: 5 | annotations: 6 | flux.weave.works/automated: "false" 7 | name: demo-helm-operator 8 | namespace: flux 9 | spec: 10 | chart: 11 | name: helm-operator 12 | repository: https://charts.fluxcd.io 13 | version: 1.0.2 14 | releaseName: demo-helm-operator 15 | targetNamespace: demo 16 | values: 17 | image: 18 | repository: docker.io/fluxcd/helm-operator 19 | tag: 1.1.0 20 | allowNamespace: demo 21 | helm: 22 | versions: "v3" 23 | rbac: 24 | create: true 25 | clusterRole: 26 | create: true 27 | prometheus: 28 | enabled: true 29 | -------------------------------------------------------------------------------- /kustomize/base/helm-operators/common/default.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: helm.fluxcd.io/v1 3 | kind: HelmRelease 4 | metadata: 5 | annotations: 6 | flux.weave.works/automated: "false" 7 | name: default-helm-operator 8 | namespace: flux 9 | spec: 10 | chart: 11 | name: helm-operator 12 | repository: https://charts.fluxcd.io 13 | version: 1.0.2 14 | releaseName: default-helm-operator 15 | targetNamespace: default 16 | values: 17 | image: 18 | repository: docker.io/fluxcd/helm-operator 19 | tag: 1.1.0 20 | allowNamespace: default 21 | helm: 22 | versions: "v3" 23 | rbac: 24 | create: true 25 | clusterRole: 26 | create: true 27 | prometheus: 28 | enabled: true 29 | -------------------------------------------------------------------------------- /kustomize/base/helm-operators/common/podinfo.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: helm.fluxcd.io/v1 3 | kind: HelmRelease 4 | metadata: 5 | annotations: 6 | flux.weave.works/automated: "false" 7 | name: podinfo-helm-operator 8 | namespace: flux 9 | spec: 10 | chart: 11 | name: helm-operator 12 | repository: https://charts.fluxcd.io 13 | version: 1.0.2 14 | releaseName: podinfo-helm-operator 15 | targetNamespace: podinfo 16 | values: 17 | image: 18 | repository: docker.io/fluxcd/helm-operator 19 | tag: 1.1.0 20 | allowNamespace: podinfo 21 | helm: 22 | versions: "v3" 23 | rbac: 24 | create: true 25 | clusterRole: 26 | create: true 27 | prometheus: 28 | enabled: true 29 | -------------------------------------------------------------------------------- /kustomize/base/helm-operators/common/custom-repositories.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: helm.fluxcd.io/v1 3 | kind: HelmRelease 4 | metadata: 5 | name: ignored-but-required 6 | spec: 7 | values: 8 | configureRepositories: 9 | enable: true 10 | repositories: 11 | - name: bitnami 12 | url: https://charts.bitnami.com 13 | - name: concourse 14 | url: https://concourse-charts.storage.googleapis.com 15 | - name: flux 16 | url: https://charts.fluxcd.io 17 | - name: jetstack 18 | url: https://charts.jetstack.io 19 | - name: podinfo 20 | url: https://stefanprodan.github.io/podinfo 21 | - name: stable 22 | url: https://kubernetes-charts.storage.googleapis.com 23 | -------------------------------------------------------------------------------- /bin/kubeval-helmreleases-for-environment: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | ENV=$1 4 | IGNORE_VALUES=false 5 | KUBE_VERSION=1.17.0 6 | HELM_VERSION=v3 7 | FLUX_PATCH_LOCATION=kustomize/${ENV}/flux-patch.yaml 8 | set -eu 9 | 10 | printf "Running hrval against all %s HelmReleases\n" "${ENV}" 11 | 12 | mkdir -p /tmp/"${ENV}" 13 | 14 | if [ -s "${FLUX_PATCH_LOCATION}" ]; then 15 | echo "Flux patch found @ ${FLUX_PATCH_LOCATION}" 16 | sed "s/\$ENV/${ENV}/" ci/kustomization-with-flux-patch-template.yaml > ./kustomization.yaml 17 | else 18 | echo "No flux patch found @ ${FLUX_PATCH_LOCATION}" 19 | sed "s/\$ENV/${ENV}/" ci/kustomization-without-flux-patch-template.yaml > ./kustomization.yaml 20 | fi 21 | 22 | 23 | kustomize build . -o /tmp/"${ENV}" 24 | rm kustomization.yaml 25 | hrval /tmp/"${ENV}"/ ${IGNORE_VALUES} ${KUBE_VERSION} ${HELM_VERSION} 26 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | CURRENT_WORKING_DIR=$(shell pwd) 2 | TOOLKIT_IMAGE := eu.gcr.io/swade1987/kubernetes-toolkit:1.17.2 3 | 4 | init: 5 | brew install k3d 6 | 7 | cluster: 8 | k3d cluster create --agents 2 gitops 9 | 10 | destroy: 11 | k3d cluster delete gitops 12 | 13 | install-flux: 14 | ./scripts/flux-init.sh 15 | 16 | check-duplicate-release-name: 17 | clear 18 | docker run --rm --name check-duplicate-release-name -v $(CURRENT_WORKING_DIR)/kustomize:/kustomize $(TOOLKIT_IMAGE) bash -c "`cat bin/check-duplicate-release-name`" 19 | 20 | kubeval-environments: 21 | clear 22 | docker run --rm --name kubeval-environments -v $(CURRENT_WORKING_DIR)/kustomize:/kustomize $(TOOLKIT_IMAGE) bash -c "`cat bin/kubeval-each-environment`" 23 | 24 | hrval-environments: 25 | clear 26 | docker run --rm --name hrval-environments -v $(CURRENT_WORKING_DIR)/kustomize:/kustomize $(TOOLKIT_IMAGE) bash -c "`cat bin/kubeval-each-environment`" 27 | 28 | deprek8-check: 29 | clear 30 | docker run --rm --name kubeval-charts -v $(CURRENT_WORKING_DIR)/kustomize:/kustomize $(TOOLKIT_IMAGE) bash -c "`cat bin/deprek8s-check`" 31 | 32 | kustomization-yaml-fix: 33 | docker run --rm --name kustomization-yaml-fix -v $(CURRENT_WORKING_DIR)/kustomize:/kustomize $(TOOLKIT_IMAGE) kustomization-yaml-fix /kustomize 34 | 35 | test-%: 36 | mkdir -p _test 37 | kustomize build kustomize/$* > _test/$*.yaml 38 | @echo 39 | @echo The output can be found at _test/$*.yaml 40 | -------------------------------------------------------------------------------- /docs/automated-image-upgrades.md: -------------------------------------------------------------------------------- 1 | # Automated Image Upgrades 2 | 3 | ## Base 4 | 5 | In the base directory we set the `podinfo` HelmRelease to have automatic image upgrades turned off. 6 | 7 | This can be seen from the following annotations: 8 | 9 | ``` 10 | fluxcd.io/automated: "false" 11 | ``` 12 | 13 | ## Environment specific 14 | 15 | For us to enable automatic image upgrades we need to uncomment the following annotations from the `dev` specific configuration. 16 | 17 | ``` 18 | fluxcd.io/automated: "true" 19 | filter.fluxcd.io/chart-image: semver:~4.0 20 | ``` 21 | 22 | Now when flux next reconciles it and looks for new images in the image registry it will find the latest one and upgrade the deployment. 23 | 24 | An example of this can be seen below in the flux logs. 25 | 26 | ``` 27 | ts=2020-08-09T14:50:02.657447155Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads" 28 | ts=2020-08-09T14:50:02.780931146Z caller=images.go:111 component=sync-loop workload=podinfo:helmrelease/podinfo container=chart-image repo=stefanprodan/podinfo pattern=semver:~4.0 current=stefanprodan/podinfo:4.0.3 info="added update to automation run" new=stefanprodan/podinfo:4.0.6 reason="latest 4.0.6 (2020-06-26 10:47:57.011325031 +0000 UTC) > current 4.0.3 (2020-06-06 11:37:05.183233795 +0000 UTC)" 29 | ``` 30 | 31 | ## Writing back to Git 32 | 33 | After the upgrade has been successful you will see a new commit in GitHub showing the change. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # GitOps (using Kustomize) 2 | 3 | An example repo structure for GitOps with: 4 | 5 | - [Flux](https://github.com/fluxcd/flux) 6 | - [Helm Operator](https://github.com/fluxcd/helm-operator) 7 | - [Kustomize](https://github.com/kubernetes-sigs/kustomize) 8 | 9 | ## Directory structure 10 | 11 | ``` 12 | kustomize 13 | ├── base # base kustomizations 14 | │  ├── cluster # logical grouping of resources 15 | ├── dev # Directory per env which pulls from base and extends/overrides helmreleases 16 | ``` 17 | 18 | Resources are organised per environment in the `kustomize` directory. 19 | 20 | ## Pre-requisites 21 | 22 | A list of pre-requisites can be found [here](docs/pre-reqs.md). 23 | 24 | ## Setup 25 | 26 | 1. To configure this to work with your repository first read the steps [here](docs/configuration.md). 27 | 28 | 2. Create a cluster using `make cluster` 29 | 30 | 3. Install [Flux](https://github.com/fluxcd/flux) and the [Helm Operator](https://github.com/fluxcd/helm-operator) using `make install-flux` 31 | 32 | 4. After following the prompts, flux will establish a connection to your repository and start reconciling. 33 | 34 | ## Continuous Integration 35 | 36 | A deep-dive into running checks locally and the CircleCI configuration, read [here](docs/ci.md). 37 | 38 | ## Kustomize directory structure 39 | 40 | A deep-dive into the kustomize setup can be found [here](docs/kustomize-setup.md). 41 | 42 | ## Automatic image upgrades 43 | 44 | An example of automated image upgrades with `HelmReleases` resources can be found [here](docs/automated-image-upgrades.md). -------------------------------------------------------------------------------- /docs/kustomize-setup.md: -------------------------------------------------------------------------------- 1 | # Kustomize setup 2 | 3 | The following explains in depth the reasoning behind the structure of the `kustomize` directory. 4 | 5 | ## kustomize/base 6 | 7 | The base directory contains all non-environment specific configuration. This could be: 8 | 9 | - Any manifests which are deployed to every environment (e.g. PSPs, Storage Classes, RBAC etc) 10 | - `HelmRelease` manifests without the environment specific details 11 | 12 | ### kustomize/base/cluster 13 | 14 | This directory should contain all the default cluster resources, examples include: 15 | 16 | - Default namespaces 17 | - Priority classes 18 | - Default Pod Security Policies 19 | - Default RBAC permissions 20 | - Storage classes 21 | 22 | In this demo repository its only a subset of these things. 23 | 24 | ### kustomize/base/helm-operators 25 | 26 | This directory is a directory to contain all Helm Operators you may want to deploy to a given cluster. 27 | 28 | However, there is an important thing to note within their configuration. 29 | 30 | The `HelmRelease` manifests themselves are deployed to the `flux` namespace to be reconciled by the default Helm Operator running within that namespace. 31 | 32 | However, each of the Helm Operator pods are actually deployed to specific namespaces and are scoped to only reconcile `HelmRelease` resources in their namespace. 33 | 34 | This is made possible by the following: 35 | 36 | ``` 37 | spec.targetNamespace: cert-manager 38 | spec.values.allowNamespace: cert-manager 39 | ``` 40 | 41 | This setup allows for faster reconciliation of `HelmRelease` manifests. 42 | 43 | For more information on `targetNamespace` I would recommend reading [this](https://docs.fluxcd.io/projects/helm-operator/en/1.0.0-rc9/references/helmrelease-custom-resource.html#helmrelease-custom-resource). -------------------------------------------------------------------------------- /.circleci/config.yml: -------------------------------------------------------------------------------- 1 | version: 2.1 2 | 3 | defaults: &defaults 4 | working_directory: ~/project 5 | docker: 6 | - image: eu.gcr.io/swade1987/kubernetes-toolkit:1.17.2 7 | 8 | jobs: 9 | 10 | kubeval: 11 | <<: *defaults 12 | steps: 13 | - checkout 14 | - setup_remote_docker 15 | - run: 16 | name: kubeval environments 17 | command: bash -c "`cat bin/kubeval-each-environment`" 18 | 19 | check-for-duplicate-helmrelease-names: 20 | <<: *defaults 21 | steps: 22 | - checkout 23 | - setup_remote_docker 24 | - run: 25 | name: check for duplicate helmreleases 26 | command: bash -c "`cat bin/check-duplicate-release-name`" 27 | 28 | deprek8s-check: 29 | <<: *defaults 30 | steps: 31 | - checkout 32 | - run: 33 | name: verify against deprecated API versions 34 | command: bash -c "`cat bin/deprek8s-check`" 35 | 36 | hrval-for-dev: 37 | <<: *defaults 38 | steps: 39 | - checkout 40 | - run: 41 | name: kubeval helmreleases for dev 42 | command: bash -c "bin/kubeval-helmreleases-for-environment dev" 43 | 44 | kustomization-yaml-check: 45 | <<: *defaults 46 | steps: 47 | - checkout 48 | - run: 49 | name: kustomize edit fix 50 | command: kustomization-yaml-fix kustomize 51 | - run: 52 | name: test git is clean 53 | command: test "$(git status --porcelain | wc -l)" = "0" 54 | 55 | workflows: 56 | version: 2 57 | flow: 58 | jobs: 59 | - kubeval 60 | - check-for-duplicate-helmrelease-names 61 | - deprek8s-check 62 | - kustomization-yaml-check 63 | - hrval-for-dev: 64 | filters: 65 | branches: 66 | ignore: 67 | - master -------------------------------------------------------------------------------- /scripts/flux-init.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -e 4 | 5 | if [[ ! -x "$(command -v kubectl)" ]]; then 6 | echo "kubectl not found" 7 | exit 1 8 | fi 9 | 10 | if [[ ! -x "$(command -v helm)" ]]; then 11 | echo "helm not found" 12 | exit 1 13 | fi 14 | 15 | if [[ ! -x "$(command -v fluxctl)" ]]; then 16 | echo "fluxctl not found" 17 | exit 1 18 | fi 19 | 20 | REPO_GIT_INIT_PATHS="kustomize/dev" 21 | REPO_ROOT=$(git rev-parse --show-toplevel) 22 | REPO_URL=${1:-git@github.com:swade1987/gitops-with-kustomize} 23 | REPO_BRANCH=master 24 | TEMP=${REPO_ROOT}/temp 25 | 26 | rm -rf ${TEMP} && mkdir ${TEMP} 27 | 28 | helm repo add fluxcd https://charts.fluxcd.io 29 | 30 | echo ">>> Installing Flux for ${REPO_URL} only watching the ${REPO_GIT_INIT_PATHS} directory" 31 | kubectl create ns flux || true 32 | helm upgrade -i flux fluxcd/flux --wait \ 33 | --set git.url=${REPO_URL} \ 34 | --set git.branch=${REPO_BRANCH} \ 35 | --set git.path=${REPO_GIT_INIT_PATHS} \ 36 | --set git.pollInterval=1m \ 37 | --set manifestGeneration=true \ 38 | --set registry.pollInterval=1m \ 39 | --set sync.state=secret \ 40 | --set syncGarbageCollection.enabled=true \ 41 | --namespace flux 42 | 43 | echo ">>> Installing Helm Operator" 44 | kubectl apply -f https://raw.githubusercontent.com/fluxcd/helm-operator/master/deploy/crds.yaml 45 | helm upgrade -i helm-operator fluxcd/helm-operator --wait \ 46 | --set git.ssh.secretName=flux-git-deploy \ 47 | --set helm.versions=v3 \ 48 | --namespace flux 49 | 50 | echo ">>> GitHub deploy key" 51 | kubectl -n flux logs deployment/flux | grep identity.pub | cut -d '"' -f2 52 | 53 | # wait until flux is able to sync with repo 54 | echo ">>> Waiting on user to add above deploy key to Github with write access" 55 | until kubectl logs -n flux deployment/flux | grep event=refreshed 56 | do 57 | sleep 5 58 | done 59 | echo ">>> Github deploy key is ready" 60 | 61 | echo ">>> Cluster bootstrap done!" -------------------------------------------------------------------------------- /docs/ci.md: -------------------------------------------------------------------------------- 1 | # Continuous Integration 2 | 3 | All tasks use the docker container from [https://github.com/swade1987/kubernetes-toolkit](https://github.com/swade1987/kubernetes-toolkit). 4 | 5 | If you would like changes made to the above image please feel free to open an issue or pull request. 6 | 7 | ## 1. Local usage 8 | 9 | There are a number of useful `Make` tasks available for you to validate your manifests and overall Kustomize setup. 10 | 11 | These can be found within the `Makefile` in the root of this repository. 12 | 13 | ### check-duplicate-release-name 14 | 15 | As the name of each `HelmRelease` resources needs to be unique in a cluster this makes sure this is the case. 16 | 17 | ### kubeval-environments 18 | 19 | This runs [kubeval](https://github.com/instrumenta/kubeval) against each `HelmRelease` constructed during the output of `kustomize build` for each environment. 20 | 21 | This makes sure that resources being deployed match strictly to the schemas for the version of Kubernetes specified by the toolkit container image. 22 | 23 | At present a container image is only available for Kubernetes v1.17.2, if you would like a newer image please open a PR. 24 | 25 | ### deprek8-check 26 | 27 | This uses [conftest](https://github.com/open-policy-agent/conftest) to make sure the resources being applied to the cluster are not using deprecated API versions. 28 | 29 | This uses the policies located [here](https://github.com/swade1987/kubernetes-toolkit/tree/master/policies/deprecations). 30 | 31 | ### kustomization-yaml-fix 32 | 33 | This simply makes sure each of your `kustomization.yaml` files are accurate against the `Kustomize` specification. 34 | 35 | ## 2. Circle CI 36 | 37 | The scripts available for execution locally can also be executed as part of CI. 38 | 39 | The configuration for these can be seen within `.circle/config.yml`. 40 | 41 | ### kubeval-helmreleases-for-environment 42 | 43 | This script (`bin/kubeval-helmreleases-for-environment`) is expected to only run within CI and not locally due to the way its execute. 44 | 45 | It runs [hrval](https://github.com/stefanprodan/hrval-action) against each `HelmRelease` constructed during the output of `kustomize build` for a given environment. 46 | 47 | The steps `hrval` executes for each `HelmRelease` are as follows: 48 | 49 | - extracts the chart source with yq 50 | - downloads the chart from the Helm or Git repository 51 | - extracts the Helm Release values with yq 52 | - runs `helm template` for the extracted values 53 | - validates the YAMLs using [kubeval](https://github.com/instrumenta/kubeval) in strict mode 54 | 55 | ## 3. GitHub Actions 56 | 57 | There is one GitHub action that runs as part of CI at present. 58 | 59 | ### kustomize-diff 60 | 61 | This actions uses [eeveebank/github-action-kustomize-diff](https://github.com/eeveebank/github-action-kustomize-diff). 62 | 63 | Shout outs to [@benhartley](https://github.com/benhartley) for implementing this. 64 | 65 | The steps the action takes are as follows: 66 | 67 | - Store the output of `kustomize build` (for each environment) on the current branch. 68 | - Store the output of `kustomize build` (for each environment) on the `master` branch. 69 | - Based on the two outputs above it performs a `git diff` and stores the output in a variable called `escaped_output`. 70 | 71 | This action is combined with [unsplash/comment-on-pr](https://github.com/unsplash/comment-on-pr) to comment the output to the PR. 72 | 73 | The complete action can be found in [.github/workflows/kustomize-diff.yml](../.github/workflows/kustomize-diff.yml). 74 | 75 | An example output can be seen in [https://github.com/swade1987/gitops-with-kustomize/pull/6](https://github.com/swade1987/gitops-with-kustomize/pull/6). --------------------------------------------------------------------------------