├── .gitignore ├── README.md ├── argocd ├── app.yaml ├── k8sgpt.yaml ├── prometheus-crds.yaml ├── prometheus.yaml └── trivy-operator.yaml ├── k8sgpt-resources ├── k8sgpt-resource.yaml └── k8sgpt-secret.yaml ├── notes └── digital-ocean-resources │ ├── resources-to-install-app-do │ ├── provider-for-da-app.tf │ ├── terraform-do.tfvars │ └── variables-do.tf │ └── terraform-DO-cluster │ ├── cluster.tf │ ├── outputs.tf │ ├── provider.tf │ ├── terraform.tfvars │ └── variables.tf ├── terraform-app ├── argo-helm.tf ├── provider.tf ├── terraform.tfvars └── variables.tf └── terraform-kind-cluster ├── cluster.tf └── provider.tf /.gitignore: -------------------------------------------------------------------------------- 1 | terraform-infra/*terraform* 2 | terraform-app/*terraform* 3 | 4 | !terraform-app/terraform.tfvars 5 | !terraform-infra/terraform.tfvars -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # GitOps to secure your software supply chain 2 | 3 | GitOps is the principle of defining all of your resources in Git. 4 | It has become popular throughout the past year. Additionally, projects like ArgoCD and Crossplane are not serving anymore just a niche market but are goining more and more traction through production use cases. 5 | 6 | The main benefit of GitOps is gaining higher insights into the deployment process. Knwoing what got deployed when and by whom as well as any of the changed or affected services will make it much easier to identify root causes during an incident. 7 | However, the advantages of using GitOps go much further and extent to cloud native security. 8 | 9 | This tutorial will detail how following GitOps best practices generates higher scan coverage for your resources. 10 | 11 | ## GitOps vs. other deployment tools 12 | 13 | Lots of tools, such as those focused on platform engineering, try to make the deployment process as "easy as possible" by peviding dashboards and interfaces that allow you to deploy containers and other resources through a few clicks. While this can be great to test out running workloads, it does not scale well nor is it providing production-like environments that would make it possible to stress-test our resources and similar. 14 | 15 | ## Setting up your GitOps workflow 16 | 17 | We are going to set up an example workflow through ArgoCD to demonstrate how an application would be deployed and managed. 18 | 19 | ### Prerequisites 20 | 21 | * kubectl installaed 22 | * the Terraform CLI installed 23 | 24 | ### Intall the Kubernetes cluster 25 | 26 | In this example, I have set up the Terraform configuration to create a kind Kubernetes cluster. If you are on a Mac, this won't work and you will have to create a cluster manually. Make sure that you have kind installed and then run: 27 | ``` 28 | kind create cluster --name demo 29 | ``` 30 | 31 | The configuration can be found in the [./terraform-infra](./terraform-kind-cluster/) directory 32 | * provider.tf -- contains the Kind Provider, you could use any other cloud provider or provider that helps you create a Kubernetes cluster 33 | * cluster.tf -- this file containes the cluster Resources that will be created through the provider 34 | 35 | Next, we can go into the directory and initialise the Terraform state: *I think this is how you say it 36 | ``` 37 | cd terraform-infra 38 | terraform init 39 | ``` 40 | 41 | This will provide us with the terraform specific resources. 42 | 43 | Next, you can either do first a 'terraform plan' to view the resources that will be created -- in our case this is just going to be the Digital Ocean Kubernetes cluster -- or you can go ahead and run the apply command: 44 | ``` 45 | terraform apply 46 | ``` 47 | 48 | If you are not in the terraform-infra directory, you will have to specify the directory in which you want to run the command. 49 | 50 | Once you run `terraform apply` it will show you the resources that will be created, changed, or deleted -- review the list and if you are happy with it, then type `yes` and press enter. 51 | 52 | The creation of the Kubernetes cluster might take several minutes depending on which cloud provider you are using. 53 | 54 | ### Installing ArgoCD 55 | 56 | Next, we are going ahead and installing ArgoCD inside of our cluster through Terraform. 57 | 58 | This is going to be the same process as with the cluster installation. 59 | 60 | The terraform-app directory contains the following files: 61 | * provider.tf -- this file requires multiple providers; the Kubernetes provider to install our argo-cd namespace, the Helm provider to install the ArgoCD Helm chart and the DigitalOcean provider to connect to our cluster in the first place 62 | * argo-helm.tf -- this file specifies the Helm installation for ArgoCD 63 | * terraform.tfvars -- this file contains the digital ocean token as well as the output from the `terraform-infra apply stage`. The information will be used to connect to the Digital Ocean account and access the previously created cluster. 64 | * variables.tf -- here we define the variables that we are using in this module 65 | 66 | Next, you can either do first a 'terraform plan' to view the resources that will be created -- in our case this is just going to be the argocd namespace and the argocd Helm Chart within -- or you can go ahead and run the apply command: 67 | ``` 68 | terraform apply 69 | ``` 70 | 71 | If you are not in the terraform-app directory, you will have to specify the directory in which you want to run the command. 72 | 73 | Once you run `terraform apply` it will show you the resources that will be created, changed, or deleted -- review the list and if you are happy with it, then type `yes` and press enter. 74 | 75 | The creation of the Helm Chart might take a couple of minutes (2 or 3). 76 | 77 | ## Installing our application resources through ArgoCD 78 | 79 | You can now view the ArgoCD Helm Chart in the argocd namespace 80 | 81 | We have specified two Application installations that will be managed by ArgoCD in the argocd directory: 82 | 83 | * app.yaml -- this file has the details to our Application installation 84 | * trivy-operator.yaml -- this file defines how the Trivy Operator will be installed 85 | * k8sgpt.yaml -- this file defines how the K8sGPT Operator will be installed 86 | * promtheus -- this file defines how the Kube Prometheus Stack Operator will be installed 87 | 88 | We have already defined in the manifests that those resources will be given to ArgoCD in the argocd namespace. Lastly, we can apply the resources through the following command: 89 | ``` 90 | kubectl apply -f ./argocd 91 | ``` 92 | 93 | This will install all the resources at once. You then have to makes sure that they are synced in ArgoCD. 94 | This can be done in the UI. 95 | 96 | First, access the ArgoCD secret: 97 | ``` 98 | kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d 99 | ``` 100 | 101 | and then doing a port-forward on the ArgoCD service: 102 | ``` 103 | kubectl port-forward svc/argocd-server -n argocd 8080:443 104 | ``` 105 | 106 | the username is `admin` and the password is the secret from the command before. 107 | 108 | More information can be found in the [ArgoCD docs]([ArgoCD CLI](https://argo-cd.readthedocs.io/en/stable/getting_started/). 109 | 110 | ## Security Benefits 111 | 112 | Now that we have seen how an application can be deployed and managed through GitOps best practices, let's look at some of the advanteges this brings to our security scanning. 113 | 114 | We could be using something like Crossplane to provision our infrastructure resources such as our Kubernetes cluster. However in that case, we would not be able to scan our infrastructure resources for misconfiguration issues. 115 | 116 | Several security scanners, including Trivy are able to scan Terraform resources for misconfiguration issues. 117 | 118 | More information can be found on the [Trivy documentation](https://aquasecurity.github.io/trivy/). 119 | 120 | ## Other applications used in this demo 121 | 122 | K8sGPT: https://k8sgpt.ai/ 123 | 124 | K8sGPT gives Kubernetes Superpowers to everyone 125 | 126 | k8sgpt is a tool for scanning your kubernetes clusters, diagnosing and triaging issues in simple english. It has SRE experience codified into its analyzers and helps to pull out the most relevant information to enrich it with AI. -------------------------------------------------------------------------------- /argocd/app.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: argoproj.io/v1alpha1 2 | kind: Application 3 | metadata: 4 | name: website 5 | namespace: argocd 6 | spec: 7 | project: default 8 | source: 9 | repoURL: https://github.com/Cloud-Native-Security/website 10 | path: cns-website 11 | directory: 12 | recurse: true 13 | destination: 14 | server: https://kubernetes.default.svc 15 | namespace: app 16 | syncPolicy: 17 | automated: 18 | prune: true 19 | selfHeal: true 20 | syncOptions: 21 | - CreateNamespace=true -------------------------------------------------------------------------------- /argocd/k8sgpt.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: argoproj.io/v1alpha1 2 | kind: Application 3 | metadata: 4 | name: k8sgpt 5 | namespace: argocd 6 | spec: 7 | project: default 8 | source: 9 | chart: k8sgpt-operator 10 | repoURL: https://charts.k8sgpt.ai/ 11 | targetRevision: 0.0.17 12 | helm: 13 | values: | 14 | serviceMonitor: 15 | enabled: true 16 | GrafanaDashboard: 17 | enabled: true 18 | destination: 19 | server: https://kubernetes.default.svc 20 | namespace: k8sgpt-operator-system 21 | syncPolicy: 22 | automated: 23 | prune: true 24 | selfHeal: true 25 | syncOptions: 26 | - CreateNamespace=true -------------------------------------------------------------------------------- /argocd/prometheus-crds.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: argoproj.io/v1alpha1 2 | kind: Application 3 | metadata: 4 | name: crds 5 | namespace: argocd 6 | spec: 7 | project: default 8 | source: 9 | repoURL: https://github.com/prometheus-community/helm-charts.git 10 | path: charts/kube-prometheus-stack/crds/ 11 | targetRevision: kube-prometheus-stack-46.6.0 12 | directory: 13 | recurse: true 14 | destination: 15 | server: https://kubernetes.default.svc 16 | namespace: monitoring 17 | syncPolicy: 18 | automated: 19 | prune: true 20 | selfHeal: true 21 | syncOptions: 22 | - Replace=true -------------------------------------------------------------------------------- /argocd/prometheus.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: argoproj.io/v1alpha1 2 | kind: Application 3 | metadata: 4 | name: prom 5 | namespace: argocd 6 | spec: 7 | destination: 8 | namespace: monitoring 9 | server: https://kubernetes.default.svc 10 | project: default 11 | source: 12 | chart: kube-prometheus-stack 13 | repoURL: https://prometheus-community.github.io/helm-charts 14 | targetRevision: 46.6.0 15 | helm: 16 | skipCrds: true 17 | values: | 18 | prometheus: 19 | prometheusSpec: 20 | serviceMonitorSelectorNilUsesHelmValues: false 21 | serviceMonitorSelector: {} 22 | serviceMonitorNamespaceSelector: {} 23 | destination: 24 | server: https://kubernetes.default.svc 25 | namespace: monitoring 26 | syncPolicy: 27 | automated: 28 | prune: true 29 | selfHeal: true 30 | syncOptions: 31 | - CreateNamespace=true -------------------------------------------------------------------------------- /argocd/trivy-operator.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: argoproj.io/v1alpha1 2 | kind: Application 3 | metadata: 4 | name: trivy-operator 5 | namespace: argocd 6 | spec: 7 | project: default 8 | source: 9 | chart: trivy-operator 10 | repoURL: https://aquasecurity.github.io/helm-charts/ 11 | targetRevision: 0.14.0 12 | helm: 13 | values: | 14 | serviceMonitor: 15 | enabled: true # has to be false if you do not have Prometheus already installed 16 | trivy: 17 | ignoreUnfixed: true 18 | destination: 19 | server: https://kubernetes.default.svc 20 | namespace: trivy-system 21 | syncPolicy: 22 | automated: 23 | prune: true 24 | selfHeal: true 25 | syncOptions: 26 | - CreateNamespace=true -------------------------------------------------------------------------------- /k8sgpt-resources/k8sgpt-resource.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: core.k8sgpt.ai/v1alpha1 2 | kind: K8sGPT 3 | metadata: 4 | name: k8sgpt-sample 5 | namespace: k8sgpt-operator-system 6 | spec: 7 | model: gpt-3.5-turbo 8 | backend: openai 9 | noCache: false 10 | version: v0.3.5 11 | enableAI: true 12 | # filters: 13 | # - Ingress 14 | # extraOptions: 15 | # backstage: 16 | # enabled: true 17 | secret: 18 | name: k8sgpt-sample-secret 19 | key: openai-api-key -------------------------------------------------------------------------------- /k8sgpt-resources/k8sgpt-secret.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Secret 3 | metadata: 4 | name: k8sgpt-sample-secret 5 | namespace: k8sgpt-operator-system 6 | type: kubernetes.io/ 7 | data: 8 | # You can include additional key value pairs as you do with Opaque Secrets 9 | openai-api-key: c2stOGRicHMyT2gxMlk1RWVrMXhMQkRUM0JsYmtGSmpaRktvV1M3b1dpZlZTVWxISEpCCg== -------------------------------------------------------------------------------- /notes/digital-ocean-resources/resources-to-install-app-do/provider-for-da-app.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | digitalocean = { 4 | source = "digitalocean/digitalocean" 5 | version = ">= 2.4.0" 6 | } 7 | kubernetes = { 8 | source = "hashicorp/kubernetes" 9 | version = ">= 2.7.0" 10 | } 11 | helm = { 12 | source = "hashicorp/helm" 13 | version = ">= 2.0.1" 14 | } 15 | } 16 | } 17 | 18 | # Configure the DigitalOcean Provider 19 | provider "digitalocean" { 20 | token = var.do_token 21 | } 22 | 23 | data "digitalocean_kubernetes_cluster" "demo" { 24 | name = var.cluster_name 25 | } 26 | 27 | provider "kubernetes" { 28 | host = data.digitalocean_kubernetes_cluster.demo.endpoint 29 | token = data.digitalocean_kubernetes_cluster.demo.kube_config[0].token 30 | cluster_ca_certificate = base64decode( 31 | data.digitalocean_kubernetes_cluster.demo.kube_config[0].cluster_ca_certificate 32 | ) 33 | } 34 | 35 | provider "helm" { 36 | kubernetes { 37 | host = data.digitalocean_kubernetes_cluster.demo.endpoint 38 | token = data.digitalocean_kubernetes_cluster.demo.kube_config[0].token 39 | cluster_ca_certificate = base64decode( 40 | data.digitalocean_kubernetes_cluster.demo.kube_config[0].cluster_ca_certificate 41 | ) 42 | } 43 | } 44 | -------------------------------------------------------------------------------- /notes/digital-ocean-resources/resources-to-install-app-do/terraform-do.tfvars: -------------------------------------------------------------------------------- 1 | do_token = "" 2 | cluster_id = "" 3 | cluster_name = "demo" -------------------------------------------------------------------------------- /notes/digital-ocean-resources/resources-to-install-app-do/variables-do.tf: -------------------------------------------------------------------------------- 1 | variable "cluster_name" { 2 | type = string 3 | default = "demo" 4 | } 5 | 6 | variable "do_token" { 7 | type = string 8 | } 9 | 10 | variable "cluster_id" { 11 | type = string 12 | } -------------------------------------------------------------------------------- /notes/digital-ocean-resources/terraform-DO-cluster/cluster.tf: -------------------------------------------------------------------------------- 1 | resource "digitalocean_kubernetes_cluster" "demo" { 2 | name = "demo" 3 | region = "lon1" 4 | # Grab the latest version slug from `doctl kubernetes options versions` 5 | version = "1.24.4-do.0" 6 | 7 | node_pool { 8 | name = "worker-pool" 9 | size = "s-2vcpu-2gb" 10 | node_count = 3 11 | } 12 | } -------------------------------------------------------------------------------- /notes/digital-ocean-resources/terraform-DO-cluster/outputs.tf: -------------------------------------------------------------------------------- 1 | output "cluster_id" { 2 | value = digitalocean_kubernetes_cluster.demo.id 3 | } 4 | 5 | output "cluster_name" { 6 | value = digitalocean_kubernetes_cluster.demo.name 7 | } -------------------------------------------------------------------------------- /notes/digital-ocean-resources/terraform-DO-cluster/provider.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | digitalocean = { 4 | source = "digitalocean/digitalocean" 5 | version = "~> 2.0" 6 | } 7 | } 8 | } 9 | 10 | # Configure the DigitalOcean Provider 11 | provider "digitalocean" { 12 | token = var.do_token 13 | } 14 | 15 | -------------------------------------------------------------------------------- /notes/digital-ocean-resources/terraform-DO-cluster/terraform.tfvars: -------------------------------------------------------------------------------- 1 | do_token = "DGUWGDUWQDBjqadsfhuHAHSDUIHWAIUSDASHALS" 2 | -------------------------------------------------------------------------------- /notes/digital-ocean-resources/terraform-DO-cluster/variables.tf: -------------------------------------------------------------------------------- 1 | variable "do_token" { 2 | type = string 3 | } -------------------------------------------------------------------------------- /terraform-app/argo-helm.tf: -------------------------------------------------------------------------------- 1 | resource "helm_release" "argocd" { 2 | name = "argocd-helm" 3 | namespace = "argocd" 4 | create_namespace = true 5 | 6 | repository = "https://argoproj.github.io/argo-helm" 7 | chart = "argo-cd" 8 | 9 | # depends_on = [kubernetes_namespace.argocd_namespace] 10 | } -------------------------------------------------------------------------------- /terraform-app/provider.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | kubernetes = { 4 | source = "hashicorp/kubernetes" 5 | version = "2.21.1" 6 | } 7 | helm = { 8 | source = "hashicorp/helm" 9 | version = "2.10.1" 10 | } 11 | } 12 | } 13 | 14 | provider "kubernetes" { 15 | config_path = "~/.kube/config" 16 | config_context = var.cluster_name 17 | } 18 | 19 | provider "helm" { 20 | kubernetes { 21 | config_path = "~/.kube/config" 22 | config_context = var.cluster_name 23 | } 24 | } 25 | -------------------------------------------------------------------------------- /terraform-app/terraform.tfvars: -------------------------------------------------------------------------------- 1 | namespace = "argocd" 2 | cluster_name = "kind-devopsdays" -------------------------------------------------------------------------------- /terraform-app/variables.tf: -------------------------------------------------------------------------------- 1 | variable "namespace" { 2 | type = string 3 | } 4 | 5 | variable "cluster_name" { 6 | type = string 7 | } -------------------------------------------------------------------------------- /terraform-kind-cluster/cluster.tf: -------------------------------------------------------------------------------- 1 | resource "kind_cluster" "default" { 2 | name = "demo-cluster" 3 | wait_for_ready = "true" 4 | kubeconfig = "true" 5 | # node_image = "kindest/node:v1.16.1" 6 | } -------------------------------------------------------------------------------- /terraform-kind-cluster/provider.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | kind = { 4 | source = "unicell/kind" 5 | version = "0.0.2-u2" 6 | } 7 | } 8 | } 9 | 10 | provider "kind" { 11 | # Configuration options 12 | } --------------------------------------------------------------------------------