├── ApplicationSet.yaml ├── Demo.md ├── ReadMe.md ├── argo-app.yaml ├── argocd-values.yaml ├── charts └── hello-world-0.1.0 │ ├── .argocd-allow-concurrency │ ├── .helmignore │ ├── Chart.yaml │ ├── README.md │ ├── templates │ ├── NOTES.txt │ ├── _helpers.tpl │ ├── deployment.yaml │ ├── service.yaml │ └── serviceaccount.yaml │ └── values.yaml ├── clusters ├── cluster-1 │ ├── app-1-namespace │ │ ├── Application.yaml │ │ ├── limitrange.yaml │ │ ├── namespace.yaml │ │ └── resourcequota.yaml │ ├── app-1 │ │ ├── Application.yaml │ │ └── values.yaml │ ├── app-2-namespace │ │ ├── Application.yaml │ │ ├── limitrange.yaml │ │ ├── namespace.yaml │ │ └── resourcequota.yaml │ └── app-2 │ │ ├── Application.yaml │ │ └── values.yaml ├── cluster-2 │ ├── app-1-namespace │ │ ├── Application.yaml │ │ ├── limitrange.yaml │ │ ├── namespace.yaml │ │ └── resourcequota.yaml │ └── app-1 │ │ ├── Application.yaml │ │ └── values.yaml └── in-cluster │ ├── argocd │ ├── Application.yaml │ └── values.yaml │ └── ingress-nginx │ ├── Application.yaml │ └── values.yaml ├── commonValues └── app-1-values.yaml └── kind-cluster-setup.sh /ApplicationSet.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: argoproj.io/v1alpha1 2 | kind: ApplicationSet 3 | metadata: 4 | name: cluster-apps 5 | spec: 6 | goTemplate: true 7 | syncPolicy: 8 | preserveResourcesOnDeletion: true 9 | generators: 10 | - git: 11 | repoURL: https://github.com/burhanuguz/advanced-argocd-gitops.git 12 | revision: HEAD 13 | files: 14 | - path: 'clusters/*/*/Application.yaml' 15 | template: 16 | metadata: 17 | name: '{{ index .path.segments 1 }}-{{ .appName }}' 18 | namespace: argo-cd 19 | annotations: 20 | argocd.argoproj.io/sync-wave: '{{ .syncOrder }}' 21 | argocd.argoproj.io/manifest-generate-paths: '.' 22 | labels: 23 | clustername: '{{ index .path.segments 1 }}' 24 | app: '{{ default "" .chartName }}' 25 | spec: 26 | syncPolicy: 27 | syncOptions: 28 | - ServerSideApply=true 29 | # automated: {} 30 | destination: 31 | name: '{{ index .path.segments 1 }}' 32 | namespace: '{{ .namespace }}' 33 | project: '{{ .argoProject }}' 34 | source: 35 | repoURL: '{{ .repo }}' 36 | targetRevision: '{{ .branch }}' 37 | path: '{{ .path.path }}' 38 | plugin: 39 | env: 40 | - name: pluginName 41 | value: '{{ .plugin }}' 42 | - name: AVP_SECRET 43 | value: '{{ default "" .keyVault }}' 44 | # These values are needed when both local and remote helm plugin is used 45 | - name: chartName 46 | value: '{{ default "" .chartName }}' 47 | - name: chartReleaseName 48 | value: '{{ default "" .appName | trunc 53 }}' 49 | ## chartRepository and chartVersion values are needed when helm remote plugin is used 50 | - name: chartRepository 51 | value: '{{ default "" .chartRepository }}' 52 | - name: chartVersion 53 | value: '{{ default "" .chartVersion }}' 54 | ## You can put extra args with spaces. 55 | - name: extraArgs 56 | value: '{{ default "" .extraArgs }}' -------------------------------------------------------------------------------- /Demo.md: -------------------------------------------------------------------------------- 1 | ## Demo 2 | 3 | Quick demo on kind clusters. I had created '*.local.gd' tls certificate with a Root CA I created. In the ArgoCD values yaml file, I use the key and crt I store in Azure Key Vault. You can do the changes for your own cluster. 4 | 5 | Script for the installation is in the git repo as well. 6 | 7 | ### ArgoCD and Ingress Installation 8 | 9 | - The first part is to install a master cluster. NodePorts will be used by Nginx Ingress and will make 80 and 443 to be available on the host. 10 | 11 | ```bash 12 | ## Create the master cluster 13 | cat << EOF | kind create cluster --config - 14 | kind: Cluster 15 | apiVersion: kind.x-k8s.io/v1alpha4 16 | name: master 17 | nodes: 18 | - role: control-plane 19 | extraPortMappings: 20 | - containerPort: 30080 21 | hostPort: 80 22 | - containerPort: 30443 23 | hostPort: 443 24 | EOF 25 | ## Wait for the master Kubernetes cluster to get ready 26 | kubectl --context "kind-master" wait --namespace kube-system --for=condition=ready pod --all --timeout=120s 27 | ``` 28 | 29 | - Next, install ingress and argocd 30 | 31 | ```bash 32 | ## Install Kubernetes Nginx Ingress 33 | helm upgrade --install ingress-nginx ingress-nginx --kube-context "kind-master" \ 34 | --repo https://kubernetes.github.io/ingress-nginx \ 35 | --namespace ingress-nginx --create-namespace \ 36 | --set controller.ingressClassResource.default=true \ 37 | --set controller.service.type=NodePort \ 38 | --set controller.service.nodePorts.http=30080 \ 39 | --set controller.service.nodePorts.https=30443 \ 40 | --set controller.extraArgs.enable-ssl-passthrough='' 41 | 42 | ## Wait for ingress to get ready 43 | kubectl --context "kind-master" wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/mponent=controller --timeout=120s 44 | 45 | ## Install ArgoCD cluster. I use the same values YAML as the values files inside the GitHub repository 46 | helm upgrade --install argocd argo-cd --kube-context "kind-master" \ 47 | --repo https://argoproj.github.io/argo-helm \ 48 | --namespace=argocd --create-namespace \ 49 | -f argocd-values.yaml 50 | 51 | ## Wait for argocd to get ready 52 | kubectl --context "kind-master" wait --namespace argocd --for=condition=ready pod --all --timeout=120s 53 | ``` 54 | 55 | - After that, you will be able to login into the cluster. Apply argo-app yaml that has ApplicationSet. With it, you will be able to make cluster deployments in the next section. 56 | 57 | ```bash 58 | ## Get the initial secret and then apply the argo-appy.yaml to install the Application Set to the cluster. 59 | kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d 60 | kubectl apply -f argo-app.yaml 61 | ``` 62 | 63 | - In this gif, you will find installation up to this point 64 | 65 | ![1](https://user-images.githubusercontent.com/59168275/193477170-41fc9b43-0958-4803-a9a1-dbcd669dcfa3.gif) 66 | 67 | ### Logging into the ArgoCD instance, using ArgoCD Vault Plugin, and managing ArgoCD with ArgoCD 68 | 69 | - The second gif consists of several steps. I have used argocd-vault-plugin-helm-remote-repo and pulled the secret from my own Azure Key Vault. 70 | - I created a service principal in Azure, then I gave a Key Vault reader role with Azure RBAC. I have changed the Key Vault access policy to RBAC as well. Here is my Azure Key Vault Backed Yaml secret YAML as a template. 71 | 72 | ```yaml 73 | apiVersion: v1 74 | stringData: 75 | AZURE_TENANT_ID: '' 76 | AZURE_CLIENT_ID: '' 77 | AZURE_CLIENT_SECRET: '' 78 | AVP_TYPE: azurekeyvault 79 | kind: Secret 80 | metadata: 81 | name: argocd-vault-plugin-azure-credentials-in-cluster 82 | namespace: argocd 83 | ``` 84 | 85 | - I showed logon in this gif. First, it is insecure and also in-cluster-argocd gives an error because the secret was not applied. I applied it and refreshed the Application resource, then the automated sync worked, and made the changes. 86 | - My ArgoCD instance started working with my own certificate defined in it, and that certificate information comes from Azure Key Vault. 87 | - I commented on ArgoCD with TLS configuration lines. You can integrate your Vault and make use of it. 88 | - At last, you will see ApplicationSet is degraded because it can not find cluster-1 and cluster-2. Let's create those clusters and manage them as well. 89 | 90 | ![2](https://user-images.githubusercontent.com/59168275/193478206-27b794f4-df4f-4275-9da9-9fe20117b458.gif) 91 | 92 | ### Creating cluster-1 and cluster-2 and managing them 93 | 94 | - Create kind clusters first. 95 | 96 | ```bash 97 | ## Create other two clusters 98 | kind create cluster --name=cluster-1 99 | kind create cluster --name=cluster-2 100 | ``` 101 | 102 | - Then copy their kubeconfig files. We will combine master, cluster-1, and cluster-2 kubeconfig files and use them to add those clusters to ArgoCD. Unfortunately, we have to do these in the master-control-plane container because Windows WSL can not reach to clusters' control plane without reaching to container network. 103 | - Also, download the argocd-cli only from the ArgoCD instance itself. It always uses the latest version. New Kubernetes versions do not create service account secret tokens, but ArgoCD needs service account secret tokens. The New ArgoCD client solves it via creating the secret manually and adding annotation manually while adding the remote clusters into the main ArgoCD instance. 104 | 105 | ```bash 106 | ## Copy their kubeconfig YAML file. We also will use combine three conf, and their use is the same, we need to change the username in kubeconfig YAMLs 107 | docker cp cluster-1-control-plane:/etc/kubernetes/admin.conf ../cluster-1-admin.conf; sed -i 's/kubernetes-admin/kubernetes-admin-1/' ../cluster-1-admin.conf 108 | docker cp cluster-2-control-plane:/etc/kubernetes/admin.conf ../cluster-2-admin.conf; sed -i 's/kubernetes-admin/kubernetes-admin-2/' ../cluster-2-admin.conf 109 | wget --no-check-certificate https://argocd.local.gd/download/argocd-linux-amd64 -O ../argocd-linux-amd64 110 | ``` 111 | 112 | - Next, copy the kubeconfig files and argocd-cli to master-control-plane 113 | 114 | ```bash 115 | ## Copy argocd cli and kubeconfig files to the kind-master cluster. Windows WSL2 can not communicate 116 | ## with containers. That's why it has to be dealt with inside the container. 117 | docker cp ../cluster-1-admin.conf master-control-plane:/root/ 118 | docker cp ../cluster-2-admin.conf master-control-plane:/root/ 119 | docker cp ../argocd-linux-amd64 master-control-plane:/root/argocd 120 | ``` 121 | 122 | - Make the argocd cli as an executable 123 | 124 | ```bash 125 | ## Run commands below to login and then and other clusters to argocd to manage 126 | docker exec -it master-control-plane chmod +x /root/argocd 127 | ``` 128 | 129 | - Login to the ArgoCD instance inside of the master-control-plane. 130 | 131 | ```bash 132 | argoPass=$(kubectl --context kind-master -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d) 133 | docker exec -it master-control-plane /root/argocd login argocd.local.gd:30443 --insecure --username=admin --password="${argoPass}" 134 | ``` 135 | 136 | - The last stage is to add those clusters with the commands below. 137 | 138 | ```bash 139 | ## Combining all three cluster in one kubeconfig and using them for each command 140 | kindconfig='/etc/kubernetes/admin.conf:/root/cluster-1-admin.conf:/root/cluster-2-admin.conf' 141 | docker exec -it master-control-plane bash -c "KUBECONFIG=${kindconfig} /root/argocd cluster add --name cluster-1 --insecure kubernetes-admin-1@cluster-1 --yes" 142 | docker exec -it master-control-plane bash -c "KUBECONFIG=${kindconfig} /root/argocd cluster add --name cluster-2 --insecure kubernetes-admin-2@cluster-2 --yes" 143 | ``` 144 | 145 | - Now, you can log in to ArgoCD and refresh the Application Set object. You will see that applications will be deployed. 146 | 147 | ![3](https://user-images.githubusercontent.com/59168275/193477580-5e6e1021-7392-4ef2-941b-aa303f1e70d4.gif) 148 | -------------------------------------------------------------------------------- /ReadMe.md: -------------------------------------------------------------------------------- 1 | # ArgoCD [ApplicationSet](https://argocd-applicationset.readthedocs.io/en/stable/) + [Vault Plugin](https://argocd-vault-plugin.readthedocs.io/en/stable/) Advanced Use Case with Helm 2 | 3 | TL;DR: Helm local/remote with external value deployments + Kubernetes Manifests Deployments + With/Without Vault Plugins and you can dig into one app of a cluster at any time. 4 | 5 | In this demo, you will see a custom solution for managing clusters with Kubernetes manifests and helm packages within a mono or multi-repo while keeping your secrets in a vault that [ArgoCD Vault Plugin](https://argocd-vault-plugin.readthedocs.io/en/stable/backends/) supports as backend. 6 | 7 | With the correct YAML manifests and helm packages, you can deploy any resource to any cloud or on-premise Kubernetes platform. In this way, you can even deploy operators, and their instances with YAML manifests. 8 | 9 | With the plugin customizations, it is possible to easily add two external value files(one base, and one cluster-specific) to remote, and local helm repositories. You can increase the value files by changing the ApplicationSet YAML file. 10 | 11 | With just one ApplicationSet Object, child Application objects will be created and they will install in the clusters with definition files created under the mono or multi repo. 12 | 13 | I prevented child Application objects generated deployment or any other resources from being deleted. 14 | 15 | ## Architecture 16 | 17 | The architecture has quite complicated sides in it. Because it is making all changes in just one object, and from it, child objects will be created for each cluster. 18 | 19 | Originally [**app of apps pattern**](https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/#app-of-apps-pattern) was the idea that came to my mind, but after discovering ApplicationSet and its generators, I could invent an idea to solve for almost any case and apply it to numerous clusters within(or with many) repos, and the biggest advantage is to use argocd-vault-plugin to deploy applications without putting their secrets to git. 20 | 21 | Let's check the ApplicationSet.yaml to understand the concept better. 22 | 23 | ```yaml 24 | apiVersion: argoproj.io/v1alpha1 25 | kind: ApplicationSet 26 | metadata: 27 | name: cluster-apps 28 | spec: 29 | goTemplate: true 30 | # This option will prevent child Applications to be deleted. 31 | syncPolicy: 32 | preserveResourcesOnDeletion: true 33 | generators: 34 | # This is the most important part of the architecture itself. In the Application.yaml files, 35 | # there are definitions for both what # will be deployed in to the cluster, and how it will 36 | # be. Under the clusters folder, for each app, you will have a folder. For each app folder, 37 | # you will put a declarative Application.yaml file which holds information about how the app 38 | # will be deployed. As a local or remote helm chart, or just standard YAML manifests 39 | - git: 40 | repoURL: https://github.com/burhanuguz/advanced-argocd-gitops.git 41 | revision: HEAD 42 | files: 43 | - path: 'clusters/*/*/Application.yaml' 44 | template: 45 | # Application Objects will be created with cluster name and application name concatenated. 46 | metadata: 47 | name: '{{ index .path.segments 1 }}-{{ .appName }}' 48 | namespace: argo-cd 49 | annotations: 50 | # There is an option for syncOrder as well, to sync applications in a more professional 51 | # way. Unfortunately, this feature can not be used with Applications that are created 52 | # with Application Sets in the current version of ArgoCD. I put that anyway if it could 53 | # be used in the future. # There are some issues opened for this one. It will be much 54 | # better option once they are GA. It will be possible to bootstrap a cluster very easily 55 | # in the future with application dependencies. 56 | argocd.argoproj.io/sync-wave: '{{ .syncOrder }}' 57 | # There are high availability considerations for monorepos. I tried to put best practices 58 | # as much as I can. This one prevents Application syncs if no file is changed inside the 59 | # Application's own folder. It is especially useful with monorepo's. 60 | # Read more at 61 | # https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#monorepo-scaling-considerations 62 | argocd.argoproj.io/manifest-generate-paths: '.' 63 | labels: 64 | clustername: '{{ index .path.segments 1 }}' 65 | app: '{{ default "" .chartName }}' 66 | spec: 67 | # The automated sync policy can be placed, but it will sync all of the apps that are created 68 | # with Application Set. I tried to make it declarative as well, but you can not declare fields 69 | # other than String fields and we need object field. For now I would sugget you to create your 70 | # own solution with argocd cli or use the solution in the other branch that uses helm chart 71 | # which adds another layer for creating Application objects in cluster. But you can declare 72 | # anything with it 73 | # 74 | # There are some issues opened as Advanced Application Set templating, when that happens, 75 | # I will change the definitions here. You will be able to declare anything to App Objects. 76 | syncPolicy: 77 | syncOptions: 78 | - ServerSideApply=true 79 | # automated: {} 80 | # You can also create projects for ArgoCD to limit which resources can be deployed into 81 | # the cluster. This is a better solution when Application Developers try to do deployments. 82 | # It will limit which resources they can create and what they will see in the UI as well. 83 | destination: 84 | name: '{{ index .path.segments 1 }}' 85 | namespace: '{{ .namespace }}' 86 | project: '{{ .argoProject }}' 87 | source: 88 | # Here is the magic part that lets you define the multiple repos for each # application. 89 | # You can use it to store your manifests, helm charts, or external value files in a 90 | # different repo. 91 | repoURL: '{{ .repo }}' 92 | targetRevision: '{{ .branch }}' 93 | # Here is the folder that application manifests or helm values will be stored for each 94 | # application. 95 | path: '{{ .path.path }}' 96 | plugin: 97 | env: 98 | # Determines whether helm local or remote or just application manifest to be deployed 99 | - name: pluginName 100 | value: '{{ .plugin }}' 101 | # Determines which vault will be used for an app. 102 | # Read at https://argocd-vault-plugin.readthedocs.io/en/stable/backends/ 103 | - name: AVP_SECRET 104 | value: '{{ default "" .keyVault }}' 105 | # These values are needed when either local or remote helm plugin is used 106 | - name: chartName 107 | value: '{{ default "" .chartName }}' 108 | - name: chartReleaseName 109 | value: '{{ default "" .appName | trunc 53 }}' 110 | ## chartRepository and chartVersion values are needed when helm remote plugin is used 111 | - name: chartRepository 112 | value: '{{ default "" .chartRepository }}' 113 | - name: chartVersion 114 | value: '{{ default "" .chartVersion }}' 115 | ## You can put extra args with spaces. 116 | - name: extraArgs 117 | value: '{{ default "" .extraArgs }}' 118 | ``` 119 | 120 | And here is the application manifest file that will be deployed. Here is where you can do the trick for mono or multi repo 121 | 122 | ```yaml 123 | # Can be in different repo, but you have to keep the same folder structure in the repo you define 124 | repo: https://github.com/burhanuguz/advanced-argocd-gitops.git 125 | # Three types of plugins are defined. argocd-vault-plugin, argocd-vault-plugin-helm-local-repo 126 | # and argocd-vault-plugin-helm-remote-repo. 127 | plugin: 'argocd-vault-plugin-helm-remote-repo' 128 | # As explained, you can use it to limit which resources are to be deployed. 129 | argoProject: 'default' 130 | # Although it does not work with Application Set for now, syncOrder will be an important parameter. 131 | # in the future. It will do the sync with the order. It will useful when you have some dependencies 132 | # and sync to be at first. It will make it much easier to bootstrap clusters in the future. 133 | syncOrder: '0' 134 | # The branch can be defined as well. 135 | branch: 'main' 136 | # This will be the release name. 137 | appName: 'argocd' 138 | ## These values are needed when the helm plugin is used. All values are 139 | ## already explains itself. 140 | chartName: 'argo-cd' 141 | chartRepository: 'https://argoproj.github.io/argo-helm' 142 | # It is not needed if the chart is in the repository. 143 | chartVersion: '5.5.6' 144 | # Which namespace to deploy. 145 | namespace: 'argocd' 146 | # Which keyVault secret, i.e which Vault will be used for the app? Not necessary to put. 147 | # You can delete it if you don't use it. 148 | keyVault: '' 149 | ``` 150 | 151 | This is all done with a customized plugin below. I have added a plugin as a sidecar and added this YAML file to **configMap** because it is not an actual Kubernetes resource. There is a value YAML inside of this repository you can check. 152 | You can add this and the argocd-vault-plugin binary to the sidecar's image and use it like that. That solution will make your configuration support on-premise environments as well. 153 | Read more at: [Configure plugin via sidecar](https://argo-cd.readthedocs.io/en/stable/user-guide/config-management-plugins/#option-2-configure-plugin-via-sidecar) 154 | 155 | ```yaml 156 | apiVersion: argoproj.io/v1alpha1 157 | kind: ConfigManagementPlugin 158 | metadata: 159 | name: argocd-vault-plugin 160 | spec: 161 | lockRepo: false 162 | allowConcurrency: true 163 | discover: 164 | ## Discover command catches if the application will be created with the plugin or not. 165 | find: 166 | command: 167 | - bash 168 | - "-c" 169 | - "[[ ${ARGOCD_ENV_pluginName} =~ ^argocd-vault-plugin(|-helm-local-repo|-helm-remote-repo)$ ]] && echo 'OK'" 170 | generate: 171 | command: 172 | - bash 173 | - "-c" 174 | - | 175 | rootFolder=$(pwd | cut -d "/" -f-4) 176 | 177 | # The vault plugin binary will use the secret name defined in the Application.yaml 178 | # If the keyVault value is empty, the command will not be defined and vault plugin 179 | # not be used. It will just generate manifest files. 180 | [[ -n "${ARGOCD_ENV_AVP_SECRET}" ]] && avpCommand="| argocd-vault-plugin generate -s ${ARGOCD_ENV_AVP_SECRET} -" 181 | 182 | # Values YAML's in the order will be checked if they are there or not. If not, they 183 | # will not be added in the externalYamlFiles parameter. You can declare two values 184 | # YAML. A common/base YAML file for all clusters, and cluster specific YAML file 185 | # which would be inside of the Application's own path. 186 | for valuesYaml in "${rootFolder}/commonValues/${ARGOCD_ENV_chartReleaseName}/values.yaml" "values.yaml"; do 187 | [[ -f ${valuesYaml} ]] && externalYamlFiles="$externalYamlFiles --values ${valuesYaml}" 188 | done 189 | 190 | # Helm base command for both a chart in it's own remote helmrepo, or in a git repository. 191 | helmBaseCommand="helm template --name-template ${ARGOCD_ENV_chartReleaseName} --namespace ${ARGOCD_APP_NAMESPACE} --kube-version ${KUBE_VERSION} --api-versions ${KUBE_API_VERSIONS//,/ --api-versions } ${externalYamlFiles} ${ARGOCD_ENV_extraArgs}" 192 | # Helm chart location when helm chart in the git repository. 193 | helmLocalChart="${rootFolder}/charts/$ARGOCD_ENV_chartName/" 194 | # Remote helm charts need releaseName, repo, and version defined to template them 195 | helmRemoteChart="${ARGOCD_ENV_chartName} --repo ${ARGOCD_ENV_chartRepository} --version ${ARGOCD_ENV_chartVersion}" 196 | 197 | # Dependency update should be done for local charts if they have dependencies 198 | helmLocalDependency="helm dependency update ${helmLocalChart} 2>&1 >/dev/null;" 199 | 200 | # If-else structure to determine which command should be executed 201 | if [[ "${ARGOCD_ENV_pluginName}" == 'argocd-vault-plugin' ]]; then 202 | command='find . -regex .*\.ya?ml ! -name Application.y*ml -exec bash -c "cat {}; echo; echo ---" \;' 203 | elif [[ "${ARGOCD_ENV_pluginName}" == 'argocd-vault-plugin-helm-local-repo' ]]; then 204 | command="${helmLocalDependency} ${helmBaseCommand} ${helmLocalChart}" 205 | elif [[ "${ARGOCD_ENV_pluginName}" == 'argocd-vault-plugin-helm-remote-repo' ]]; then 206 | command="${helmBaseCommand} ${helmRemoteChart}" 207 | fi 208 | 209 | # Evaluate the command 210 | eval ${command} ${avpCommand} 211 | ``` 212 | 213 | Here is the folder hierarchy. It has local and remote helm repositories, Kubernetes manifest YAML files, and deploying to specific clusters examples in it all at once. 214 | 215 | ```bash 216 | 📦advanced-argocd-gitops/ # 📦Git Folder 217 | ├── 📜ApplicationSet.yaml ## ├── 📜ApplicationSet manifest file that creates a child Application object for each definition will be done under the app folder. 218 | ├── 📂charts ## ├── 📂Charts folder for apps helm repos. Add the helm chart here as a folder 219 | │ └── 📂hello-world-0.1.0 ## │ └── 📂Hello World helm chart added as an example with version 220 | │ ├── 📜.argocd-allow-concurrency ## │ ├── 📜Add .argocd-allow-concurrency for best practice. Read here https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#enable-concurrent-processing 221 | │ ├── 📂.......... ## │ ├── 📂Helm Chart specific files/folders 222 | │ └── 📜.......... ## │ └── 📜Helm Chart specific files/folders 223 | ├── 📂clusters ## ├── 📂Add clusters and app definitions here. They will be generated for each cluster by ApplicationSet 224 | │ ├── 📂cluster-1 ## │ ├── 📂'cluster-1' will get deployments defined in the folders under it. 225 | │ │ ├── 📂app-1 ## │ │ ├── 📂Remote Helm Chart example. Base value will be used for this app. Check commonValues below. 226 | │ │ │ ├── 📜Application.yaml ## │ │ │ ├── 📜Application.yaml file that holds the information about the helloworld remote Helm Chart. 227 | │ │ │ └── 📜values.yaml ## │ │ │ └── 📜app-1 helm values yaml for cluster-1 228 | │ │ ├── 📂app-1-namespace ## │ │ ├── 📂Creating namespace for newly created helloworld app with limit ranges and quota 229 | │ │ │ ├── 📜Application.yaml ## │ │ │ ├── 📜Application.yaml file that holds the information for applying manifests to the clusters 230 | │ │ │ ├── 📜limitrange.yaml ## │ │ │ ├── 📜app-1-namespace cluster-1 specific manifest yamls 231 | │ │ │ ├── 📜namespace.yaml ## │ │ │ ├── 📜app-1-namespace cluster-1 specific manifest yamls 232 | │ │ │ └── 📜resourcequota.yaml ## │ │ │ └── 📜app-1-namespace cluster-1 specific manifest yamls 233 | │ │ ├── 📂app-2 ## │ │ ├── 📂Remote Helm Chart example. Base value will be used for this app. Check commonValues below. 234 | │ │ │ ├── 📜Application.yaml ## │ │ │ ├── 📜Application.yaml file that holds the information about the helloworld remote Helm Chart. 235 | │ │ │ └── 📜values.yaml ## │ │ │ └── 📜app-2 helm values yaml for cluster-1 236 | │ │ └── 📂app-2-namespace ## │ │ └── 📂Creating namespace for newly created helloworld app with limit ranges and quota 237 | │ │ ├── 📜Application.yaml ## │ │ ├── 📜Application.yaml file that holds the information for applying manifests to the clusters 238 | │ │ ├── 📜limitrange.yaml ## │ │ ├── 📜app-2-namespace cluster-1 specific manifest yamls 239 | │ │ ├── 📜namespace.yaml ## │ │ ├── 📜app-2-namespace cluster-1 specific manifest yamls 240 | │ │ └── 📜resourcequota.yaml ## │ │ └── 📜app-2-namespace cluster-1 specific manifest yamls 241 | │ ├── 📂cluster-2 ## │ ├── 📂'cluster-2' will get deployments defined in the folders under it. 242 | │ │ ├── 📂app-1 ## │ │ ├── 📂Remote Helm Chart example. Base value will be used for this app. Check commonValues below. 243 | │ │ │ ├── 📜Application.yaml ## │ │ │ ├── 📜Application.yaml file that holds the information about the helloworld remote Helm Chart. 244 | │ │ │ └── 📜values.yaml ## │ │ │ └── 📜app-1 helm values yaml for cluster-2 245 | │ │ └── 📂app-1-namespace ## │ │ └── 📂Creating namespace for newly created helloworld app with limit ranges and quota 246 | │ │ ├── 📜Application.yaml ## │ │ ├── 📜Application.yaml file that holds the information for applying manifests to the clusters 247 | │ │ ├── 📜limitrange.yaml ## │ │ ├── 📜app-1-namespace cluster-2 specific manifest yamls 248 | │ │ ├── 📜namespace.yaml ## │ │ ├── 📜app-1-namespace cluster-2 specific manifest yamls 249 | │ │ └── 📜resourcequota.yaml ## │ │ └── 📜app-1-namespace cluster-2 specific manifest yamls 250 | │ └── 📂in-cluster ## │ └── 📂I will maintain ArgoCD and other master cluster resources from here. In case anything happens to the master cluster, it will help us quickly install everything again with minimal downtime. 251 | │ ├── 📂argocd ## │ ├── 📂Remote Argo CD Helm Chart example to manage ArgoCD within ArgoCD 252 | │ │ ├── 📜Application.yaml ## │ │ ├── 📜Application.yaml file that holds the information about the ArgoCD remote Helm Chart. 253 | │ │ └── 📜values.yaml ## │ │ └── 📜All ArgoCD values 254 | │ └── 📂ingress-nginx ## │ └── 📂Nginx ingress controller remote Helm Chart 255 | │ ├── 📜Application.yaml ## │ ├── 📜Application.yaml file that holds the information about the Nginx ingress controller remote Helm Chart. 256 | │ └── 📜values.yaml ## │ └── 📜Nginx ingress controller values 257 | └── 📂commonValues ## └── 📂Add base values of applications for all clusters here 258 | └── 📂app-1 ## └──📂app-1 commonValues for all clusters. 259 | └── 📜values.yaml ## └── 📜values.yaml that holds common values. 260 | ``` 261 | 262 | ## Demo 263 | 264 | Quick demo setting up the example in this repo. 265 | 266 | https://github.com/burhanuguz/advanced-argocd-gitops/blob/master/Demo.md 267 | 268 | ## Summary 269 | 270 | In summary, you can add numerous clusters in mono/multi mix git repositories and manage them with the vault plugin as well. 271 | I absolutely recommend you have one vault backed for the master cluster, and put cluster credentials and other keyVault secrets into the vault as well. Apply those credentials and secrets via Vault, and you will never need to redefine it even if you lose the clusters because you already defined it and manage it via GitOps :) 272 | Also, you can manage even Cilium, OPA, service mesh, and other tools via ArgoCD. 273 | 274 | You can separate Vaults, and each app repo for each cluster you want to deploy. You can create ArgoCD projects and declare them in Application.yaml files. You would give access to very few resources in that project. And inside of the Application.yaml, you can declare another repo that application developers would use. Application developers can add their manifests or values yaml and can't create resources they are not allowed to. You could make it in automated way and keep clusters state in the main repo only, and get the values outside of that repo. 275 | 276 | Helm local/remote charts with external value deployments + Kubernetes Manifests Deployments + With/Without Vault Plugins and you can dig into one app of a cluster at any time. 277 | 278 | It is just to show how powerful ArgoCD can be. Hope it can inspire and be useful to readers. 279 | 280 | Cheers :) 281 | -------------------------------------------------------------------------------- /argo-app.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: argoproj.io/v1alpha1 2 | kind: Application 3 | metadata: 4 | name: all 5 | namespace: argocd 6 | annotations: 7 | argocd.argoproj.io/manifest-generate-paths: '/ApplicationSet.yaml' 8 | spec: 9 | syncPolicy: 10 | automated: {} 11 | syncOptions: 12 | - Replace=true 13 | destination: 14 | namespace: argocd 15 | server: https://kubernetes.default.svc 16 | project: default 17 | source: 18 | path: . 19 | repoURL: https://github.com/burhanuguz/advanced-argocd-gitops.git 20 | targetRevision: HEAD 21 | directory: 22 | include: 'ApplicationSet.yaml' 23 | -------------------------------------------------------------------------------- /argocd-values.yaml: -------------------------------------------------------------------------------- 1 | global: 2 | image: 3 | repository: &avpCmpImage quay.io/argoproj/argocd 4 | 5 | # redis-ha: 6 | # enabled: true 7 | 8 | extraObjects: 9 | - apiVersion: v1 10 | kind: ConfigMap 11 | metadata: 12 | name: cmp-plugin 13 | data: 14 | argocd-vault-plugin.yaml: | 15 | --- 16 | apiVersion: argoproj.io/v1alpha1 17 | kind: ConfigManagementPlugin 18 | metadata: 19 | name: argocd-vault-plugin 20 | spec: 21 | lockRepo: false 22 | allowConcurrency: true 23 | discover: 24 | find: 25 | command: 26 | - bash 27 | - "-c" 28 | - "[[ ${ARGOCD_ENV_pluginName} =~ ^argocd-vault-plugin(|-helm-local-repo|-helm-remote-repo)$ ]] && echo 'OK'" 29 | generate: 30 | command: 31 | - bash 32 | - "-c" 33 | - | 34 | rootFolder=$(pwd | cut -d "/" -f-4) 35 | 36 | [[ -n "${ARGOCD_ENV_AVP_SECRET}" ]] && avpCommand="| argocd-vault-plugin generate -s ${ARGOCD_ENV_AVP_SECRET} -" 37 | 38 | for valuesYaml in "${rootFolder}/commonValues/${ARGOCD_ENV_chartReleaseName}/values.yaml" "values.yaml"; do 39 | [[ -f ${valuesYaml} ]] && externalYamlFiles="$externalYamlFiles --values ${valuesYaml}" 40 | done 41 | 42 | helmBaseCommand="helm template --name-template ${ARGOCD_ENV_chartReleaseName} --namespace ${ARGOCD_APP_NAMESPACE} --kube-version ${KUBE_VERSION} --api-versions ${KUBE_API_VERSIONS//,/ --api-versions } ${externalYamlFiles} ${ARGOCD_ENV_extraArgs}" 43 | helmLocalChart="${rootFolder}/charts/$ARGOCD_ENV_chartName/" 44 | helmRemoteChart="${ARGOCD_ENV_chartName} --repo ${ARGOCD_ENV_chartRepository} --version ${ARGOCD_ENV_chartVersion}" 45 | 46 | helmLocalDependency="helm dependency update ${helmLocalChart} 2>&1 >/dev/null;" 47 | 48 | if [[ "${ARGOCD_ENV_pluginName}" == 'argocd-vault-plugin' ]]; then 49 | command='find . -regex .*\.ya?ml ! -name Application.y*ml -exec bash -c "cat {}; echo; echo ---" \;' 50 | elif [[ "${ARGOCD_ENV_pluginName}" == 'argocd-vault-plugin-helm-local-repo' ]]; then 51 | command="${helmLocalDependency} ${helmBaseCommand} ${helmLocalChart}" 52 | elif [[ "${ARGOCD_ENV_pluginName}" == 'argocd-vault-plugin-helm-remote-repo' ]]; then 53 | command="${helmBaseCommand} ${helmRemoteChart}" 54 | fi 55 | 56 | eval ${command} ${avpCommand} 57 | 58 | server: 59 | # autoscaling: 60 | # enabled: true 61 | # minReplicas: 2 62 | # clusterAdminAccess: 63 | # enabled: false 64 | ingress: 65 | enabled: true 66 | # Add annotations below if you want to secure your cluster 67 | # annotations: 68 | # nginx.ingress.kubernetes.io/force-ssl-redirect: "true" 69 | # nginx.ingress.kubernetes.io/ssl-passthrough: "true" 70 | hosts: 71 | - argocd.local.gd 72 | # Add values like below if you want to secure your cluster 73 | # tls: 74 | # - secretName: argocd-secret 75 | # hosts: 76 | # - argocd.local.gd 77 | # Change http from false to true 78 | # https: true 79 | https: false 80 | 81 | repoServer: 82 | # autoscaling: 83 | # enabled: true 84 | # minReplicas: 2 85 | # clusterAdminAccess: 86 | # enabled: false 87 | initContainers: 88 | - name: download-tools 89 | image: alpine:3.8 90 | command: [sh, -c] 91 | # Don't forget to update this to whatever the stable release version is 92 | # Note the lack of the `v` prefix unlike the git tag 93 | env: 94 | - name: AVP_VERSION 95 | value: "1.13.0" 96 | args: 97 | - >- 98 | wget -O argocd-vault-plugin 99 | https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v${AVP_VERSION}/argocd-vault-plugin_${AVP_VERSION}_linux_amd64 && 100 | chmod +x argocd-vault-plugin && 101 | mv argocd-vault-plugin /custom-tools/ 102 | volumeMounts: 103 | - mountPath: /custom-tools 104 | name: custom-tools 105 | extraContainers: 106 | - name: argocd-vault-plugin 107 | command: [/var/run/argocd/argocd-cmp-server] 108 | image: *avpCmpImage 109 | securityContext: 110 | runAsNonRoot: true 111 | runAsUser: 999 112 | volumeMounts: 113 | - mountPath: /var/run/argocd 114 | name: var-files 115 | - mountPath: /home/argocd/cmp-server/plugins 116 | name: plugins 117 | 118 | # Register plugins into sidecar 119 | - mountPath: /home/argocd/cmp-server/config/plugin.yaml 120 | subPath: argocd-vault-plugin.yaml 121 | name: cmp-plugin 122 | 123 | # Important: Mount tools into $PATH 124 | - name: custom-tools 125 | subPath: argocd-vault-plugin 126 | mountPath: /usr/local/bin/argocd-vault-plugin 127 | 128 | volumes: 129 | - configMap: 130 | name: cmp-plugin 131 | name: cmp-plugin 132 | - name: custom-tools 133 | emptyDir: {} 134 | serviceAccount: 135 | automountServiceAccountToken: true 136 | rbac: 137 | - apiGroups: 138 | - "" 139 | resources: 140 | - secrets 141 | verbs: 142 | - get 143 | - list 144 | - watch 145 | 146 | # applicationSet: 147 | # replicas: 2 148 | 149 | configs: 150 | params: 151 | server.insecure: true 152 | # Change values like below if you want to secure your cluster 153 | # params: 154 | # server.insecure: false 155 | # secret: 156 | # argocdServerTlsConfig: 157 | # crt: | 158 | # 159 | # key: | 160 | # 161 | -------------------------------------------------------------------------------- /charts/hello-world-0.1.0/.argocd-allow-concurrency: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/burhanuguz/advanced-argocd-gitops/e3e532a68a08223e80139e25d059d3af06cebbdc/charts/hello-world-0.1.0/.argocd-allow-concurrency -------------------------------------------------------------------------------- /charts/hello-world-0.1.0/.helmignore: -------------------------------------------------------------------------------- 1 | # Patterns to ignore when building packages. 2 | # This supports shell glob matching, relative path matching, and 3 | # negation (prefixed with !). Only one pattern per line. 4 | .DS_Store 5 | # Common VCS dirs 6 | .git/ 7 | .gitignore 8 | .bzr/ 9 | .bzrignore 10 | .hg/ 11 | .hgignore 12 | .svn/ 13 | # Common backup files 14 | *.swp 15 | *.bak 16 | *.tmp 17 | *.orig 18 | *~ 19 | # Various IDEs 20 | .project 21 | .idea/ 22 | *.tmproj 23 | .vscode/ 24 | -------------------------------------------------------------------------------- /charts/hello-world-0.1.0/Chart.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v2 2 | name: hello-world 3 | description: A Helm chart for Kubernetes 4 | 5 | # A chart can be either an 'application' or a 'library' chart. 6 | # 7 | # Application charts are a collection of templates that can be packaged into versioned archives 8 | # to be deployed. 9 | # 10 | # Library charts provide useful utilities or functions for the chart developer. They're included as 11 | # a dependency of application charts to inject those utilities and functions into the rendering 12 | # pipeline. Library charts do not define any templates and therefore cannot be deployed. 13 | type: application 14 | 15 | # This is the chart version. This version number should be incremented each time you make changes 16 | # to the chart and its templates, including the app version. 17 | # Versions are expected to follow Semantic Versioning (https://semver.org/) 18 | version: 0.1.0 19 | 20 | # This is the version number of the application being deployed. This version number should be 21 | # incremented each time you make changes to the application. Versions are not expected to 22 | # follow Semantic Versioning. They should reflect the version the application is using. 23 | # It is recommended to use it with quotes. 24 | appVersion: "1.16.0" 25 | -------------------------------------------------------------------------------- /charts/hello-world-0.1.0/README.md: -------------------------------------------------------------------------------- 1 | # Hello World 2 | -------------------------------------------------------------------------------- /charts/hello-world-0.1.0/templates/NOTES.txt: -------------------------------------------------------------------------------- 1 | 1. Get the application URL by running these commands: 2 | {{- if contains "NodePort" .Values.service.type }} 3 | export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "hello-world.fullname" . }}) 4 | export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}") 5 | echo http://$NODE_IP:$NODE_PORT 6 | {{- else if contains "LoadBalancer" .Values.service.type }} 7 | NOTE: It may take a few minutes for the LoadBalancer IP to be available. 8 | You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "hello-world.fullname" . }}' 9 | export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "hello-world.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}") 10 | echo http://$SERVICE_IP:{{ .Values.service.port }} 11 | {{- else if contains "ClusterIP" .Values.service.type }} 12 | export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "hello-world.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}") 13 | export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}") 14 | echo "Visit http://127.0.0.1:8080 to use your application" 15 | kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT 16 | {{- end }} 17 | -------------------------------------------------------------------------------- /charts/hello-world-0.1.0/templates/_helpers.tpl: -------------------------------------------------------------------------------- 1 | {{/* 2 | Expand the name of the chart. 3 | */}} 4 | {{- define "hello-world.name" -}} 5 | {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} 6 | {{- end }} 7 | 8 | {{/* 9 | Create a default fully qualified app name. 10 | We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). 11 | If release name contains chart name it will be used as a full name. 12 | */}} 13 | {{- define "hello-world.fullname" -}} 14 | {{- if .Values.fullnameOverride }} 15 | {{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} 16 | {{- else }} 17 | {{- $name := default .Chart.Name .Values.nameOverride }} 18 | {{- if contains $name .Release.Name }} 19 | {{- .Release.Name | trunc 63 | trimSuffix "-" }} 20 | {{- else }} 21 | {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} 22 | {{- end }} 23 | {{- end }} 24 | {{- end }} 25 | 26 | {{/* 27 | Create chart name and version as used by the chart label. 28 | */}} 29 | {{- define "hello-world.chart" -}} 30 | {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} 31 | {{- end }} 32 | 33 | {{/* 34 | Common labels 35 | */}} 36 | {{- define "hello-world.labels" -}} 37 | helm.sh/chart: {{ include "hello-world.chart" . }} 38 | {{ include "hello-world.selectorLabels" . }} 39 | {{- if .Chart.AppVersion }} 40 | app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} 41 | {{- end }} 42 | app.kubernetes.io/managed-by: {{ .Release.Service }} 43 | {{- end }} 44 | 45 | {{/* 46 | Selector labels 47 | */}} 48 | {{- define "hello-world.selectorLabels" -}} 49 | app.kubernetes.io/name: {{ include "hello-world.name" . }} 50 | app.kubernetes.io/instance: {{ .Release.Name }} 51 | {{- end }} 52 | 53 | {{/* 54 | Create the name of the service account to use 55 | */}} 56 | {{- define "hello-world.serviceAccountName" -}} 57 | {{- if .Values.serviceAccount.create }} 58 | {{- default (include "hello-world.fullname" .) .Values.serviceAccount.name }} 59 | {{- else }} 60 | {{- default "default" .Values.serviceAccount.name }} 61 | {{- end }} 62 | {{- end }} 63 | -------------------------------------------------------------------------------- /charts/hello-world-0.1.0/templates/deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: {{ include "hello-world.fullname" . }} 5 | labels: 6 | {{- include "hello-world.labels" . | nindent 4 }} 7 | spec: 8 | replicas: {{ .Values.replicaCount }} 9 | selector: 10 | matchLabels: 11 | {{- include "hello-world.selectorLabels" . | nindent 6 }} 12 | template: 13 | metadata: 14 | labels: 15 | {{- include "hello-world.selectorLabels" . | nindent 8 }} 16 | spec: 17 | serviceAccountName: {{ include "hello-world.serviceAccountName" . }} 18 | containers: 19 | - name: {{ .Chart.Name }} 20 | image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" 21 | imagePullPolicy: {{ .Values.image.pullPolicy }} 22 | ports: 23 | - name: http 24 | containerPort: 80 25 | protocol: TCP 26 | livenessProbe: 27 | httpGet: 28 | path: / 29 | port: http 30 | readinessProbe: 31 | httpGet: 32 | path: / 33 | port: http 34 | -------------------------------------------------------------------------------- /charts/hello-world-0.1.0/templates/service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: {{ include "hello-world.fullname" . }} 5 | labels: 6 | {{- include "hello-world.labels" . | nindent 4 }} 7 | spec: 8 | type: {{ .Values.service.type }} 9 | ports: 10 | - port: {{ .Values.service.port }} 11 | targetPort: http 12 | protocol: TCP 13 | name: http 14 | selector: 15 | {{- include "hello-world.selectorLabels" . | nindent 4 }} 16 | -------------------------------------------------------------------------------- /charts/hello-world-0.1.0/templates/serviceaccount.yaml: -------------------------------------------------------------------------------- 1 | {{- if .Values.serviceAccount.create -}} 2 | apiVersion: v1 3 | kind: ServiceAccount 4 | metadata: 5 | name: {{ include "hello-world.serviceAccountName" . }} 6 | labels: 7 | {{- include "hello-world.labels" . | nindent 4 }} 8 | {{- with .Values.serviceAccount.annotations }} 9 | annotations: 10 | {{- toYaml . | nindent 4 }} 11 | {{- end }} 12 | {{- end }} 13 | -------------------------------------------------------------------------------- /charts/hello-world-0.1.0/values.yaml: -------------------------------------------------------------------------------- 1 | # Default values for hello-world. 2 | # This is a YAML-formatted file. 3 | # Declare variables to be passed into your templates. 4 | 5 | replicaCount: 1 6 | 7 | image: 8 | repository: nginx 9 | pullPolicy: IfNotPresent 10 | # Overrides the image tag whose default is the chart appVersion. 11 | tag: "" 12 | 13 | nameOverride: "" 14 | fullnameOverride: "" 15 | 16 | serviceAccount: 17 | # Specifies whether a service account should be created 18 | create: true 19 | # Annotations to add to the service account 20 | annotations: {} 21 | # The name of the service account to use. 22 | # If not set and create is true, a name is generated using the fullname template 23 | name: "" 24 | 25 | service: 26 | type: ClusterIP 27 | port: 80 28 | -------------------------------------------------------------------------------- /clusters/cluster-1/app-1-namespace/Application.yaml: -------------------------------------------------------------------------------- 1 | repo: https://github.com/burhanuguz/advanced-argocd-gitops.git 2 | plugin: 'argocd-vault-plugin' 3 | argoProject: 'default' 4 | syncOrder: '0' 5 | branch: 'master' 6 | namespace: 'app-1-namespace' 7 | appName: 'app-1-namespace' -------------------------------------------------------------------------------- /clusters/cluster-1/app-1-namespace/limitrange.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: LimitRange 3 | metadata: 4 | name: app-1-limitrange 5 | namespace: app-1-namespace 6 | spec: 7 | limits: 8 | - type: Container 9 | default: 10 | cpu: 250m 11 | memory: 900Mi 12 | defaultRequest: 13 | cpu: 100m 14 | memory: 600Mi 15 | max: 16 | cpu: 1 17 | memory: 2100Mi -------------------------------------------------------------------------------- /clusters/cluster-1/app-1-namespace/namespace.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Namespace 3 | metadata: 4 | name: app-1-namespace -------------------------------------------------------------------------------- /clusters/cluster-1/app-1-namespace/resourcequota.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ResourceQuota 3 | metadata: 4 | name: app-1-resourcequota 5 | namespace: app-1-namespace 6 | spec: 7 | hard: 8 | requests.storage: "15Gi" 9 | requests.cpu: "2" 10 | requests.memory: 2Gi 11 | limits.cpu: "4" 12 | limits.memory: 4Gi -------------------------------------------------------------------------------- /clusters/cluster-1/app-1/Application.yaml: -------------------------------------------------------------------------------- 1 | repo: https://github.com/burhanuguz/advanced-argocd-gitops.git 2 | plugin: 'argocd-vault-plugin-helm-remote-repo' 3 | argoProject: 'default' 4 | syncOrder: '0' 5 | branch: 'master' 6 | appName: 'app-1' 7 | chartName: 'hello-world' 8 | chartRepository: 'https://helm.github.io/examples' 9 | chartVersion: '0.1.0' 10 | namespace: 'app-1-namespace' -------------------------------------------------------------------------------- /clusters/cluster-1/app-1/values.yaml: -------------------------------------------------------------------------------- 1 | image: 2 | tag: "1.22.0" -------------------------------------------------------------------------------- /clusters/cluster-1/app-2-namespace/Application.yaml: -------------------------------------------------------------------------------- 1 | repo: https://github.com/burhanuguz/advanced-argocd-gitops.git 2 | plugin: 'argocd-vault-plugin' 3 | argoProject: 'default' 4 | syncOrder: '0' 5 | branch: 'master' 6 | namespace: 'app-2-namespace' 7 | appName: 'app-2-namespace' -------------------------------------------------------------------------------- /clusters/cluster-1/app-2-namespace/limitrange.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: LimitRange 3 | metadata: 4 | name: app-2-limitrange 5 | namespace: app-2-namespace 6 | spec: 7 | limits: 8 | - type: Container 9 | default: 10 | cpu: 250m 11 | memory: 900Mi 12 | defaultRequest: 13 | cpu: 100m 14 | memory: 600Mi 15 | max: 16 | cpu: 1 17 | memory: 2100Mi -------------------------------------------------------------------------------- /clusters/cluster-1/app-2-namespace/namespace.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Namespace 3 | metadata: 4 | name: app-2-namespace -------------------------------------------------------------------------------- /clusters/cluster-1/app-2-namespace/resourcequota.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ResourceQuota 3 | metadata: 4 | name: app-2-resourcequota 5 | namespace: app-2-namespace 6 | spec: 7 | hard: 8 | requests.storage: "15Gi" 9 | requests.cpu: "2" 10 | requests.memory: 2Gi 11 | limits.cpu: "4" 12 | limits.memory: 4Gi -------------------------------------------------------------------------------- /clusters/cluster-1/app-2/Application.yaml: -------------------------------------------------------------------------------- 1 | repo: https://github.com/burhanuguz/advanced-argocd-gitops.git 2 | plugin: 'argocd-vault-plugin-helm-local-repo' 3 | argoProject: 'default' 4 | syncOrder: '0' 5 | branch: 'master' 6 | appName: 'app-2' 7 | chartName: 'hello-world-0.1.0' 8 | namespace: 'app-2-namespace' -------------------------------------------------------------------------------- /clusters/cluster-1/app-2/values.yaml: -------------------------------------------------------------------------------- 1 | replicaCount: 1 2 | image: 3 | repository: nginx 4 | pullPolicy: IfNotPresent 5 | # Overrides the image tag whose default is the chart appVersion. 6 | tag: "latest" -------------------------------------------------------------------------------- /clusters/cluster-2/app-1-namespace/Application.yaml: -------------------------------------------------------------------------------- 1 | repo: https://github.com/burhanuguz/advanced-argocd-gitops.git 2 | plugin: 'argocd-vault-plugin' 3 | argoProject: 'default' 4 | syncOrder: '0' 5 | branch: 'master' 6 | namespace: 'app-1-namespace' 7 | appName: 'app-1-namespace' -------------------------------------------------------------------------------- /clusters/cluster-2/app-1-namespace/limitrange.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: LimitRange 3 | metadata: 4 | name: app-1-limitrange 5 | namespace: app-1-namespace 6 | spec: 7 | limits: 8 | - type: Container 9 | default: 10 | cpu: 250m 11 | memory: 900Mi 12 | defaultRequest: 13 | cpu: 100m 14 | memory: 600Mi 15 | max: 16 | cpu: 1 17 | memory: 2100Mi -------------------------------------------------------------------------------- /clusters/cluster-2/app-1-namespace/namespace.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Namespace 3 | metadata: 4 | name: app-1-namespace -------------------------------------------------------------------------------- /clusters/cluster-2/app-1-namespace/resourcequota.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ResourceQuota 3 | metadata: 4 | name: app-1-resourcequota 5 | namespace: app-1-namespace 6 | spec: 7 | hard: 8 | requests.storage: "15Gi" 9 | requests.cpu: "2" 10 | requests.memory: 2Gi 11 | limits.cpu: "4" 12 | limits.memory: 4Gi -------------------------------------------------------------------------------- /clusters/cluster-2/app-1/Application.yaml: -------------------------------------------------------------------------------- 1 | repo: https://github.com/burhanuguz/advanced-argocd-gitops.git 2 | plugin: 'argocd-vault-plugin-helm-remote-repo' 3 | argoProject: 'default' 4 | syncOrder: '0' 5 | branch: 'master' 6 | appName: 'app-1' 7 | chartName: 'hello-world' 8 | chartRepository: 'https://helm.github.io/examples' 9 | chartVersion: '0.1.0' 10 | namespace: 'app-1-namespace' -------------------------------------------------------------------------------- /clusters/cluster-2/app-1/values.yaml: -------------------------------------------------------------------------------- 1 | image: 2 | tag: "stable" -------------------------------------------------------------------------------- /clusters/in-cluster/argocd/Application.yaml: -------------------------------------------------------------------------------- 1 | repo: https://github.com/burhanuguz/advanced-argocd-gitops.git 2 | plugin: 'argocd-vault-plugin-helm-remote-repo' 3 | argoProject: 'default' 4 | syncOrder: '0' 5 | branch: 'master' 6 | appName: 'argocd' 7 | chartName: 'argo-cd' 8 | chartRepository: 'https://argoproj.github.io/argo-helm' 9 | chartVersion: '5.16.4' 10 | namespace: 'argocd' 11 | # keyVault: argocd-vault-plugin-azure-credentials-in-cluster -------------------------------------------------------------------------------- /clusters/in-cluster/argocd/values.yaml: -------------------------------------------------------------------------------- 1 | global: 2 | image: 3 | repository: &avpCmpImage quay.io/argoproj/argocd 4 | 5 | # redis-ha: 6 | # enabled: true 7 | 8 | extraObjects: 9 | - apiVersion: v1 10 | kind: ConfigMap 11 | metadata: 12 | name: cmp-plugin 13 | data: 14 | argocd-vault-plugin.yaml: | 15 | --- 16 | apiVersion: argoproj.io/v1alpha1 17 | kind: ConfigManagementPlugin 18 | metadata: 19 | name: argocd-vault-plugin 20 | spec: 21 | lockRepo: false 22 | allowConcurrency: true 23 | discover: 24 | find: 25 | command: 26 | - bash 27 | - "-c" 28 | - "[[ ${ARGOCD_ENV_pluginName} =~ ^argocd-vault-plugin(|-helm-local-repo|-helm-remote-repo)$ ]] && echo 'OK'" 29 | generate: 30 | command: 31 | - bash 32 | - "-c" 33 | - | 34 | rootFolder=$(pwd | cut -d "/" -f-4) 35 | 36 | [[ -n "${ARGOCD_ENV_AVP_SECRET}" ]] && avpCommand="| argocd-vault-plugin generate -s ${ARGOCD_ENV_AVP_SECRET} -" 37 | 38 | for valuesYaml in "${rootFolder}/commonValues/${ARGOCD_ENV_chartReleaseName}/values.yaml" "values.yaml"; do 39 | [[ -f ${valuesYaml} ]] && externalYamlFiles="$externalYamlFiles --values ${valuesYaml}" 40 | done 41 | 42 | helmBaseCommand="helm template --name-template ${ARGOCD_ENV_chartReleaseName} --namespace ${ARGOCD_APP_NAMESPACE} --kube-version ${KUBE_VERSION} --api-versions ${KUBE_API_VERSIONS//,/ --api-versions } ${externalYamlFiles} ${ARGOCD_ENV_extraArgs}" 43 | helmLocalChart="${rootFolder}/charts/$ARGOCD_ENV_chartName/" 44 | helmRemoteChart="${ARGOCD_ENV_chartName} --repo ${ARGOCD_ENV_chartRepository} --version ${ARGOCD_ENV_chartVersion}" 45 | 46 | helmLocalDependency="helm dependency update ${helmLocalChart} 2>&1 >/dev/null;" 47 | 48 | if [[ "${ARGOCD_ENV_pluginName}" == 'argocd-vault-plugin' ]]; then 49 | command='find . -regex .*\.ya?ml ! -name Application.y*ml -exec bash -c "cat {}; echo; echo ---" \;' 50 | elif [[ "${ARGOCD_ENV_pluginName}" == 'argocd-vault-plugin-helm-local-repo' ]]; then 51 | command="${helmLocalDependency} ${helmBaseCommand} ${helmLocalChart}" 52 | elif [[ "${ARGOCD_ENV_pluginName}" == 'argocd-vault-plugin-helm-remote-repo' ]]; then 53 | command="${helmBaseCommand} ${helmRemoteChart}" 54 | fi 55 | 56 | eval ${command} ${avpCommand} 57 | 58 | server: 59 | # autoscaling: 60 | # enabled: true 61 | # minReplicas: 2 62 | # clusterAdminAccess: 63 | # enabled: false 64 | ingress: 65 | enabled: true 66 | # Add annotations below if you want to secure your cluster 67 | # annotations: 68 | # nginx.ingress.kubernetes.io/force-ssl-redirect: "true" 69 | # nginx.ingress.kubernetes.io/ssl-passthrough: "true" 70 | hosts: 71 | - argocd.local.gd 72 | # Add values like below if you want to secure your cluster 73 | # tls: 74 | # - secretName: argocd-secret 75 | # hosts: 76 | # - argocd.local.gd 77 | # Change http from false to true 78 | # https: true 79 | https: false 80 | 81 | repoServer: 82 | # autoscaling: 83 | # enabled: true 84 | # minReplicas: 2 85 | # clusterAdminAccess: 86 | # enabled: false 87 | initContainers: 88 | - name: download-tools 89 | image: alpine:3.8 90 | command: [sh, -c] 91 | # Don't forget to update this to whatever the stable release version is 92 | # Note the lack of the `v` prefix unlike the git tag 93 | env: 94 | - name: AVP_VERSION 95 | value: "1.13.0" 96 | args: 97 | - >- 98 | wget -O argocd-vault-plugin 99 | https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v${AVP_VERSION}/argocd-vault-plugin_${AVP_VERSION}_linux_amd64 && 100 | chmod +x argocd-vault-plugin && 101 | mv argocd-vault-plugin /custom-tools/ 102 | volumeMounts: 103 | - mountPath: /custom-tools 104 | name: custom-tools 105 | extraContainers: 106 | - name: argocd-vault-plugin 107 | command: [/var/run/argocd/argocd-cmp-server] 108 | image: *avpCmpImage 109 | securityContext: 110 | runAsNonRoot: true 111 | runAsUser: 999 112 | volumeMounts: 113 | - mountPath: /var/run/argocd 114 | name: var-files 115 | - mountPath: /home/argocd/cmp-server/plugins 116 | name: plugins 117 | 118 | # Register plugins into sidecar 119 | - mountPath: /home/argocd/cmp-server/config/plugin.yaml 120 | subPath: argocd-vault-plugin.yaml 121 | name: cmp-plugin 122 | 123 | # Important: Mount tools into $PATH 124 | - name: custom-tools 125 | subPath: argocd-vault-plugin 126 | mountPath: /usr/local/bin/argocd-vault-plugin 127 | 128 | volumes: 129 | - configMap: 130 | name: cmp-plugin 131 | name: cmp-plugin 132 | - name: custom-tools 133 | emptyDir: {} 134 | serviceAccount: 135 | automountServiceAccountToken: true 136 | rbac: 137 | - apiGroups: 138 | - "" 139 | resources: 140 | - secrets 141 | verbs: 142 | - get 143 | - list 144 | - watch 145 | 146 | # applicationSet: 147 | # replicas: 2 148 | 149 | configs: 150 | params: 151 | server.insecure: true 152 | # Change values like below if you want to secure your cluster 153 | # params: 154 | # server.insecure: false 155 | # secret: 156 | # argocdServerTlsConfig: 157 | # You can add secrets from vault 158 | # crt: | 159 | # 160 | # You can add secrets from vault 161 | # key: | 162 | # 163 | -------------------------------------------------------------------------------- /clusters/in-cluster/ingress-nginx/Application.yaml: -------------------------------------------------------------------------------- 1 | repo: https://github.com/burhanuguz/advanced-argocd-gitops.git 2 | plugin: 'argocd-vault-plugin-helm-remote-repo' 3 | argoProject: 'default' 4 | syncOrder: '0' 5 | branch: 'master' 6 | appName: 'ingress-nginx' 7 | chartName: 'ingress-nginx' 8 | chartRepository: 'https://kubernetes.github.io/ingress-nginx' 9 | chartVersion: '4.2.5' 10 | namespace: 'ingress-nginx' -------------------------------------------------------------------------------- /clusters/in-cluster/ingress-nginx/values.yaml: -------------------------------------------------------------------------------- 1 | controller: 2 | ingressClassResource: 3 | default: true 4 | service: 5 | type: NodePort 6 | nodePorts: 7 | http: 30080 8 | https: 30443 9 | extraArgs: 10 | enable-ssl-passthrough: '' -------------------------------------------------------------------------------- /commonValues/app-1-values.yaml: -------------------------------------------------------------------------------- 1 | replicaCount: 1 2 | image: 3 | repository: nginx 4 | pullPolicy: IfNotPresent 5 | # Overrides the image tag whose default is the chart appVersion. 6 | tag: "latest" -------------------------------------------------------------------------------- /kind-cluster-setup.sh: -------------------------------------------------------------------------------- 1 | ## Create the master cluster 2 | cat << EOF | kind create cluster --config - 3 | kind: Cluster 4 | apiVersion: kind.x-k8s.io/v1alpha4 5 | name: master 6 | nodes: 7 | - role: control-plane 8 | extraPortMappings: 9 | - containerPort: 30080 10 | hostPort: 80 11 | - containerPort: 30443 12 | hostPort: 443 13 | EOF 14 | 15 | ## Wait for the master Kubernetes cluster to get ready 16 | kubectl --context "kind-master" wait --namespace kube-system --for=condition=ready pod --all --timeout=120s 17 | 18 | ## Install Kubernetes Nginx Ingress 19 | helm upgrade --install ingress-nginx ingress-nginx --kube-context "kind-master" \ 20 | --repo https://kubernetes.github.io/ingress-nginx \ 21 | --namespace ingress-nginx --create-namespace \ 22 | --set controller.ingressClassResource.default=true \ 23 | --set controller.service.type=NodePort \ 24 | --set controller.service.nodePorts.http=30080 \ 25 | --set controller.service.nodePorts.https=30443 \ 26 | --set controller.extraArgs.enable-ssl-passthrough='' 27 | 28 | ## Wait for ingress to get ready 29 | kubectl --context "kind-master" wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/mponent=controller --timeout=120s 30 | 31 | ## Install ArgoCD cluster. I use the same values YAML as the values files inside the GitHub repository 32 | helm upgrade --install argocd argo-cd --kube-context "kind-master" \ 33 | --repo https://argoproj.github.io/argo-helm \ 34 | --namespace=argocd --create-namespace \ 35 | -f argocd-values.yaml 36 | 37 | ## Wait for argocd to get ready 38 | kubectl --context "kind-master" wait --namespace argocd --for=condition=ready pod --all --timeout=120s 39 | 40 | ## Get the initial secret and then apply the argo-appy.yaml to install the Application Set to the cluster. 41 | kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d 42 | kubectl apply -f argo-app.yaml 43 | 44 | ## Create other two clusters 45 | kind create cluster --name=cluster-1 46 | kind create cluster --name=cluster-2 47 | 48 | ## Copy their kubeconfig YAML file. We also will use combine three conf, and their use is the same, we need to change the username in kubeconfig YAMLs 49 | docker cp cluster-1-control-plane:/etc/kubernetes/admin.conf ../cluster-1-admin.conf; sed -i 's/kubernetes-admin/kubernetes-admin-1/' ../cluster-1-admin.conf 50 | docker cp cluster-2-control-plane:/etc/kubernetes/admin.conf ../cluster-2-admin.conf; sed -i 's/kubernetes-admin/kubernetes-admin-2/' ../cluster-2-admin.conf 51 | wget --no-check-certificate https://argocd.local.gd/download/argocd-linux-amd64 -O ../argocd-linux-amd64 52 | 53 | ## Copy argocd cli and kubeconfig files to kind-master cluster. Windows WSL2 can not communicate 54 | ## with containers. That's why it has to be dealt with inside the container. 55 | docker cp ../cluster-1-admin.conf master-control-plane:/root/ 56 | docker cp ../cluster-2-admin.conf master-control-plane:/root/ 57 | docker cp ../argocd-linux-amd64 master-control-plane:/root/argocd 58 | 59 | ## Run commands below to login and then and other clusters to argocd to manage 60 | docker exec -it master-control-plane chmod +x /root/argocd 61 | 62 | argoPass=$(kubectl --context kind-master -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d) 63 | docker exec -it master-control-plane /root/argocd login argocd.local.gd:30443 --insecure --username=admin --password="${argoPass}" 64 | 65 | ## Combining all three cluster in one kubeconfig and using them for each command 66 | kindconfig='/etc/kubernetes/admin.conf:/root/cluster-1-admin.conf:/root/cluster-2-admin.conf' 67 | docker exec -it master-control-plane bash -c "KUBECONFIG=${kindconfig} /root/argocd cluster add --name cluster-1 --insecure kubernetes-admin-1@cluster-1 --yes" 68 | docker exec -it master-control-plane bash -c "KUBECONFIG=${kindconfig} /root/argocd cluster add --name cluster-2 --insecure kubernetes-admin-2@cluster-2 --yes" --------------------------------------------------------------------------------