├── .gitignore ├── environments ├── base │ └── apps │ │ ├── kustomization.yaml │ │ └── logging-claim.yaml └── dev │ ├── env │ ├── kustomization.yaml │ ├── namespace.yaml │ ├── provider-config.yaml │ └── cluster-config.json │ └── apps │ ├── kustomization.yaml │ └── logging-claim.yaml ├── docs ├── images │ ├── argo-ui-helm.png │ ├── managed-resource.png │ ├── composition-and-xrd.png │ ├── crossplane-vs-helm.png │ ├── crossplane-in-gitops.png │ └── managed-resource-template.png ├── using-crossplane-in-gitops-1.md ├── using-crossplane-in-gitops-2.md └── using-crossplane-in-gitops-3.md ├── config ├── capabilities │ ├── helm │ │ └── logging │ │ │ ├── values.yaml │ │ │ ├── templates │ │ │ ├── subscription.yaml │ │ │ ├── kibana.yaml │ │ │ ├── elasticsearch.yaml │ │ │ └── post-uninstall-job.yaml │ │ │ └── Chart.yaml │ ├── helm-composition │ │ └── logging │ │ │ ├── values.yaml │ │ │ ├── templates │ │ │ └── composition.yaml │ │ │ └── Chart.yaml │ ├── crossplane-helm │ │ └── logging │ │ │ ├── values.yaml │ │ │ ├── templates │ │ │ ├── subscription.yaml │ │ │ ├── clusterserviceversion.yaml │ │ │ ├── kibana.yaml │ │ │ ├── elasticsearch.yaml │ │ │ └── post-install-job.yaml │ │ │ └── Chart.yaml │ ├── managed-resources │ │ └── logging │ │ │ ├── subscription.yaml │ │ │ ├── kibana.yaml │ │ │ ├── clusterserviceversion.yaml │ │ │ └── elasticsearch.yaml │ └── composition │ │ └── logging │ │ ├── definition.yaml │ │ └── composition.yaml └── shared │ ├── argocd │ ├── argocd-admin.yaml │ └── argocd-cm.yaml │ ├── crossplane-ext │ └── capabilities-configuration.yaml │ ├── argocd.yaml │ ├── olm.yaml │ ├── crossplane-ext.yaml │ ├── sealed-secrets.yaml │ └── crossplane.yaml ├── scripts ├── kind.yaml ├── config.sh └── install.sh ├── LICENSE └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | */.cache/ 2 | */.work/ 3 | environments/*/env/cluster-config.yaml 4 | -------------------------------------------------------------------------------- /environments/base/apps/kustomization.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | resources: 3 | - logging-claim.yaml 4 | -------------------------------------------------------------------------------- /docs/images/argo-ui-helm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/morningspace/capabilities-shim-gitops/HEAD/docs/images/argo-ui-helm.png -------------------------------------------------------------------------------- /docs/images/managed-resource.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/morningspace/capabilities-shim-gitops/HEAD/docs/images/managed-resource.png -------------------------------------------------------------------------------- /environments/dev/env/kustomization.yaml: -------------------------------------------------------------------------------- 1 | resources: 2 | - cluster-config.json 3 | - provider-config.yaml 4 | - namespace.yaml 5 | -------------------------------------------------------------------------------- /docs/images/composition-and-xrd.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/morningspace/capabilities-shim-gitops/HEAD/docs/images/composition-and-xrd.png -------------------------------------------------------------------------------- /docs/images/crossplane-vs-helm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/morningspace/capabilities-shim-gitops/HEAD/docs/images/crossplane-vs-helm.png -------------------------------------------------------------------------------- /docs/images/crossplane-in-gitops.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/morningspace/capabilities-shim-gitops/HEAD/docs/images/crossplane-in-gitops.png -------------------------------------------------------------------------------- /docs/images/managed-resource-template.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/morningspace/capabilities-shim-gitops/HEAD/docs/images/managed-resource-template.png -------------------------------------------------------------------------------- /environments/dev/apps/kustomization.yaml: -------------------------------------------------------------------------------- 1 | namespace: dev 2 | 3 | bases: 4 | - ../../base/apps 5 | 6 | patchesStrategicMerge: 7 | - logging-claim.yaml 8 | -------------------------------------------------------------------------------- /environments/dev/env/namespace.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: Namespace 4 | metadata: 5 | name: dev 6 | spec: 7 | finalizers: 8 | - kubernetes 9 | -------------------------------------------------------------------------------- /config/capabilities/helm/logging/values.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | metadata: 3 | name: my-logging-stack 4 | namespace: default 5 | spec: 6 | parameters: 7 | esVersion: 7.13.3 8 | kibanaVersion: 7.13.3 9 | -------------------------------------------------------------------------------- /config/capabilities/helm-composition/logging/values.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | metadata: 3 | name: my-logging-stack 4 | namespace: default 5 | spec: 6 | parameters: 7 | esVersion: 7.13.3 8 | kibanaVersion: 7.13.3 9 | -------------------------------------------------------------------------------- /scripts/kind.yaml: -------------------------------------------------------------------------------- 1 | kind: Cluster 2 | apiVersion: kind.x-k8s.io/v1alpha4 3 | nodes: 4 | - role: control-plane 5 | extraPortMappings: 6 | - containerPort: 30443 7 | hostPort: 9443 8 | listenAddress: "0.0.0.0" 9 | -------------------------------------------------------------------------------- /config/capabilities/crossplane-helm/logging/values.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | metadata: 3 | name: my-logging-stack 4 | namespace: dev 5 | spec: 6 | parameters: 7 | esVersion: 7.13.3 8 | kibanaVersion: 7.13.3 9 | providerConfigRef: 10 | name: provider-config-dev 11 | -------------------------------------------------------------------------------- /environments/dev/env/provider-config.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: kubernetes.crossplane.io/v1alpha1 2 | kind: ProviderConfig 3 | metadata: 4 | name: provider-config-dev 5 | spec: 6 | credentials: 7 | source: Secret 8 | secretRef: 9 | namespace: dev 10 | name: cluster-config 11 | key: kubeconfig 12 | -------------------------------------------------------------------------------- /environments/dev/apps/logging-claim.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: capabilities.morningspace.io/v1alpha1 2 | kind: LoggingClaim 3 | metadata: 4 | annotations: 5 | capabilities.morningspace.io/provider-config: provider-config-dev 6 | name: my-logging-stack 7 | spec: 8 | parameters: 9 | esVersion: 7.15.0 10 | kibanaVersion: 7.15.0 -------------------------------------------------------------------------------- /environments/base/apps/logging-claim.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: capabilities.morningspace.io/v1alpha1 2 | kind: LoggingClaim 3 | metadata: 4 | name: my-logging-stack 5 | spec: 6 | parameters: 7 | esVersion: 7.13.3 8 | kibanaVersion: 7.13.3 9 | compositionSelector: 10 | matchLabels: 11 | capability: logging 12 | provider: olm 13 | -------------------------------------------------------------------------------- /config/shared/argocd/argocd-admin.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: rbac.authorization.k8s.io/v1 3 | kind: RoleBinding 4 | metadata: 5 | name: argocd-admin 6 | namespace: argocd 7 | roleRef: 8 | apiGroup: rbac.authorization.k8s.io 9 | kind: ClusterRole 10 | name: admin 11 | subjects: 12 | - kind: ServiceAccount 13 | name: argocd-application-controller 14 | namespace: argocd 15 | -------------------------------------------------------------------------------- /config/capabilities/helm/logging/templates/subscription.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: operators.coreos.com/v1alpha1 3 | kind: Subscription 4 | metadata: 5 | namespace: operators 6 | name: {{.Values.metadata.name}} 7 | annotations: 8 | argocd.argoproj.io/sync-wave: "100" 9 | spec: 10 | channel: stable 11 | name: elastic-cloud-eck 12 | source: operatorhubio-catalog 13 | sourceNamespace: olm 14 | -------------------------------------------------------------------------------- /scripts/config.sh: -------------------------------------------------------------------------------- 1 | # The version of kind 2 | KIND_VERSION=${KIND_VERSION:-v0.11.1} 3 | # The version of kubectl 4 | KUBECTL_VERSION=${KUBECTL_VERSION:-v1.17.11} 5 | # The version of argocd 6 | ARGOCD_VERSION=${ARGOCD_VERSION:-v2.1.5} 7 | # The version of argocd cli 8 | ARGOCD_CLI_VERSION=${ARGOCD_CLI_VERSION:-v2.1.5} 9 | # The version of kubeseal cli 10 | KUBESEAL_CLI_VERSION=${KUBESEAL_CLI_VERSION:-v0.16.0} 11 | -------------------------------------------------------------------------------- /config/capabilities/helm/logging/templates/kibana.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: kibana.k8s.elastic.co/v1 3 | kind: Kibana 4 | metadata: 5 | name: {{.Values.metadata.name}} 6 | namespace: {{.Values.metadata.namespace}} 7 | annotations: 8 | argocd.argoproj.io/sync-wave: "200" 9 | spec: 10 | version: {{.Values.spec.parameters.kibanaVersion}} 11 | elasticsearchRef: 12 | name: {{.Values.metadata.name}} 13 | count: 1 14 | -------------------------------------------------------------------------------- /config/shared/crossplane-ext/capabilities-configuration.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: pkg.crossplane.io/v1 3 | kind: Configuration 4 | metadata: 5 | name: capabilities-shim 6 | spec: 7 | ignoreCrossplaneConstraints: false 8 | package: quay.io/moyingbj/capabilities-shim:v0.0.1 9 | packagePullPolicy: IfNotPresent 10 | revisionActivationPolicy: Automatic 11 | revisionHistoryLimit: 0 12 | skipDependencyResolution: false 13 | -------------------------------------------------------------------------------- /config/capabilities/helm/logging/templates/elasticsearch.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: elasticsearch.k8s.elastic.co/v1 3 | kind: Elasticsearch 4 | metadata: 5 | name: {{.Values.metadata.name}} 6 | namespace: {{.Values.metadata.namespace}} 7 | annotations: 8 | argocd.argoproj.io/sync-wave: "200" 9 | spec: 10 | version: {{.Values.spec.parameters.esVersion}} 11 | nodeSets: 12 | - name: default 13 | count: 1 14 | config: 15 | node.store.allow_mmap: false 16 | -------------------------------------------------------------------------------- /config/shared/argocd.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: argoproj.io/v1alpha1 3 | kind: Application 4 | metadata: 5 | name: argocd-app 6 | namespace: argocd 7 | annotations: 8 | argocd.argoproj.io/sync-wave: "100" 9 | spec: 10 | destination: 11 | namespace: argocd 12 | server: https://kubernetes.default.svc 13 | project: default 14 | source: 15 | repoURL: https://github.com/morningspace/capabilities-shim-gitops 16 | path: config/shared/argocd 17 | targetRevision: HEAD 18 | syncPolicy: 19 | automated: 20 | prune: true 21 | selfHeal: true 22 | -------------------------------------------------------------------------------- /config/shared/olm.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: argoproj.io/v1alpha1 3 | kind: Application 4 | metadata: 5 | name: olm-app 6 | namespace: argocd 7 | annotations: 8 | argocd.argoproj.io/sync-wave: "100" 9 | spec: 10 | destination: 11 | namespace: olm 12 | server: https://kubernetes.default.svc 13 | project: default 14 | source: 15 | repoURL: https://github.com/operator-framework/operator-lifecycle-manager 16 | path: deploy/upstream/quickstart 17 | targetRevision: HEAD 18 | syncPolicy: 19 | automated: 20 | prune: true 21 | selfHeal: true 22 | -------------------------------------------------------------------------------- /config/shared/crossplane-ext.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: argoproj.io/v1alpha1 2 | kind: Application 3 | metadata: 4 | name: crossplane-ext-app 5 | namespace: argocd 6 | annotations: 7 | argocd.argoproj.io/sync-wave: "150" 8 | spec: 9 | destination: 10 | namespace: argocd 11 | server: https://kubernetes.default.svc 12 | project: default 13 | source: 14 | repoURL: https://github.com/morningspace/capabilities-shim-gitops 15 | path: config/shared/crossplane-ext 16 | targetRevision: HEAD 17 | syncPolicy: 18 | automated: 19 | prune: true 20 | selfHeal: true 21 | -------------------------------------------------------------------------------- /config/capabilities/managed-resources/logging/subscription.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: kubernetes.crossplane.io/v1alpha1 3 | kind: Object 4 | metadata: 5 | name: sub-my-logging-stack 6 | spec: 7 | forProvider: 8 | manifest: 9 | apiVersion: operators.coreos.com/v1alpha1 10 | kind: Subscription 11 | metadata: 12 | namespace: operators 13 | name: my-logging-stack 14 | spec: 15 | channel: stable 16 | name: elastic-cloud-eck 17 | source: operatorhubio-catalog 18 | sourceNamespace: olm 19 | providerConfigRef: 20 | name: provider-config-dev 21 | -------------------------------------------------------------------------------- /config/capabilities/crossplane-helm/logging/templates/subscription.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: kubernetes.crossplane.io/v1alpha1 3 | kind: Object 4 | metadata: 5 | name: sub-{{.Values.metadata.name}} 6 | spec: 7 | forProvider: 8 | manifest: 9 | apiVersion: operators.coreos.com/v1alpha1 10 | kind: Subscription 11 | metadata: 12 | namespace: operators 13 | name: {{.Values.metadata.name}} 14 | spec: 15 | channel: stable 16 | name: elastic-cloud-eck 17 | source: operatorhubio-catalog 18 | sourceNamespace: olm 19 | providerConfigRef: 20 | name: {{.Values.spec.providerConfigRef.name}} 21 | -------------------------------------------------------------------------------- /config/shared/sealed-secrets.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: argoproj.io/v1alpha1 3 | kind: Application 4 | metadata: 5 | name: sealed-secrets-controller 6 | namespace: argocd 7 | annotations: 8 | argocd.argoproj.io/sync-wave: "100" 9 | spec: 10 | destination: 11 | namespace: argocd 12 | server: https://kubernetes.default.svc 13 | project: default 14 | source: 15 | repoURL: https://bitnami-labs.github.io/sealed-secrets 16 | targetRevision: 1.16.1 17 | chart: sealed-secrets 18 | helm: 19 | values: |- 20 | # https://github.com/argoproj/argo-cd/issues/5991 21 | commandArgs: 22 | - "--update-status" 23 | syncPolicy: 24 | automated: 25 | prune: true 26 | selfHeal: true 27 | -------------------------------------------------------------------------------- /config/capabilities/managed-resources/logging/kibana.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: kubernetes.crossplane.io/v1alpha1 3 | kind: Object 4 | metadata: 5 | name: kibana-my-logging-stack 6 | spec: 7 | references: 8 | - fromObject: 9 | apiVersion: kubernetes.crossplane.io/v1alpha1 10 | kind: Object 11 | name: csv-my-logging-stack 12 | fieldPath: status.atProvider.manifest.status.phase 13 | forProvider: 14 | manifest: 15 | apiVersion: kibana.k8s.elastic.co/v1 16 | kind: Kibana 17 | metadata: 18 | name: my-logging-stack 19 | namespace: default 20 | spec: 21 | version: 7.13.3 22 | elasticsearchRef: 23 | name: my-logging-stack 24 | count: 1 25 | providerConfigRef: 26 | name: provider-config-dev 27 | -------------------------------------------------------------------------------- /config/capabilities/managed-resources/logging/clusterserviceversion.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: kubernetes.crossplane.io/v1alpha1 3 | kind: Object 4 | metadata: 5 | name: csv-my-logging-stack 6 | annotations: 7 | kubernetes.crossplane.io/managementType: "ObservableAndDeletable" 8 | spec: 9 | references: 10 | - fromObject: 11 | apiVersion: kubernetes.crossplane.io/v1alpha1 12 | kind: Object 13 | name: sub-my-logging-stack 14 | fieldPath: status.atProvider.manifest.status.currentCSV 15 | toFieldPath: spec.forProvider.manifest.metadata.name 16 | forProvider: 17 | manifest: 18 | apiVersion: operators.coreos.com/v1alpha1 19 | kind: ClusterServiceVersion 20 | metadata: 21 | namespace: operators 22 | providerConfigRef: 23 | name: provider-config-dev 24 | -------------------------------------------------------------------------------- /config/shared/crossplane.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: Namespace 4 | metadata: 5 | name: crossplane-system 6 | spec: 7 | finalizers: 8 | - kubernetes 9 | --- 10 | apiVersion: argoproj.io/v1alpha1 11 | kind: Application 12 | metadata: 13 | name: crossplane-app 14 | namespace: argocd 15 | annotations: 16 | argocd.argoproj.io/sync-wave: "100" 17 | spec: 18 | destination: 19 | namespace: crossplane-system 20 | server: https://kubernetes.default.svc 21 | project: default 22 | source: 23 | repoURL: https://charts.crossplane.io/stable 24 | chart: crossplane 25 | targetRevision: 1.4.1 26 | ignoreDifferences: 27 | - group: rbac.authorization.k8s.io 28 | jsonPointers: 29 | - /rules 30 | kind: ClusterRole 31 | syncPolicy: 32 | automated: 33 | prune: true 34 | selfHeal: true 35 | -------------------------------------------------------------------------------- /config/capabilities/crossplane-helm/logging/templates/clusterserviceversion.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: kubernetes.crossplane.io/v1alpha1 3 | kind: Object 4 | metadata: 5 | name: csv-{{.Values.metadata.name}} 6 | annotations: 7 | kubernetes.crossplane.io/managementType: "ObservableAndDeletable" 8 | spec: 9 | references: 10 | - fromObject: 11 | apiVersion: kubernetes.crossplane.io/v1alpha1 12 | kind: Object 13 | name: sub-{{.Values.metadata.name}} 14 | fieldPath: status.atProvider.manifest.status.currentCSV 15 | toFieldPath: spec.forProvider.manifest.metadata.name 16 | forProvider: 17 | manifest: 18 | apiVersion: operators.coreos.com/v1alpha1 19 | kind: ClusterServiceVersion 20 | metadata: 21 | namespace: operators 22 | providerConfigRef: 23 | name: {{.Values.spec.providerConfigRef.name}} 24 | -------------------------------------------------------------------------------- /config/capabilities/managed-resources/logging/elasticsearch.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: kubernetes.crossplane.io/v1alpha1 3 | kind: Object 4 | metadata: 5 | name: elasticsearch-my-logging-stack 6 | spec: 7 | references: 8 | - fromObject: 9 | apiVersion: kubernetes.crossplane.io/v1alpha1 10 | kind: Object 11 | name: csv-my-logging-stack 12 | fieldPath: status.atProvider.manifest.status.phase 13 | forProvider: 14 | manifest: 15 | apiVersion: elasticsearch.k8s.elastic.co/v1 16 | kind: Elasticsearch 17 | metadata: 18 | name: my-logging-stack 19 | namespace: default 20 | spec: 21 | version: 7.13.3 22 | nodeSets: 23 | - name: default 24 | count: 1 25 | config: 26 | node.store.allow_mmap: false 27 | providerConfigRef: 28 | name: provider-config-dev 29 | -------------------------------------------------------------------------------- /config/capabilities/crossplane-helm/logging/templates/kibana.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: kubernetes.crossplane.io/v1alpha1 3 | kind: Object 4 | metadata: 5 | name: kibana-{{.Values.metadata.name}} 6 | spec: 7 | references: 8 | - fromObject: 9 | apiVersion: kubernetes.crossplane.io/v1alpha1 10 | kind: Object 11 | name: csv-{{.Values.metadata.name}} 12 | fieldPath: status.atProvider.manifest.status.phase 13 | forProvider: 14 | manifest: 15 | apiVersion: kibana.k8s.elastic.co/v1 16 | kind: Kibana 17 | metadata: 18 | name: {{.Values.metadata.name}} 19 | namespace: {{.Values.metadata.namespace}} 20 | spec: 21 | version: {{.Values.spec.parameters.kibanaVersion}} 22 | elasticsearchRef: 23 | name: {{.Values.metadata.name}} 24 | count: 1 25 | providerConfigRef: 26 | name: {{.Values.spec.providerConfigRef.name}} 27 | -------------------------------------------------------------------------------- /config/capabilities/crossplane-helm/logging/templates/elasticsearch.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: kubernetes.crossplane.io/v1alpha1 3 | kind: Object 4 | metadata: 5 | name: elasticsearch-{{.Values.metadata.name}} 6 | spec: 7 | references: 8 | - fromObject: 9 | apiVersion: kubernetes.crossplane.io/v1alpha1 10 | kind: Object 11 | name: csv-{{.Values.metadata.name}} 12 | fieldPath: status.atProvider.manifest.status.phase 13 | forProvider: 14 | manifest: 15 | apiVersion: elasticsearch.k8s.elastic.co/v1 16 | kind: Elasticsearch 17 | metadata: 18 | name: {{.Values.metadata.name}} 19 | namespace: {{.Values.metadata.namespace}} 20 | spec: 21 | version: {{.Values.spec.parameters.esVersion}} 22 | nodeSets: 23 | - name: default 24 | count: 1 25 | config: 26 | node.store.allow_mmap: false 27 | providerConfigRef: 28 | name: {{.Values.spec.providerConfigRef.name}} 29 | -------------------------------------------------------------------------------- /config/capabilities/helm-composition/logging/templates/composition.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: operators.coreos.com/v1alpha1 3 | kind: Subscription 4 | metadata: 5 | namespace: operators 6 | name: {{.Values.metadata.name}} 7 | spec: 8 | channel: stable 9 | name: elastic-cloud-eck 10 | source: operatorhubio-catalog 11 | sourceNamespace: olm 12 | --- 13 | apiVersion: elasticsearch.k8s.elastic.co/v1 14 | kind: Elasticsearch 15 | metadata: 16 | name: {{.Values.metadata.name}} 17 | namespace: {{.Values.metadata.namespace}} 18 | spec: 19 | version: {{.Values.spec.parameters.esVersion}} 20 | nodeSets: 21 | - name: default 22 | count: 1 23 | config: 24 | node.store.allow_mmap: false 25 | --- 26 | apiVersion: kibana.k8s.elastic.co/v1 27 | kind: Kibana 28 | metadata: 29 | name: {{.Values.metadata.name}} 30 | namespace: {{.Values.metadata.namespace}} 31 | spec: 32 | version: {{.Values.spec.parameters.kibanaVersion}} 33 | elasticsearchRef: 34 | name: {{.Values.metadata.name}} 35 | count: 1 36 | -------------------------------------------------------------------------------- /config/capabilities/composition/logging/definition.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apiextensions.crossplane.io/v1 2 | kind: CompositeResourceDefinition 3 | metadata: 4 | name: loggings.capabilities.morningspace.io 5 | spec: 6 | group: capabilities.morningspace.io 7 | names: 8 | kind: Logging 9 | plural: loggings 10 | claimNames: 11 | kind: LoggingClaim 12 | plural: loggingclaims 13 | versions: 14 | - name: v1alpha1 15 | served: true 16 | referenceable: true 17 | schema: 18 | openAPIV3Schema: 19 | type: object 20 | properties: 21 | spec: 22 | type: object 23 | properties: 24 | parameters: 25 | type: object 26 | properties: 27 | esVersion: 28 | description: The Elasticsearch version 29 | type: string 30 | kibanaVersion: 31 | description: The Kibana version 32 | type: string 33 | required: 34 | - esVersion 35 | - kibanaVersion -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 MorningSpace 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /config/capabilities/helm/logging/Chart.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v2 3 | name: capabilities-shim 4 | description: Capabilities Shim 5 | 6 | # A chart can be either an 'application' or a 'library' chart. 7 | # 8 | # Application charts are a collection of templates that can be packaged into versioned archives 9 | # to be deployed. 10 | # 11 | # Library charts provide useful utilities or functions for the chart developer. They're included as 12 | # a dependency of application charts to inject those utilities and functions into the rendering 13 | # pipeline. Library charts do not define any templates and therefore cannot be deployed. 14 | type: application 15 | 16 | # This is the chart version. This version number should be incremented each time you make changes 17 | # to the chart and its templates, including the app version. 18 | # Versions are expected to follow Semantic Versioning (https://semver.org/) 19 | version: 0.1.0 20 | 21 | # This is the version number of the application being deployed. This version number should be 22 | # incremented each time you make changes to the application. Versions are not expected to 23 | # follow Semantic Versioning. They should reflect the version the application is using. 24 | appVersion: "1.0" 25 | -------------------------------------------------------------------------------- /config/capabilities/crossplane-helm/logging/Chart.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v2 3 | name: capabilities-shim 4 | description: Capabilities Shim 5 | 6 | # A chart can be either an 'application' or a 'library' chart. 7 | # 8 | # Application charts are a collection of templates that can be packaged into versioned archives 9 | # to be deployed. 10 | # 11 | # Library charts provide useful utilities or functions for the chart developer. They're included as 12 | # a dependency of application charts to inject those utilities and functions into the rendering 13 | # pipeline. Library charts do not define any templates and therefore cannot be deployed. 14 | type: application 15 | 16 | # This is the chart version. This version number should be incremented each time you make changes 17 | # to the chart and its templates, including the app version. 18 | # Versions are expected to follow Semantic Versioning (https://semver.org/) 19 | version: 0.1.0 20 | 21 | # This is the version number of the application being deployed. This version number should be 22 | # incremented each time you make changes to the application. Versions are not expected to 23 | # follow Semantic Versioning. They should reflect the version the application is using. 24 | appVersion: "1.0" 25 | -------------------------------------------------------------------------------- /config/capabilities/helm-composition/logging/Chart.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v2 3 | name: capabilities-shim 4 | description: Capabilities Shim 5 | 6 | # A chart can be either an 'application' or a 'library' chart. 7 | # 8 | # Application charts are a collection of templates that can be packaged into versioned archives 9 | # to be deployed. 10 | # 11 | # Library charts provide useful utilities or functions for the chart developer. They're included as 12 | # a dependency of application charts to inject those utilities and functions into the rendering 13 | # pipeline. Library charts do not define any templates and therefore cannot be deployed. 14 | type: application 15 | 16 | # This is the chart version. This version number should be incremented each time you make changes 17 | # to the chart and its templates, including the app version. 18 | # Versions are expected to follow Semantic Versioning (https://semver.org/) 19 | version: 0.1.0 20 | 21 | # This is the version number of the application being deployed. This version number should be 22 | # incremented each time you make changes to the application. Versions are not expected to 23 | # follow Semantic Versioning. They should reflect the version the application is using. 24 | appVersion: "1.0" 25 | -------------------------------------------------------------------------------- /config/capabilities/helm/logging/templates/post-uninstall-job.yaml: -------------------------------------------------------------------------------- 1 | # Ensuring all namespace and cluster scoped resources are cleaned up 2 | apiVersion: batch/v1 3 | kind: Job 4 | metadata: 5 | name: {{ .Release.Name }}-post-uninstall-job 6 | namespace: argocd 7 | labels: 8 | app.kubernetes.io/managed-by: {{ .Release.Service | quote }} 9 | app.kubernetes.io/instance: {{ .Release.Name | quote }} 10 | app.kubernetes.io/version: "{{ .Chart.AppVersion }}" 11 | helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" 12 | annotations: 13 | # hooks are defined here 14 | "helm.sh/hook": post-delete 15 | spec: 16 | template: 17 | metadata: 18 | name: {{ .Release.Name }}-post-uninstall-job 19 | labels: 20 | app.kubernetes.io/managed-by: {{ .Release.Service | quote }} 21 | app.kubernetes.io/instance: {{ .Release.Name | quote }} 22 | helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" 23 | spec: 24 | containers: 25 | - name: clean-resources 26 | image: quay.io/bitnami/kubectl:latest 27 | command: 28 | - /bin/bash 29 | - -c 30 | - | 31 | set -x 32 | 33 | result=0 34 | 35 | # clean csv 36 | csvs=($(kubectl get csv -n operators -o name)) 37 | if [[ -n ${csvs[@]} ]]; then 38 | kubectl delete ${csvs[@]} -n operators 39 | (( result+=$? )) 40 | fi 41 | 42 | exit "${result}" 43 | restartPolicy: Never 44 | serviceAccountName: argocd-application-controller 45 | backoffLimit: 1 46 | -------------------------------------------------------------------------------- /config/capabilities/crossplane-helm/logging/templates/post-install-job.yaml: -------------------------------------------------------------------------------- 1 | # Ensuring all CSVs in target namespace are ready 2 | apiVersion: batch/v1 3 | kind: Job 4 | metadata: 5 | name: {{ .Release.Name }}-post-install-job 6 | namespace: argocd 7 | labels: 8 | app.kubernetes.io/managed-by: {{ .Release.Service | quote }} 9 | app.kubernetes.io/instance: {{ .Release.Name | quote }} 10 | app.kubernetes.io/version: "{{ .Chart.AppVersion }}" 11 | helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" 12 | annotations: 13 | argocd.argoproj.io/hook: PostSync 14 | spec: 15 | template: 16 | metadata: 17 | name: {{ .Release.Name }}-post-install-job 18 | labels: 19 | app.kubernetes.io/managed-by: {{ .Release.Service | quote }} 20 | app.kubernetes.io/instance: {{ .Release.Name | quote }} 21 | helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" 22 | spec: 23 | containers: 24 | - name: check-csv 25 | image: quay.io/bitnami/kubectl:latest 26 | command: 27 | - /bin/sh 28 | - -c 29 | - | 30 | set -x 31 | 32 | current_seconds=0 33 | limit_seconds=$(( $(date +%s) + 600 )) 34 | installing=0 35 | 36 | while [ ${current_seconds} -lt ${limit_seconds} ]; do 37 | if [ $(kubectl get csv -n "{{.Values.metadata.namespace}}" -o name | grep -c clusterserviceversion) -eq 0 ] || \ 38 | [ $(kubectl get csv -n "{{.Values.metadata.namespace}}" --no-headers | grep -vc Succeeded) -gt 0 ]; then 39 | installing=1 40 | echo "INFO: ClusterServiceVersion resources still installing." 41 | sleep 1 42 | else 43 | installing=0 44 | break 45 | fi 46 | current_seconds=$(( $(date +%s) )) 47 | done 48 | 49 | if [ ${installing} -eq 0 ]; then 50 | echo "INFO: All ClusterServiceVersion resources are ready." 51 | else 52 | echo "ERROR: ClusterServiceVersion resources still not ready." 53 | exit 1 54 | fi 55 | restartPolicy: Never 56 | serviceAccountName: argocd-application-controller 57 | backoffLimit: 1 58 | -------------------------------------------------------------------------------- /config/shared/argocd/argocd-cm.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: ConfigMap 4 | metadata: 5 | name: argocd-cm 6 | namespace: argocd 7 | labels: 8 | app.kubernetes.io/name: argocd-cm 9 | app.kubernetes.io/part-of: argocd 10 | data: 11 | resource.customizations.health.operators.coreos.com_Subscription: | 12 | hs = {} 13 | hs.status = "Progressing" 14 | hs.message = "" 15 | if obj.status ~= nil then 16 | hs.status = "Healthy" 17 | hs.message = obj.status.state 18 | end 19 | return hs 20 | resource.customizations.health.operators.coreos.com_ClusterServiceVersion: | 21 | hs = {} 22 | hs.status = "Progressing" 23 | hs.message = "" 24 | if obj.status ~= nil then 25 | if obj.status.phase == "Succeeded" then 26 | hs.status = "Healthy" 27 | end 28 | hs.message = obj.status.message 29 | end 30 | return hs 31 | resource.customizations.health.elasticsearch.k8s.elastic.co_Elasticsearch: | 32 | hs = {} 33 | hs.status = "Progressing" 34 | hs.message = "" 35 | if obj.status ~= nil and obj.status.health ~= nil and obj.status.phase ~= nil then 36 | if obj.status.health == "green" and obj.status.phase == "Ready" then 37 | hs.status = "Healthy" 38 | end 39 | hs.message = "health = " .. obj.status.health .. " phase = " .. obj.status.phase 40 | end 41 | return hs 42 | resource.customizations.health.kibana.k8s.elastic.co_Kibana: | 43 | hs = {} 44 | hs.status = "Progressing" 45 | hs.message = "" 46 | if obj.status ~= nil and obj.status.health ~= nil and obj.status.associationStatus ~= nil then 47 | if obj.status.health == "green" and obj.status.associationStatus == "Established" then 48 | hs.status = "Healthy" 49 | end 50 | hs.message = "health = " .. obj.status.health .. " associationStatus = " .. obj.status.associationStatus 51 | end 52 | return hs 53 | resource.customizations.health.kubernetes.crossplane.io_Object: | 54 | hs = {} 55 | hs.status = "Progressing" 56 | hs.message = "" 57 | if obj.status ~= nil and obj.status.atProvider ~= nil then 58 | kind = obj.spec.forProvider.manifest.kind 59 | res = obj.status.atProvider.manifest 60 | if res ~= nil then 61 | if kind == "Subscription" then 62 | if res.status ~= nil then 63 | hs.status = "Healthy" 64 | hs.message = res.status.state 65 | end 66 | elseif kind == "ClusterServiceVersion" then 67 | if res.status ~= nil then 68 | if res.status.phase == "Succeeded" then 69 | hs.status = "Healthy" 70 | end 71 | hs.message = res.status.message 72 | end 73 | elseif kind == "Elasticsearch" then 74 | if res.status ~= nil and res.status.health ~= nil and res.status.phase ~= nil then 75 | if res.status.health == "green" and res.status.phase == "Ready" then 76 | hs.status = "Healthy" 77 | end 78 | hs.message = "health = " .. res.status.health .. " phase = " .. res.status.phase 79 | end 80 | elseif kind == "Kibana" then 81 | if res.status ~= nil and res.status.health ~= nil and res.status.associationStatus ~= nil then 82 | if res.status.health == "green" and res.status.associationStatus == "Established" then 83 | hs.status = "Healthy" 84 | end 85 | hs.message = "health = " .. res.status.health .. " associationStatus = " .. res.status.associationStatus 86 | end 87 | end 88 | end 89 | end 90 | return hs 91 | -------------------------------------------------------------------------------- /config/capabilities/composition/logging/composition.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apiextensions.crossplane.io/v1 2 | kind: Composition 3 | metadata: 4 | name: loggings.capabilities.olm.morningspace.io 5 | labels: 6 | capability: logging 7 | provider: olm 8 | spec: 9 | compositeTypeRef: 10 | apiVersion: capabilities.morningspace.io/v1alpha1 11 | kind: Logging 12 | resources: 13 | - base: 14 | apiVersion: kubernetes.crossplane.io/v1alpha1 15 | kind: Object 16 | spec: 17 | forProvider: 18 | manifest: 19 | apiVersion: operators.coreos.com/v1alpha1 20 | kind: Subscription 21 | metadata: 22 | namespace: operators 23 | spec: 24 | channel: stable 25 | name: elastic-cloud-eck 26 | source: operatorhubio-catalog 27 | sourceNamespace: olm 28 | patches: 29 | - fromFieldPath: metadata.annotations[capabilities.morningspace.io/provider-config] 30 | toFieldPath: spec.providerConfigRef.name 31 | - fromFieldPath: metadata.name 32 | transforms: 33 | - type: string 34 | string: 35 | fmt: 'sub-%s-eck' 36 | - fromFieldPath: metadata.name 37 | toFieldPath: spec.forProvider.manifest.metadata.name 38 | transforms: 39 | - type: string 40 | string: 41 | fmt: '%s-eck' 42 | readinessChecks: 43 | - type: NonEmpty 44 | fieldPath: status.atProvider.manifest.status.currentCSV 45 | - base: 46 | apiVersion: kubernetes.crossplane.io/v1alpha1 47 | kind: Object 48 | metadata: 49 | annotations: 50 | kubernetes.crossplane.io/managementType: "ObservableAndDeletable" 51 | spec: 52 | references: 53 | - fromObject: 54 | apiVersion: kubernetes.crossplane.io/v1alpha1 55 | kind: Object 56 | fieldPath: status.atProvider.manifest.status.currentCSV 57 | toFieldPath: spec.forProvider.manifest.metadata.name 58 | forProvider: 59 | manifest: 60 | apiVersion: operators.coreos.com/v1alpha1 61 | kind: ClusterServiceVersion 62 | metadata: 63 | namespace: operators 64 | patches: 65 | - fromFieldPath: metadata.annotations[capabilities.morningspace.io/provider-config] 66 | toFieldPath: spec.providerConfigRef.name 67 | - fromFieldPath: metadata.name 68 | transforms: 69 | - type: string 70 | string: 71 | fmt: 'csv-%s-eck' 72 | - fromFieldPath: metadata.name 73 | toFieldPath: spec.references[0].fromObject.name 74 | transforms: 75 | - type: string 76 | string: 77 | fmt: 'sub-%s-eck' 78 | readinessChecks: 79 | - type: MatchString 80 | fieldPath: status.atProvider.manifest.status.phase 81 | matchString: Succeeded 82 | - base: 83 | apiVersion: kubernetes.crossplane.io/v1alpha1 84 | kind: Object 85 | spec: 86 | references: 87 | - fromObject: 88 | apiVersion: kubernetes.crossplane.io/v1alpha1 89 | kind: Object 90 | fieldPath: status.atProvider.manifest.status.phase 91 | forProvider: 92 | manifest: 93 | apiVersion: elasticsearch.k8s.elastic.co/v1 94 | kind: Elasticsearch 95 | metadata: {} 96 | spec: 97 | nodeSets: 98 | - name: default 99 | count: 1 100 | config: 101 | node.store.allow_mmap: false 102 | patches: 103 | - fromFieldPath: metadata.annotations[capabilities.morningspace.io/provider-config] 104 | toFieldPath: spec.providerConfigRef.name 105 | - fromFieldPath: metadata.name 106 | transforms: 107 | - type: string 108 | string: 109 | fmt: 'elasticsearch-%s' 110 | - fromFieldPath: metadata.name 111 | toFieldPath: spec.references[0].fromObject.name 112 | transforms: 113 | - type: string 114 | string: 115 | fmt: 'csv-%s-eck' 116 | - fromFieldPath: metadata.name 117 | toFieldPath: spec.forProvider.manifest.metadata.name 118 | - fromFieldPath: spec.claimRef.namespace 119 | toFieldPath: spec.forProvider.manifest.metadata.namespace 120 | - fromFieldPath: spec.parameters.esVersion 121 | toFieldPath: spec.forProvider.manifest.spec.version 122 | readinessChecks: 123 | - type: MatchString 124 | fieldPath: status.atProvider.manifest.status.phase 125 | matchString: Ready 126 | - type: MatchString 127 | fieldPath: status.atProvider.manifest.status.health 128 | matchString: green 129 | - base: 130 | apiVersion: kubernetes.crossplane.io/v1alpha1 131 | kind: Object 132 | spec: 133 | references: 134 | - fromObject: 135 | apiVersion: kubernetes.crossplane.io/v1alpha1 136 | kind: Object 137 | fieldPath: status.atProvider.manifest.status.phase 138 | forProvider: 139 | manifest: 140 | apiVersion: kibana.k8s.elastic.co/v1 141 | kind: Kibana 142 | metadata: {} 143 | spec: 144 | count: 1 145 | patches: 146 | - fromFieldPath: metadata.annotations[capabilities.morningspace.io/provider-config] 147 | toFieldPath: spec.providerConfigRef.name 148 | - fromFieldPath: metadata.name 149 | transforms: 150 | - type: string 151 | string: 152 | fmt: 'kibana-%s' 153 | - fromFieldPath: metadata.name 154 | toFieldPath: spec.references[0].fromObject.name 155 | transforms: 156 | - type: string 157 | string: 158 | fmt: 'csv-%s-eck' 159 | - fromFieldPath: metadata.name 160 | toFieldPath: spec.forProvider.manifest.metadata.name 161 | - fromFieldPath: spec.claimRef.namespace 162 | toFieldPath: spec.forProvider.manifest.metadata.namespace 163 | - fromFieldPath: spec.parameters.kibanaVersion 164 | toFieldPath: spec.forProvider.manifest.spec.version 165 | - fromFieldPath: metadata.name 166 | toFieldPath: spec.forProvider.manifest.spec.elasticsearchRef.name 167 | readinessChecks: 168 | - type: MatchString 169 | fieldPath: status.atProvider.manifest.status.associationStatus 170 | matchString: Established 171 | - type: MatchString 172 | fieldPath: status.atProvider.manifest.status.health 173 | matchString: green 174 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # capabilities-shim-gitops 2 | 3 | This repository is used to demonstrate how to effectively adopt Crossplane in GitOps by using a sample application. The sample application is based on another demo project [capabilities-shim](https://github.com/morningspace/capabilities-shim) that I created to prove the technical feasibility of composing variant existing software capabilities using Crossplane to come up with solution that meets user specific needs. 4 | 5 | ![](docs/images/crossplane-in-gitops.png) 6 | 7 | ## What it covers? 8 | 9 | The repository includes: 10 | 11 | * A bunch of well-organized YAML manifests: Represents variant software modules that can be synchronized and deployed by Argo CD to target cluster. This can be taken as a reference for people to evaluate what a GitOps repository can look like when using Crossplane. 12 | * Documents in `docs` folder: Discuss best practices, common considerations, and lessons learned that you might experience as well when use Crossplane in GitOps. 13 | * A shell script: Help you launch a GitOps demo environment in minutes including a KIND cluster with Argo CD installed. 14 | 15 | Follow below instructions to luanch the demo environment. After it is up and running, you can explore the repository, follow the documents in `docs` folder, and experiment to get hands-on experience of using Crossplane in GitOps. 16 | 17 | ## Play with the demo environment 18 | 19 | ### Launch the demo environment 20 | 21 | Please fork this repository to your own GitHub account, then follow below steps to launch the GitOps demo environment. 22 | 23 | Go to project root folder and run below command to bring up the environment. 24 | 25 | ```shell 26 | ./scripts/install.sh up 27 | ``` 28 | 29 | It can help you launch a KIND cluster with an Argo CD instance installed. Besides that, it will also install all the necessary command line tools that you may need when you experiment with GitOps, e.g.: argocd CLI, kubeseal CLI. 30 | 31 | At the end of the install process, it will print all softwares with their versions installed on your demo environment. Such as below: 32 | 33 | ```console 34 | 👏 Congratulations! The GitOps demo environment is available! 35 | It launched a kind cluster, installed following tools and applitions: 36 | - argocd cli v2.1.5 37 | - kubeseal cli v0.16.0 38 | 39 | To access Argo CD UI, open https://localhost:9443 in browser. 40 | - username: admin 41 | - password: **************** 42 | 43 | For tools you want to run anywhere, create links in a directory defined in your PATH, e.g: 44 | ln -s -f /root/Code/capabilities-shim-gitops/scripts/.cache/tools/linux_x86_64/kubectl-v1.17.11 /usr/local/bin/kubectl 45 | ln -s -f /root/Code/capabilities-shim-gitops/scripts/.cache/tools/linux_x86_64/kind-v0.11.1 /usr/local/bin/kind 46 | ln -s -f /root/Code/capabilities-shim-gitops/scripts/.cache/tools/linux_x86_64/argocd-v2.1.5 /usr/local/bin/argocd 47 | ln -s -f /root/Code/capabilities-shim-gitops/scripts/.cache/tools/linux_x86_64/kubeseal-v0.16.0 /usr/local/bin/kubeseal 48 | ``` 49 | 50 | It also prints the Argo CD UI address, the username and password that you can use to login. 51 | 52 | ### Argocd cli login for cli operations 53 | ```shell 54 | argocd login localhost:9443 --username admin 55 | Proceed insecurely (y/n)? y 56 | Password: 57 | ``` 58 | 59 | ### Install everything else using Argo CD 60 | 61 | Login to Argo CD from UI, then create an Argo Application using the following values: 62 | 63 | | Field | Value | 64 | | ---------------- | -------------------------------------------------------- | 65 | | Application Name | shared-apps | 66 | | Path | config/shared | 67 | | Project | default | 68 | | Sync policy | Automatic | 69 | | Self Heal | true | 70 | | Repository URL | https://github.com/morningspace/capabilities-shim-gitops | 71 | | Revision | HEAD | 72 | | Cluster URL | https://kubernetes.default.svc | 73 | 74 | You can also do the same thing using argocd CL from command line as below: 75 | 76 | ```shell 77 | argocd app create shared-app --repo https://github.com/morningspace/capabilities-shim-gitops.git \ 78 | --path config/shared \ 79 | --dest-namespace default \ 80 | --dest-server https://kubernetes.default.svc 81 | ``` 82 | 83 | This will install Crossplane with its providers, OLM, Sealed Secrets Controller, etc. to your demo environment. 84 | 85 | After everything is up, run below command to generate and update the `cluster-config` secret encrypted at local by kubeseal for your demo environment, then check it in git. This is required for the Crossplane provider to access to the KIND cluster. 86 | 87 | ```shell 88 | ./scripts/install.sh cluster-config 89 | git add environments/dev/env/cluster-config.json 90 | git commit -m "Update cluster-config.json" 91 | git push 92 | ``` 93 | 94 | Then create an Argo Application that represents the environment from UI using the following values: 95 | 96 | | Field | Value | 97 | | ---------------- | -------------------------------------------------------- | 98 | | Application Name | dev-env | 99 | | Path | environments/dev/env | 100 | | Project | default | 101 | | Sync policy | Automatic | 102 | | Self Heal | true | 103 | | Repository URL | https://github.com/morningspace/capabilities-shim-gitops | 104 | | Revision | HEAD | 105 | | Cluster URL | https://kubernetes.default.svc | 106 | 107 | 108 | Congratulations! Now you can explore the repository, follow the documents in `docs` folder, and experiment with this environment to get hands-on experience of using Crossplane in GitOps. 109 | 110 | ### Troubleshooting 111 | 112 | When install Argo CD, some pod may fail to install due to `ImagePullBackOff` error. This is because the image comes from Docker Hub and you do not specify image pull secret. This can be fixed by run below command: 113 | 114 | ```shell 115 | ./scripts/install.sh patch-pull-secret 116 | ``` 117 | 118 | ### Clean up 119 | 120 | Run below command to delete the demo environment completely from your local. 121 | 122 | ```shell 123 | ./scripts/install.sh down 124 | ``` 125 | -------------------------------------------------------------------------------- /environments/dev/env/cluster-config.json: -------------------------------------------------------------------------------- 1 | { 2 | "kind": "SealedSecret", 3 | "apiVersion": "bitnami.com/v1alpha1", 4 | "metadata": { 5 | "name": "cluster-config", 6 | "namespace": "dev", 7 | "creationTimestamp": null 8 | }, 9 | "spec": { 10 | "template": { 11 | "metadata": { 12 | "name": "cluster-config", 13 | "namespace": "dev", 14 | "creationTimestamp": null 15 | }, 16 | "data": null 17 | }, 18 | "encryptedData": { 19 | "kubeconfig": "AgAD0WVFu4bFnIrmmkV1aFGaToQ5GqxLvw9tuHdP/oq74lnIVYm7O6fj0N9HWMOKnvnYrLjdSPXhUsJVAUhn8VuagqdvklbTiVzg+s75HslfodVILff5g+A+2EdwZYrwUKrVwTmh0MbScZYGtN/eD/8RqcFm1IIPkKFV1IGfstx+trOMcElmJv/O7BSqzatTJunrW18DDMnu2XumCTZdM1iAttZvQm2z586BJIbPL1HJadUhv9YXQ94O9BY3hCoCXFhpd8Tlqip2iVsPy/IKqgu8xKbRXFPXVZUWSv6iZ+G6p0vC6ACtcp+msugAsBgyT1LLavr4TmlFYfO2T8tswdXGH4tmcuU+mSOxUzNWl+GzIjXGAjvrq+ZjgNv9n9COF49cc1NZjvDMmcxkX2opX7xXxurR3zISe2N6oiee8Njr+Nj8/DkDlH1sxgjSeyyWhaU+mpiRZtjNJgY6hU6LDUpts2M23KiVzak/AUCM8faUGd3WP8aCcNRtgn3n1kmz0Fu8MW3x7FGkZ9D0gLcmnU9KTfkY15/iT1TGxgKqUkwGSikAgZtKwfns4tpnt94ZR4fNqBhO+1MlZzopNaMQWFZ+0nsvKkLLOE+FA7I8NIf4ucQwnixiCgqFGhaUls5ae9RH+3sSayVwok8kARAO9JP/PLyjWTVzPSfpprpSEEewhSHeMT7uEW4W3uoiABtwZ0FZ6xE8wc7x+ATArjriLsxXGlbKqV3aU+HxK3tjWZze4tfqzvsKn4yDxu/OJIXd9L1MWPo63fD/tXnR42OHcgSPHrfAGh5rY12jcdCkbq8fgctQnLh3iMfNXJX0E2Ysu+bz1IKJ94EGKlu/QBvYAtCp8x/v8mXhcSHqfrk7Lh6M7mTi0JGolAQBOwtptmZtGMTJVZE14lcslRXnLjZvnm34qaJDLDW2/8vHeitwZGgwdYQFqsHw2ISlqBhPPxuAXnyqS70d0MlReQ/I12JcyBbSA5/9snaR0XwGsoBJqCGvImTz8k5wpPsoqQ5fxq2jaob3L5EqF2BhgvOyPgocFqrb4IpA0FKo+O96aqkSN/Uu49v1la379SPKC1dIl4bTS0JKh/98Po8pkTpD1cIHyRxzXn0FJY/uTBdrTORFSBYcQuPpI+YYRodDZ1w5t20IVVmCUxjrpO+Ym0b1sgWDweXf7S2YDK09wA1cLDJaEh+kPVxL7xxSX+ay9L9gHkjwGL6XkiAiPUOdV4H+3wH4WzhZrLpt9ln2kI+kVZ2PbWoJdu6dM1WqtYAHxK/+bndZwa3/1dyJqO5DztHBPvkSZR7x7BSQ9tKP1T+Fv3Rl1aZbl0s2OhsOsqcIJC70lvu9E8Gop0nqzwrl+phNyvlaHhK3I0p3sTZAqso5a/xgTgxAORhPy2qu7UC5Ds9QE9e1mKtoZ1AQGyMkV31LtQ94icOeG5hAE3+asXN61gFPff0bDVKS4YNyY26gwIVF0UhWcM5ZvLQgHHudOYJWnDOdBMxukvzxBjYLHuZ5DAi5QWX7hQ7tIOMmW0AeJmMty3sAsTLKV/CMZxiroUBPKpl4McCr0hrUN2GPImYVHksARv/3E6IzIRwB62HR4obdrovschq7ql6E+MN8RCf0NR898u1jLYjj+4QC54d6P3Kna4XxnsOwwRm+UB8jF5ckbPgZb+cipA96TLN4iZsj4QspsQHsnwXxbnfLxeHnDk7Cf0+yVkb3QqLDdOWHZ3u/0w8kd9Ws+27NrwRB1onTWMQ9Po2JyTFafBXxMz5hJT1+irPLLkRR6ptbQHCF4yoDpYeUqFVNrM9UyqsnDP3dNLrlVSkjXPf7mXubcJrFjHjN1VXwXY9vbQxbM5HpL0eiS5QNNBcfsPTjCRAl1tSRbyBdvhG5uP37Ru9rv7lHaJKvOHNIbwiui7dwug0sdCL14jACQTMeG5n35Kqsrx9hQpj622HVAuvaReBtMcKVTDnR8tKTabkZT3lRaQaYrLzbIRil12Y46ihBp9LRdT9CffIUKFVNpY+0tGxjxGg9/uNkK93OWrzRG6FAAPAIjkRX5UcKu1vQ9VIHCR98fH7XKOykHFGmhWVuuhkabc8xE4vGVfRkPz+VBpjesFCKNxSjeo/VZaxdHd7DfM4No88d+HSMM2kxm+iucSrmq4+kf8Mics4TL87RX2db+v9mYuqzYMDxB7vg4o4euxePufoZM0zI+TVusA36uaUHUQfsYWiSuaugbzABECtdhr1Cc1ZBGs0lBIU9g/TpMtosVB7kN3CLaskHnHgbT9zvnUfOlkS7Tt7SuXJpdoiN2tzuiRR0WDvEE13H6c00sgmIL1KwMQcqBDOybbbfnrWbVX0tvqNkO9FgfsNj+4QFYCmpwp0TJEwZor5cKKeX/lZYrdszsCd9pg3GdL04MO8OcahWv4AoWhIkS+A5Oau87mqz3QvoOL5mGJRq/9cYQRjcrXsEqrvbRJGImyPuFMd7zlRclq0+UdcwKEgXCPfQCKnOAmmoB1tfTZzz4iPEwxr7gJoDceUksYrkiY6vilrQt5ZL+wYlVZzjjFHURhVIbWb6blKYasS3dFtu1gF+0jj0ioZopZg27PYWf0nYvOYr8OSoV03yhiDFJIrE6qQz0DZIdMIdfQe4f45dNd1205UVajEyQTf7q1HYkgeEw9/xR7sgYaDgrHxxMHH8NykMc2cwnD5yhfUTLvzw4fdBkDzx7To+eHDsW8AR8HI+WQ55PoqIJkcEKb41Y1PbxBQr5zbpyarRQRq9pm6VQfmRdEbx5OWTHRT9jWxfZz9Ypx8Gq5pa6SYRLbTQ7OZkJUAb87bHbkkyoD0Fdp+47yNDon74gRh+zUbO+2G5wMWBatefX3dblBi6D7kAdxTeQlIvL7i4Gu8s4p0flGSAZG8kWRGCowbseq5UK7fyuL1NS1zlN8P+y8dtJ8pJco2ZqQbkk/yfjTICymO95aTOXpMJulvqL56ot11JR4Xs8rhvztzF+y3E1uBlP/fpL3pNjdVtWSR7IduRE1DMpVdLo/leGgm36mB/U51CD6OUHTOXRr53bafhbzRefhiHkkuCBd9AtKbFaAXOLAJ3cMtN84T/4Pka0HZWF/Ategh/Sqh4SdvEoFDk73AXs/1bHy2Q85PcVdYs6pTJvM4oWZxpXw9pJ5laDl+oO6Cut2Nmg2icO2V2V+0RTwePdNffPhjEJnsGQ0Dz85Jun65GZ8Z7IsDvn0e/yR835NNx4dyU0pmpxF14qYahXuG2V9EmRXf/hipErpHlKBCibWXgxBgelF8bC7Z2xkmZhN6nk9g7tUpNmqorjH6NUjGi53PfRYUOnvUGBOUUV9jPrdiymK3cccz1m0khGv/fRYzDKiR95h9wR8IhMWn7DomiiqDa+M1ijLF6iymc6DasVisnXIEBAyEyv+iOPzeTtq2xB7xFKE2ViUvL/dM5A6ehKvn5VRJ56Mqpjifhed+WxnEM3ZYcWRv6RhBSiI+V7aynr5J5713dMVamIB94PANXCNQ43NdaZqW2Baht6bziUOrZR8oUfL6NHM+7caTqQIj7/B7iKTyn7PSSYgGDbzdi8kP+rIurISnah0Kg5wbDoGwP0U/hE37CLfu5FT1Qc2et1jQcyiFPiRPc4ukjvVMLKtt7JxegSe3nrDn/MeaZf6LLGDaiOAtjKYwnFf9+/7R/9rw3pI23QYtTKzvWnO1XpfSnap6XtyJRJ2Eve7PjNgCXlj8WAnUZP8M+1dqfyV+kT2mvQbrX4JXMO9dqwJ+tcv6oEeHBHYrKafYnoHTqwuDmV5ZZkd05eE+qz+OalJPgFKOL5FXUToOPBS65f2QW7cqJnHI4EMEFTb7c/GN6gaEclC1kurrERzIhhoWU0QGWlYyBuPNa6LhQxG7+XLS89kX33VTKlthY4NKY8sQ1qo4MqV+Fnysc8+6a4jgESXAYnyDOdDOmU7V625jZg4wz1emkfHQT/htWNyLeS6AT98/ifqWP8YgH3+JK22yTAPkxyfNOLXQQJukLlq+IiTrJd1ZOHJv5x3cOeAzpldynTMwqG2oy+DQkb9uowKSFWQ9oqcktSZxF0qP7gWJh2MHGXgAcXYg1D3eDQX7GsFzXzZBg5HwTnbiEz8jeVQSPfzbC0RXxrVhL+UzkLhzq/UdOwds8HzGZJTHvOsjdCvxf5Dk489keaIQjegcmwMWdrRoDwXoPtAu7FHh1jDsxOyOOmXxQpFB82BhxGkFC2d6872sEjyoa1wcG+4QsDDwVP6m/vK2FXvXOyRF40YqbPXKIkKZnkoLa6E6MxbmiLSSYvk0U1jBXvlG8NRpBIzCdltPEU7IjJfactLq5FLiF56Jw0trXZ1Xw8+K+gj9VDiRsyLi+G3ljkjpsJFeAm7MzdE8IUdkn32Ts2ypk8q/IBaKEOVhrhs+2OR9nlP6e0awwVy8BzEC0xH0MlG2cngDRK+jLknxpDcq7BW1KTK+VWC3nI7g+g4MaAXSui/FHJV0g9xPN/WOK5VF3faKzpgh9eIh8k7rXI5zS0+h1esB4gIsEJfNBB5EYHunQ/BBXe2cubh4YePI9f1SZnTMC8S2K6e4xh7yhEsb4I49QII6++VpeR/3uIUFGg1o2Cop3KNlqD0vk/La+rhpVcslxT7shOHEUGMVWOEWoiBHr99vJgtAhKB0gawiasfiD5tykA0G9fU9zhz9wQxHgih9+UasubG5Rkrgv+E/bW17/N3odrEWNU62dEQonJLLbPZpoq5vCtipH3/2v8FL4LIUelTZ8dftkW/8vP/IholNbEDZ+2aCqRExrlCe/S+U39LG9XhfJZD+JuPgVN+HSWemjFx6xpEjiHmnxtg+Vc2kX0bJ321+wl0UCEGj2jFS/Ylo+IM5n5LanyryTS44FyHK35U1bcE8XnMbui4zLQy4/UnxAbLw4LS8SY1PKKAph5TkwA1T7iL2l3PBMWjRKp+rD8tS67daruzRs21YVENdlb5rCbyjj5XDccygGdnuMUn34SMMxvzCT2FToUqnGtj+ao0/S57I2WR3Qx7f7jj9/XIa7EmTkNJ99YOEoYzHAESmo0wsyk+J4C7M21Qlj96OYLL10XICsDCYiagOcHOapSVQZXV1fEcIJ6zAbbi5QgBEZgd2zXDykKJUruuWdq1jPRgEbCppmvDSASJeS1YQf9eqpF4yux6u5OSDnwAUI357+dKgfUozc5YTC8DrslnmF1aEJ/14PitAi9kL3TpU34gOpAQQLnrY189bR6lF0R046xXWzoAWs2aLm311JDjMWoB0G8Fe/MsPqi+Ks2+jBq6Ha4T/a0PMIq62pvvtdGX1+GXAV3cAY0nkXPnY5GwKkRc00p+llDzjueXb3Dk29ra6bnZ0uaiWYC4YpfH7FkZkJW8PbXmgEt0K1l3lAX4u1ny+ztUwGtqz9amE1kdc7PX51IIVRb6TIk3IfgqDnup/rGjo9y8UpOHiWRb3Docrt/M0tVIOFj5ejT5IXGeLAw/UGsvFZAcYSnPxsfZgIeLPJqXRb4w9042s3XKUHgb0nO5jpAXCdemy+Y9ZDQhLY/R6Id9arQ2C/Ee7xPqFt2W1HozExxs6Tyj/JDBXIKrAuT2nTZwP4QwHx5yKOUqTtu+CLs41Hr7gQvdh0e36ZWEsV5AjeAehhuhoz5emCLhP3TvdHoyiOGKEBsU2fN5sizyOErSWAjrMIZ4+eQoYqkhvHB+/lCsCHYLaAyAX4Mjhp54Dlop2fapjGtl18/nLvpoLj2kDi2RWJP2tk7LWR1gYRbAR4CERziAaOjWF9hjvgI+aX/+OjA9rOJhoy3ZyAihBIJvUsjXqZF4rgrNPJCrUjkbMjXe6TUmozRiwzC2yOfclmBHwUblcxikAOMYLNOmI6a+Jcpgb6O+mCnxYzDywwuwUXS3s6C4j8qft/svSzqQlmCL+/+HQ5MbKZHFjCrFrE4MvLAMUH78teZ3rYuF5FfecPPFrY/9NUgae38L1MYedEK2hSqPmCKWBpEv0/EkX0Dc4yDplqYCOd4eoonRT+13zl/jITvLLbd/nYfiTSKN9SpoUFZw76tRNO4ECNSWxLbQ2DmEGNpCM6J4uJZOHOd2oLkjqcgo6X+xAB7Wu32mmCH70T3gURgvEVFdGcijpo22frzQedoIT8laQiOEv7Tp26jA0YCHTRr8vZMZ91o3ECf/+JBpRTnMpeRts75QewbOBwd1gRhfCN+P1ul5k7GLh+/JiXuf/NH+IypTOHmtU/dwxtrgFzHznVdOhPYmISwcnsyQTp2HSX4iak2ITBWGOiUArxUCxDKItdx0iFNzpr4D6SUEo+2IdXqjHssA2LhI3TJQuwVbez5U8kJKu4JtF3v65q5H68wPg5kr0YAA/ysgfsqQXi/uf88iOetBLnEz45YkR70LNJhWMqpMjOziSg++9YZh7ymtevgoYg7Ayic8dbNYFSj7Fr28uvLDBABJq7bSnttr8pEj2n+UcDcDb1lPOYZGhdOpA07wjIlDEG6zqUIO62ldynfBI60DMQ6NVsXJ36FalO5NxyLbkYTlNrdrx6EL6PXONghSjVbh4yBSKA9uEH0hDTYNx3HYY3WBQZQXmqjYt1nYDRs4cjG6QmES0xt8Cttf8+V62oSM+eUV3pQcK9/eeJcXdOOLdS+ly6rhHSIn8L6nR1PwAzTZT7NC0xBnpMerhBWx4zqzd9n70q5eUPuIm0Pqni5dblVtWvUl6ovgUGuSZB1t8ddU1EQZ6iuzMtj0BWHpN1WCdsgkzofb9WIhdD8eXLZe0DzaX3mUjjrGtQsKnHMCB/mNQszEJM4uxTuOyf34UI0NUVn9pCcJweNQOR6gNEOLK0GRPfwSyi8srqcTYTPTMfaY9CszINblOgY5kyu5xyKesNaMmf8if7qpmKdu+uVNWjHN70kcJEFDYX4jmA/DbIn2Tum0GteG8mcCHdCOgwZalEkEkbuUaFgBFGD3mjxaPyPpRyIL00fxGI4vjyduAkrlVFpXTvlbiV1ySmVINpQ4OvWW07h1ywWUpchZuWXPof+pZJYbCy4byLYjCYD9JZNOlbsmG/O0mK3EUYycGydwN/TLLMt5QrdQkOh75no+ZeFrZ9IZVavQs0SRVkzmH0/4s9KlEeDXyt5cxZvpM/q7zt5XltLzPHotuWgG61mDOww9+C156NoCElBIWz43TDIHZzIM5QQzIOVAn1kySEwNCKXcIMgQ2bgX7BzzQZ8URyrmsvLWPkZmNxVxe9aaLC8tnSCw/hDbAT2fQ25fEzfuvqBOcDj+BiOCLWDp1f+NOj4qNSiYh2W/XKDCOoHuoUzVs7UoGnFbZ97axsgQuCwp0+keohVF6vOiXKMuLE+6m1A1/MnqJTKOrAXms8/rCF0hKbYL7X9laQn4givKOfwIgn9kYDEfj36xQEmbJxb4rf25seAMwk0ndm50IHey3m+Xgk9tb1OgEge2Sk+vcAFJDEJVLow0sKhGdE9dhGkUHHW84K6/y2U+kwNEkqur9S3wAtNi2OcEMDPisM7/UDhSmfX+Uh9nccmoyVd8Hx0EE0uudQu6ZncMrqHXnHp08m6WUAI7/+PQG20/klMPW7gfoJSFnDcaFxzrQjjE/8vWPBCEr3GNwAS5CNVBvcwpb5OSDIA1LMZENzhm5fCaKoJ+GD1xxu5QxOutkmy0fBbQWqlmlGE6983rAWO1eyVX8VkfH333+DBSomQcsrB7Yii+9D1yRFIPglpauepOAmm1fxgHganSsIRREbdPvtEZaQMx0Ncp2VJkCDCrqMqssvVIInYV4JiJpuqb8xGqAxvRfwZztDgHCUjbIhDXZf65SipLRUZevqiVi7Hw8Gz33IOnx8Zus5dMWocweShxNEZ9eRYIO8TIcSQyNa7dMvia1TCkvE2BZGJJshKe2WJzxl1VAx853ZiXy4WSuRNbcaLYdlHwi4dYfto3tbrLd7pQC7em+Ka5hSLA3Va8zkaz3WW3JwdgF66W5VX4FLEDJmbk44SfPUwx7HEeKXoiPrElRBgWfvXtGZdD7TdSvoU3aaO1AEjFXOzwIJcLE7i5IC3NrjXEiAqgG1YgqwZacmRhOyqbclrwmAuUz7srOxyzK7I0Ds9A+vvdwncoIGATydyklM8WMY4U749PKeLwGXhy+LbLczMIU3GOnW5rY842RzKvAIhq72o9OL7RoadqZgAk4cWtf8fIGmiLLXQqSD6d7nVeaLsgsYu7tf4C6y8gza0H5TeBxVZUqEOnHl5NQU59LIPCn1n+Eq7Gum+Q==" 20 | } 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /docs/using-crossplane-in-gitops-1.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | **Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* 4 | 5 | - [Using Crossplane in GitOps, Part I](#using-crossplane-in-gitops-part-i) 6 | - [Why Crossplane?](#why-crossplane) 7 | - [Bootstrap: Deploy Crossplane](#bootstrap-deploy-crossplane) 8 | - [Setup ProviderConfig](#setup-providerconfig) 9 | 10 | 11 | 12 | # Using Crossplane in GitOps, Part I: Bootstrap 13 | 14 | In this series of articles, I will share my recent study on using Crossplane in GitOps. I will use Argo CD as the GitOps tool to demonstrate how Crossplane can work with it to provision applications from git to target cluster. Meanwhile, I will also explore some best practices, common considerations, and lessons learned that you might experience as well when use Crossplane in GitOps. 15 | 16 | This article particularly focuses on why Crossplane fits into GitOps and how to bootstrap Crossplane using GitOps. You can find the demo project on GitHub at: https://github.com/morningspace/capabilities-shim-gitops. 17 | 18 | ## Why Crossplane? 19 | 20 | GitOps is essentially a way to host a git repository that contains declarative descriptions of target system as desired state and an automated process, usually driven by a GitOps tool such as Argo CD or Flux CD, to make the target system match the desired state in the repository, so that when you deploy a new application or update an existing one, you just need to update the repository, then the automated process handles everything else. 21 | 22 | Crossplane, on the other side, is a Kubernetes add-on that enables you to assemble infrastructure from multiple vendors and applications on top of that to expose them as high level self-service APIs for people to consume. 23 | 24 | There are a few reasons that I can see how Crossplane can fit into GitOps realm quit well: 25 | 26 | * Since Crossplane is designed to assemble infrastructure from multiple vendors, it makes it a lot easier to practice GitOps for application deployment across different vendors, typically public cloud vendors such as Amazon AWS, Google Cloud, Azure, IBM Cloud, etc. in a consistent manner. 27 | 28 | * With the help of its powerful composition engine, Crossplane allows people to compose different modules from infrastructure, service, to application as needed in a declarative way, where we can check these declarative descriptions into git for GitOps tools to pick up easily. 29 | 30 | * Crossplane allows people to extend its capabilities using Provider that can interact with different backends. There is a large amount of providers available in community and it is still actively evolving. By using variant providers, we can turn many different backends into something that are Kubernetes friendly, so that the desired state can be described using Kubernetes custom resource, then check into git and driven by GitOps tools. 31 | 32 | > A good example such as Terraform provider allows people to integrate existing Terraform automation assets into Crossplane and modeled as Kubernetes custom resource. See: https://github.com/crossplane-contrib/provider-terraform 33 | 34 | Turn everything into Kubernetes API no matter what kind of backend it is so that can be handled consistently using Kubernetes native way. As a result, those Kubernetes native objects are what we store in git which drive the GitOps flow. Using Crossplane opens the door to **gitopsifying** everything for us! 35 | 36 | ## Bootstrap: Deploy Crossplane 37 | 38 | When using Crossplane, usually you will have Crossplane with a set of providers and optionally configuration packages installed on your cluster. Since Crossplane works as a control plane to drive infrastructure and application provisioning and composing, it can be co-located with Argo CD on the same cluster as a GitOps control plane. 39 | 40 | Interestingly, if you have already installed Argo CD, you can take Argo CD as a top level "installer" which can help to install everything else including Crossplane itself. The only thing you need to do is to create an Argo `Application` that points to the helm repository where hosts the Crossplane helm charts. Check below YAML manifest into git, you will see Argo CD drives the Crossplane installation automatically. 41 | 42 | ```yaml 43 | --- 44 | apiVersion: v1 45 | kind: Namespace 46 | metadata: 47 | name: crossplane-system 48 | spec: 49 | finalizers: 50 | - kubernetes 51 | --- 52 | apiVersion: argoproj.io/v1alpha1 53 | kind: Application 54 | metadata: 55 | name: crossplane-app 56 | namespace: argocd 57 | spec: 58 | destination: 59 | namespace: crossplane-system 60 | server: https://kubernetes.default.svc 61 | project: default 62 | source: 63 | repoURL: https://charts.crossplane.io/stable 64 | chart: crossplane 65 | targetRevision: 1.4.1 66 | syncPolicy: 67 | automated: 68 | prune: true 69 | selfHeal: true 70 | ``` 71 | 72 | For provider installing, it can also be handled as such if the provider has been packaged and published so that it can be described using Crossplane manifest. 73 | 74 | Here I'm using [capabilities-shim](https://github.com/morningspace/capabilities-shim) as a demo project to demonstrate the Crossplane use in GitOps. There is a Crossplane configuration package which includes some pre-defined Composition and CompositeResourceDefinition manifests. It also depends on [an enhanced version](https://github.com/morningspace/provider-kubernetes) of [provider-kubernetes](https://github.com/crossplane-contrib/provider-kubernetes). In order to install both the configuration package and the provider, just need to check a `Configuration` resource in git as below: 75 | 76 | ```yaml 77 | apiVersion: pkg.crossplane.io/v1 78 | kind: Configuration 79 | metadata: 80 | name: capabilities-shim 81 | spec: 82 | ignoreCrossplaneConstraints: false 83 | package: quay.io/moyingbj/capabilities-shim:v0.0.1 84 | packagePullPolicy: IfNotPresent 85 | revisionActivationPolicy: Automatic 86 | revisionHistoryLimit: 0 87 | skipDependencyResolution: false 88 | ``` 89 | 90 | Crossplane will then extract the package, install all resources included, along with the dependent provider all automatically. 91 | 92 | ### Setup ProviderConfig 93 | 94 | For most Crossplane providers, you need to setup ProviderConfig before it can connect to the remote backends. In our case, provider-kubernetes needs below `ProviderConfig` in order to connect to the target cluster to provision the actual applications. 95 | 96 | ```yaml 97 | apiVersion: kubernetes.crossplane.io/v1alpha1 98 | kind: ProviderConfig 99 | metadata: 100 | name: provider-config-dev 101 | spec: 102 | credentials: 103 | source: Secret 104 | secretRef: 105 | namespace: dev 106 | name: cluster-config 107 | key: kubeconfig 108 | ``` 109 | 110 | This can also be checked into git. However, there are couple of things need to note. 111 | 112 | The above `ProviderConfig` requires a secret referenced by `spec.credentials.secretRef` to be created beforehand. The secret includes kubeconfig that maps local or remote clusters for the provider to connect to. This needs to be checked in git too if you want Argo CD to help you create it rather than manually create by yourself. 113 | 114 | You should never commit raw secrets in git. Before the secret can be stored in git, it needs to be encrypted at first. This can be done by sealed secrets introduced by the [Bitnami Sealed Secrets Controller](https://engineering.bitnami.com/articles/sealed-secrets.html). I will skip the detailed introduction about sealed secret here but you can google and learn how it works. Again, Sealed Secrets Controller can be installed using Argo CD by checking below Argo `Application` into git: 115 | 116 | ```yaml 117 | --- 118 | apiVersion: argoproj.io/v1alpha1 119 | kind: Application 120 | metadata: 121 | name: sealed-secrets-controller 122 | namespace: argocd 123 | spec: 124 | destination: 125 | namespace: argocd 126 | server: https://kubernetes.default.svc 127 | project: default 128 | source: 129 | repoURL: https://bitnami-labs.github.io/sealed-secrets 130 | targetRevision: 1.16.1 131 | chart: sealed-secrets 132 | helm: 133 | values: |- 134 | # https://github.com/argoproj/argo-cd/issues/5991 135 | commandArgs: 136 | - "--update-status" 137 | syncPolicy: 138 | automated: 139 | prune: true 140 | selfHeal: true 141 | ``` 142 | 143 | Then you need to generate a secret including the kubeconfig information for your target cluster, and use the `kubeseal` command line tool to encrypt it into a sealed secret. For example: 144 | 145 | ```console 146 | kubectl create secret generic cluster-config --from-literal=kubeconfig="`cat path/to/your/kubeconfig`" --dry-run -o yaml > cluster-config.yaml 147 | kubeseal -n dev --controller-namespace argocd < cluster-config.yaml > cluster-config.json 148 | ``` 149 | 150 | After the encrypted secret is generated, check in git and let Argo CD synchronize it to target cluster. 151 | 152 | > There are alternative approaches to handle secrets in GitOps, e.g. store secrets in external storage such as HashiCorp Vault, then store the secret key in git. It is not covered in this article. 153 | 154 | Now that you have launched the environment with the prerequisites ready including Crossplane, its providers, and all the necessary configuration, you can start to check application manifests in git to trigger the application provisioning driven by GitOps. In next article, I will explore several ways that I experimented when using Crossplane and Argo CD to provision applications. 155 | 156 | *(To be continued)* 157 | -------------------------------------------------------------------------------- /scripts/install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | #################### 4 | # Settings 5 | #################### 6 | 7 | # OS and arch settings 8 | HOSTOS=$(uname -s | tr '[:upper:]' '[:lower:]') 9 | HOSTARCH=$(uname -m) 10 | SAFEHOSTARCH=${HOSTARCH} 11 | if [[ ${HOSTOS} == darwin ]]; then 12 | SAFEHOSTARCH=amd64 13 | fi 14 | if [[ ${HOSTARCH} == x86_64 ]]; then 15 | SAFEHOSTARCH=amd64 16 | fi 17 | HOST_PLATFORM=${HOSTOS}_${HOSTARCH} 18 | SAFEHOSTPLATFORM=${HOSTOS}-${SAFEHOSTARCH} 19 | 20 | # Directory settings 21 | ROOT_DIR=$(cd -P $(dirname $0) >/dev/null 2>&1 && pwd) 22 | DEPLOY_LOCAL_WORKDIR=${ROOT_DIR}/.work/local/localdev 23 | TOOLS_HOST_DIR=${ROOT_DIR}/.cache/tools/${HOST_PLATFORM} 24 | 25 | mkdir -p ${DEPLOY_LOCAL_WORKDIR} 26 | mkdir -p ${TOOLS_HOST_DIR} 27 | 28 | # Custom settings 29 | . ${ROOT_DIR}/config.sh 30 | 31 | #################### 32 | # Utility functions 33 | #################### 34 | 35 | CYAN="\033[0;36m" 36 | NORMAL="\033[0m" 37 | RED="\033[0;31m" 38 | 39 | function info { 40 | echo -e "${CYAN}INFO ${NORMAL}$@" >&2 41 | } 42 | 43 | function error { 44 | echo -e "${RED}ERROR ${NORMAL}$@" >&2 45 | } 46 | 47 | function wait-deployment { 48 | local object=$1 49 | local ns=$2 50 | echo -n "Waiting for deployment $object in $ns namespace ready " 51 | retries=600 52 | until [[ $retries == 0 ]]; do 53 | echo -n "." 54 | local result=$(${KUBECTL} get deploy $object -n $ns -o jsonpath='{.status.readyReplicas}' 2>/dev/null) 55 | if [[ $result == 1 ]]; then 56 | echo " Done" 57 | break 58 | fi 59 | sleep 1 60 | retries=$((retries - 1)) 61 | done 62 | [[ $retries == 0 ]] && echo 63 | } 64 | 65 | #################### 66 | # Preflight check 67 | #################### 68 | 69 | function preflight-check { 70 | if ! command -v docker >/dev/null 2>&1; then 71 | error "docker not installed, exit." 72 | exit 1 73 | fi 74 | } 75 | 76 | #################### 77 | # Install kind 78 | #################### 79 | 80 | KIND=${TOOLS_HOST_DIR}/kind-${KIND_VERSION} 81 | 82 | function install-kind { 83 | info "Installing kind ${KIND_VERSION} ..." 84 | 85 | if [[ ! -f ${KIND} ]]; then 86 | curl -fsSLo ${KIND} https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-${SAFEHOSTPLATFORM} || exit -1 87 | chmod +x ${KIND} 88 | else 89 | echo "kind ${KIND_VERSION} detected." 90 | fi 91 | 92 | info "Installing kind ${KIND_VERSION} ... OK" 93 | } 94 | 95 | #################### 96 | # Install kubectl 97 | #################### 98 | 99 | KUBECTL=${TOOLS_HOST_DIR}/kubectl-${KUBECTL_VERSION} 100 | 101 | function install-kubectl { 102 | info "Installing kubectl ${KUBECTL_VERSION} ..." 103 | 104 | if [[ ! -f ${KUBECTL} ]]; then 105 | curl -fsSLo ${KUBECTL} https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/${HOSTOS}/${SAFEHOSTARCH}/kubectl || exit -1 106 | chmod +x ${KUBECTL} 107 | else 108 | echo "kubectl ${KUBECTL_VERSION} detected." 109 | fi 110 | 111 | info "Installing kubectl ${KUBECTL_VERSION} ... OK" 112 | } 113 | 114 | #################### 115 | # Launch kind 116 | #################### 117 | 118 | # The cluster information 119 | KIND_CLUSTER_NAME=capabilities-gitops-demo 120 | 121 | function kind-up { 122 | info "kind up ..." 123 | 124 | KIND_CONFIG_FILE=${ROOT_DIR}/kind.yaml 125 | ${KIND} get kubeconfig --name ${KIND_CLUSTER_NAME} >/dev/null 2>&1 || ${KIND} create cluster --name=${KIND_CLUSTER_NAME} --config="${KIND_CONFIG_FILE}" 126 | 127 | info "kind up ... OK" 128 | } 129 | 130 | function kind-down { 131 | info "kind down ..." 132 | 133 | ${KIND} delete cluster --name=${KIND_CLUSTER_NAME} 134 | 135 | info "kind down ... OK" 136 | } 137 | 138 | #################### 139 | # Install Argo CD 140 | #################### 141 | 142 | ARGOCD_CLI=${TOOLS_HOST_DIR}/argocd-${ARGOCD_CLI_VERSION} 143 | 144 | function install-argocd { 145 | info "Installing Argo CD ${ARGOCD_VERSION} ..." 146 | 147 | ${KUBECTL} get ns -o name | grep -q argocd || ${KUBECTL} create namespace argocd 148 | ${KUBECTL} apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/${ARGOCD_VERSION}/manifests/install.yaml 149 | 150 | wait-deployment argocd-server argocd 151 | 152 | ${KUBECTL} patch service/argocd-server -n argocd -p '{"spec": {"type": "NodePort", "ports": [{"name":"https", "nodePort": 30443, "port": 443}]}}' 153 | 154 | info "Installing Argo CD ${ARGOCD_VERSION} ... OK" 155 | } 156 | 157 | function install-argocd-cli { 158 | info "Installing Argo CD CLI ${ARGOCD_CLI_VERSION} ..." 159 | 160 | if [[ ! -f ${ARGOCD_CLI} ]]; then 161 | curl -fsSLo ${ARGOCD_CLI} https://github.com/argoproj/argo-cd/releases/download/${ARGOCD_CLI_VERSION}/argocd-${HOSTOS}-${SAFEHOSTARCH} || exit -1 162 | chmod +x ${ARGOCD_CLI} 163 | else 164 | echo "Argo CD CLI ${ARGOCD_CLI_VERSION} detected." 165 | fi 166 | 167 | info "Installing Argo CD CLI ${ARGOCD_CLI_VERSION} ... OK" 168 | } 169 | 170 | #################### 171 | # Install KubeSeal CLI 172 | #################### 173 | 174 | KUBESEAL_CLI=${TOOLS_HOST_DIR}/kubeseal-${KUBESEAL_CLI_VERSION} 175 | 176 | function install-kubeseal-cli { 177 | info "Installing KubeSeal CLI ${ARGOCD_CLI_VERSION} ..." 178 | 179 | if [[ ! -f ${KUBESEAL_CLI} ]]; then 180 | curl -fsSLo ${KUBESEAL_CLI} https://github.com/bitnami-labs/sealed-secrets/releases/download/${KUBESEAL_CLI_VERSION}/kubeseal-${HOSTOS}-${SAFEHOSTARCH} || exit -1 181 | chmod +x ${KUBESEAL_CLI} 182 | else 183 | echo "KubeSeal CLI ${KUBESEAL_CLI_VERSION} detected." 184 | fi 185 | 186 | info "Installing KubeSeal CLI ${KUBESEAL_CLI_VERSION} ... OK" 187 | } 188 | 189 | #################### 190 | # Print summary after install 191 | #################### 192 | 193 | function print-summary { 194 | cat << EOF 195 | 196 | 👏 Congratulations! The GitOps demo environment is available! 197 | It launched a kind cluster, installed following tools and applitions: 198 | - kind ${KIND_VERSION} 199 | - kubectl ${KUBECTL_VERSION} 200 | - argocd ${ARGOCD_VERSION} 201 | - argocd cli ${ARGOCD_CLI_VERSION} 202 | - kubeseal cli ${KUBESEAL_CLI_VERSION} 203 | 204 | $(print-console) 205 | 206 | For tools you want to run anywhere, create links in a directory defined in your PATH, e.g: 207 | ln -s -f ${KUBECTL} /usr/local/bin/kubectl 208 | ln -s -f ${KIND} /usr/local/bin/kind 209 | ln -s -f ${ARGOCD_CLI} /usr/local/bin/argocd 210 | ln -s -f ${KUBESEAL_CLI} /usr/local/bin/kubeseal 211 | 212 | EOF 213 | } 214 | 215 | function print-console { 216 | ARGOCD_PASSWORD="$(${KUBECTL} -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)" 217 | 218 | cat << EOF 219 | To access Argo CD UI, open https://$(hostname):9443 in browser. 220 | - username: admin 221 | - password: ${ARGOCD_PASSWORD} 222 | EOF 223 | } 224 | 225 | #################### 226 | # Generate cluster info 227 | #################### 228 | 229 | function gen-cluster-config { 230 | info "Generating cluster information ..." 231 | 232 | local ns=${1:-dev} 233 | 234 | CLUSTER_CONFIG_PATH=$(cd -P ${ROOT_DIR}/../environments/${ns}/env >/dev/null 2>&1 && pwd) 235 | 236 | KUBESVC_IP=$(${KUBECTL} get service kubernetes -o jsonpath='{.spec.clusterIP}') 237 | CLUSTER_CONFIG=$(${KIND} get kubeconfig --name ${KIND_CLUSTER_NAME} | sed -e "s|server:\s*.*$|server: https://${KUBESVC_IP}|g") 238 | ${KUBECTL} create secret generic cluster-config --from-literal=kubeconfig="${CLUSTER_CONFIG}" --dry-run -o yaml > ${CLUSTER_CONFIG_PATH}/cluster-config.yaml 239 | ${KUBESEAL_CLI} -n ${ns} --controller-namespace argocd < ${CLUSTER_CONFIG_PATH}/cluster-config.yaml > ${CLUSTER_CONFIG_PATH}/cluster-config.json.tmp 240 | if [[ $? == 0 ]]; then 241 | mv ${CLUSTER_CONFIG_PATH}/cluster-config.json{.tmp,} 242 | echo "The file ${CLUSTER_CONFIG_PATH}/cluster-config.json is updated, please check in to git." 243 | else 244 | rm ${CLUSTER_CONFIG_PATH}/cluster-config.json.tmp 245 | exit 1 246 | fi 247 | # rm -f ${CLUSTER_CONFIG_PATH}/cluster-config.yaml 248 | 249 | info "Generating cluster information ... OK" 250 | } 251 | 252 | #################### 253 | # Patch image pull secret 254 | #################### 255 | 256 | function patch-pull-secret { 257 | namespace='argocd' 258 | positional=('deploy/argocd-redis') 259 | 260 | while [[ $# -gt 0 ]]; do 261 | case "$1" in 262 | -n|--namespace) 263 | namespace="$2"; shift; shift ;; 264 | *) 265 | positional+=("$1"); shift ;; 266 | esac 267 | done 268 | 269 | echo -n "Enter docker username: " 270 | read DOCKER_USERNAME 271 | 272 | echo -n "Enter docker password: " 273 | read -s DOCKER_PASSWORD 274 | echo 275 | 276 | ${KUBECTL} create secret docker-registry docker-pull --docker-server=docker.io --docker-username=${DOCKER_USERNAME} --docker-password=${DOCKER_PASSWORD} -n $namespace 277 | ${KUBECTL} patch ${positional[@]} -n $namespace -p '{"spec": {"template": {"spec": {"imagePullSecrets":[{"name":"docker-pull"}]}}}}' 278 | } 279 | 280 | #################### 281 | # Print help 282 | #################### 283 | 284 | function print-help { 285 | cat << EOF 286 | Usage: $0 up 287 | $0 down 288 | $0 cluster-config 289 | $0 patch-pull-secret -n 290 | $0 console 291 | 292 | Examples: 293 | # Bring up the demo environment on your machine 294 | $0 up 295 | 296 | # Take down the demo environment on your machine 297 | $0 down 298 | 299 | # Generate and update the cluster-config secret encrypted by kubeseal for the demo environment 300 | # default to dev if omitted 301 | $0 cluster-config 302 | 303 | # Patch image pull secret for specific deployment from docker hub 304 | # default to deployment/argocd-redis if omitted 305 | # default to argocd if omitted 306 | $0 patch-pull-secret 307 | 308 | # Print Argo CD UI Console access information 309 | $0 console 310 | EOF 311 | } 312 | 313 | #################### 314 | # Main entrance 315 | #################### 316 | 317 | case $1 in 318 | "down") 319 | install-kind 320 | kind-down 321 | ;; 322 | "up") 323 | install-kind 324 | install-kubectl 325 | kind-up 326 | install-argocd 327 | install-argocd-cli 328 | install-kubeseal-cli 329 | print-summary 330 | ;; 331 | "cluster-config") 332 | install-kubeseal-cli 333 | gen-cluster-config ${@:2} 334 | ;; 335 | "patch-pull-secret") 336 | patch-pull-secret ${@:2} 337 | ;; 338 | "console") 339 | print-console 340 | ;; 341 | *) 342 | print-help 343 | ;; 344 | esac 345 | -------------------------------------------------------------------------------- /docs/using-crossplane-in-gitops-2.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | **Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* 4 | 5 | - [Using Crossplane in GitOps, Part II](#using-crossplane-in-gitops-part-ii) 6 | - [Store Manifests in Git Repository](#store-manifests-in-git-repository) 7 | - [Managed Resources](#managed-resources) 8 | - [Using Kustomize](#using-kustomize) 9 | - [Managed Resources As Template](#managed-resources-as-template) 10 | - [Composition and CompositeResourceDefinition](#composition-and-compositeresourcedefinition) 11 | - [When to Check Composition and CompositeResourceDefinition in Git](#when-to-check-composition-and-compositeresourcedefinition-in-git) 12 | - [Using Kustomize](#using-kustomize-1) 13 | - [Crossplane vs. Helm](#crossplane-vs-helm) 14 | - [Crossplane Composition vs. Helm Templates](#crossplane-composition-vs-helm-templates) 15 | - [CompositeResource(Claim), CompositeResourceDefinition vs. values.yaml](#compositeresourceclaim-compositeresourcedefinition-vs-valuesyaml) 16 | 17 | 18 | 19 | # Using Crossplane in GitOps, Part II: What to Check in Git 20 | 21 | In this series of articles, I will share my recent study on using Crossplane in GitOps. I will use Argo CD as the GitOps tool to demonstrate how Crossplane can work with it to provision applications from git to target cluster. Meanwhile, I will also explore some best practices, common considerations, and lessons learned that you might experience as well when use Crossplane in GitOps. 22 | 23 | This article particularly focuses on what to store in git for Crossplane and a side-by-side comparison between Crossplane and Helm as deployment tools. You can find the demo project on GitHub at: https://github.com/morningspace/capabilities-shim-gitops. 24 | 25 | ## Store Manifests in Git Repository 26 | 27 | Now that you have launched the environment with the prerequisites ready including Crossplane, its providers, and all the necessary configuration, you can start to check application manifests in git to trigger the application provisioning driven by GitOps. There are several ways that we can consider. 28 | 29 | ### Managed Resources 30 | 31 | In Crossplane, managed resource is Kubernetes custom resource defined and handled by Crossplane Provider. You can think of Crossplane with its provider equivalent to Kubernetes controller or operator. As such, when you check managed resource as manifest in git, it will be synchronized by Argo CD from git to target cluster, then detected by the provider and drive the actual application provisioning. 32 | 33 | ![](images/managed-resource.png) 34 | 35 | For example, check below managed resource in git, you will be able to deploy an Elasticsearch instance with the help of provider kubernetes: 36 | 37 | ```yaml 38 | --- 39 | apiVersion: kubernetes.crossplane.io/v1alpha1 40 | kind: Object 41 | metadata: 42 | name: elasticsearch-my-logging-stack 43 | spec: 44 | references: 45 | - fromObject: 46 | apiVersion: kubernetes.crossplane.io/v1alpha1 47 | kind: Object 48 | name: csv-my-logging-stack 49 | fieldPath: status.atProvider.manifest.status.phase 50 | forProvider: 51 | manifest: 52 | apiVersion: elasticsearch.k8s.elastic.co/v1 53 | kind: Elasticsearch 54 | metadata: 55 | name: my-logging-stack 56 | namespace: default 57 | spec: 58 | version: 7.13.3 59 | nodeSets: 60 | - name: default 61 | count: 1 62 | config: 63 | node.store.allow_mmap: false 64 | providerConfigRef: 65 | name: provider-config-dev 66 | ``` 67 | 68 | This approach is very straightforward, but it does not allow you to customize the configuration. For example, in our case, if you want to deploy Elasticsearch using another version instead of the one defined in the above managed resource, you need a way to override the default configuration. 69 | 70 | Another example is the per-environment deployment. When you deploy application to multiple clusters where some clusters may have specific configuration than others, you may need per-environment configuration. Of course, if you have one folder for each environment in git, you can copy and paste all the manifests to each folder that maps to the specific environment and do the environment specific modifications there. This may lead to duplication that is hard to maintain when the repository grows. 71 | 72 | #### Using Kustomize 73 | 74 | Per environment configuration can be done by [Kustomize](https://kustomize.io/). By using Kustomize, you can have the manifests with their default configuration at `base` layer, then specify the custom settings at `overlays` layer to override the base one. 75 | 76 | However, it should not be overused too much. The reason is that it obfuscates the understanding of what is actually deployed. If there are many kustomize-based versions of the same application manifests for different clusters, you have to assemble the final YAML in your head or using `kustomize` CLI to understand what is actually deployed in each cluster. In such a case, a templated framework like Helm would help. 77 | 78 | ### Managed Resources As Template 79 | 80 | Helm dynamically generates the configuration based on functions and parameters. It results in more reusable manifests stored in git. By using Helm, you can extract customizable configuration out of the managed resources, and put them into `values.yaml` with default values provided. With that, the managed resources stored in git will be templated manifests. 81 | 82 | ![](images/managed-resource-template.png) 83 | 84 | The good news is that Argo CD supports Helm very well. You can override the configuration defined in `values.yaml` when you define Argo `Application` resource for your application to be deployed. As an example, in the below Argo `Application`, we customized the name of the capabilities to be deployed via `spec.source.helm.parameters`. 85 | 86 | ```yaml 87 | apiVersion: argoproj.io/v1alpha1 88 | kind: Application 89 | metadata: 90 | name: capabilities-logging-app 91 | namespace: argocd 92 | spec: 93 | destination: 94 | namespace: dev 95 | server: 'https://kubernetes.default.svc' 96 | source: 97 | path: config/capabilities/crossplane-helm/logging 98 | repoURL: 'https://github.com/morningspace/capabilities-shim-gitops' 99 | targetRevision: HEAD 100 | helm: 101 | parameters: 102 | - name: metadata.name 103 | value: dev-env-logging-stack 104 | project: default 105 | ``` 106 | 107 | You can even override the configuration via Argo CD UI when you create or update the Argo `Application`. 108 | 109 | ![](images/argo-ui-helm.png) 110 | 111 | ### Composition and CompositeResourceDefinition 112 | 113 | Crossplane has a powerful composition engine. By defining `Composition` resource, it can compose multiple resources at different level from infrastructure to application all driven by providers at backend from different vendors. 114 | 115 | It also supports `CompositeResourceDefinition` (XRD) resource, which is extracted from the resources to be composed, and exposed as configurable settings with well-defined type and schema. 116 | 117 | Both `Composition` and `CompositeResourceDefinition` resources can be checked into git, so they can be synchronized by GitOps tool from git to target cluster. Based on that, you can define `CompositeResource` (XR) or `CompositeResourceClaim` (XRC), which is usually environment specific, and check it in git as well. After it is synchronized from git to target cluster, it will trigger Crossplane to generate the corresponding managed resources which will be detected by the providers and drive the actual application provisioning. 118 | 119 | ![](images/composition-and-xrd.png) 120 | 121 | In our case, since we have already synchronized and installed the Crossplane configuration package which includes all `Composition` and `CompositeResourceDefinition` resources to the target cluster by checking the `configurations.pkg.crossplane.io` resource in git, you do not have to check these packaged resources in git any more. The only thing you need to check in git is the `CompositeResourceClaim` resource such as below: 122 | 123 | ```yaml 124 | apiVersion: capabilities.morningspace.io/v1alpha1 125 | kind: LoggingClaim 126 | metadata: 127 | name: my-logging-stack 128 | spec: 129 | parameters: 130 | esVersion: 7.13.3 131 | kibanaVersion: 7.13.3 132 | compositionSelector: 133 | matchLabels: 134 | capability: logging 135 | provider: olm 136 | ``` 137 | 138 | #### When to Check Composition and CompositeResourceDefinition in Git 139 | 140 | It depends on whether or not you want to expose these details to Ops or SREs. For example, if you want them to understand how the managed resources are organized, to be able to modify these compositions, or even define their own compositions as needed, then you should check Composition and CompositeResourceDefinition in git. 141 | 142 | #### Using Kustomize 143 | 144 | The `CompositeResourceClaim` resource is usually environment specific. That means you can put it into environment specific folder in git. However, if you want the resource to be reusable and only override it partially per environment, you can also use Kustomize. Below is a sample folder structure: 145 | 146 | ``` 147 | └── environments 148 | ├── base 149 | │ ├── logging-claim.yaml 150 | │ └── kustomization.yaml 151 | └── overlays 152 | └── dev 153 | ├── logging-claim.yaml 154 | └── kustomization.yaml 155 | ``` 156 | 157 | There is a `logging-claim.yaml` in `base` folder, and a customized version in `overlays/dev` folder to override the base for environment dev. 158 | 159 | ```yaml 160 | apiVersion: capabilities.morningspace.io/v1alpha1 161 | kind: LoggingClaim 162 | metadata: 163 | annotations: 164 | capabilities.morningspace.io/provider-config: provider-config-dev 165 | name: my-logging-stack 166 | spec: 167 | parameters: 168 | esVersion: 7.15.0 169 | kibanaVersion: 7.15.0 170 | ``` 171 | 172 | Here we changed the version of Elasticsearch and Kibana to 7.15.0 as opposed to the default value in base, 7.13.3. We also added the annotation to specify which ProviderConfig we are going to use. 173 | 174 | ## Crossplane vs. Helm 175 | 176 | When use Composition, you may notice that it is very similar to Helm templates since essentially they both compose a set of Kubernetes resources. From application deployment point of view, Crossplane, as a deployment tool, provides some building blocks that are very similar to what Helm does, but they also have differences. In this section, I will explore them and make side-by-side comparison between Crossplane and Helm. 177 | 178 | Before that, there is one thing you may need to know: Crossplane and Helm are not mutual exclusive. Instead, they can be combined together. For example, you have already seen that Crossplane managed resources can be made as template using Helm. Especially, when you only use Crossplane providers and do not use its composition feature, the Crossplane runtime with the provider is very similar to a Kubernetes controller or an operator. In such a case, using Helm to render Kubernetes resources managed by the controller or operator is a very common practice. 179 | 180 | ### Crossplane Composition vs. Helm Templates 181 | 182 | A Crossplane `Composition` resource defines way of composing a set of Kubernetes resources. It is equivalent to Helm templates which include a set of template files and each file maps to one or more Kubernetes resources. A Crossplane Configuration package typically includes a set of Compositions, which map to multiple charts or sub-charts in Helm. The difference is that Composition organizes resources in a monolithic way where all resources are defined in the same file. But for Helm templates, they are separate files in the same folder or different folders. Certainly you can combine these files together as a single file to make it very similar to a Crossplane Composition resource, but they do have some differences in nature as below: 183 | 184 | * Instead of templating, Crossplane renders `Composition` resource by extracting values from `CompositeResource` (XR) or `CompositeResourceClaim` (XRC) resource and patching them to specific fields on managed resources. This is very similar to Kustomize. 185 | * At a much higher level, we usually see Crossplane Composition is used to composing modules from infrastructure, service, to application in more coarse grained way. On the other hand, Helm usually focuses on "composing" modules at application level in more fine grained way. But this does not mean you cannot compose infrastructure using Helm. For example, with the combined use of Crossplane Provider and Helm, you can compose infrastructure, service, and application too. 186 | 187 | ### CompositeResource(Claim), CompositeResourceDefinition vs. values.yaml 188 | 189 | The Crossplane `CompositeResource` (XR) or `CompositeResourceClaim` (XRC) resource is essentially equivalent to the `values.yaml` file in Helm. Just like `values.yaml`, XR/XRC is aimed to extract the configurable settings out of the original resources for people to consume. 190 | 191 | As an example, in our demo project, there is also a folder including all manifests used to provision the demo application using Helm. If you look at the `values.yaml` inside the folder, you will see it is very similar to the `CompositeResourceClaim` resource that we defined previously: 192 | 193 | ```yaml 194 | metadata: 195 | name: my-logging-stack 196 | namespace: default 197 | spec: 198 | parameters: 199 | esVersion: 7.13.3 200 | kibanaVersion: 7.13.3 201 | ``` 202 | 203 | The major difference between the two representations is that, Crossplane uses a more well-defined data structure to organize these configurable settings. That is `CompositeResourceDefinition` (XRD). By defining XRD, you can restrict user input with these settings in a more controlled manner. For example, each field has a type and can be required or optional. All user input verification happens at server side. This is very different from what Helm does. Also, each field can have a description so that the field can be well documented and self explained. 204 | 205 | Another apparent difference is that, GitOps tool such as Argo CD has integrated with Helm very well. You can specify custom settings in Argo `Application` resource, even from its UI, without touching `values.yaml` directly. Crossplane on the other side has no such level of integration yet. 206 | 207 | Below is a table that summarizes all above differences that we explored. 208 | 209 | | Crossplane | Helm | Description 210 | |:----------------------------|:------------|:----- 211 | | Composition | Templates | Both to compose a set of Kubernetes resources, but Composition uses patch to override while Helm uses template. 212 | | CompositeResource(Claim) | values.yaml | Both to allow user input as configurable settings. Argo CD has better support on Helm, e.g: to specify values in Argo `Application` resource. 213 | | CompositeResourceDefinition | n/a | CompositeResourceDefinition as a schema has better user input control. 214 | 215 | And a diagram to depict the side-by-side comparison results. 216 | 217 | ![](images/crossplane-vs-helm.png) 218 | 219 | In next article, I will explore some common considerations when you practice Crossplane in GitOps such as synchronization order, hooks, health check, git repository folder structure, and so on. 220 | 221 | *(To be continued)* -------------------------------------------------------------------------------- /docs/using-crossplane-in-gitops-3.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | **Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* 4 | 5 | - [Using Crossplane in GitOps, Part III](#using-crossplane-in-gitops-part-iii) 6 | - [Deploying Order](#deploying-order) 7 | - [Argo CD Syncwave](#argo-cd-syncwave) 8 | - [Dependency Resolving by Crossplane](#dependency-resolving-by-crossplane) 9 | - [Uninstall in Reversed Order](#uninstall-in-reversed-order) 10 | - [Using Hooks](#using-hooks) 11 | - [Argo CD Hooks](#argo-cd-hooks) 12 | - [Helm Hooks](#helm-hooks) 13 | - [Health Check](#health-check) 14 | - [Organizing Configuration](#organizing-configuration) 15 | 16 | 17 | 18 | # Using Crossplane in GitOps, Part III: Common Considerations 19 | 20 | In this series of articles, I will share my recent study on using Crossplane in GitOps. I will use Argo CD as the GitOps tool to demonstrate how Crossplane can work with it to provision applications from git to target cluster. Meanwhile, I will also explore some best practices, common considerations, and lessons learned that you might experience as well when use Crossplane in GitOps. 21 | 22 | This article particularly focuses on some common considerations that you may have when using Crossplane in GitOps such as deploying order, using hooks, health check, organizing configuration. You can find the demo project on GitHub at: https://github.com/morningspace/capabilities-shim-gitops. 23 | 24 | ## Deploying Order 25 | 26 | ### Argo CD Syncwave 27 | 28 | It is a common case where an application may be composed of multiple modules and some modules depend on other modules. When GitOps tool synchronizes these modules to target cluster, they need to be applied in a specific order. As an example, the deployment of Crossplane needs to be finished prior to the deployment of a Crossplane configuration package. This is because the configuration package requires some Crossplane CRDs available before it can be deployed. These CRDs come from the Crossplane deployment. 29 | 30 | In Argo CD, this can be solved by using syncwave. Syncwaves are used to order how manifests are applied or synchronized by Argo CD to the cluster. You can have one or more waves, that allows you to ensure certain resources are healthy before subsequent resources are synchronized. 31 | 32 | Syncwave for a certain resource can be specified by using the annotation `argocd.argoproj.io/sync-wave` with a number. Resource annotated with lower number will be synchronized first. 33 | 34 | For example, in our case, we created Argo `Application` for Crossplane, OLM, and Sealed Secret Controller and they are all annotated with the same syncwave value. This is because they do not have dependencies with each other and can be deployed in parallel. 35 | 36 | ```yaml 37 | apiVersion: argoproj.io/v1alpha1 38 | kind: Application 39 | metadata: 40 | name: crossplane-app 41 | namespace: argocd 42 | annotations: 43 | argocd.argoproj.io/sync-wave: "100" 44 | ... 45 | --- 46 | apiVersion: argoproj.io/v1alpha1 47 | kind: Application 48 | metadata: 49 | name: olm-app 50 | namespace: argocd 51 | annotations: 52 | argocd.argoproj.io/sync-wave: "100" 53 | ... 54 | --- 55 | apiVersion: argoproj.io/v1alpha1 56 | kind: Application 57 | metadata: 58 | name: sealed-secrets-controller 59 | namespace: argocd 60 | annotations: 61 | argocd.argoproj.io/sync-wave: "100" 62 | ... 63 | ``` 64 | 65 | But for the Argo `Application` that represents the deployment of Crossplane configuration package, it needs to be applied after Crossplane is running. This is why its syncwave is a number larger the above ones. 66 | 67 | ```yaml 68 | --- 69 | apiVersion: argoproj.io/v1alpha1 70 | kind: Application 71 | metadata: 72 | name: crossplane-ext-app 73 | namespace: argocd 74 | annotations: 75 | argocd.argoproj.io/sync-wave: "150" 76 | ``` 77 | 78 | ### Dependency Resolving by Crossplane 79 | 80 | Using syncwave to guarantee the resource synchronization order is a very useful practice. But it also makes your GitOps solution coupled with a specific GitOps tool. For example, you may need to figure out a similar approach if you switch from Argo CD to Flux CD as Flux does not recognize the Argo CD syncwave annotations. 81 | 82 | Crossplane, on the other hand, might help in this case. This is because it can provide a way to define the deploying order for a set of composable resources while remains neutral to GitOps tools. Although this has not been there yet, there is a design proposal in community called [Generic References](https://github.com/crossplane/crossplane/pull/2385) which is currently under discussion. This design can help us define resource dependency and do more than that. 83 | 84 | The major goal of this design is to address the scenario where managed resource requires information that is not available when user creates the resource. For example, to create a Subnet, you need to supply a VPC ID. If you create the VPC at the same time, then the ID will be available only after the VPC creation. According to the proposal, all managed resources will have a top-level spec field that lists the generic references. They will be resolved in order. Failure of any one will cause reconciliation to fail. Below is a sample snippet for a managed resource which has a reference to another resource: 85 | 86 | ```yaml 87 | spec: 88 | patches: 89 | - fromObject: 90 | apiVersion: v1 91 | kind: ConfigMap 92 | name: common-settings 93 | namespace: crossplane-system 94 | fieldPath: data.region 95 | toFieldPath: spec.forProvider.region 96 | ``` 97 | 98 | By defining references, we can ask Crossplane to help us resolve dependencies and patch missing fields as above if needed. Thus, we can keep deploying order without depending on Argo CD. Argo CD can simply synchronize all the resources in one go, then Crossplane will handle the deploying order for these resources properly. 99 | 100 | Although this design has not been closed yet, a similar idea has been implemented in [the enhanced version](https://github.com/morningspace/provider-kubernetes) of provider-kubernetes. When an `Object` resource is define, the managed resource defined and handled by provider-kubernetes, it allows you to specify one or more references in `spec.references` using exactly the same syntax as it is defined in the above design. 101 | 102 | For example, in our case, the `Object` resource for `ClusterServiceVersion` has a reference to the `Object` resource for `Subscription`, so the `ClusterServiceVersion` name can be resolved from the dependent `Object` via field path `status.atProvider.manifest.status.currentCSV` and be applied to the referencing `Object` at `spec.forProvider.manifest.metadata.name`: 103 | 104 | ```yaml 105 | --- 106 | apiVersion: kubernetes.crossplane.io/v1alpha1 107 | kind: Object 108 | metadata: 109 | name: csv-my-logging-stack 110 | annotations: 111 | kubernetes.crossplane.io/managementType: "ObservableAndDeletable" 112 | spec: 113 | references: 114 | - fromObject: 115 | apiVersion: kubernetes.crossplane.io/v1alpha1 116 | kind: Object 117 | name: sub-my-logging-stack 118 | fieldPath: status.atProvider.manifest.status.currentCSV 119 | toFieldPath: spec.forProvider.manifest.metadata.name 120 | forProvider: 121 | manifest: 122 | apiVersion: operators.coreos.com/v1alpha1 123 | kind: ClusterServiceVersion 124 | metadata: 125 | namespace: operators 126 | providerConfigRef: 127 | name: provider-config-dev 128 | ``` 129 | 130 | This approach has been adopted in the GitOps demo project: [capabilities-shim-gitops](https://github.com/morningspace/capabilities-shim-gitops). For more information on the enhanced version of provider-kubernetes, please check [this document](https://github.com/morningspace/capabilities-shim/blob/main/docs/enhanced-provider-k8s.md). 131 | 132 | ### Uninstall in Reversed Order 133 | 134 | Although it may not be very often in a real customer environment where people want to uninstall the whole application stack completely, when you delete the resources from target cluster, usually it needs to be in a reversed order as opposed to the order used when the resources are applied. 135 | 136 | Interestingly, Argo CD supports this case for its syncwave feature. There is [an issue](https://github.com/argoproj/argo-cd/issues/3211) reported for this as an enhancement and it has been implemented since v1.7. So, if I have two resources and specify the syncwave that resource a needs to be synchronized before resource b. When deleting these resources, it should delete resource b first, and then resource a. 137 | 138 | On the other hand, Crossplane does not support this at the time of writing. It is also not covered in the [Generic References](https://github.com/crossplane/crossplane/pull/2385) design proposal. But [the enhanced version](https://github.com/morningspace/provider-kubernetes) of provider-kubernetes has implemented that. It is achieved by applying a set of finalizers to the managed resources sequentially. More on this can be found [in this section](https://github.com/morningspace/capabilities-shim/blob/main/docs/enhanced-provider-k8s.md#uninstall-order) from its design document. 139 | 140 | ## Using Hooks 141 | 142 | To define and check the desired state of target cluster in git then have GitOps tool to synchronize it over is great. But in some cases, to run everything declaratively may not be realistic in real world. You may still have to prepare something imperatively using script before the synchronization starts or do some cleanup after it is completed. 143 | 144 | ### Argo CD Hooks 145 | 146 | In Argo CD, this can be configured using [Resource Hooks](https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/). Hooks are ways to run scripts before, during, and after a synchronization happens. A hook is simply a Kubernetes resource, typically a pod or job, annotated with `argocd.argoproj.io/hook`. As an example, in our case, we defined below job as `PostSync` hook to make sure all ClusterServiceVersion resources are succeeded after synchronization is finished. 147 | 148 | ```yaml 149 | apiVersion: batch/v1 150 | kind: Job 151 | metadata: 152 | name: post-install-job 153 | namespace: argocd 154 | annotations: 155 | argocd.argoproj.io/hook: PostSync 156 | spec: 157 | template: 158 | metadata: 159 | name: post-install-job 160 | spec: 161 | containers: 162 | - name: check-csv 163 | image: quay.io/bitnami/kubectl:latest 164 | command: 165 | - /bin/sh 166 | - -c 167 | - | 168 | current_seconds=0 169 | limit_seconds=$(( $(date +%s) + 600 )) 170 | installing=0 171 | 172 | while [ ${current_seconds} -lt ${limit_seconds} ]; do 173 | if [ $(kubectl get csv -n "{{.Values.metadata.namespace}}" -o name | grep -c clusterserviceversion) -eq 0 ] || \ 174 | [ $(kubectl get csv -n "{{.Values.metadata.namespace}}" --no-headers | grep -vc Succeeded) -gt 0 ]; then 175 | installing=1 176 | echo "INFO: ClusterServiceVersion resources still installing." 177 | sleep 1 178 | else 179 | installing=0 180 | break 181 | fi 182 | current_seconds=$(( $(date +%s) )) 183 | done 184 | 185 | if [ ${installing} -eq 0 ]; then 186 | echo "INFO: All ClusterServiceVersion resources are ready." 187 | else 188 | echo "ERROR: ClusterServiceVersion resources still not ready." 189 | exit 1 190 | fi 191 | restartPolicy: Never 192 | serviceAccountName: argocd-application-controller 193 | backoffLimit: 1 194 | ``` 195 | 196 | ### Helm Hooks 197 | 198 | Helm also supports hooks. It allows chart developers to intervene at certain points when a Helm release is deployed. Just like Argo CD, Helm uses special annotation `helm.sh/hook` to indicate whether a template, e.g.: a pod or job, will be treated as a hook. 199 | 200 | Interestingly, Argo CD supports Helm hooks by mapping the Helm annotations onto Argo CD's own hook annotations. When Argo CD synchronizes Helm charts, a Helm hook can usually be transformed to an Argo CD hook without additional work. For example, the pre-install hook in Helm is equivalent to the PreSync hook in Argo CD, and the post-install hook in Helm is equivalent to the PostSync hook in Argo CD. For more details, please check the [Argo CD document](https://argo-cd.readthedocs.io/en/stable/user-guide/helm/#helm-hooks) where it includes a mapping table between Helm hooks and Argo CD hooks. 201 | 202 | However, one thing to note is that not all Helm hooks have equivalents in Argo CD. For example, there's no equivalent in Argo CD for Helm post-delete hook. That means if you define some job as post-delete hook in Helm, they will never be triggered in Argo CD. [An issue](https://github.com/argoproj/argo-cd/issues/7575) on this was opened in Argo CD community. 203 | 204 | In Crossplane, there is no concept such as hooks. So, if you want to do something before or after a certain synchronization, you may still resort to Helm or Argo CD. 205 | 206 | ## Health Check 207 | 208 | To check the desired state in git and ask Argo CD synchronize it is just one side. The other side is to ensure what you deployed is healthy by checking the actual state in target cluster. 209 | 210 | Argo CD provides built-in health assessment for several kubernetes resources. It can be further customized by writing your own health checks in Lua code. This is useful if you have a custom resource for which Argo CD does not have a built-in health check. You may find the more you rely on Argo CD for resource health check, the less you go back to check that by using kubectl from command line. 211 | 212 | It is very common in Crossplane for its providers which have their own managed resources and do not have built-in health checks support in Argo CD. Thus, you should define your own health checks for Crossplane providers. 213 | 214 | In our demo project, we defined custom health checks for those resource types handled by OLM and operators launched by OLM, also the checks for the managed resource type `Object` handled by provider kubernetes. All these custom health checks should be added to a ConfigMap called `argocd-cm` in the namespace where Argo CD is deployed. Again, this ConfigMap can be checked in git so that can be synchronized by Argo CD as well. 215 | 216 | For example, below is a sample snippet for the ConfigMap which includes the custom health checks for the resource type `ClusterServiceVersion` handled by OLM and the custom resource `Elasticsearch` handled by Elasticsearch operator. Typically, you need to assess the resource healthiness by querying sub-fields under the resource status field. You can also construct a status message as below to reveal more detailed information: 217 | 218 | ```yaml 219 | --- 220 | apiVersion: v1 221 | kind: ConfigMap 222 | metadata: 223 | name: argocd-cm 224 | namespace: argocd 225 | labels: 226 | app.kubernetes.io/name: argocd-cm 227 | app.kubernetes.io/part-of: argocd 228 | data: 229 | resource.customizations.health.operators.coreos.com_ClusterServiceVersion: | 230 | hs = {} 231 | hs.status = "Progressing" 232 | hs.message = "" 233 | if obj.status ~= nil then 234 | if obj.status.phase == "Succeeded" then 235 | hs.status = "Healthy" 236 | end 237 | hs.message = obj.status.message 238 | end 239 | return hs 240 | resource.customizations.health.elasticsearch.k8s.elastic.co_Elasticsearch: | 241 | hs = {} 242 | hs.status = "Progressing" 243 | hs.message = "" 244 | if obj.status ~= nil and obj.status.health ~= nil and obj.status.phase ~= nil then 245 | if obj.status.health == "green" and obj.status.phase == "Ready" then 246 | hs.status = "Healthy" 247 | end 248 | hs.message = "health = " .. obj.status.health .. " phase = " .. obj.status.phase 249 | end 250 | return hs 251 | ``` 252 | 253 | For the health check of `Object` resource handled by provider kubernetes, it is a bit more complex. 254 | 255 | ```yaml 256 | resource.customizations.health.kubernetes.crossplane.io_Object: | 257 | hs = {} 258 | hs.status = "Progressing" 259 | hs.message = "" 260 | if obj.status ~= nil and obj.status.atProvider ~= nil then 261 | kind = obj.spec.forProvider.manifest.kind 262 | res = obj.status.atProvider.manifest 263 | if res ~= nil then 264 | if kind == "ClusterServiceVersion" then 265 | if res.status ~= nil then 266 | if res.status.phase == "Succeeded" then 267 | hs.status = "Healthy" 268 | end 269 | hs.message = res.status.message 270 | end 271 | elseif kind == "Elasticsearch" then 272 | if res.status ~= nil and res.status.health ~= nil and res.status.phase ~= nil then 273 | if res.status.health == "green" and res.status.phase == "Ready" then 274 | hs.status = "Healthy" 275 | end 276 | hs.message = "health = " .. res.status.health .. " phase = " .. res.status.phase 277 | end 278 | end 279 | end 280 | end 281 | return hs 282 | ``` 283 | 284 | ## Organizing Configuration 285 | 286 | As we all know, in GitOps, the git repository used to store desired state for target system is the single source of truth, but there is no single answer on how the desired state should be organized in git repository. Although each team may have its own consideration on this for variant reasons, there are some common suggestions as following: 287 | 288 | - Place common configuration that can be applied to all environments in one single place. This may include the common infrastructure configuration, shared applications and their settings that are applicable to all environments. For example, in the demo project, we use `config/` folder to store all configuration required to deploy Crossplane and its provider, OLM, Sealed Secret Controller, as well as the Argo CD customization settings and RBAC settings. 289 | 290 | - Place per-environment configuration in separate places and one place for each environment. Some environment may have its unique configuration which can be put in a place represents that specific environment. For example, there can be separate folders for dev, staging, and product environment. In our demo project, we use `environments/` folder to host all environment specific configuration, where we place the Crossplane ProviderConfig, the encrypted secret that represents the target cluster kubeconfig credentials, and so on. These are all configuration unique to each cluster. 291 | 292 | - Use branch to track per-release configuration. It is a very common practice for developer to track code changes among different releases using git release branch, especially when you have multiple releases that need to be maintained at the same time. The same rules apply to configuration changes in GitOps. When you have multiple releases to support and the configuration keeps changing from release to release, you can create branch for each release. 293 | Of course, it will bring additional effort to merge changes among branches and resolve merge conflicts when needed. The good thing is, all branching and merging practices that you have already been familiar with when dealing with code changes can also be applied to the configuration changes. 294 | 295 | - Host multiple applications manifests in a monolithic repository. This is a very effective way to manage applications in a small project where you put all applications configuration manifests in a single repository. It can be stored in separate folders, one folder for each application. 296 | 297 | - Host multiple applications manifests separately in multiple repositories. This is suitable for a large project in which you may have multiple products, each product has its own set of applications, and maintained by different team. By putting configuration manifests for these applications in separate repositories, you can leverage the sophisticated organization or repository membership and access control capabilities provided by the git infrastructure provider such as GitHub, to manage application deployment for each team differently in a fine grained manner. 298 | 299 | - Use Helm to parameterize deployment manifests and turn into reusable templates. In some cases, you may want your application manifests to be configurable, e.g.: to allow Ops or SREs to choose which version to install, which storage to pick up, or which namespace to apply, etc. Helm as a deployment tool for Kubernetes application is widely used. It can help you extract parameters out of deployment manifest, and turn the manifest into a reusable template. Helm can also be used together with Crossplane, especially when you only use Crossplane providers to provision applications and, do not use its composition. In such case, Crossplane plays very similar role as Kubernetes controller or Operator does with its wide range of providers that extend the scope of what you can manage using GitOps. 300 | 301 | 302 | - Use Kustomize to override deployment manifests as a base for a specific environment. In some cases, you may want your application manifests to be customized on a per environment basis. Instead of Helm, this can also be achieved by using Kustomize. You can put the manifests with default configuration in a folder as a base layer, then put environment specific manifest pieces in a separate folder that represents a specific environment to override the base layer. This can also be applied when you use Crossplane, especially when using composition. For example, you can define the default CompositeResourceClaim (XRC) at base layer, then override it at environment specific layer. 303 | 304 | - Use App of Apps pattern when use Argo CD to mange a set of applications. In Argo CD, an `Application` resource is a unit that deploys a set of manifests. Since an Application is a Kubernetes resource, it can be managed by Argo CD too. Furthermore, an Application resource can manage multiple other Application resources. That is called App of Apps pattern. By using this pattern, you can deploy a set of applications in one go. Also, increasing or decreasing Application resources can be done by adding or removing manifests to git repository instead of operating Argo CD via its Web UI or command line. 305 | 306 | Below is a sample structure for a git repository to demonstrate how folders can be organized base on the above discussion: 307 | 308 | ```console 309 | ├── config 310 | │ ├── argocd-apps 311 | │ ├── infra 312 | │ ├── services 313 | │ └── apps 314 | └── environments 315 | ├── base 316 | └── overlays 317 | ├── dev 318 | ├── staging 319 | └── prod 320 | ``` 321 | --------------------------------------------------------------------------------