├── docs ├── images │ ├── promotion-flow.png │ ├── kalypso-detailed.png │ ├── next-environment.png │ ├── source-branches.png │ ├── cicd-tutorial-setup.png │ ├── gh-promotion-flow.png │ ├── kalypso-highlevel.png │ ├── manifests-branches.png │ ├── under-construction.png │ ├── cicd-tutorial-dev-pr.png │ ├── cicd-tutorial-gitops.png │ ├── cicd-build-commit-status.png │ ├── cicd-tutorial-config-pr.png │ ├── cicd-tutorial-greetings.png │ ├── kalypso-non-containerized.png │ ├── cicd-deployed-commit-status.png │ ├── cicd-tutorial-post-deployment.png │ ├── gitops-CD-Simple-Flow.drawio.png │ ├── gitops-CD-Detailed-Flow.drawio.png │ ├── cicd-promoted-stage-commit-status.png │ └── cicd-tutorial-done-commit-status.png ├── run-books │ ├── images │ │ ├── eu-config.png │ │ ├── revert-pr.png │ │ ├── workload.png │ │ ├── ci-pipeline.png │ │ ├── k3scluster.png │ │ ├── new-ring-pr.png │ │ ├── pr-promoted.png │ │ ├── pr-deployment.png │ │ ├── pr-to-src-code.png │ │ ├── update-config.png │ │ ├── gitops-commit-id.png │ │ ├── mayberry-cluster.png │ │ ├── new-environment.png │ │ ├── next-environment.png │ │ ├── next_ring.drawio.png │ │ ├── not-promoted-pr.png │ │ ├── k3scluster-gitops.png │ │ ├── platform-gitops-pr.png │ │ ├── pr-deployment-stage.png │ │ ├── promote-flow.drawio.png │ │ ├── scheduling-policy.png │ │ ├── revert-config-change.png │ │ ├── workload-registration.png │ │ ├── config-change-dashboard.png │ │ ├── empty-environment-branch.png │ │ ├── environment-definition.png │ │ ├── new-branch-ring-created.png │ │ ├── next_environment.drawio.png │ │ ├── not-promoted-pr-details.png │ │ ├── update-app-config-values.png │ │ ├── next-new-ring-environment.png │ │ ├── deployment-observability-edit.png │ │ ├── deployment-observability-hub.png │ │ ├── deployment-observability-save.png │ │ ├── deployment-observability-edit-colors.png │ │ ├── deployment-observability-hw-deployed.png │ │ └── deployment-observability-hw-deployed-stage.png │ ├── README.md │ ├── application-team-onboards-a-new-application.md │ ├── platform-team-manages-platform-configuration.md │ ├── application-team-manages-application-configuration.md │ ├── platform-team-onboards-a-new-cluster.md │ ├── application-team-creates-a-new-application-ring.md │ ├── platform-team-creates-a-new-environment.md │ ├── platform-team-schedules-applications-for-deployment.md │ └── application-team-promotes-a-change-through-environments.md ├── use-cases │ ├── platform-team-provides-configuration-value.md │ ├── platform-team-onboards-workload.md │ ├── application-team-provides-configuration-value.md │ ├── platform-team-defines-deployment-targets.md │ ├── application-team-defines-deployment-targets.md │ ├── platform-team-defines-cluster-type.md │ ├── platform-team-scheduled-application.md │ ├── platform-team-config-values.md │ ├── application-team-updates-application.md │ └── platform-team-adds-platform-service.md ├── cd-concept.md └── bootstrap │ ├── quickstart.md │ └── prerequisites.md ├── scripts └── bootstrap │ ├── templates │ ├── gitops │ │ ├── main │ │ │ └── README.md │ │ └── dev │ │ │ └── .github │ │ │ └── workflows │ │ │ └── check-promote.yaml │ ├── control-plane │ │ ├── dev │ │ │ ├── gitops-repo.yaml │ │ │ ├── base-repo.yaml │ │ │ ├── cluster-types │ │ │ │ └── dev.yaml │ │ │ ├── configs │ │ │ │ └── dev-config.yaml │ │ │ └── scheduling-policies │ │ │ │ └── dev-policy.yaml │ │ └── main │ │ │ ├── .environments │ │ │ └── dev.yaml │ │ │ ├── .github │ │ │ └── workflows │ │ │ │ ├── utils │ │ │ │ ├── baserepo-template.yaml │ │ │ │ ├── save-promoted-commit.sh │ │ │ │ ├── update-status.sh │ │ │ │ └── wait-for-deployment.sh │ │ │ │ ├── ci.yaml │ │ │ │ ├── cd.yaml │ │ │ │ └── check-promote.yaml │ │ │ ├── workloads │ │ │ └── sample-workload-registration.yaml │ │ │ └── templates │ │ │ ├── configmap.yaml │ │ │ ├── namespace.yaml │ │ │ ├── argocd.yaml │ │ │ └── arc-flux.yaml │ └── config.yaml │ ├── .shellcheckrc │ ├── README.md │ └── lib │ └── install.sh ├── .gitmodules ├── package.json ├── CODE_OF_CONDUCT.md ├── SUPPORT.md ├── .github └── workflows │ ├── templates │ ├── utils │ │ ├── generate-deployment-descriptor.sh │ │ ├── update-status.sh │ │ ├── get-tracking-info.sh │ │ ├── generate-manifests.sh │ │ ├── create-pr.sh │ │ └── wait-for-deployment.sh │ ├── notify-on-pr.yml │ ├── notify-on-config-change.yml │ ├── ci.yml │ ├── deploy.yml │ └── post-deployment.yml │ └── pr.yaml ├── cspell.json ├── cspell-cse.txt ├── LICENSE ├── .markdownlinkcheck.json ├── SECURITY.md ├── cicd ├── readme.md ├── setup.md ├── tutorial │ ├── deploy.sh │ └── cicd-tutorial.md └── setup.sh └── .markdownlint.yaml /docs/images/promotion-flow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/promotion-flow.png -------------------------------------------------------------------------------- /docs/images/kalypso-detailed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/kalypso-detailed.png -------------------------------------------------------------------------------- /docs/images/next-environment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/next-environment.png -------------------------------------------------------------------------------- /docs/images/source-branches.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/source-branches.png -------------------------------------------------------------------------------- /docs/images/cicd-tutorial-setup.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/cicd-tutorial-setup.png -------------------------------------------------------------------------------- /docs/images/gh-promotion-flow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/gh-promotion-flow.png -------------------------------------------------------------------------------- /docs/images/kalypso-highlevel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/kalypso-highlevel.png -------------------------------------------------------------------------------- /docs/images/manifests-branches.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/manifests-branches.png -------------------------------------------------------------------------------- /docs/images/under-construction.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/under-construction.png -------------------------------------------------------------------------------- /docs/run-books/images/eu-config.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/eu-config.png -------------------------------------------------------------------------------- /docs/run-books/images/revert-pr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/revert-pr.png -------------------------------------------------------------------------------- /docs/run-books/images/workload.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/workload.png -------------------------------------------------------------------------------- /docs/images/cicd-tutorial-dev-pr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/cicd-tutorial-dev-pr.png -------------------------------------------------------------------------------- /docs/images/cicd-tutorial-gitops.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/cicd-tutorial-gitops.png -------------------------------------------------------------------------------- /docs/run-books/images/ci-pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/ci-pipeline.png -------------------------------------------------------------------------------- /docs/run-books/images/k3scluster.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/k3scluster.png -------------------------------------------------------------------------------- /docs/run-books/images/new-ring-pr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/new-ring-pr.png -------------------------------------------------------------------------------- /docs/run-books/images/pr-promoted.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/pr-promoted.png -------------------------------------------------------------------------------- /docs/images/cicd-build-commit-status.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/cicd-build-commit-status.png -------------------------------------------------------------------------------- /docs/images/cicd-tutorial-config-pr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/cicd-tutorial-config-pr.png -------------------------------------------------------------------------------- /docs/images/cicd-tutorial-greetings.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/cicd-tutorial-greetings.png -------------------------------------------------------------------------------- /docs/run-books/images/pr-deployment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/pr-deployment.png -------------------------------------------------------------------------------- /docs/run-books/images/pr-to-src-code.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/pr-to-src-code.png -------------------------------------------------------------------------------- /docs/run-books/images/update-config.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/update-config.png -------------------------------------------------------------------------------- /docs/images/kalypso-non-containerized.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/kalypso-non-containerized.png -------------------------------------------------------------------------------- /docs/run-books/images/gitops-commit-id.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/gitops-commit-id.png -------------------------------------------------------------------------------- /docs/run-books/images/mayberry-cluster.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/mayberry-cluster.png -------------------------------------------------------------------------------- /docs/run-books/images/new-environment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/new-environment.png -------------------------------------------------------------------------------- /docs/run-books/images/next-environment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/next-environment.png -------------------------------------------------------------------------------- /docs/run-books/images/next_ring.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/next_ring.drawio.png -------------------------------------------------------------------------------- /docs/run-books/images/not-promoted-pr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/not-promoted-pr.png -------------------------------------------------------------------------------- /docs/images/cicd-deployed-commit-status.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/cicd-deployed-commit-status.png -------------------------------------------------------------------------------- /docs/images/cicd-tutorial-post-deployment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/cicd-tutorial-post-deployment.png -------------------------------------------------------------------------------- /docs/images/gitops-CD-Simple-Flow.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/gitops-CD-Simple-Flow.drawio.png -------------------------------------------------------------------------------- /docs/run-books/images/k3scluster-gitops.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/k3scluster-gitops.png -------------------------------------------------------------------------------- /docs/run-books/images/platform-gitops-pr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/platform-gitops-pr.png -------------------------------------------------------------------------------- /docs/run-books/images/pr-deployment-stage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/pr-deployment-stage.png -------------------------------------------------------------------------------- /docs/run-books/images/promote-flow.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/promote-flow.drawio.png -------------------------------------------------------------------------------- /docs/run-books/images/scheduling-policy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/scheduling-policy.png -------------------------------------------------------------------------------- /docs/images/gitops-CD-Detailed-Flow.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/gitops-CD-Detailed-Flow.drawio.png -------------------------------------------------------------------------------- /docs/run-books/images/revert-config-change.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/revert-config-change.png -------------------------------------------------------------------------------- /docs/run-books/images/workload-registration.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/workload-registration.png -------------------------------------------------------------------------------- /docs/images/cicd-promoted-stage-commit-status.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/cicd-promoted-stage-commit-status.png -------------------------------------------------------------------------------- /docs/images/cicd-tutorial-done-commit-status.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/images/cicd-tutorial-done-commit-status.png -------------------------------------------------------------------------------- /docs/run-books/images/config-change-dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/config-change-dashboard.png -------------------------------------------------------------------------------- /docs/run-books/images/empty-environment-branch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/empty-environment-branch.png -------------------------------------------------------------------------------- /docs/run-books/images/environment-definition.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/environment-definition.png -------------------------------------------------------------------------------- /docs/run-books/images/new-branch-ring-created.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/new-branch-ring-created.png -------------------------------------------------------------------------------- /docs/run-books/images/next_environment.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/next_environment.drawio.png -------------------------------------------------------------------------------- /docs/run-books/images/not-promoted-pr-details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/not-promoted-pr-details.png -------------------------------------------------------------------------------- /docs/run-books/images/update-app-config-values.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/update-app-config-values.png -------------------------------------------------------------------------------- /docs/run-books/images/next-new-ring-environment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/next-new-ring-environment.png -------------------------------------------------------------------------------- /docs/run-books/images/deployment-observability-edit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/deployment-observability-edit.png -------------------------------------------------------------------------------- /docs/run-books/images/deployment-observability-hub.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/deployment-observability-hub.png -------------------------------------------------------------------------------- /docs/run-books/images/deployment-observability-save.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/deployment-observability-save.png -------------------------------------------------------------------------------- /scripts/bootstrap/templates/gitops/main/README.md: -------------------------------------------------------------------------------- 1 | # GitOps Repository 2 | 3 | This repository receives deployment manifests generated by Kalypso Scheduler. 4 | -------------------------------------------------------------------------------- /docs/run-books/images/deployment-observability-edit-colors.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/deployment-observability-edit-colors.png -------------------------------------------------------------------------------- /docs/run-books/images/deployment-observability-hw-deployed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/deployment-observability-hw-deployed.png -------------------------------------------------------------------------------- /docs/run-books/images/deployment-observability-hw-deployed-stage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/kalypso/HEAD/docs/run-books/images/deployment-observability-hw-deployed-stage.png -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/dev/gitops-repo.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: scheduler.kalypso.io/v1alpha1 2 | kind: GitOpsRepo 3 | metadata: 4 | name: dev 5 | spec: 6 | repo: ${GITOPS_REPO_URL} 7 | branch: dev 8 | path: . 9 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/dev/base-repo.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: scheduler.kalypso.io/v1alpha1 2 | kind: BaseRepo 3 | metadata: 4 | name: main 5 | spec: 6 | repo: ${CONTROL_PLANE_REPO_URL} 7 | branch: main 8 | path: . 9 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "scheduler"] 2 | path = scheduler 3 | url = https://github.com/microsoft/kalypso-scheduler 4 | [submodule "observability-hub"] 5 | path = observability-hub 6 | url = https://github.com/microsoft/kalypso-observability-hub 7 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/dev/cluster-types/dev.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: scheduler.kalypso.io/v1alpha1 2 | kind: ClusterType 3 | metadata: 4 | name: dev 5 | spec: 6 | reconciler: arc-flux 7 | namespaceService: default 8 | configType: configmap 9 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/dev/configs/dev-config.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: dev-config 5 | labels: 6 | platform-config: "true" 7 | data: 8 | ENVIRONMENT: Dev 9 | SOME_DEV_ENVIRONMENT_VARIABLE: "true" -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "scripts": { 3 | "lint": "markdownlint --config .markdownlint.yaml **/*.md", 4 | "cspell": "cspell docs/**/*.md" 5 | }, 6 | "dependencies": { 7 | "cspell": "^6.12.0", 8 | "markdownlint-cli": "^0.32.2" 9 | } 10 | } 11 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/main/.environments/dev.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: scheduler.kalypso.io/v1alpha1 2 | kind: Environment 3 | metadata: 4 | name: dev 5 | spec: 6 | controlPlane: 7 | repo: ${CONTROL_PLANE_REPO_URL} 8 | branch: dev 9 | path: . 10 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/main/.github/workflows/utils/baserepo-template.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: scheduler.kalypso.io/v1alpha1 2 | kind: BaseRepo 3 | metadata: 4 | name: $branch 5 | spec: 6 | repo: https://github.com/$repo 7 | branch: $branch 8 | path: . 9 | commit: $promoted_commit 10 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/dev/scheduling-policies/dev-policy.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: scheduler.kalypso.io/v1alpha1 2 | kind: SchedulingPolicy 3 | metadata: 4 | name: dev-policy 5 | spec: 6 | deploymentTargetSelector: 7 | labelSelector: 8 | matchLabels: {} 9 | clusterTypeSelector: 10 | labelSelector: {} -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/main/.github/workflows/utils/save-promoted-commit.sh: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License. 3 | 4 | #!/bin/bash 5 | echo $1 6 | 7 | baserepo_template="baserepo-template.yaml" 8 | 9 | set -euo pipefail 10 | cat $baserepo_template | envsubst > $1 11 | 12 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/main/workloads/sample-workload-registration.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: scheduler.kalypso.io/v1alpha1 2 | kind: WorkloadRegistration 3 | metadata: 4 | name: sample-workload 5 | spec: 6 | workload: 7 | repo: https://github.com/example/sample-app-manifests 8 | branch: main 9 | path: ./workload 10 | workspace: sample-workspace 11 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/main/templates/configmap.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: scheduler.kalypso.io/v1alpha1 2 | kind: Template 3 | metadata: 4 | name: configmap 5 | spec: 6 | type: config 7 | manifests: 8 | - | 9 | apiVersion: v1 10 | kind: ConfigMap 11 | metadata: 12 | name: platform-config 13 | namespace: {{ .DeploymentTargetName}} 14 | data: 15 | {{ stringify .ConfigData | indent 2}} 16 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Microsoft Open Source Code of Conduct 2 | 3 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 4 | 5 | Resources: 6 | 7 | - [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) 8 | - [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) 9 | - Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns 10 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/main/.github/workflows/utils/update-status.sh: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License. 3 | 4 | #!/bin/bash 5 | set -eo pipefail # fail on error 6 | 7 | STATUS=$1 8 | DESCRIPTION=$2 9 | CONTEXT=$3 10 | 11 | gh api \ 12 | --method POST \ 13 | -H "Accept: application/vnd.github+json" \ 14 | /repos/$GITHUB_REPOSITORY/statuses/$PROMOTED_COMMIT_ID \ 15 | -f state=$STATUS \ 16 | -f target_url='' \ 17 | -f description="$DESCRIPTION" \ 18 | -f context="$CONTEXT" 19 | -------------------------------------------------------------------------------- /SUPPORT.md: -------------------------------------------------------------------------------- 1 | # Support 2 | 3 | ## How to file issues and get help 4 | 5 | This project uses GitHub Issues to track bugs and feature requests. Please search the existing 6 | issues before filing new issues to avoid duplicates. For new issues, file your bug or 7 | feature request as a new Issue. 8 | 9 | For help and questions about using this project, please add a new [GitHub Discussion](https://github.com/microsoft/kalypso/discussions). 10 | 11 | ## Microsoft Support Policy 12 | 13 | Support for this project is limited to the resources listed above. 14 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/main/.github/workflows/ci.yaml: -------------------------------------------------------------------------------- 1 | name: ci 2 | 3 | on: 4 | push: 5 | branches: 6 | - main 7 | 8 | permissions: write-all 9 | 10 | jobs: 11 | ci: 12 | runs-on: ubuntu-latest 13 | steps: 14 | - name: Checkout Source 15 | uses: actions/checkout@v2.3.4 16 | - name: Do Some Checks 17 | run: | 18 | echo "Do Some Checks" 19 | - name: Start CD 20 | run: | 21 | gh workflow run cd.yaml -f environment=dev -f promoted_commit_id=$GITHUB_SHA 22 | env: 23 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} -------------------------------------------------------------------------------- /.github/workflows/templates/utils/generate-deployment-descriptor.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Usage: 4 | # generate-deployment-descriptor.sh DEPLOYMENT_TARGET DEPLOYMENT_TARGET_MANIFESTS_PATH APPLICATION OUTPUT_FILE TEMPLATE_FILE 5 | 6 | export DEPLOYMENT_TARGET=$1 7 | export DEPLOYMENT_TARGET_MANIFESTS_PATH=$2 8 | export APPLICATION=$3 9 | OUTPUT_FILE=$4 10 | TEMPLATE_FILE=$5 11 | 12 | echo $DEPLOYMENT_TARGET 13 | echo $DEPLOYMENT_TARGET_MANIFESTS_PATH 14 | echo $APPLICATION 15 | echo $OUTPUT_FILE 16 | echo $TEMPLATE_FILE 17 | 18 | 19 | set -eo pipefail 20 | 21 | envsubst <"$TEMPLATE_FILE" > "$OUTPUT_FILE" 22 | 23 | 24 | 25 | -------------------------------------------------------------------------------- /.github/workflows/templates/utils/update-status.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Updates git commit status for the commit in $GITHUB_REPOSITORY for the commit in $PROMOTED_COMMIT_ID 4 | 5 | # Usage: 6 | # update-status.sh STATUS DESCRIPTION CONTEXT 7 | # Example: 8 | # update-status.sh "success" "Tested" "d2" 9 | 10 | 11 | set -eo pipefail # fail on error 12 | 13 | STATUS=$1 14 | DESCRIPTION=$2 15 | CONTEXT=$3 16 | 17 | gh api \ 18 | --method POST \ 19 | -H "Accept: application/vnd.github+json" \ 20 | /repos/$GITHUB_REPOSITORY/statuses/$PROMOTED_COMMIT_ID \ 21 | -f state=$STATUS \ 22 | -f target_url='' \ 23 | -f description="$DESCRIPTION" \ 24 | -f context="$CONTEXT" 25 | -------------------------------------------------------------------------------- /scripts/bootstrap/.shellcheckrc: -------------------------------------------------------------------------------- 1 | # ShellCheck configuration for Kalypso bootstrap scripts 2 | # See: https://www.shellcheck.net/wiki/ 3 | 4 | # Disable specific checks that are not relevant for this project 5 | 6 | # SC1090: Can't follow non-constant source 7 | # We use dynamic sourcing for library files 8 | disable=SC1090 9 | 10 | # SC1091: Not following sourced files 11 | # Library files are checked separately 12 | disable=SC1091 13 | 14 | # SC2034: Variable appears unused 15 | # Some variables are exported for use in other scripts 16 | disable=SC2034 17 | 18 | # SC2153: Possible misspelling 19 | # Variables are set by other modules (config.sh sets cluster variables) 20 | disable=SC2153 21 | -------------------------------------------------------------------------------- /docs/use-cases/platform-team-provides-configuration-value.md: -------------------------------------------------------------------------------- 1 | # Platform team provides a configuration value for a service deployment target 2 | 3 | *Preconditions*: 4 | 5 | - Platform Service Config repo exists 6 | 7 | *Postconditions*: 8 | 9 | - Configuration value for the deployment target is set in the Platform Service Config repo 10 | 11 | Platform Service Config repo contains configuration branches with configuration values for each environment. Each configuration branch contain folders representing deployment targets. Platform Team creates a file in a corresponding folder of the configuration branch with the config values. These values are used by the service CI/CD pipeline to generate manifests for this deployment target. 12 | -------------------------------------------------------------------------------- /docs/use-cases/platform-team-onboards-workload.md: -------------------------------------------------------------------------------- 1 | # Platform Team Onboards a workload 2 | 3 | *Preconditions*: 4 | 5 | - Control Plane and Platform GitOps repositories exist 6 | - Workload GitOps repository exists 7 | 8 | *Postconditions*: 9 | 10 | - Information on the new workload is stored in the control plane 11 | - Promotion flow has started promoting the workload across environments 12 | 13 | Platform Team submits a PR to the control plane *main* branch with the information on the workload. Once the PR is reviewed and merged, the promotional flow starts promoting the application across environment branches in the GitOps repo. 14 | 15 | > Refer to the runbook [Platform Team Schedules Applications for Deployment](../run-books/platform-team-schedules-applications-for-deployment.md). 16 | -------------------------------------------------------------------------------- /docs/use-cases/application-team-provides-configuration-value.md: -------------------------------------------------------------------------------- 1 | # Application team provides a configuration value for a deployment target 2 | 3 | *Preconditions*: 4 | 5 | - Application Config repo exists 6 | 7 | *Postconditions*: 8 | 9 | - Configuration value for the deployment target is set in the Application Config repo 10 | 11 | Application Config repo contains configuration branches with configuration values for each environment. Each configuration branch contain folders representing deployment targets. Application Team creates a file in a corresponding folder of the configuration branch with the config values. These values are used by the application CI/CD pipeline to generate manifests for this deployment target. 12 | 13 | > Refer to the runbook [Application Team Manages Application Configuration](../run-books/application-team-manages-application-configuration.md). 14 | -------------------------------------------------------------------------------- /cspell.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "./node_modules/@cspell/cspell-types/cspell.schema.json", 3 | "version": "0.2", 4 | "language": "en-US", 5 | "flagWords": [ 6 | "hte" 7 | ], 8 | "ignorePaths": [ 9 | ".git", 10 | ".gitignore", 11 | ".vscodeignore", 12 | ".gitattributes", 13 | "*.lock", 14 | "*.log", 15 | "cspell.json", 16 | "package-lock.json" 17 | ], 18 | "ignoreWords": [], 19 | "dictionaryDefinitions": [ 20 | { 21 | "name": "cse", 22 | "path": "cspell-cse.txt", 23 | "addWords": false 24 | } 25 | ], 26 | "dictionaries": [ 27 | "cse" 28 | ], 29 | "languageSettings": [ 30 | { 31 | "languageId": "*", 32 | "allowCompoundWords": false 33 | } 34 | ], 35 | "useGitignore": true 36 | } -------------------------------------------------------------------------------- /docs/use-cases/platform-team-defines-deployment-targets.md: -------------------------------------------------------------------------------- 1 | # Platform team defines service deployment targets 2 | 3 | *Preconditions*: 4 | 5 | - Platform Service Source repo exists 6 | 7 | *Postconditions*: 8 | 9 | - Service deployment targets are defined in the Platform Service Source repo 10 | 11 | Platform Team creates a file in their service source repository containing a list of deployment targets where the service might be deployed in the fleet. Each deployment target references a rollout environment (e.g. *dev*, *stage*) and a place where the manifests for this deployment target are stored. It can be plain K8s manifests, Helm charts, or any other deployment descriptors, stored in a repo or OCI storage or any other storage that a reconciler on the clusters can understand. Each deployment target is marked with a set of custom labels to automatically schedule the deployment target on the clusters in the fleet. 12 | -------------------------------------------------------------------------------- /docs/use-cases/application-team-defines-deployment-targets.md: -------------------------------------------------------------------------------- 1 | # Application team defines application deployment targets 2 | 3 | *Preconditions*: 4 | 5 | - Application Source repo exists 6 | 7 | *Postconditions*: 8 | 9 | - Application deployment targets are defined in the Application Source repo 10 | 11 | Application Team creates a file in their source repository containing a list of deployment targets where the application might be deployed during its lifecycle. Each deployment target references a rollout environment (e.g. *dev*, *stage*) and a place where the manifests for this deployment target are stored. It can be plain K8s manifests, Helm charts, or any other deployment descriptors, stored in a repo or OCI storage or any other storage that a reconciler on the clusters can understand. Each deployment target is marked with a set of custom labels that the Platform Team uses to schedule the deployment target on the clusters in the fleet. 12 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/main/templates/namespace.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: scheduler.kalypso.io/v1alpha1 2 | kind: Template 3 | metadata: 4 | labels: 5 | app.kubernetes.io/name: default 6 | name: default 7 | spec: 8 | type: namespace 9 | manifests: 10 | - | 11 | apiVersion: v1 12 | kind: Namespace 13 | metadata: 14 | name: "{{ .DeploymentTargetName}}" 15 | labels: 16 | environment: "{{ .Environment}}" 17 | workspace: "{{ .Workspace}}" 18 | workload: "{{ .Workload}}" 19 | deploymentTarget: "{{ .DeploymentTargetName}}" 20 | - | 21 | apiVersion: v1 22 | kind: ServiceAccount 23 | metadata: 24 | name: "{{ .Workload}}-sa" 25 | namespace: "{{ .DeploymentTargetName}}" 26 | {{if .ConfigData.imagePullSecrets }} 27 | imagePullSecrets: 28 | - name: {{ .ConfigData.imagePullSecrets }} 29 | {{end}} 30 | 31 | -------------------------------------------------------------------------------- /.github/workflows/templates/utils/get-tracking-info.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Usage: 4 | # get-tracking-info.sh 5 | 6 | 7 | # Reads the tracking information (PROMOTED_COMMIT_ID, IMAGE_NAME) from the .github/tracking folder and 8 | # "promoted" label from the GitOps PR 9 | # Sets the environment variables for the GitHub Actions workflow to use. 10 | 11 | PROMOTED_COMMIT_ID=$(cat .github/tracking/Promoted_Commit_Id) 12 | echo "PROMOTED_COMMIT_ID=$PROMOTED_COMMIT_ID" >> $GITHUB_ENV 13 | 14 | IMAGE_TAG=$(cat .github/tracking/Image_tag) 15 | echo "IMAGE_TAG=$IMAGE_TAG" >> $GITHUB_ENV 16 | VERSION=$(cat .github/tracking/Version) 17 | echo "VERSION=$VERSION" >> $GITHUB_ENV 18 | 19 | gh pr list --search $COMMIT_ID --state merged --label promoted 20 | 21 | promoted=$(gh pr list --search $COMMIT_ID --state merged --label promoted) 22 | if [[ -z "$promoted" ]]; then 23 | PROMOTION='n' 24 | echo "PROMOTION=$PROMOTION" >> $GITHUB_ENV 25 | fi 26 | 27 | echo $PROMOTION -------------------------------------------------------------------------------- /docs/use-cases/platform-team-defines-cluster-type.md: -------------------------------------------------------------------------------- 1 | # Platform team defines a cluster type 2 | 3 | *Preconditions*: 4 | 5 | - Control Plane and Platform GitOps repositories exist and have environment branches 6 | 7 | *Postconditions*: 8 | 9 | - Information on the new cluster type is stored in the control plane environment branches 10 | - Platform GitOps repository environment branches contain folders representing the cluster type with the subfolders representing deployment targets scheduled on the cluster type 11 | 12 | Platform Team submits a PR to the control plane environment branch (e.g. *dev* or *stage*) with the information on the cluster type. Once the PR is reviewed and merged, the scheduler analyzes what deployment targets should be assigned to this cluster type. It generates assignment manifests and creates a PR to the environment branch of the Platform GitOps repository. 13 | 14 | > Refer to the runbook [Platform Team Onboards a New Cluster](../run-books/platform-team-onboards-a-new-cluster.md). 15 | -------------------------------------------------------------------------------- /cspell-cse.txt: -------------------------------------------------------------------------------- 1 | CWEP 2 | TPM 3 | TPMs 4 | CSE's 5 | CSE 6 | Shewchuk 7 | Kiran 8 | Naidu 9 | Finden 10 | Abbosh 11 | Rubbelke 12 | Shirey 13 | Hopcroft 14 | Mansel 15 | Komandur 16 | Luttrell 17 | Zerega 18 | Pantoja 19 | Schnieder 20 | DAST 21 | OWASP 22 | SAST 23 | SIEM 24 | SDLC 25 | Upskilling 26 | Finster 27 | preconfigured 28 | toolchains 29 | Crory 30 | OLAP 31 | Pseudonymization 32 | Anonymization 33 | MLOPs 34 | TDSP 35 | Faiza 36 | OLTP 37 | Ghazanfar 38 | automations 39 | Hyperautomation 40 | BPMS 41 | AzDO 42 | RSAT 43 | TFVC 44 | Devs 45 | exfiltration 46 | chatbots 47 | chatbot 48 | Dataverse 49 | Customizer 50 | VMSS 51 | ORAS 52 | Exploitability 53 | runtimes 54 | CISA 55 | ADLS 56 | sociotechnical 57 | Explainability 58 | Hyperparameters 59 | interpretability 60 | OARP 61 | operationalization 62 | operationalized 63 | socio 64 | unidimensional 65 | Otel 66 | OTLP 67 | preprocess 68 | gnzg 69 | westus 70 | Kusto 71 | Kalypso 72 | Fluentbit 73 | Kubefed 74 | Karmada 75 | CAPI 76 | Crossplane 77 | Istio 78 | onboards 79 | onboarded 80 | commitid 81 | lifecycles 82 | Kustomization -------------------------------------------------------------------------------- /docs/use-cases/platform-team-scheduled-application.md: -------------------------------------------------------------------------------- 1 | # Platform team schedules an application on cluster types 2 | 3 | *Preconditions*: 4 | 5 | - Control Plane and Platform GitOps repositories exist and have environment branches 6 | 7 | *Postconditions*: 8 | 9 | - Scheduling policy is configured in the control plane 10 | - Application deployment targets are assigned to the cluster types in the Platform GitOps repo 11 | 12 | Platform Team submits a PR to the control plane environment branch (e.g. *dev* or *stage*) with a scheduling policy, that defines rules of how deployment targets should be scheduled to cluster types. The rules might be trivial, based on label matching, or they can include score-based configurations defined by OCM Placement API. Once the PR is reviewed and merged, the scheduler analyzes what deployment targets should be assigned to what cluster types in the environment. It generates assignment manifests and creates a PR to the environment branch of the Platform GitOps repository. 13 | 14 | > Refer to the runbook [Platform Team Schedules Applications for Deployment](../run-books/platform-team-schedules-applications-for-deployment.md). 15 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/main/templates/argocd.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: scheduler.kalypso.io/v1alpha1 2 | kind: Template 3 | metadata: 4 | labels: 5 | someLabel: "true" 6 | name: argocd 7 | spec: 8 | type: reconciler 9 | manifests: 10 | - | 11 | apiVersion: argoproj.io/v1alpha1 12 | kind: Application 13 | metadata: 14 | name: "{{ .Manifests.branch}}.{{ .Workload}}.{{ .DeploymentTargetName}}" 15 | namespace: argocd 16 | spec: 17 | destination: 18 | server: https://kubernetes.default.svc 19 | namespace: "{{ .DeploymentTargetName}}" 20 | project: default 21 | source: 22 | path: "{{ .Manifests.path}}" 23 | repoURL: "{{ .Manifests.repo}}" 24 | targetRevision: "{{ .Manifests.branch}}" 25 | directory: 26 | recurse: true 27 | include: '*.yaml' 28 | exclude: 'kustomization.yaml' 29 | syncPolicy: 30 | automated: 31 | prune: true 32 | selfHeal: true 33 | allowEmpty: false 34 | syncOptions: 35 | - CreateNamespace=true 36 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Microsoft Corporation. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE 22 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/gitops/dev/.github/workflows/check-promote.yaml: -------------------------------------------------------------------------------- 1 | name: checkpromote 2 | 3 | on: 4 | push: 5 | branches: 6 | - '*' 7 | - '!main' 8 | paths-ignore: 9 | - '.github/**' 10 | 11 | jobs: 12 | Check_deployment_in_control_plane: 13 | runs-on: ubuntu-latest 14 | steps: 15 | - name: Checkout Gitops 16 | uses: actions/checkout@v3 17 | - name: Get Promoted Label 18 | run: | 19 | promoted=$(gh pr list --search $COMMIT_ID --state merged --label promoted) 20 | if [[ ! -z "$promoted" ]]; then 21 | PROMOTED_COMMIT_ID=$(cat .github/tracking/Promoted_Commit_Id) 22 | echo "PROMOTED_COMMIT_ID=$PROMOTED_COMMIT_ID" >> $GITHUB_ENV 23 | echo $PROMOTED_COMMIT_ID 24 | fi 25 | env: 26 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 27 | COMMIT_ID: ${{ github.sha }} 28 | - name: Check and Promote 29 | run: | 30 | gh workflow run check-promote.yaml -f environment=$GITHUB_REF_NAME -f promoted_commit_id=$PROMOTED_COMMIT_ID -f commit_id=$GITHUB_SHA -r main -R ${CONTROL_PLANE_REPO_URL} 31 | env: 32 | GH_TOKEN: ${{ secrets.CONTROL_PLANE_TOKEN }} 33 | -------------------------------------------------------------------------------- /docs/use-cases/platform-team-config-values.md: -------------------------------------------------------------------------------- 1 | # Platform team provides configuration values for a cluster type 2 | 3 | *Preconditions*: 4 | 5 | - Control Plane and Platform GitOps repositories exist and have environment branches 6 | 7 | *Postconditions*: 8 | 9 | - Configuration values are stored in the control plane 10 | - Deployment target subfolders in the Platform GitOps repository contain consolidated config maps with all platform config values available on the cluster type 11 | 12 | Platform Team submits a PR to the control plane environment branch (e.g. *dev* or *stage*) with the a config map containing platform config values. The config map is marked with a set of custom labels. Once the PR is reviewed and merged, the scheduler scans all config maps in the environment and collects values for each cluster type basing on the label matching. Then, it creates a PR to the Platform GitOps repository with a consolidated config map in every deployment target folder. The config map contains all platform configuration values, that the workload can use on this cluster type in this environment. 13 | 14 | > Refer to the runbook [Platform Team Manages Platform Configuration](../run-books/platform-team-manages-platform-configuration.md). 15 | -------------------------------------------------------------------------------- /.github/workflows/templates/notify-on-pr.yml: -------------------------------------------------------------------------------- 1 | # Notifies orchestrator (source repo workflows) that GitOps PR has been merged 2 | # So that the orchestrator can proceed with the next step in the pipeline 3 | 4 | name: notify-on-pr 5 | 6 | on: 7 | push: 8 | branches: 9 | - '*' 10 | - '!main' 11 | paths-ignore: 12 | - '.github/**' 13 | 14 | jobs: 15 | notify-on-pr: 16 | runs-on: ubuntu-latest 17 | permissions: write-all 18 | steps: 19 | - name: Checkout Gitops 20 | uses: actions/checkout@v3 21 | with: 22 | ref: ${{ github.event.commit.sha }} 23 | 24 | - name: Get Tracking Info 25 | run: | 26 | .github/workflows/utils/get-tracking-info.sh 27 | env: 28 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 29 | COMMIT_ID: ${{ github.sha }} 30 | 31 | - name: Notify Orchestrator 32 | run: | 33 | gh workflow run post-deployment.yml -f environment="$GITHUB_REF_NAME" -f promoted_commit_id="$PROMOTED_COMMIT_ID" -f commit_id="$GITHUB_SHA" -f promotion="$PROMOTION" -f image_tag="$IMAGE_TAG" -f version="$VERSION" -r main -R "$SRC_REPO" 34 | env: 35 | GH_TOKEN: ${{ secrets.CD_BOOTSTRAP_TOKEN }} 36 | SRC_REPO: ${{ vars.SRC_REPO }} 37 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/main/templates/arc-flux.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: scheduler.kalypso.io/v1alpha1 2 | kind: Template 3 | metadata: 4 | labels: 5 | someLabel: "false" 6 | name: arc-flux 7 | spec: 8 | type: reconciler 9 | manifests: 10 | - | 11 | apiVersion: source.toolkit.fluxcd.io/v1beta2 12 | kind: GitRepository 13 | metadata: 14 | name: {{ .Manifests.branch}}.{{ .Workload}}.{{ .DeploymentTargetName}} 15 | namespace: flux-system 16 | spec: 17 | interval: 30s 18 | url: "{{ .Manifests.repo}}" 19 | ref: 20 | branch: "{{ .Manifests.branch}}" 21 | secretRef: 22 | name: platform-{{ .Manifests.branch}}-auth 23 | - | 24 | apiVersion: kustomize.toolkit.fluxcd.io/v1beta2 25 | kind: Kustomization 26 | metadata: 27 | name: {{ .Manifests.branch}}.{{ .Workload}}.{{ .DeploymentTargetName}} 28 | namespace: flux-system 29 | spec: 30 | interval: 30s 31 | targetNamespace: "{{ .DeploymentTargetName}}" 32 | sourceRef: 33 | kind: GitRepository 34 | name: {{ .Manifests.branch}}.{{ .Workload}}.{{ .DeploymentTargetName}} 35 | path: "{{ .Manifests.path}}" 36 | prune: true 37 | -------------------------------------------------------------------------------- /docs/use-cases/application-team-updates-application.md: -------------------------------------------------------------------------------- 1 | # Application team updates the application 2 | 3 | *Preconditions*: 4 | 5 | - Application Source repo exists 6 | - Application ConfigSource repo exists 7 | - Application Manifest Storage exists 8 | 9 | *Postconditions*: 10 | 11 | - Application is rebuilt 12 | - Manifests for the deployment targets are generated and put in the corresponding storage places 13 | 14 | Application Team submits a PR with the source code change to the *main* branch in the Application Source repository. Once the PR is merged to *main*, it starts a CI/CD workflow. The workflow performs standard code quality and security checks, builds application Docker images and pushes them to the container registry. Then, it generates manifests for the deployment targets defined for the first environment in the rollout chain (e.g. *dev*) and places them to the Manifest Storage. Once the Application Team is satisfied with the deployment to the first environment, they promote the change to the next environment, so the CD workflow goes ahead and generates manifests for the deployment targets in that environment. 15 | 16 | > Refer to the runbook [Application Team Promotes a Change Through Environments](../run-books/application-team-promotes-a-change-through-environments.md). 17 | -------------------------------------------------------------------------------- /docs/use-cases/platform-team-adds-platform-service.md: -------------------------------------------------------------------------------- 1 | # Platform team adds a platform service 2 | 3 | *Preconditions*: 4 | 5 | - Platform Service Source repo exists 6 | - Platform Service Manifest Storage exists 7 | 8 | *Postconditions*: 9 | 10 | - Platform Service Manifests and images are pulled and checked 11 | - Manifests for the deployment targets are generated and put in the corresponding storage places 12 | 13 | Platform team submits a PR to the *main* branch in the Service Source repository with a file, that contains a reference to the external manifests storage. For example, a reference to a Grafana Helm chart in . Once the PR is merged to *main*, it starts a CI/CD workflow. The workflow pulls the manifests from the external storage, pulls service images and performs standard security scanning. Then, it generates manifests for the deployment targets defined for the first environment in the rollout chain (e.g. *dev*) and places them to the Manifest Storage, such as an OCI storage or a local Helm repo, or just a Git repo. The manifests are generated with the config values provided for the deployment target in the config branch of the Service Source repository. Once the Platform Team is satisfied with the deployment to the first environment, they promote the change to the next environment, so the CD workflow goes ahead and generates manifests for the deployment targets in that environment. 14 | -------------------------------------------------------------------------------- /docs/run-books/README.md: -------------------------------------------------------------------------------- 1 | # Run Books 2 | 3 | This collection of run books provides general guidelines for executing common use cases against the Kalypso control plane and application repositories. 4 | 5 | ## Run Books 6 | 7 | | Team | Run Book | 8 | | ----------- | ----------------------------------------------------------------------------------------------------- | 9 | | Platform | [Onboard a New Cluster](./platform-team-onboards-a-new-cluster.md) | 10 | | Platform | [Create a New Environment](./platform-team-creates-a-new-environment.md) | 11 | | Platform | [Schedule Applications for Deployment](./platform-team-schedules-applications-for-deployment.md) | 12 | | Platform | [Manage Platform Configuration](./platform-team-manages-platform-configuration.md) | 13 | | Application | [Onboard a New Application](./application-team-onboards-a-new-application.md) | 14 | | Application | [Create a New Application Ring](./application-team-creates-a-new-application-ring.md) | 15 | | Application | [Promote a Change Through Environments](./application-team-promotes-a-change-through-environments.md) | 16 | | Application | [Manage Application Configuration](./application-team-manages-application-configuration.md) | 17 | -------------------------------------------------------------------------------- /.github/workflows/templates/notify-on-config-change.yml: -------------------------------------------------------------------------------- 1 | name: notify-on-config-change 2 | 3 | # When thh configuration has changed, the workflow notifies the orchestrator (soource repo workflows) 4 | # to start the CD workflow, so the manifests are regenerated and PRed to the target environment. 5 | 6 | on: 7 | push: 8 | branches: 9 | - 'dev*' 10 | - 'qa*' 11 | - 'test*' 12 | - 'stage*' 13 | - 'prod*' 14 | - '!main' 15 | 16 | jobs: 17 | notify-on-config-change: 18 | runs-on: ubuntu-latest 19 | permissions: write-all 20 | steps: 21 | - name: Checkout GitOps 22 | uses: actions/checkout@v2.3.4 23 | with: 24 | repository: ${{ vars.MANIFESTS_REPO }} 25 | token: ${{ secrets.CD_BOOTSTRAP_TOKEN }} 26 | ref: ${{ github.ref_name }} 27 | path: gitops 28 | 29 | - name: Get Tracking Info 30 | run: | 31 | .github/workflows/utils/get-tracking-info.sh 32 | working-directory: gitops 33 | env: 34 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 35 | COMMIT_ID: ${{ github.sha }} 36 | 37 | - name: Start CD 38 | run: | 39 | gh workflow run deploy.yml -f environment="$ACTIVE_ENVIRONMENT" -f commit_id="$PROMOTED_COMMIT_ID" -f promotion=n -f image_tag="$IMAGE_TAG" -f version="$VERSION" -r main --repo "$SRC_REPO" 40 | env: 41 | GH_TOKEN: ${{ secrets.CD_BOOTSTRAP_TOKEN }} 42 | ACTIVE_ENVIRONMENT: ${{ github.ref_name }} 43 | SRC_REPO: ${{ vars.SRC_REPO }} 44 | -------------------------------------------------------------------------------- /.github/workflows/templates/ci.yml: -------------------------------------------------------------------------------- 1 | name: ci 2 | 3 | on: 4 | push: 5 | branches: ["main"] 6 | workflow_dispatch: 7 | 8 | env: 9 | SEM_VER: 0.0.1 10 | IMAGE_NAME: ghcr.io/${{ github.repository }}/${{ github.event.repository.name }} 11 | 12 | jobs: 13 | Build_Push_Image: 14 | runs-on: ubuntu-latest 15 | permissions: write-all 16 | 17 | steps: 18 | - name: Checkout Source 19 | uses: actions/checkout@v3 20 | - name: Login to ghcr 21 | uses: docker/login-action@v1 22 | with: 23 | registry: ghcr.io 24 | username: ${{ github.actor }} 25 | password: ${{ secrets.GITHUB_TOKEN }} 26 | - name: Generate Image Tag 27 | id: generate_image_tag 28 | run: | 29 | IMAGE_TAG=${{ env.SEM_VER }}-${{ github.run_number }} 30 | echo "IMAGE_TAG=$IMAGE_TAG" >> $GITHUB_ENV 31 | 32 | VERSION=${{ env.SEM_VER }}-${{ github.run_number }} 33 | echo "VERSION=$VERSION" >> $GITHUB_ENV 34 | - name: Build and Push to ghcr 35 | uses: docker/build-push-action@v2 36 | with: 37 | push: true 38 | context: . 39 | tags: ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}, ${{ env.IMAGE_NAME }}:latest 40 | 41 | - name: Start CD 42 | run: | 43 | gh workflow run deploy.yml -f environment="$START_ENVIRONMENT" -f commit_id="$PROMOTED_COMMIT_ID" -f image_tag="$IMAGE_TAG" -f version="$VERSION" 44 | env: 45 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 46 | PROMOTED_COMMIT_ID: ${{ github.sha }} 47 | START_ENVIRONMENT: ${{ vars.START_ENVIRONMENT }} 48 | 49 | -------------------------------------------------------------------------------- /.markdownlinkcheck.json: -------------------------------------------------------------------------------- 1 | { 2 | "ignorePatterns": [ 3 | { 4 | "pattern": "^https://csefy19.visualstudio.com" 5 | }, 6 | { 7 | "pattern": "^https://dev.azure.com/csefy19" 8 | }, 9 | { 10 | "pattern": "^https://dev.azure.com/CSECodeHub" 11 | }, 12 | { 13 | "pattern": "^https://github.com/cse-labs" 14 | }, 15 | { 16 | "pattern": "^https://github.com/orgs/cse-labs" 17 | }, 18 | { 19 | "pattern": "^https://opensource.org/licenses" 20 | }, 21 | { 22 | "pattern": "^https://cse-labs.github.io" 23 | }, 24 | { 25 | "pattern": "^https://aka.ms/SolutionOps" 26 | }, 27 | { 28 | "pattern": "^https://aka.ms/solutionops/dataops-playbook" 29 | }, 30 | { 31 | "pattern": "^https://aka.ms/fusionops" 32 | }, 33 | { 34 | "pattern": "^https://aka.ms/fusionopsrepo" 35 | }, 36 | { 37 | "pattern": "^https://aka.ms/fusionopsbacklog" 38 | }, 39 | { 40 | "pattern": "^https://aka.ms/opensource" 41 | }, 42 | { 43 | "pattern": "^https://github.com/microsoft" 44 | }, 45 | { 46 | "pattern": "^https://github.com/Azure" 47 | }, 48 | { 49 | "pattern": "^https://linkedin.github.io" 50 | }, 51 | { 52 | "pattern": "^https://aka.ms/solutionops" 53 | }, 54 | { 55 | "pattern": "^http://localhost" 56 | } 57 | ], 58 | "httpHeaders": [ 59 | { 60 | "urls": [ 61 | "https://docs.github.com" 62 | ], 63 | "headers": { 64 | "Accept-Encoding": "deflate, compress, gzip, br" 65 | } 66 | } 67 | ], 68 | "timeout": "20s", 69 | "retryOn429": true, 70 | "retryCount": 3, 71 | "fallbackRetryDelay": "30s", 72 | "aliveStatusCodes": [ 73 | 200, 74 | 206 75 | ] 76 | } -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/main/.github/workflows/cd.yaml: -------------------------------------------------------------------------------- 1 | name: cd 2 | 3 | on: 4 | workflow_dispatch: 5 | inputs: 6 | promoted_commit_id: 7 | description: 'Promoted Commit Id from Main' 8 | required: true 9 | type: string 10 | environment: 11 | required: true 12 | type: string 13 | 14 | permissions: write-all 15 | 16 | jobs: 17 | promote_to_environment: 18 | runs-on: ubuntu-latest 19 | steps: 20 | - name: Checkout Source 21 | uses: actions/checkout@v2.3.4 22 | 23 | - name: Save Promoted Commit Id 24 | run: | 25 | # echo "$COMMIT_ID" > Promoted_Commit_Id 26 | ./save-promoted-commit.sh base-repo.yaml 27 | cat base-repo.yaml 28 | env: 29 | promoted_commit: ${{ inputs.promoted_commit_id }} 30 | repo: ${{ github.repository }} 31 | branch: ${{ github.ref_name }} 32 | working-directory: .github/workflows/utils 33 | 34 | - name: Push Promoted Commit Id to Environment branch 35 | uses: dmnemec/copy_file_to_another_repo_action@main 36 | env: 37 | API_TOKEN_GITHUB: ${{ secrets.GITOPS_REPO_TOKEN }} 38 | with: 39 | source_file: '.github/workflows/utils/base-repo.yaml' 40 | destination_repo: ${{ github.repository }} 41 | destination_folder: '.' 42 | destination_branch: ${{ inputs.environment }} 43 | user_email: 'agent@gitops.com' 44 | user_name: 'Git Ops' 45 | commit_message: 'Promoting Commit_Id to the environment' 46 | 47 | - name: Update Commit Status 48 | run: | 49 | .github/workflows/utils/update-status.sh "pending" "Promoted" "$ENVIRONMENT environment" 50 | env: 51 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 52 | PROMOTED_COMMIT_ID: ${{ inputs.promoted_commit_id }} 53 | ENVIRONMENT: ${{ inputs.environment }} 54 | -------------------------------------------------------------------------------- /docs/run-books/application-team-onboards-a-new-application.md: -------------------------------------------------------------------------------- 1 | # Application Team Onboards a New Application 2 | 3 | - [Application Team Onboards a New Application](#application-team-onboards-a-new-application) 4 | - [Overview](#overview) 5 | - [Prerequisites](#prerequisites) 6 | - [1. Identify Git Repository](#1-identify-git-repository) 7 | - [Steps](#steps) 8 | - [1. Setup a GitOps CI/CD flow](#1-setup-a-gitops-cicd-flow) 9 | - [Next Steps](#next-steps) 10 | 11 | ## Overview 12 | 13 | This run book describes how to onboard a new workload to an existing Kalypso system. This workload will include 3 repositories containing source code, configuration, and gitops manifests respectively and will follow the GitOps CI/CD flow using GitHub as described in [GitOps CI/CD with GitHub](https://github.com/microsoft/kalypso/blob/main/cicd/readme.md). 14 | 15 | ## Prerequisites 16 | 17 | ### 1. Identify Git Repository 18 | 19 | A workload consists of 3 git repositories: 20 | 21 | - a **source** repository for application source code and deployment templates 22 | - a **config** repository for application configurations in branches organized by rings and environments 23 | - a **gitops** repository for application manifests that are deployed into clusters 24 | 25 | Create or identify the **source** repository for your workload and make sure you have admin access as this run book will require managing the GitHub environments and other configurations. 26 | 27 | The **config** and **gitops** repositories will be automatically created using scripts in this run book. 28 | 29 | ## Steps 30 | 31 | ### 1. Setup a GitOps CI/CD flow 32 | 33 | The guide for setting up a new application is documented in [setup.md](../../cicd/setup.md). Follow those instructions for your workload using the source repository identified in [prerequisite 1](#1-identify-git-repository). 34 | 35 | ## Next Steps 36 | 37 | To add more rings to the application deployment, see [Application Team Creates a New Application Ring](./application-team-creates-a-new-application-ring.md). 38 | 39 | To provide configuration for different rings and environments to your application, see [Application Team Manages Application Configuration](./application-team-manages-application-configuration.md) and [Platform Team Manages Platform Configuration](./platform-team-manages-platform-configuration.md). 40 | 41 | To deploy the application through environments and rings, see [Application Team Promotes a Change Through Environments](./application-team-promotes-a-change-through-environments.md). 42 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/config.yaml: -------------------------------------------------------------------------------- 1 | # Kalypso Bootstrap Configuration Template 2 | # 3 | # This file provides a template for configuring the Kalypso bootstrap script. 4 | # Copy this file and customize it for your environment. 5 | # 6 | # Usage: ./bootstrap.sh --config 7 | 8 | # Cluster Configuration 9 | cluster: 10 | # Whether to create a new cluster or use an existing one 11 | # Set to true to create a new cluster, false to use existing 12 | # If omitted, defaults to false (use existing) 13 | create: true 14 | 15 | # Name of the AKS cluster 16 | # Required for both creating new clusters and using existing ones 17 | name: kalypso-cluster 18 | 19 | # Azure resource group name 20 | # Required - will be created if it doesn't exist when creating a new cluster 21 | resourceGroup: kalypso-rg 22 | 23 | # Azure region/location (required only for new clusters) 24 | # Examples: eastus, westus2, northeurope, southeastasia 25 | location: eastus 26 | 27 | # Number of nodes in the cluster (for new clusters only) 28 | # Recommended: 3-5 for production, 1 for development 29 | # Default: 1 30 | nodeCount: 1 31 | 32 | # Azure VM size for cluster nodes (for new clusters only) 33 | # Examples: 34 | # - Standard_B2s (2 vCPU, 4 GiB) - Burstable, good for dev 35 | # - Standard_DS2_v2 (2 vCPU, 7 GiB) - General purpose (default) 36 | # - Standard_DS3_v2 (4 vCPU, 14 GiB) - More resources 37 | # - Standard_D4s_v3 (4 vCPU, 16 GiB) - Latest generation 38 | nodeSize: Standard_DS2_v2 39 | 40 | # Repository Configuration 41 | repositories: 42 | # Whether to create new repositories or use existing ones 43 | # Set to true to create new repositories, false to use existing 44 | # If omitted, defaults to false (use existing) 45 | create: true 46 | 47 | # Control plane repository 48 | # When create=true: provide repository name (e.g., "my-control-plane") 49 | # When create=false: provide full URL (e.g., "https://github.com/myorg/kalypso-control-plane") 50 | # Leave empty to use default name "kalypso-control-plane" when creating 51 | controlPlane: "my-control-plane" 52 | 53 | # GitOps repository 54 | # When create=true: provide repository name (e.g., "my-gitops") 55 | # When create=false: provide full URL (e.g., "https://github.com/myorg/kalypso-gitops") 56 | # Leave empty to use default name "kalypso-gitops" when creating 57 | gitops: "my-gitops" 58 | 59 | # GitHub Configuration 60 | github: 61 | # GitHub organization name (optional) 62 | # Leave empty to create repositories in your personal account 63 | # Provide organization name to create repositories in an organization 64 | # Example: my-organization 65 | org: "my-organization" 66 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Security 4 | 5 | Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). 6 | 7 | If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below. 8 | 9 | ## Reporting Security Issues 10 | 11 | **Please do not report security vulnerabilities through public GitHub issues.** 12 | 13 | Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report). 14 | 15 | If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey). 16 | 17 | You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc). 18 | 19 | Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: 20 | 21 | * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) 22 | * Full paths of source file(s) related to the manifestation of the issue 23 | * The location of the affected source code (tag/branch/commit or direct URL) 24 | * Any special configuration required to reproduce the issue 25 | * Step-by-step instructions to reproduce the issue 26 | * Proof-of-concept or exploit code (if possible) 27 | * Impact of the issue, including how an attacker might exploit the issue 28 | 29 | This information will help us triage your report more quickly. 30 | 31 | If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs. 32 | 33 | ## Preferred Languages 34 | 35 | We prefer all communications to be in English. 36 | 37 | ## Policy 38 | 39 | Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd). 40 | 41 | 42 | -------------------------------------------------------------------------------- /cicd/readme.md: -------------------------------------------------------------------------------- 1 | # GitOps CI/CD with GitHub 2 | 3 | ## Promotional Flow 4 | 5 | The promotional flow is implemented with a chain of GiHub Actions Workflows: 6 | 7 | ![promotion-flow](../docs/images/gh-promotion-flow.png) 8 | 9 | ### CI 10 | 11 | [The `CI` workflow](../.github/workflows/templates/ci.yml) is triggered on a commit to the *main* branch. It performs standard CI procedures, such as linting, security scanning, code analysis, unit testing, building and pushing Docker images.At the very end it invokes the CD part which starts with `Deploy` workflow. 12 | 13 | ### Deploy 14 | 15 | [The `Deploy` workflow](../.github/workflows/templates/deploy.yml) takes the Helm manifest templates from the source repo according to the source commit id. It generates K8s manifests applying config values from the Config Git repo. It takes the config values from a branch corresponding to the current environment (e.g. `dev`). Having manifests generated, it creates a PR to the GitOps repository on the processed environment/ring branch (e.g. `dev`). The workflow updates the Git commit status specifying that the change has been promoted to the environment. 16 | 17 | ### On PR Merge 18 | 19 | The `On PR Merge` workflow (defined in [notify-on-pr.yml](../.github/workflows/templates/notify-on-pr.yml) in GitOps repo) is triggered when the PR with the manifests is merged to the environment/ring branch in the GitOps repository. It notifies the source repository, which serves as the main orchestrator of the CD flow, by invoking the `Post Deploy` workflow. 20 | 21 | ### Post Deploy 22 | 23 | [The `Post Deploy` workflow](../.github/workflows/templates/post-deployment.yml) checks if the deployment is successful across all clusters subscribed on this change. This workflow polls Azure Resource Graph, waiting until all registered [GitOps configurations](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/conceptual-gitops-flux2) in the subscription are in compliance with the GitOps commit id. 24 | 25 | If one of the clusters reports failure, the workflow fails. It updates the Git commit status in the source repository as failed and the whole promotion flow stops. 26 | 27 | Once all configurations are compliant, the workflow updates the source commit id with `ENV Deployed` status. 28 | 29 | The workflow checks if the change that started the CD should be promoted (e.g. originated in `main`) to the next environment and if the next environment is configured, it starts a new loop by invoking the `deploy` workflow for the next environment. The next environment name is stored in `NEXT_ENVIRONMENT` GH environment variable. The whole process stops when there is no next environment in the chain. 30 | 31 | ## Tutorial 32 | 33 | In order to try out the promotional flow, go through the [tutorial](./tutorial/cicd-tutorial.md) that provides an end-to-end sample with the the guidance on deploying, promoting and configuring and application in GitOps fashion. 34 | 35 | ## Setup 36 | 37 | Follow the [setup](./setup.md) instructions to bootstrap the GitOps CI/CD for your application. 38 | -------------------------------------------------------------------------------- /.github/workflows/templates/utils/generate-manifests.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Usage: 4 | # generate-manifests.sh FOLDER_WITH_MANIFESTS FOLDER_WITH_CONFIGS GENERATED_MANIFESTS_FOLDER 5 | 6 | # Example: 7 | # generate-manifests.sh "/manifests" "/configs" "/generated-manifests" 8 | 9 | # Generates manifests from Helm charts in the FOLDER_WITH_MANIFESTS using values.yaml files in the FOLDER_WITH_CONFIGS and 10 | # saves them in the GENERATED_MANIFESTS_FOLDER 11 | 12 | FOLDER_WITH_MANIFESTS=$1 13 | FOLDER_WITH_CONFIGS=$2 14 | GENERATED_MANIFESTS_FOLDER=$3 15 | 16 | echo $FOLDER_WITH_MANIFESTS 17 | echo $FOLDER_WITH_CONFIGS 18 | echo $GENERATED_MANIFESTS_FOLDER 19 | 20 | set -euo pipefail 21 | 22 | gen_manifests_file_name='gen_manifests.yaml' 23 | values_file_name='values.yaml' 24 | 25 | 26 | mkdir -p $GENERATED_MANIFESTS_FOLDER 27 | 28 | # Substitute env variables in Helm yaml files in the manifest folder 29 | # e.g. "image_name: $IMAGE_NAME" -> "image_name: gitops.azurecr.io/agent:1.0.0" 30 | for file in `find $FOLDER_WITH_MANIFESTS -type f \( -name "values.yaml" \)`; do envsubst <"$file" > "$file"1 && mv "$file"1 "$file"; done 31 | 32 | 33 | cd $FOLDER_WITH_CONFIGS 34 | for dir in `find . -type d \( ! -name . \)`; do 35 | # Generate manifests for every leaf folder with values.yaml 36 | # All values.yaml files in the path to the leaf folder are merged into one values.yaml 37 | if [ -z "$(find $dir -mindepth 1 -type d \( ! -name . \))" ] && [ -f $dir/$values_file_name ]; then 38 | manifests_dir=$GENERATED_MANIFESTS_FOLDER/$dir 39 | mkdir -p $manifests_dir 40 | path=$dir 41 | while [[ $path != $FOLDER_WITH_CONFIGS ]]; 42 | do 43 | # if there is any values.yaml in $path flush its content to manifests_dir 44 | if [ -f $path/$values_file_name ]; then 45 | touch $manifests_dir/$values_file_name 46 | cat $path/$values_file_name $manifests_dir/$values_file_name > tmp_val && cat tmp_val > $manifests_dir/$values_file_name && rm tmp_val 47 | echo >> $manifests_dir/$values_file_name 48 | fi 49 | path="$(readlink -f "$path"/..)" 50 | done 51 | # Generate manifests out of helm chart 52 | envsubst <"$manifests_dir/$values_file_name" > "$manifests_dir/$values_file_name"1 && mv "$manifests_dir/$values_file_name"1 "$manifests_dir/$values_file_name" 53 | deployment_target=$(echo $manifests_dir | rev | cut -d'/' -f1 | rev) 54 | helm template "$deployment_target" "$FOLDER_WITH_MANIFESTS" -f $manifests_dir/$values_file_name > $manifests_dir/$gen_manifests_file_name && \ 55 | cat $manifests_dir/$gen_manifests_file_name 56 | if [ $? -gt 0 ] 57 | then 58 | echo "Could not render manifests" 59 | exit 1 60 | fi 61 | 62 | pushd $manifests_dir 63 | 64 | # Generate kustomization.yaml 65 | kustomize create --autodetect 66 | popd 67 | 68 | # # Generate deployment descriptor 69 | 70 | 71 | # mkdir -p $manifests_dir/descriptor 72 | # $GITHUB_WORKSPACE/.github/workflows/utils/generate-deployment-descriptor.sh $deployment_target $manifests_dir/descriptor/$deployment_descriptor_file_name $GITHUB_WORKSPACE/$deployment_descriptor_template 73 | 74 | rm $manifests_dir/$values_file_name 75 | fi 76 | done 77 | 78 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/main/.github/workflows/check-promote.yaml: -------------------------------------------------------------------------------- 1 | name: checkpromote 2 | 3 | on: 4 | workflow_dispatch: 5 | inputs: 6 | promoted_commit_id: 7 | description: 'Promoted Commit Id from Main' 8 | required: false 9 | type: string 10 | environment: 11 | required: true 12 | type: string 13 | commit_id: 14 | required: true 15 | type: string 16 | 17 | permissions: write-all 18 | 19 | jobs: 20 | check_and_promote: 21 | environment: ${{ inputs.environment }} 22 | runs-on: ubuntu-latest 23 | steps: 24 | 25 | - name: Checkout Source 26 | uses: actions/checkout@v2.3.4 27 | 28 | 29 | - name: Update Commit Status 30 | if: inputs.promoted_commit_id != '' 31 | run: | 32 | .github/workflows/utils/update-status.sh pending Deploying "$ENVIRONMENT environment" 33 | env: 34 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 35 | PROMOTED_COMMIT_ID: ${{ inputs.promoted_commit_id }} 36 | ENVIRONMENT: ${{ inputs.environment }} 37 | 38 | # - name: Wait for deployment to complete 39 | # run: | 40 | # .github/workflows/utils/wait-for-deployment.sh -r https://github.com/$GITOPS_REPO -b $ENVIRONMENT -c $COMMIT_ID 41 | # env: 42 | # GITOPS_REPO: ${{ secrets.GITOPS_REPO }} 43 | # ENVIRONMENT: ${{ inputs.environment }} 44 | # COMMIT_ID: ${{ inputs.commit_id }} 45 | 46 | 47 | - name: Run postdeployment activities 48 | run: | 49 | echo "Runinig..." 50 | env: 51 | NEXT_ENVIRONMENT: ${{ secrets.NEXT_ENVIRONMENT }} 52 | PROMOTED_COMMIT_ID: ${{ inputs.promoted_commit_id }} 53 | 54 | - name: Update Pending Commit Status 55 | if: inputs.promoted_commit_id != '' && env.NEXT_ENVIRONMENT != '' 56 | run: | 57 | .github/workflows/utils/update-status.sh "pending" "Deployed" "$ENVIRONMENT environment" 58 | env: 59 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 60 | PROMOTED_COMMIT_ID: ${{ inputs.promoted_commit_id }} 61 | ENVIRONMENT: ${{ inputs.environment }} 62 | 63 | - name: Update Positive Commit Status 64 | if: inputs.promoted_commit_id != '' && env.NEXT_ENVIRONMENT == '' 65 | run: | 66 | .github/workflows/utils/update-status.sh "success" "Deployed" "$ENVIRONMENT environment" 67 | env: 68 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 69 | PROMOTED_COMMIT_ID: ${{ inputs.promoted_commit_id }} 70 | ENVIRONMENT: ${{ inputs.environment }} 71 | 72 | - name: Update Negative Commit Status 73 | if: failure() && inputs.promoted_commit_id != '' 74 | run: | 75 | .github/workflows/utils/update-status.sh "failure" "Failed to deploy" "$ENVIRONMENT environment" 76 | env: 77 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 78 | PROMOTED_COMMIT_ID: ${{ inputs.promoted_commit_id }} 79 | ENVIRONMENT: ${{ inputs.environment }} 80 | 81 | 82 | - name: Start CD 83 | if: inputs.promoted_commit_id != '' && env.NEXT_ENVIRONMENT != '' 84 | run: | 85 | gh workflow run cd.yaml -f environment=$NEXT_ENVIRONMENT -f promoted_commit_id=$PROMOTED_COMMIT_ID 86 | env: 87 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 88 | NEXT_ENVIRONMENT: ${{ secrets.NEXT_ENVIRONMENT }} 89 | PROMOTED_COMMIT_ID: ${{ inputs.promoted_commit_id }} 90 | 91 | 92 | -------------------------------------------------------------------------------- /docs/run-books/platform-team-manages-platform-configuration.md: -------------------------------------------------------------------------------- 1 | # Platform Team Manages Platform Configuration 2 | 3 | - [Platform Team Manages Platform Configuration](#platform-team-manages-platform-configuration) 4 | - [Overview](#overview) 5 | - [Prerequisites](#prerequisites) 6 | - [1. Access to Platform Control Plane and GitOps Repos](#1-access-to-platform-control-plane-and-gitops-repos) 7 | - [Steps](#steps) 8 | - [1. Create/Update ConfigMaps](#1-createupdate-configmaps) 9 | - [2. Review and Merge GitOps Pull Request](#2-review-and-merge-gitops-pull-request) 10 | - [Next Steps](#next-steps) 11 | 12 | ## Overview 13 | 14 | Platform configuration is any value that is specific to an environment and cluster. For example, a database endpoint will be different for the same application deployed in 2 different factories or 2 different environments. 15 | 16 | In Kalypso, Platform configuration is managed through the Kalypso Control Plane repository. Review the root [README.md in the example Kalypso Control Plane repository](https://github.com/microsoft/kalypso-control-plane) for details on how CI/CD works for platform configuration using the promotional flow. 17 | 18 | > Any value that does not depend on environment or cluster should instead be managed through [Application Team Manages Application Configuration](./application-team-manages-application-configuration.md). These values are specific to the application or deployment ring, not tied to the hosting platform. 19 | 20 | ## Prerequisites 21 | 22 | ### 1. Access to Platform Control Plane and GitOps Repos 23 | 24 | This run book is intended to be completed by the Platform Engineer with contributor access to the Kalypso Control Plane and Platform GitOps repositories. 25 | 26 | For reference, here are the example [Control Plane](https://github.com/microsoft/kalypso-control-plane) and [Platform GitOps](https://github.com/microsoft/kalypso-gitops) repositories from Kalypso. 27 | 28 | ## Steps 29 | 30 | ### 1. Create/Update ConfigMaps 31 | 32 | Find or create the relevant ConfigMap in the correct branch & folder in the Platform Control Plane repository. Configs are provided to applications based on templating and label matching. 33 | 34 | | Variable | Description | 35 | | --------------- | ----------------------------------------------------------------------------- | 36 | | `NAME` | an identifier for the config map | 37 | | `CLUSTER_TYPE` | (Optional) specify specific cluster types that the config map should apply to | 38 | | `WORKLOAD_NAME` | (Optional) specify specific workloads that the config map should apply to | 39 | 40 | Additional labels for custom label matching can be provided as necessary. 41 | 42 | ```yaml 43 | apiVersion: v1 44 | kind: ConfigMap 45 | metadata: 46 | name: 47 | labels: 48 | platform-config: "true" 49 | 50 | # Optional labels 51 | # cluster-type: 52 | # workload: 53 | 54 | # Add any other custom labels as desired 55 | 56 | data: 57 | # Whatever values you'd like. Keys and values must be strings. 58 | exampleConfigKey: exampleConfigValue 59 | ``` 60 | 61 | The following example demonstrates a ConfigMap with platform configurations for all clusters in the EU region. This is determined by `region: eu` label. 62 | 63 | ![eu-config](./images/eu-config.png) 64 | 65 | All configurations that apply to every cluster will be consolidated int final platform-config ConfigMaps in the Platform GitOps repository. 66 | 67 | ### 2. Review and Merge GitOps Pull Request 68 | 69 | Once the configuration values have been updated in the platform control plane repositry on the appropriate git branch, an automated pull request will be created against the Platform GitOps repository. 70 | 71 | Check the Platform GitOps repository for these pull requests. Review the changes and merge when ready to deploy. 72 | 73 | ![platform-gitops-pr](./images/platform-gitops-pr.png) 74 | 75 | ## Next Steps 76 | 77 | To troubleshoot deployments, see the Kalypso Observability Hub dashboards that provide an overview of what applications are deployed into what clusters. 78 | 79 | It may be necessary to work with the specific application development teams to troubleshoot broken deployments. 80 | -------------------------------------------------------------------------------- /.github/workflows/templates/utils/create-pr.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Usage: 4 | # create-pr.sh -s SOURCE_FOLDER -d DEST_FOLDER -r DEST_REPO -b DEST_BRANCH -i IMAGE_NAME -t TOKEN -r ENV_NAME -m AUTO_MERGE -l LABEL 5 | 6 | # Example: 7 | # create-pr.sh -s "/manifests" -d "/vsu" -r "https://github.com/GM-SDV/rs-dispatcher-framework-gitops" -b "d2" -i "a210298d2musea2crglobal.azurecr.io/rs-dispatcher-framework:0.0.150-a91af7de6e31afc3c1d342a9b8659ee3976c71aa" -t "token" -r "d2" -m "N" -l "promoted" 8 | 9 | 10 | # Creates a PR from the SOURCE_FOLDER to the DEST_FOLDER in the DEST_REPO in the DEST_BRANCH. It also creates a label on the PR, if specified. 11 | # The script saves tracking information, such as PROMOTED_COMMIT_ID, VERSION and INAGE_NAME in the .github/tracking folder in the DEST_REPO. 12 | 13 | 14 | while getopts "s:d:r:b:i:t:e:m:l:" option; 15 | do 16 | case "$option" in 17 | s ) SOURCE_FOLDER=${OPTARG};; 18 | d ) DEST_FOLDER=${OPTARG};; 19 | r ) DEST_REPO=${OPTARG};; 20 | b ) DEST_BRANCH=${OPTARG};; 21 | i ) IMAGE_NAME=${OPTARG};; 22 | t ) TOKEN=${OPTARG};; 23 | e ) ENV_NAME=${OPTARG};; 24 | m ) AUTO_MERGE=${OPTARG};; 25 | l ) LABEL=${OPTARG};; 26 | esac 27 | done 28 | echo "List input params" 29 | echo $SOURCE_FOLDER 30 | echo $DEST_FOLDER 31 | echo $DEST_REPO 32 | echo $DEST_BRANCH 33 | echo $IMAGE_NAME 34 | echo $ENV_NAME 35 | echo $LABEL 36 | echo "end of list" 37 | 38 | set -eo pipefail # fail on error 39 | 40 | pr_user_name="Git Ops" 41 | pr_user_email="agent@gitops.com" 42 | 43 | git config --global user.email $pr_user_email 44 | git config --global user.name $pr_user_name 45 | 46 | # Clone manifests repo 47 | echo "Clone manifests repo" 48 | repo_url="${DEST_REPO#http://}" 49 | repo_url="${DEST_REPO#https://}" 50 | repo_url="https://automated:$TOKEN@$repo_url" 51 | 52 | echo "git clone $repo_url -b $DEST_BRANCH --depth 1 --single-branch" 53 | git clone $repo_url -b $DEST_BRANCH --depth 1 --single-branch 54 | repo=${DEST_REPO##*/} 55 | repo_name=${repo%.*} 56 | cd "$repo_name" 57 | echo "git status" 58 | git status 59 | 60 | # Create a new branch 61 | deploy_branch_name=deploy/$PROMOTED_COMMIT_ID/$ENV_NAME 62 | 63 | echo "Create a new branch $deploy_branch_name" 64 | git checkout -b $deploy_branch_name 65 | 66 | # Add generated manifests to the new deploy branch 67 | mkdir -p $DEST_FOLDER 68 | cp -r $SOURCE_FOLDER/* $DEST_FOLDER/ 69 | 70 | # Add tracking information 71 | mkdir -p .github/tracking 72 | echo "$PROMOTED_COMMIT_ID" > .github/tracking/Promoted_Commit_Id 73 | echo "$CONFIG_COMMIT_ID" > .github/tracking/Config_Commit_Id 74 | echo "$VERSION" > .github/tracking/Version 75 | echo "$IMAGE_TAG" > .github/tracking/Image_tag 76 | 77 | git add -A 78 | git status 79 | # If there are changes, commit them 80 | if [[ `git status --porcelain | head -1` ]]; then 81 | git commit -m "deployment $VERSION" 82 | 83 | # In case the deploy branch already exists, merge it with the current changes 84 | echo "Pull the deploy branch $deploy_branch_name" 85 | echo "git pull $repo_url $deploy_branch_name -s ours" 86 | git config pull.rebase false 87 | git pull $repo_url $deploy_branch_name -s ours || true 88 | 89 | # Push to the deploy branch 90 | echo "Push to the deploy branch $deploy_branch_name" 91 | echo "git push --set-upstream $repo_url $deploy_branch_name" 92 | git push --set-upstream $repo_url $deploy_branch_name 93 | 94 | # Create a PR 95 | echo "Create a PR to $DEST_BRANCH" 96 | 97 | owner_repo="${DEST_REPO#https://github.com/}" 98 | owner_repo="${owner_repo%.*}" 99 | echo $owner_repo 100 | GITHUB_TOKEN=$TOKEN 101 | echo $GITHUB_TOKEN | gh auth login --with-token 102 | 103 | if gh pr view $deploy_branch_name -R $owner_repo --json state | grep OPEN; then 104 | echo "PR already exists" 105 | else 106 | pr_response=$(gh pr create --base $DEST_BRANCH --head $deploy_branch_name --title "Deploy to '$ENV_NAME' '$VERSION'" --body "Deploy to '$ENV_NAME' '$VERSION'") 107 | echo $pr_response 108 | pr_num="${pr_response##*pull/}" 109 | echo $pr_num 110 | # Add a label to the PR 111 | if [[ ! -z "$LABEL" ]]; then 112 | gh issue edit $pr_num --add-label $LABEL 113 | fi 114 | # If auto merge is specified, merge the PR 115 | if [[ "$AUTO_MERGE" == "Y" ]]; then 116 | gh pr merge $pr_num -m -d --repo $repo_url 117 | fi 118 | 119 | fi 120 | fi 121 | -------------------------------------------------------------------------------- /.github/workflows/templates/deploy.yml: -------------------------------------------------------------------------------- 1 | name: deploy 2 | 3 | # The `deploy` workflow takes the Helm manifest templates from the source repo according to the source commit id. 4 | # It generates K8s manifests applying config values from the Config Git repo. 5 | # It takes the config values from a branch corresponding to the current environment. 6 | # Having manifests generated, it creates a PR to the GitOps repository on the processed environment/ring branch. 7 | # The workflow updates the Git commit status specifying that the change has been promoted to the environment. 8 | 9 | 10 | on: 11 | workflow_dispatch: 12 | inputs: 13 | commit_id: 14 | required: true 15 | type: string 16 | description: 'Source commit_id (e.g. 62d1d965735a80afca35da9ec8d8aeb39db2da6c)' 17 | environment: 18 | required: false 19 | default: dev 20 | type: string 21 | description: 'Environment to deploy to (e.g. d2)' 22 | promotion: 23 | required: false 24 | type: string 25 | default: y 26 | description: 'If the change is a subject for the promotion (e.g. Y)' 27 | image_tag: 28 | required: true 29 | type: string 30 | description: 'Image tag' 31 | version: 32 | required: true 33 | type: string 34 | description: 'Version (e.g. 0.0.1-15)' 35 | 36 | 37 | jobs: 38 | Deploy: 39 | name: "Deploy" 40 | runs-on: ubuntu-latest 41 | permissions: write-all 42 | environment: ${{ inputs.environment }} 43 | steps: 44 | 45 | - name: Checkout 46 | uses: actions/checkout@v3 47 | with: 48 | ref: ${{ inputs.commit_id }} 49 | - name: Checkout Configs 50 | uses: actions/checkout@v2.3.4 51 | with: 52 | repository: ${{ vars.CONFIGS_REPO }} 53 | ref: ${{ inputs.environment }} 54 | path: configs 55 | token: ${{ secrets.CD_BOOTSTRAP_TOKEN }} 56 | 57 | - name: Get Config Commit Id 58 | run: | 59 | CONFIG_COMMIT_ID=$(git rev-parse HEAD) 60 | echo "CONFIG_COMMIT_ID=$CONFIG_COMMIT_ID" >> $GITHUB_ENV 61 | working-directory: configs 62 | 63 | - name: Set Image tag 64 | run: | 65 | echo "IMAGE_TAG=$IMAGE_TAG" >> $GITHUB_ENV 66 | echo "VERSION=$VERSION" >> $GITHUB_ENV 67 | env: 68 | IMAGE_TAG: ${{ inputs.image_tag }} 69 | VERSION: ${{ inputs.version }} 70 | 71 | - name: Generate Manifests 72 | run: | 73 | .github/workflows/utils/generate-manifests.sh "$HELM_PATH" "$CONFIGS_PATH" "$MANIFESTS_PATH" 74 | env: 75 | HELM_PATH: ${{ github.workspace }}/helm 76 | CONFIGS_PATH: ${{ github.workspace }}/configs 77 | MANIFESTS_PATH: ${{ github.workspace }}/manifests 78 | WORKLOAD: ${{ github.event.repository.name }} 79 | ENVIRONMENT: ${{ inputs.environment }} 80 | SOURCE_REPO: ${{ github.event.repository.full_name }} 81 | SOURCE_BRANCH: ${{ github.event.repository.default_branch }} 82 | GITOPS_REPO: ${{ vars.MANIFESTS_REPO }} 83 | VERSION: ${{ env.VERSION }} 84 | SRC_COMMIT: ${{ inputs.commit_id }} 85 | CONFIG_REVISION: ${{ env.CONFIG_COMMIT_ID}} 86 | IMAGE_NAME: ghcr.io/${{ github.repository }}/${{ github.event.repository.name }} 87 | 88 | 89 | 90 | - name: Create PR 91 | run: | 92 | promoted=$(if [ ${{ inputs.promotion }} == "y" ]; then echo "promoted"; fi) 93 | .github/workflows/utils/create-pr.sh -s "$MANIFESTS_PATH" -d . -r "$MANIFESTS_REPO" -b "$ACTIVE_ENVIRONMENT" -i "$IMAGE_TAG" -t "$TOKEN" -e "$ACTIVE_ENVIRONMENT" -m N -l $promoted 94 | env: 95 | PROMOTED_COMMIT_ID: ${{ inputs.commit_id }} 96 | CONFIG_COMMIT_ID: ${{ env.CONFIG_COMMIT_ID }} # from the Get Config Commit Id step 97 | MANIFESTS_PATH: ${{ github.workspace }}/manifests 98 | MANIFESTS_REPO: https://github.com/${{ vars.MANIFESTS_REPO }} 99 | TOKEN: ${{ secrets.CD_BOOTSTRAP_TOKEN }} 100 | ACTIVE_ENVIRONMENT: ${{ inputs.environment }} 101 | 102 | - name: Update Commit Status 103 | run: | 104 | .github/workflows/utils/update-status.sh "pending" "Promoted $VERSION" "$ACTIVE_ENVIRONMENT" 105 | env: 106 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 107 | PROMOTED_COMMIT_ID: ${{ inputs.commit_id }} 108 | ACTIVE_ENVIRONMENT: ${{ inputs.environment }} 109 | 110 | 111 | -------------------------------------------------------------------------------- /docs/run-books/application-team-manages-application-configuration.md: -------------------------------------------------------------------------------- 1 | # Application Team Manages Application Configuration 2 | 3 | - [Application Team Manages Application Configuration](#application-team-manages-application-configuration) 4 | - [Overview](#overview) 5 | - [Prerequisites](#prerequisites) 6 | - [1. Verify Access to Application Repositories](#1-verify-access-to-application-repositories) 7 | - [Steps](#steps) 8 | - [1. Submit Configuration Change](#1-submit-configuration-change) 9 | - [2. Deploy Configuration Change via GitOps](#2-deploy-configuration-change-via-gitops) 10 | - [3. Verify Deployment](#3-verify-deployment) 11 | - [Next Steps](#next-steps) 12 | - [Rolling Back](#rolling-back) 13 | 14 | ## Overview 15 | 16 | This run book describes how to manage configuration at the application level. 17 | 18 | Application configuration includes any values that do not depend on what cluster or factory the application is deployed to (ex: logging verbosity, image tags, etc.) 19 | 20 | Configuration that does depend on what cluster or factory the application is deployed to (ex: database endpoints, KeyVault secrets, etc.) is considered platform configuration. See [Platform Team Manages Platform Configuration](./platform-team-manages-platform-configuration.md) for more information. 21 | 22 | ## Prerequisites 23 | 24 | ### 1. Verify Access to Application Repositories 25 | 26 | Each application consists of 3 git repositories. A source repository, a configuration repository, and a GitOps repository. Identify all 3 repositories and make sure you have contributor access to all 3. 27 | 28 | Make sure the **config repository** is cloned locally as this is the repository you will submit a change to. 29 | 30 | ## Steps 31 | 32 | ### 1. Submit Configuration Change 33 | 34 | Locate the appropriate branch and values file inside the configuration repository. The branch name should identify the environment and ring that this application configuration belongs to. 35 | 36 | Update the values file to add/remove/modify your desired configuration values. Then, create a PR the appropriate branch. Once the PR is merged, it will trigger the deployment to the matching environment. 37 | 38 | ![update-app-config-values](./images/update-app-config-values.png) 39 | 40 | ### 2. Deploy Configuration Change via GitOps 41 | 42 | After a change to the configuration repository has been made, a new Pull Request is automatically created against the GitOps repository. 43 | 44 | Locate the appropriate pull request in your application's GitOps repository. 45 | 46 | - It will be created shortly after the configuration change has been submitted. 47 | - It will note the current version of your application. 48 | - It should **not** be marked `promoted`. 49 | - It will target a merge into the branch of the repository that matches the configuration branch that was updated. 50 | 51 | Review and merge this pull request to deploy it to the target environment. If you find any issues, or otherwise wish not to deploy it, simply close the pull request without merging. 52 | 53 | > Pull requests labeled `promoted` indicate changes that are eligible for promotion through environments. 54 | > 55 | > Changes to configuration are not subjects for promotion and are not labeled as such. These PRs are generated from [Application Team Promotes a Change Through Environments](./application-team-promotes-a-change-through-environments.md). 56 | > 57 | > ![not-promoted-pr](./images/not-promoted-pr.png) 58 | 59 | ![not-promoted-pr-details](./images/not-promoted-pr-details.png) 60 | 61 | ### 3. Verify Deployment 62 | 63 | Once the pull request has been merged, we can verify it from the deployment observability dashboards. Navigate to the Cluster State dashboard. The GitOps commit hash you see should match the GitOps commit of the PR that was merged in [step 2](#2-deploy-configuration-change-via-gitops). 64 | 65 | Here is a screenshot of the `hello-world` deployment history to the `dev-gray` environment. Note the latest commit hash: `sha1:161b2623...` 66 | ![config-change-dashboard](./images/config-change-dashboard.png) 67 | 68 | When we find the GitOps commit for the recently merged pull request, the commit matches: `sha1:161b2623...`. This means the correct commit has been deployed to the cluster. 69 | ![gitops-commit-id](./images/gitops-commit-id.png) 70 | 71 | ## Next Steps 72 | 73 | ### Rolling Back 74 | 75 | If an issue was discovered in a deployment, and it needs to be reverted, GitOps makes that easy. 76 | 77 | 1. Locate the GitOps PR for the appropriate deployment & environment. 78 | 2. Create a new GitOps PR that reverts the deployment using the "Revert" button. 79 | 3. Review and merge the resulting PR to roll back the release. 80 | 81 | ![revert-config-change](./images/revert-config-change.png) 82 | -------------------------------------------------------------------------------- /.github/workflows/templates/post-deployment.yml: -------------------------------------------------------------------------------- 1 | name: post-deployment 2 | 3 | # The `post-deployment` workflow checks if the deployment is successful across all clusters subscribed on this change. 4 | # This workflow polls Azure Resource Graph, waiting until all registered [GitOps configurations](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/conceptual-gitops-flux2) 5 | # in the subscription are in compliance with the GitOps commit id. 6 | 7 | # If one of the clusters reports failure, the workflow fails. 8 | # It updates the Git commit status in the source repository as failed and the whole promotion flow stops. 9 | 10 | # Once all configurations are compliant, the workflow updates the source commit id with `ENV Deployed` status. 11 | 12 | 13 | on: 14 | workflow_dispatch: 15 | inputs: 16 | promoted_commit_id: 17 | description: 'Promoted Commit Id from source repo (e.g. 62d1d965735a80afca35da9ec8d8aeb39db2da6c)' 18 | required: true 19 | type: string 20 | environment: 21 | required: true 22 | type: string 23 | description: 'Environment where the deployment happened (e.g. d2)' 24 | commit_id: 25 | required: true 26 | type: string 27 | description: 'GitOps Commit Id (e.g. 42d1d965735a80afca35da9ec8d8aeb39db2da6c)' 28 | promotion: 29 | required: false 30 | type: string 31 | default: y 32 | description: 'If the change is a subject for the promotion (e.g. Y)' 33 | image_tag: 34 | required: true 35 | type: string 36 | description: 'Image tag' 37 | version: 38 | required: true 39 | type: string 40 | description: 'Version' 41 | 42 | 43 | permissions: write-all 44 | 45 | jobs: 46 | post-deployment: 47 | environment: ${{ inputs.environment }} 48 | runs-on: ubuntu-latest 49 | steps: 50 | 51 | - name: Checkout Source 52 | uses: actions/checkout@v2.3.4 53 | 54 | - name: Update Commit Status 55 | run: | 56 | .github/workflows/utils/update-status.sh pending Deploying "$ACTIVE_ENVIRONMENT" 57 | env: 58 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 59 | PROMOTED_COMMIT_ID: ${{ inputs.promoted_commit_id }} 60 | ACTIVE_ENVIRONMENT: ${{ inputs.environment }} 61 | 62 | - uses: azure/login@v1 63 | with: 64 | creds: ${{ secrets.AZURE_CREDENTIALS_SP }} 65 | 66 | - name: Wait for deployment to complete 67 | run: | 68 | .github/workflows/utils/wait-for-deployment.sh -r "$GITOPS_REPO" -b "$ACTIVE_ENVIRONMENT" -c "$COMMIT_ID" 69 | env: 70 | GITOPS_REPO: https://github.com/${{ vars.MANIFESTS_REPO }} 71 | ACTIVE_ENVIRONMENT: ${{ inputs.environment }} 72 | COMMIT_ID: ${{ inputs.commit_id }} 73 | 74 | - name: Promote to Next Environment 75 | if: inputs.promotion == 'y' && vars.NEXT_ENVIRONMENT != '' 76 | run: | 77 | gh workflow run deploy.yml -f environment="$NEXT_ENVIRONMENT" -f commit_id="$PROMOTED_COMMIT_ID" -f image_tag="$IMAGE_TAG" -f version="$VERSION" 78 | env: 79 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 80 | NEXT_ENVIRONMENT: ${{ vars.NEXT_ENVIRONMENT }} 81 | PROMOTED_COMMIT_ID: ${{ inputs.promoted_commit_id }} 82 | IMAGE_TAG : ${{ inputs.image_tag }} 83 | VERSION : ${{ inputs.version }} 84 | 85 | 86 | - name: Update Positive Deployed Commit Status 87 | if: inputs.promotion == 'y' 88 | run: | 89 | .github/workflows/utils/update-status.sh "success" "Deployed $VERSION: $COMMIT_ID" "$ACTIVE_ENVIRONMENT" 90 | env: 91 | GH_TOKEN: ${{ secrets.CD_BOOTSTRAP_TOKEN }} 92 | PROMOTED_COMMIT_ID: ${{ inputs.promoted_commit_id }} 93 | ACTIVE_ENVIRONMENT: ${{ inputs.environment }} 94 | COMMIT_ID: ${{ inputs.commit_id }} 95 | VERSION: ${{ inputs.version }} 96 | 97 | - name: Update Positive Configured Commit Status 98 | if: inputs.promotion != 'y' 99 | run: | 100 | .github/workflows/utils/update-status.sh "success" "Configured: $COMMIT_ID" "$ACTIVE_ENVIRONMENT" 101 | env: 102 | GH_TOKEN: ${{ secrets.CD_BOOTSTRAP_TOKEN }} 103 | PROMOTED_COMMIT_ID: ${{ inputs.promoted_commit_id }} 104 | ACTIVE_ENVIRONMENT: ${{ inputs.environment }} 105 | COMMIT_ID: ${{ inputs.commit_id }} 106 | 107 | 108 | - name: Update Negative Commit Status 109 | if: failure() && inputs.promoted_commit_id != '' 110 | run: | 111 | .github/workflows/utils/update-status.sh "failure" "Failed to deploy" "$ACTIVE_ENVIRONMENT" 112 | env: 113 | GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 114 | PROMOTED_COMMIT_ID: ${{ inputs.promoted_commit_id }} 115 | ACTIVE_ENVIRONMENT: ${{ inputs.environment }} 116 | 117 | -------------------------------------------------------------------------------- /scripts/bootstrap/templates/control-plane/main/.github/workflows/utils/wait-for-deployment.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Waits for the deployment to complete. 4 | # Polls ARG with the specified interval until the deployment is complete or the timeout is reached. 5 | # 6 | # Usage: 7 | # wait_for_deployment.sh flags 8 | 9 | # Flags: 10 | # -r GitOps Repository URL (e.g. https://github.com/microsoft/kalypso-gitops) 11 | # -b Environment branch (e.g. dev) 12 | # -c Commit Id (e.g. c32f8da476689f8cf309ca0e3fbbda42b3a8d387) 13 | 14 | # Example: 15 | # wait_for_deployment.sh -r https://github.com/microsoft/kalypso-gitops -b dev -c c32f8da476689f8cf309ca0e3fbbda42b3a8d387 16 | 17 | 18 | 19 | while getopts "r:b:c:" option; 20 | do 21 | case "$option" in 22 | r ) REPO_URL=${OPTARG};; 23 | b ) REPO_BRANCH=${OPTARG};; 24 | c ) COMMIT_ID=${OPTARG};; 25 | esac 26 | done 27 | 28 | total_attempts=60 # 5 minutes 29 | poll_interval=5 # seconds 30 | 31 | set -eo pipefail # fail on error 32 | az extension add --name resource-graph 33 | 34 | error() { 35 | echo $1>&2 36 | exit 1 37 | } 38 | 39 | usage() { 40 | echo $1>&2 41 | cat < 0 )); 97 | then 98 | echo "Checking for non-compliant configurations ..." 99 | non_compliant_configs=$(get_synched_configs 'Non-Compliant') 100 | echo $non_compliant_configs 101 | total_non_compliant_configs=$( echo $non_compliant_configs | jq '.total_records') 102 | 103 | if (( $total_non_compliant_configs > 0 )); 104 | then 105 | error "There are $total_non_compliant_configs Non_Compliant configurations: " + $non_compliant_configs 106 | fi 107 | 108 | echo "Checking for compliant configurations ..." 109 | compliant_configs=$(get_synched_configs 'Compliant') 110 | echo $compliant_configs 111 | total_compliant_configs=$( echo $compliant_configs | jq '.total_records') 112 | 113 | if (( $total_compliant_configs == $total_configs )); 114 | then 115 | echo "All $total_configs configurations are compliant " 116 | exit 0 117 | else 118 | echo "$total_compliant_configs out of $total_configs configurations are compliant. Keep polling... " 119 | fi 120 | 121 | sleep $poll_interval 122 | attempt=$(( $attempt + 1 )) 123 | else 124 | exit 0 125 | fi 126 | 127 | done 128 | 129 | error "$total_compliant_configs out of $total_configs configurations are compliant. Deployment is failed " 130 | 131 | 132 | } 133 | 134 | check_parameters 135 | wait_for_deployment 136 | 137 | -------------------------------------------------------------------------------- /.github/workflows/templates/utils/wait-for-deployment.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Waits for the deployment to complete. 4 | # Polls ARG with the specified interval until the deployment is complete or the timeout is reached. 5 | # 6 | # Usage: 7 | # wait_for_deployment.sh flags 8 | 9 | # Flags: 10 | # -r GitOps Repository URL (e.g. https://github.com/microsoft/kalypso-gitops) 11 | # -b Environment branch (e.g. dev) 12 | # -c Commit Id (e.g. c32f8da476689f8cf309ca0e3fbbda42b3a8d387) 13 | 14 | # Example: 15 | # wait_for_deployment.sh -r https://github.com/microsoft/kalypso-gitops -b dev -c c32f8da476689f8cf309ca0e3fbbda42b3a8d387 16 | 17 | 18 | 19 | while getopts "r:b:c:" option; 20 | do 21 | case "$option" in 22 | r ) REPO_URL=${OPTARG};; 23 | b ) REPO_BRANCH=${OPTARG};; 24 | c ) COMMIT_ID=${OPTARG};; 25 | esac 26 | done 27 | 28 | total_attempts=60 # 5 minutes 29 | poll_interval=5 # seconds 30 | 31 | set -eo pipefail # fail on error 32 | az extension add --name resource-graph 33 | 34 | error() { 35 | echo $1>&2 36 | exit 1 37 | } 38 | 39 | usage() { 40 | echo $1>&2 41 | cat < 0 )); 97 | then 98 | echo "Checking for non-compliant configurations ..." 99 | non_compliant_configs=$(get_synched_configs 'Non-Compliant') 100 | echo $non_compliant_configs 101 | total_non_compliant_configs=$( echo $non_compliant_configs | jq '.total_records') 102 | 103 | if (( $total_non_compliant_configs > 0 )); 104 | then 105 | error "There are $total_non_compliant_configs Non_Compliant configurations: " + $non_compliant_configs 106 | fi 107 | 108 | echo "Checking for compliant configurations ..." 109 | compliant_configs=$(get_synched_configs 'Compliant') 110 | echo $compliant_configs 111 | total_compliant_configs=$( echo $compliant_configs | jq '.total_records') 112 | 113 | if (( $total_compliant_configs == $total_configs )); 114 | then 115 | echo "All $total_configs configurations are compliant " 116 | exit 0 117 | else 118 | echo "$total_compliant_configs out of $total_configs configurations are compliant. Keep polling... " 119 | fi 120 | 121 | sleep $poll_interval 122 | attempt=$(( $attempt + 1 )) 123 | else 124 | exit 0 125 | fi 126 | 127 | done 128 | 129 | error "$total_compliant_configs out of $total_configs configurations are compliant. Deployment is failed " 130 | 131 | 132 | } 133 | 134 | check_parameters 135 | wait_for_deployment 136 | 137 | -------------------------------------------------------------------------------- /.github/workflows/pr.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | # PR Quality Check workflow 3 | name: PR Quality Check 4 | 5 | on: 6 | pull_request: 7 | branches: ["main"] 8 | workflow_dispatch: 9 | 10 | jobs: 11 | markdown-lint-check: 12 | name: Markdown Lint Checker 13 | runs-on: ubuntu-latest 14 | permissions: 15 | pull-requests: write 16 | steps: 17 | - name: Checkout a Git repo 18 | uses: actions/checkout@v3 19 | 20 | - name: Run Markdown Lint 21 | id: markdownlint 22 | # Official Docker image of markdownlint-cli 23 | # https://www.npmjs.com/package/markdownlint-cli 24 | uses: docker://ghcr.io/igorshubovych/markdownlint-cli:latest 25 | with: 26 | entrypoint: /usr/local/bin/markdownlint 27 | args: "--config .markdownlint.yaml --output markdown-report.txt **/*.md" 28 | 29 | - name: Prepare GitHub Summary 30 | if: ${{ always() && steps.markdownlint.conclusion == 'failure' }} 31 | run: | 32 | report=$(cat markdown-report.txt) 33 | summary=$( 34 | cat <> $GITHUB_STEP_SUMMARY 42 | 43 | - name: Report to PR as comment 44 | if: ${{ always() && steps.markdownlint.conclusion == 'failure' && github.event_name == 'pull_request' }} 45 | uses: actions/github-script@v6 46 | with: 47 | script: | 48 | let fs = require('fs'); 49 | let reportContent = fs.readFileSync('markdown-report.txt'); 50 | 51 | const body = `# Markdown Lint \`failure\` 52 |
53 | Details\n 54 | \`\`\` 55 | ` + reportContent + ` 56 | \`\`\`\n 57 |
\n 58 | Pusher: @${{ github.actor }}`; 59 | 60 | github.rest.issues.createComment({ 61 | owner: context.repo.owner, 62 | repo: context.repo.repo, 63 | issue_number: context.issue.number, 64 | body: body 65 | }) 66 | 67 | - name: Archive report 68 | if: ${{ always() && steps.markdownlint.conclusion == 'failure' }} 69 | uses: actions/upload-artifact@v4 70 | with: 71 | name: markdownlint report 72 | path: markdown-report.txt 73 | 74 | markdown-link-check: 75 | name: Markdown Link Checker 76 | runs-on: ubuntu-latest 77 | steps: 78 | - uses: actions/checkout@v3 79 | 80 | - name: Run Markdown Link Checker 81 | uses: gaurav-nelson/github-action-markdown-link-check@v1 82 | with: 83 | config-file: '.markdownlinkcheck.json' 84 | # Quiet mode only shows errors in output not successful links too 85 | use-quiet-mode: 'yes' 86 | # Specify yes to show detailed HTTP status for checked links. 87 | use-verbose-mode: 'yes' 88 | 89 | secrets-detection: 90 | name: Secrets Detection 91 | runs-on: ubuntu-latest 92 | steps: 93 | - name: Checkout repo 94 | uses: actions/checkout@v3 95 | with: 96 | fetch-depth: "0" 97 | 98 | - name: Run Gitleaks 99 | id: gitleaks 100 | uses: DariuszPorowski/github-action-gitleaks@v2 101 | with: 102 | report_format: sarif 103 | config: UDMSecretChecks.toml 104 | 105 | - name: Upload SARIF report to Workflow Artifacts 106 | uses: actions/upload-artifact@v4 107 | if: always() && steps.gitleaks.outputs.exitcode != 0 108 | with: 109 | name: gitleaks 110 | path: | 111 | ${{ steps.gitleaks.outputs.report }} 112 | 113 | - name: Upload SARIF report to Code Scanning service 114 | if: always() && steps.gitleaks.outputs.exitcode != 0 115 | uses: github/codeql-action/upload-sarif@v3 116 | with: 117 | sarif_file: ${{ steps.gitleaks.outputs.report }} 118 | category: gitleaks 119 | 120 | spell-check: 121 | name: Code Spell Checker 122 | runs-on: ubuntu-latest 123 | steps: 124 | - name: Checkout a Git repo 125 | uses: actions/checkout@v3 126 | 127 | - uses: actions/setup-node@v3 128 | with: 129 | node-version: 20 130 | cache: 'npm' 131 | 132 | - run: npm ci 133 | 134 | - run: | 135 | npm install -g cspell@latest 136 | 137 | - name: Run CSpell 138 | id: cspell 139 | run: | 140 | # cspell "docs/*.md" > cspell-report.txt 141 | cspell README.md >> cspell-report.txt 142 | 143 | - name: Prepare CSpell Summary 144 | if: ${{ always() && steps.cspell.conclusion == 'failure' }} 145 | run: | 146 | report=$(cat cspell-report.txt) 147 | summary=$( 148 | cat <> $GITHUB_STEP_SUMMARY 156 | -------------------------------------------------------------------------------- /docs/cd-concept.md: -------------------------------------------------------------------------------- 1 | # CD process with GitOps 2 | 3 | This is a conceptual overview of what a CD process looks like in the GitOps paradigm. 4 | 5 | ## Overview 6 | 7 | ![simple-flow](./images/gitops-CD-Simple-Flow.drawio.png) 8 | 9 | Application developers write their code and debug and test it both locally and on dev environment. The code is stored in the Code git repository and contains the microservice source code and descriptors describing how to build, deploy and run the microservice as a part of the whole system. Those descriptors are, normally, the Docker file and manifest templates. The Docker file is used by CI pipeline to build a Docker image. The manifest templates are used by the CD process to deploy the microservice to a chain of environments in a multistage fashion. It takes the templates from the Code repository and for each environment generates the manifests referring to the new Docker image and containing the variable values corresponding to this environment. These manifests are stored in Manifests git repository separated by branches/folders and they declaratively describe the desired state of our application in a K8s cluster in each environment. There is a GitOps operator running on a cluster that reconciles the desired cluster state by applying the manifests. When it comes to production, the deployment may be performed in a sophisticated way by implementing B/G, Canary, A/B testing deployment strategies and ring-based deployment concepts. And the whole loop is being closed by monitoring what/when/how was deployed, how it’s working, by alerts and notifications. 10 | 11 | ## Detailed Flow 12 | 13 | ![detailed-flow](./images/gitops-CD-Detailed-Flow.drawio.png) 14 | 15 | On the diagram above the output of the CI pipeline is a deployable artifact which includes Docker images, Infrastructure code, K8s manifest templates and an artifact version number. 16 | 17 | These artifacts are consumed by the CD pipeline which is triggered automatically right after the CI pipeline. The CD pipeline delivers in a multistage way K8s manifests that must be deployed for each environment and it promotes the artifacts to the next environment only after successful deployment on the previous one. For every single stage, representing an environment, the CD pipeline takes the manifest templates from the artifact, applies application configuration variables and generates K8s manifests that should be applied to the clusters in this environment. 18 | 19 | The configuration variables for each environment might be stored in variable groups (separated by environments), Azure Key Vaults, Azure App Configuration service or/and in order to strictly follow the GitOps doctrine they might be stored in a Configuration git repositories (separated by branches/folders for each environment). These configurations normally contain connection strings, ports, URIs, etc. 20 | 21 | Besides K8s manifests, the CD pipeline generates a deployment descriptor for the observability purposes. The descriptor contains information about the application, dev team, version number, Build Id, etc. 22 | 23 | The CD pipeline PRs the generated manifests and deployment descriptors to the GitOps repository which is observed by GitOps operators, such as Flux or ArgoCD. Once the PR is issued the CD pipeline is paused in agentless mode waiting for the deployment result. 24 | 25 | When the PR is approved/merged, the Deployment Observability Hub pulls the deployment descriptor from the GitOps repository and stores the information on what application version should be deployed in this environment. 26 | 27 | The GitOps operators on the environment clusters pull the manifests from the GitOps repo and apply them to the environment. With that done, clusters notify the Deployment Observability Hub with the deployment state, so the hub stores information about what clusters have successfully (or not) deployed the new application version. This information is used to notify the Azure CD pipeline about the overall deployment result, so it can be resumed and it can proceed to the post deployment activities, such as automated testing. If the deployment fails, the whole CD process is failed and the change will not be promoted any further. 28 | 29 | If the deployment and post deployment activities are successful, the CD process goes to the next stage performing the very same steps on the next environment. At the very beginning of the next stage the pipeline may have a gate analyzing test results from the previous stage. 30 | 31 | The information from the Deployment Observability Hub is consumed by the observability backends, such as Azure Monitor and/or Grafana for monitoring and alerting. 32 | 33 | Every environment may contain different clusters for different applications and purposes. E.g. a cluster for functional testing and a cluster for performance testing. Every cluster may have different configurations and platform services. All this behavior is defined in the [Control Plane](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/conceptual-workload-management#high-level-flow). It serves to manage environments, clusters, define rules of what applications should be deployed to what clusters in what environments, provide cluster configurations. 34 | 35 | ## Implementation 36 | 37 | See [GitOps CI/CD with GitHub](../cicd/readme.md) to understand the mechanics of the promotional flow implementation with GitHub Actions WOrkflows. 38 | 39 | ## Tutorial 40 | 41 | In order to try out the CI/CD process, go through the [tutorial](../cicd/tutorial/cicd-tutorial.md) that provides an end-to-end sample with the the guidance on deploying, promoting and configuring and application in GitOps fashion. 42 | 43 | ## Setup 44 | 45 | Follow the [setup](../cicd/setup.md) instructions to bootstrap the GitOps CI/CD for your application. 46 | -------------------------------------------------------------------------------- /scripts/bootstrap/README.md: -------------------------------------------------------------------------------- 1 | # Bootstrap Script Structure 2 | 3 | This directory contains the Kalypso Scheduler bootstrapping script and its supporting libraries. 4 | 5 | ## Directory Structure 6 | 7 | ```text 8 | scripts/bootstrap/ 9 | ├── bootstrap.sh # Main entry point 10 | ├── lib/ # Library modules 11 | │ ├── utils.sh # Utility functions (logging, JSON processing) 12 | │ ├── prerequisites.sh # Prerequisites and authentication validation 13 | │ ├── config.sh # Configuration management and CLI parsing 14 | │ ├── cluster.sh # AKS cluster operations 15 | │ ├── repositories.sh # GitHub repository management 16 | │ └── install.sh # Kalypso installation and verification 17 | ├── templates/ # Repository templates 18 | │ ├── control-plane/ # Control-plane repository templates 19 | │ │ ├── main/ # Main branch structure 20 | │ │ └── dev/ # Dev branch structure 21 | │ └── gitops/ # GitOps repository templates 22 | │ ├── main/ # Main branch structure 23 | │ └── dev/ # Dev branch structure 24 | └── README.md # This file 25 | ``` 26 | 27 | ## Quick Start 28 | 29 | Run the script in interactive mode: 30 | 31 | ```bash 32 | cd scripts/bootstrap 33 | ./bootstrap.sh 34 | ``` 35 | 36 | Or with a configuration file: 37 | 38 | ```bash 39 | ./bootstrap.sh --config kalypso-config.yaml 40 | ``` 41 | 42 | For detailed usage, see [../../docs/bootstrap/README.md](../../docs/bootstrap/README.md) 43 | 44 | ## Development 45 | 46 | ### Linting 47 | 48 | Use shellcheck to validate all scripts: 49 | 50 | ```bash 51 | shellcheck bootstrap.sh lib/*.sh 52 | ``` 53 | 54 | ## Library Modules 55 | 56 | ### utils.sh 57 | 58 | Core utilities used by all other modules: 59 | 60 | - Logging functions (log_error, log_warning, log_info, log_debug, log_success) 61 | - JSON processing (json_get_value) 62 | - String utilities (is_empty, trim, to_lower, to_upper) 63 | - Command utilities (command_exists, wait_for_condition) 64 | - User interaction (confirm, prompt_input) 65 | 66 | ### prerequisites.sh 67 | 68 | Prerequisites validation: 69 | 70 | - Required tool checking (kubectl, az, git, helm, gh, jq) 71 | - Optional tool checking (yq - required for YAML config files) 72 | - Version comparison 73 | - Azure authentication validation 74 | - GitHub authentication validation 75 | 76 | ### config.sh 77 | 78 | Configuration management: 79 | 80 | - CLI argument parsing 81 | - Configuration file loading (YAML, JSON, ENV formats) 82 | - Interactive prompts for missing values 83 | - Configuration validation 84 | - Resource tracking for rollback 85 | 86 | ### cluster.sh 87 | 88 | AKS cluster operations: 89 | 90 | - Cluster creation with resource group 91 | - Existing cluster validation 92 | - Kubeconfig integration 93 | - Cluster readiness checks 94 | - Namespace creation 95 | - Idempotent operations 96 | 97 | ### repositories.sh 98 | 99 | GitHub repository management: 100 | 101 | - Repository creation via GitHub API with custom names 102 | - Repository initialization with structured templates 103 | - Control-plane repository setup (main and dev branches) 104 | - GitOps repository setup (main and dev branches) 105 | - Repository validation 106 | - GitHub secrets configuration (via gh CLI) 107 | 108 | ### install.sh 109 | 110 | Kalypso installation: 111 | 112 | - Helm chart installation 113 | - Installation verification 114 | - CRD checking 115 | - Rollback functionality 116 | 117 | ## Error Handling 118 | 119 | All library functions follow these conventions: 120 | 121 | - Return 0 on success, 1 on error 122 | - Log errors using log_error function 123 | - Use log_debug for diagnostic information 124 | - Validate inputs before processing 125 | - Provide helpful error messages 126 | 127 | ## Configuration Variables 128 | 129 | Key global variables (set by config.sh): 130 | 131 | **Cluster Configuration:** 132 | 133 | - `CREATE_CLUSTER` - Boolean flag for cluster creation (default: false, uses existing) 134 | - `CLUSTER_NAME` - AKS cluster name (required) 135 | - `RESOURCE_GROUP` - Azure resource group (required) 136 | - `LOCATION` - Azure region (required for new clusters) 137 | - `NODE_COUNT` - Number of cluster nodes (default: 1) 138 | - `NODE_SIZE` - VM size for nodes (default: Standard_DS2_v2) 139 | 140 | **Repository Configuration:** 141 | 142 | - `CREATE_REPOS` - Boolean flag for repository creation 143 | - `CONTROL_PLANE_REPO` - Repository name when creating, or full URL when using existing 144 | - `GITOPS_REPO` - Repository name when creating, or full URL when using existing 145 | - `GITHUB_ORG` - GitHub organization (optional, defaults to user account) 146 | 147 | **Other:** 148 | 149 | - `KALYPSO_NAMESPACE` - Kubernetes namespace for Kalypso (default: kalypso-system) 150 | - `INTERACTIVE_MODE` - Boolean for interactive prompts (default: true) 151 | - `AUTO_ROLLBACK` - Boolean for automatic rollback on failure (default: false) 152 | 153 | ## Documentation 154 | 155 | Comprehensive documentation is available in `docs/bootstrap/`: 156 | 157 | - [README.md](../../docs/bootstrap/README.md) - Main documentation 158 | - [prerequisites.md](../../docs/bootstrap/prerequisites.md) - Prerequisites and setup 159 | - [quickstart.md](../../docs/bootstrap/quickstart.md) - Quick start guide 160 | - [troubleshooting.md](../../docs/bootstrap/troubleshooting.md) - Common issues and solutions 161 | 162 | ## Contributing 163 | 164 | When contributing to the bootstrap script: 165 | 166 | 1. Maintain consistent coding style 167 | 2. Follow existing patterns in the codebase 168 | 3. Add logging at appropriate levels 169 | 4. Handle errors gracefully 170 | 5. Update documentation 171 | 6. Add tests for new functionality 172 | 7. Run shellcheck and fix all warnings/errors 173 | 174 | ## License 175 | 176 | This script is part of the Kalypso Scheduler project and follows the same license. 177 | -------------------------------------------------------------------------------- /docs/run-books/platform-team-onboards-a-new-cluster.md: -------------------------------------------------------------------------------- 1 | # Platform Team Onboards a New Cluster 2 | 3 | - [Platform Team Onboards a New Cluster](#platform-team-onboards-a-new-cluster) 4 | - [Prerequisites](#prerequisites) 5 | - [1. Provision a Kubernetes Cluster](#1-provision-a-kubernetes-cluster) 6 | - [2. Gather GitOps Details](#2-gather-gitops-details) 7 | - [Steps](#steps) 8 | - [1. Install and Configure a GitOps Agent](#1-install-and-configure-a-gitops-agent) 9 | - [2. (Optional) Install other Components](#2-optional-install-other-components) 10 | - [Next Steps](#next-steps) 11 | 12 | ## Prerequisites 13 | 14 | ### 1. Provision a Kubernetes Cluster 15 | 16 | This run book assumes access to an existing Kubernetes cluster. This could be a managed AKS cluster in Azure, or an Arc-Enabled connected Kubernetes cluster. 17 | 18 | - For a managed AKS cluster, see [Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using Azure portal](https://learn.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-portal) 19 | - For an Arc-Enabled connected Kubernetes cluster, see [Quickstart: Connect an existing Kubernetes cluster to Azure Arc](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/quickstart-connect-cluster?tabs=azure-cli) 20 | 21 | Once the cluster is provisioned, locate and verify access to the cluster resource in the Azure portal. Verify your Entra ID access to ARM and Kubernetes resources by viewing Kubernetes resources (namespaces, workloads, etc.) 22 | 23 | ![k3scluster](./images/k3scluster.png) 24 | 25 | Collect the following values for use in this runbook. 26 | 27 | | Variable | Description | 28 | | --------------------- | -------------------------------------------------------------------------------------------------------------------- | 29 | | `RESOURCE_GROUP_NAME` | The name of the resource group that the Kubernetes cluster is in | 30 | | `SUBSCRIPTION` | The name of the subscription that the Kubernetes cluster is in | 31 | | `CLUSTER_NAME` | The name of the Kubernetes cluster resource | 32 | | `NAMESPACE` | The Kubernetes namespace to deploy GitOps configuration into | 33 | | `CLUSTER_TYPE` | `connectedClusters` if this is an Arc-connected cluster, `managedClusters` if this is an AKS cluster hosted in Azure | 34 | 35 | ### 2. Gather GitOps Details 36 | 37 | Identify the Platform GitOps repository details for this cluster. 38 | 39 | This run book assumes an existing Platform GitOps repository is set up. Gather the following details for this cluster as they will be used when configuring the GitOps agent on the cluster. These 3 variables together reference a location in git for the GitOps agent to monitor. 40 | 41 | | Varaible | Description | 42 | | --------------- | ---------------------------------------------------------------------- | 43 | | `GITOPS_REPO` | The git repository URL for the Platform GitOps repo | 44 | | `GITOPS_BRANCH` | The git branch in the Platform GitOps repo | 45 | | `GITOPS_PATH` | The file path to monitor on the git branch of the Platform GitOps repo | 46 | 47 | ## Steps 48 | 49 | ### 1. Install and Configure a GitOps Agent 50 | 51 | In order for a cluster to host workload deployments managed by Kalypso, it will need a GitOps agent. These runbooks will use the [Flux v2 AKS extension](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/conceptual-gitops-flux2) as the GitOps agent. This will ensure support for the [Kalypso Obserability Hub](https://github.com/microsoft/kalypso-observability-hub). 52 | 53 | To use another GitOps agent (like [ArgoCD](https://argo-cd.readthedocs.io/en/stable/)), work with the Platform team to ensure that proper [Reconciler templates](https://github.com/microsoft/kalypso-scheduler?tab=readme-ov-file#reconciler-template) are set up. 54 | 55 | Have a GitHub token with the readonly permissions to the Platform GitOps and application GitOps repositories. This token will be used by Flux to pull manifests to the clusters. 56 | 57 | Run the following Azure CLI command to create the Flux configuration on this cluster. Reference the variables gathered from Prerequisites. 58 | 59 | ```sh 60 | kubectl create secret generic repo-secret \ 61 | --from-literal=username=kalypso \ 62 | --from-literal=password= \ 63 | -n flux-system 64 | 65 | az k8s-configuration flux create \ 66 | --resource-group \ 67 | --cluster-name \ 68 | --name --gitops \ 69 | --namespace flux-system \ 70 | --cluster-type \ 71 | --local-auth-ref repo-secret \ 72 | --scope cluster \ 73 | --url \ 74 | --branch \ 75 | --kustomization name=platform path= prune=true \ 76 | --subscription 77 | ``` 78 | 79 | See [Tutorial: Deploy applications using GitOps with Flux v2](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/tutorial-use-gitops-flux2?tabs=azure-cli) for instructions on getting started with GitOps and Flux v2. 80 | 81 | ![k3scluster-gitops](./images/k3scluster-gitops.png) 82 | 83 | ### 2. (Optional) Install other Components 84 | 85 | This would be a good time to install and configure other platform level components for clusters in the fleet. 86 | 87 | For Kalypso, only the GitOps agent is required, but other standardized components for the fleet might include: 88 | 89 | - Kubernetes operators 90 | - Secret management components 91 | - Observability components 92 | 93 | ## Next Steps 94 | 95 | Once all steps have been completed, the cluster is ready to host applications. If the platform GitOps configuration provided in step 1 already existed, the cluster may already be pulling and running applications scheduled for the appropriate environment. 96 | 97 | Otherwise, see [Schedule Applications for Deployment onto Clusters](./platform-team-schedules-applications-for-deployment.md) for instructions on how to deploy specific applications to the new cluster. 98 | -------------------------------------------------------------------------------- /docs/bootstrap/quickstart.md: -------------------------------------------------------------------------------- 1 | # Kalypso Bootstrap Quickstart Guide 2 | 3 | Get up and running with Kalypso Scheduler in under 15 minutes. 4 | 5 | ## Prerequisites Check 6 | 7 | Before starting, verify you have: 8 | 9 | - [x] kubectl (>= 1.20.0) 10 | - [x] Azure CLI (>= 2.30.0) 11 | - [x] git (>= 2.0.0) 12 | - [x] Helm (>= 3.0.0) 13 | - [x] Azure account with subscription access 14 | - [x] GitHub personal access token 15 | 16 | See [prerequisites.md](prerequisites.md) for detailed installation instructions. 17 | 18 | ## Quick Setup (5 Minutes) 19 | 20 | ### 1. Authenticate 21 | 22 | Login to Azure: 23 | 24 | ```bash 25 | az login 26 | az account set --subscription "your-subscription-id" 27 | ``` 28 | 29 | Set GitHub token: 30 | 31 | ```bash 32 | export GITHUB_TOKEN="your-github-personal-access-token" 33 | ``` 34 | 35 | ### 2. Run Bootstrap Script 36 | 37 | Create everything new (interactive mode): 38 | 39 | ```bash 40 | cd scripts/bootstrap 41 | ./bootstrap.sh 42 | ``` 43 | 44 | The script will: 45 | 46 | 1. ✓ Check prerequisites 47 | 2. ✓ Prompt for configuration 48 | 3. ✓ Create AKS cluster (~10 minutes) 49 | 4. ✓ Create GitHub repositories 50 | 5. ✓ Install Kalypso Scheduler 51 | 6. ✓ Verify installation 52 | 53 | ### 3. Verify Installation 54 | 55 | Check that everything is running: 56 | 57 | ```bash 58 | # Get cluster credentials (if not already done) 59 | az aks get-credentials \ 60 | --resource-group kalypso-rg \ 61 | --name kalypso-cluster 62 | 63 | # Check Kalypso pods 64 | kubectl get pods -n kalypso-system 65 | 66 | # Expected output: 67 | # NAME READY STATUS RESTARTS AGE 68 | # kalypso-scheduler-controller-manager-xxxxx-xxxxx 2/2 Running 0 2m 69 | 70 | # Check CRDs 71 | kubectl get crd | grep kalypso 72 | 73 | # Expected output: 74 | # deploymenttargets.scheduler.kalypso.io 75 | # workloads.scheduler.kalypso.io 76 | # schedulingpolicies.scheduler.kalypso.io 77 | # ... (and more) 78 | ``` 79 | 80 | ## Deploy Your First Workload 81 | 82 | ### 1. Clone Control Plane Repository 83 | 84 | ```bash 85 | cd ~ 86 | git clone https://github.com/YOUR_USER/kalypso-control-plane 87 | cd kalypso-control-plane 88 | ``` 89 | 90 | ### 2. Create a Simple Workload 91 | 92 | Create a file `workloads/hello-world.yaml`: 93 | 94 | ```yaml 95 | apiVersion: scheduler.kalypso.io/v1alpha1 96 | kind: WorkloadRegistration 97 | metadata: 98 | name: helloworld 99 | labels: 100 | type: application 101 | why: sample 102 | spec: 103 | workload: 104 | repo: https://github.com/YOUR_USER/hello-world-app 105 | branch: main 106 | path: ./workload 107 | workspace: hello-world 108 | ``` 109 | 110 | ### 3. Commit and Push 111 | 112 | ```bash 113 | git add workloads/hello-world.yaml 114 | git commit -m "Add hello-world workload" 115 | git push origin main 116 | ``` 117 | 118 | ### 4. Check GitOps Repository 119 | 120 | Kalypso will create a PR with the new clusters and assignments in the GitOps repository. 121 | 122 | ```bash 123 | cd ~/kalypso-gitops 124 | git pull origin main 125 | 126 | # You should see new directories for cluster types and deployment targets 127 | ls -R clusters/ 128 | ``` 129 | 130 | ## Common Workflows 131 | 132 | ### Using Existing AKS Cluster 133 | 134 | ```bash 135 | ./bootstrap.sh \ 136 | --cluster-name my-existing-cluster \ 137 | --resource-group my-existing-rg \ 138 | --create-repos 139 | ``` 140 | 141 | ### Using Existing Repositories 142 | 143 | ```bash 144 | ./bootstrap.sh \ 145 | --create-cluster \ 146 | --control-plane-repo https://github.com/myorg/control-plane \ 147 | --gitops-repo https://github.com/myorg/gitops 148 | ``` 149 | 150 | ### Automated Setup (CI/CD) 151 | 152 | Create a config file `kalypso-config.yaml`: 153 | 154 | ```yaml 155 | cluster: 156 | create: true 157 | name: kalypso-prod 158 | resourceGroup: kalypso-prod-rg 159 | location: westus2 160 | nodeCount: 5 161 | nodeSize: Standard_DS3_v2 162 | 163 | repositories: 164 | create: true 165 | controlPlane: my-control-plane 166 | gitops: my-gitops 167 | 168 | github: 169 | org: my-organization 170 | ``` 171 | 172 | Run non-interactively: 173 | 174 | ```bash 175 | export AZURE_SUBSCRIPTION_ID="xxx" 176 | export GITHUB_TOKEN="xxx" 177 | 178 | ./bootstrap.sh --config kalypso-config.yaml --non-interactive 179 | ``` 180 | 181 | ## Troubleshooting 182 | 183 | ### Bootstrap fails during cluster creation 184 | 185 | Check Azure quotas: 186 | 187 | ```bash 188 | az vm list-usage --location eastus --output table 189 | ``` 190 | 191 | ### Kalypso pods not starting 192 | 193 | Check pod logs: 194 | 195 | ```bash 196 | kubectl logs -n kalypso-system -l app=kalypso-scheduler 197 | ``` 198 | 199 | ### Workload not being scheduled 200 | 201 | Check Kalypso logs and scheduling policies: 202 | 203 | ```bash 204 | kubectl logs -n kalypso-system deployment/kalypso-scheduler-controller-manager 205 | kubectl get schedulingpolicies -n kalypso-system 206 | ``` 207 | 208 | For more troubleshooting, see [troubleshooting.md](troubleshooting.md). 209 | 210 | ## Cleanup 211 | 212 | To remove everything created by the bootstrap script, use the automatic cleanup option: 213 | 214 | ```bash 215 | # Automatic cleanup (interactive - will prompt for confirmation) 216 | cd scripts/bootstrap 217 | ./bootstrap.sh --cleanup 218 | 219 | # Non-interactive cleanup 220 | ./bootstrap.sh --cleanup --non-interactive 221 | ``` 222 | 223 | This will delete: 224 | 225 | - Kalypso Scheduler installation (Helm release) 226 | - Namespace: kalypso-system 227 | - AKS cluster 228 | - Resource group (with confirmation) 229 | - GitHub repositories (if created by script) 230 | 231 | ## Getting Help 232 | 233 | - **Documentation**: [README.md](README.md) 234 | - **Issues**: 235 | - **Discussions**: 236 | 237 | ## Success Metrics 238 | 239 | After completing this quickstart, you should be able to: 240 | 241 | - [x] Bootstrap Kalypso infrastructure in under 15 minutes 242 | - [x] Deploy a workload using WorkloadRegistration 243 | - [x] See Kalypso schedule workloads to clusters 244 | - [x] View generated manifests in GitOps repository 245 | - [x] Understand the basic Kalypso workflow 246 | 247 | Congratulations! You now have a working Kalypso Scheduler environment. 🎉 248 | -------------------------------------------------------------------------------- /scripts/bootstrap/lib/install.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # Installation functions for Kalypso Scheduler 3 | # Handles Helm chart installation and verification 4 | 5 | ####################################### 6 | # Install Kalypso Scheduler using Helm 7 | # Globals: 8 | # CONTROL_PLANE_REPO, KALYPSO_NAMESPACE, GH_REPO_TOKEN 9 | # Arguments: 10 | # None 11 | # Returns: 12 | # 0 on success, 1 on error 13 | ####################################### 14 | install_kalypso() { 15 | local namespace="${KALYPSO_NAMESPACE:-kalypso-system}" 16 | local release_name="kalypso-scheduler" 17 | 18 | log_info "Installing Kalypso Scheduler..." "install" 19 | 20 | # Get read-only GitHub token for Kalypso 21 | local gh_repo_token="${GH_REPO_TOKEN:-}" 22 | if [[ -z "$gh_repo_token" ]]; then 23 | log_info "GH_REPO_TOKEN not set, using GITHUB_TOKEN (ensure it has read-only repo permissions)" "install" 24 | gh_repo_token="${GITHUB_TOKEN}" 25 | fi 26 | 27 | if [[ -z "$gh_repo_token" ]]; then 28 | log_error "GitHub token required for Kalypso installation" "install" 29 | log_error "Set GH_REPO_TOKEN (read-only repo) or GITHUB_TOKEN" "install" 30 | return 1 31 | fi 32 | 33 | # Extract control plane repo URL from full GitHub URL 34 | local control_plane_url="${CONTROL_PLANE_REPO}" 35 | # Remove trailing .git if present 36 | control_plane_url="${control_plane_url%.git}" 37 | 38 | # Use Kalypso Helm chart from GitHub Pages 39 | local chart_repo="https://microsoft.github.io/kalypso-scheduler" 40 | local chart_name="kalypso-scheduler" 41 | 42 | # Add Helm repository 43 | log_info "Adding Kalypso Helm repository..." "install" 44 | if ! helm repo add kalypso "$chart_repo" 2>/dev/null; then 45 | log_warning "Helm repository may already exist, continuing..." "install" 46 | fi 47 | 48 | # Update Helm repositories (don't fail if some repos are unavailable) 49 | log_info "Updating Helm repositories..." "install" 50 | helm repo update 2>/dev/null || log_warning "Some Helm repositories failed to update, continuing..." "install" 51 | 52 | # Install or upgrade Kalypso 53 | log_info "Installing Helm chart from $chart_repo..." "install" 54 | log_info "Control Plane URL: $control_plane_url" "install" 55 | log_info "Control Plane Branch: main" "install" 56 | 57 | if ! helm upgrade --devel --install "$release_name" "kalypso/$chart_name" \ 58 | --namespace "$namespace" \ 59 | --create-namespace \ 60 | --set controlPlaneURL="$control_plane_url" \ 61 | --set controlPlaneBranch="main" \ 62 | --set ghRepoToken="$gh_repo_token"; then 63 | log_error "Helm installation failed" "install" 64 | return 1 65 | fi 66 | 67 | log_success "Kalypso Scheduler installation initiated" 68 | log_info "Note: Pods may take a few minutes to become ready" "install" 69 | return 0 70 | } 71 | 72 | ####################################### 73 | # Verify Kalypso installation 74 | # Arguments: 75 | # None 76 | # Returns: 77 | # 0 if verification passes, 1 otherwise 78 | ####################################### 79 | verify_installation() { 80 | local namespace="kalypso-system" 81 | 82 | log_info "Verifying Kalypso installation..." "install" 83 | 84 | # Check if namespace exists 85 | if ! kubectl get namespace "$namespace" &> /dev/null; then 86 | log_error "Namespace $namespace not found" "install" 87 | return 1 88 | fi 89 | 90 | # Check if deployment exists 91 | local deployment_name="kalypso-scheduler-controller-manager" 92 | if ! kubectl get deployment "$deployment_name" -n "$namespace" &> /dev/null; then 93 | log_warning "Deployment $deployment_name not found" "install" 94 | return 1 95 | fi 96 | 97 | # Check if pods are running 98 | local ready_replicas 99 | ready_replicas=$(kubectl get deployment "$deployment_name" -n "$namespace" \ 100 | -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0") 101 | 102 | if [[ "$ready_replicas" -lt 1 ]]; then 103 | log_warning "No ready replicas for $deployment_name" "install" 104 | return 1 105 | fi 106 | 107 | log_success "Kalypso Scheduler is running ($ready_replicas replicas ready)" 108 | 109 | # Check CRDs 110 | log_info "Checking CRDs..." "install" 111 | local crds_found=0 112 | local expected_crds=( 113 | "workloads.scheduler.kalypso.io" 114 | "deploymenttargets.scheduler.kalypso.io" 115 | "schedulingpolicies.scheduler.kalypso.io" 116 | ) 117 | 118 | for crd in "${expected_crds[@]}"; do 119 | if kubectl get crd "$crd" &> /dev/null; then 120 | log_success "CRD found: $crd" 121 | crds_found=$((crds_found + 1)) 122 | else 123 | log_warning "CRD not found: $crd" "install" 124 | fi 125 | done 126 | 127 | if [[ $crds_found -eq 0 ]]; then 128 | log_error "No Kalypso CRDs found" "install" 129 | return 1 130 | fi 131 | 132 | return 0 133 | } 134 | 135 | ####################################### 136 | # Rollback Kalypso installation 137 | # Arguments: 138 | # None 139 | # Returns: 140 | # None 141 | ####################################### 142 | rollback_kalypso() { 143 | local namespace="kalypso-system" 144 | local release_name="kalypso-scheduler" 145 | 146 | log_warning "Rolling back Kalypso installation..." "install" 147 | 148 | # Uninstall Helm release 149 | if helm list -n "$namespace" | grep -q "$release_name"; then 150 | log_info "Uninstalling Helm release: $release_name" "install" 151 | helm uninstall "$release_name" -n "$namespace" || true 152 | fi 153 | 154 | # Delete namespace 155 | if kubectl get namespace "$namespace" &> /dev/null; then 156 | log_info "Deleting namespace: $namespace" "install" 157 | kubectl delete namespace "$namespace" --timeout=60s || true 158 | fi 159 | } 160 | 161 | ####################################### 162 | # Main rollback function for all resources 163 | # Globals: 164 | # CREATED_RESOURCES 165 | # Arguments: 166 | # None 167 | # Returns: 168 | # None 169 | ####################################### 170 | rollback_resources() { 171 | log_warning "Starting rollback of created resources..." "rollback" 172 | 173 | # Rollback in reverse order: install -> repos -> cluster 174 | rollback_kalypso 175 | 176 | if declare -f rollback_repositories &> /dev/null; then 177 | rollback_repositories 178 | fi 179 | 180 | if declare -f rollback_cluster &> /dev/null; then 181 | rollback_cluster 182 | fi 183 | 184 | log_info "Rollback completed" "rollback" 185 | } 186 | 187 | # Export functions 188 | export -f install_kalypso verify_installation rollback_kalypso rollback_resources 189 | -------------------------------------------------------------------------------- /.markdownlint.yaml: -------------------------------------------------------------------------------- 1 | # Markdownlint YAML configuration 2 | # Default source: https://github.com/DavidAnson/markdownlint/blob/main/schema/.markdownlint.yaml 3 | 4 | # Default state for all rules 5 | default: true 6 | 7 | # Path to configuration file to extend 8 | extends: null 9 | 10 | # MD001/heading-increment/header-increment - Heading levels should only increment by one level at a time 11 | MD001: true 12 | 13 | # MD002/first-heading-h1/first-header-h1 - First heading should be a top-level heading 14 | MD002: 15 | # Heading level 16 | level: 1 17 | 18 | # MD003/heading-style/header-style - Heading style 19 | MD003: 20 | # Heading style 21 | style: "consistent" 22 | 23 | # MD004/ul-style - Unordered list style 24 | MD004: 25 | # List style 26 | style: "consistent" 27 | 28 | # MD005/list-indent - Inconsistent indentation for list items at the same level 29 | MD005: true 30 | 31 | # MD006/ul-start-left - Consider starting bulleted lists at the beginning of the line 32 | MD006: true 33 | 34 | # MD007/ul-indent - Unordered list indentation 35 | MD007: 36 | # Spaces for indent 37 | indent: 2 38 | # Whether to indent the first level of the list 39 | start_indented: false 40 | # Spaces for first level indent (when start_indented is set) 41 | start_indent: 2 42 | 43 | # MD009/no-trailing-spaces - Trailing spaces 44 | MD009: 45 | # Spaces for line break 46 | br_spaces: 2 47 | # Allow spaces for empty lines in list items 48 | list_item_empty_lines: false 49 | # Include unnecessary breaks 50 | strict: false 51 | 52 | # MD010/no-hard-tabs - Hard tabs 53 | MD010: 54 | # Include code blocks 55 | code_blocks: true 56 | # Number of spaces for each hard tab 57 | spaces_per_tab: 1 58 | 59 | # MD011/no-reversed-links - Reversed link syntax 60 | MD011: true 61 | 62 | # MD012/no-multiple-blanks - Multiple consecutive blank lines 63 | MD012: 64 | # Consecutive blank lines 65 | maximum: 1 66 | 67 | # MD013/line-length - Line length 68 | MD013: false 69 | # MD013: 70 | # # Number of characters 71 | # line_length: 80 72 | # # Number of characters for headings 73 | # heading_line_length: 80 74 | # # Number of characters for code blocks 75 | # code_block_line_length: 80 76 | # # Include code blocks 77 | # code_blocks: true 78 | # # Include tables 79 | # tables: true 80 | # # Include headings 81 | # headings: true 82 | # # Include headings 83 | # headers: true 84 | # # Strict length checking 85 | # strict: false 86 | # # Stern length checking 87 | # stern: false 88 | 89 | # MD014/commands-show-output - Dollar signs used before commands without showing output 90 | MD014: true 91 | 92 | # MD018/no-missing-space-atx - No space after hash on atx style heading 93 | MD018: true 94 | 95 | # MD019/no-multiple-space-atx - Multiple spaces after hash on atx style heading 96 | MD019: true 97 | 98 | # MD020/no-missing-space-closed-atx - No space inside hashes on closed atx style heading 99 | MD020: true 100 | 101 | # MD021/no-multiple-space-closed-atx - Multiple spaces inside hashes on closed atx style heading 102 | MD021: true 103 | 104 | # MD022/blanks-around-headings/blanks-around-headers - Headings should be surrounded by blank lines 105 | MD022: 106 | # Blank lines above heading 107 | lines_above: 1 108 | # Blank lines below heading 109 | lines_below: 1 110 | 111 | # MD023/heading-start-left/header-start-left - Headings must start at the beginning of the line 112 | MD023: true 113 | 114 | # MD024/no-duplicate-heading/no-duplicate-header - Multiple headings with the same content 115 | # MD024: false 116 | MD024: 117 | # Only check sibling headings 118 | allow_different_nesting: false 119 | # Only check sibling headings 120 | siblings_only: true 121 | 122 | # MD025/single-title/single-h1 - Multiple top-level headings in the same document 123 | MD025: 124 | # Heading level 125 | level: 1 126 | # RegExp for matching title in front matter 127 | front_matter_title: "^\\s*title\\s*[:=]" 128 | 129 | # MD026/no-trailing-punctuation - Trailing punctuation in heading 130 | MD026: 131 | # Punctuation characters 132 | punctuation: ".,;:!。，；：！" 133 | 134 | # MD027/no-multiple-space-blockquote - Multiple spaces after blockquote symbol 135 | MD027: true 136 | 137 | # MD028/no-blanks-blockquote - Blank line inside blockquote 138 | MD028: true 139 | 140 | # MD029/ol-prefix - Ordered list item prefix 141 | MD029: 142 | # List style 143 | style: "one_or_ordered" 144 | 145 | # MD030/list-marker-space - Spaces after list markers 146 | MD030: 147 | # Spaces for single-line unordered list items 148 | ul_single: 1 149 | # Spaces for single-line ordered list items 150 | ol_single: 1 151 | # Spaces for multi-line unordered list items 152 | ul_multi: 1 153 | # Spaces for multi-line ordered list items 154 | ol_multi: 1 155 | 156 | # MD031/blanks-around-fences - Fenced code blocks should be surrounded by blank lines 157 | MD031: 158 | # Include list items 159 | list_items: true 160 | 161 | # MD032/blanks-around-lists - Lists should be surrounded by blank lines 162 | MD032: true 163 | 164 | # MD033/no-inline-html - Inline HTML 165 | MD033: 166 | # Allowed elements 167 | allowed_elements: ["br"] 168 | 169 | # MD034/no-bare-urls - Bare URL used 170 | MD034: true 171 | 172 | # MD035/hr-style - Horizontal rule style 173 | MD035: 174 | # Horizontal rule style 175 | style: "consistent" 176 | 177 | # MD036/no-emphasis-as-heading/no-emphasis-as-header - Emphasis used instead of a heading 178 | MD036: 179 | # Punctuation characters 180 | punctuation: ".,;:!?。，；：！？" 181 | 182 | # MD037/no-space-in-emphasis - Spaces inside emphasis markers 183 | MD037: true 184 | 185 | # MD038/no-space-in-code - Spaces inside code span elements 186 | MD038: true 187 | 188 | # MD039/no-space-in-links - Spaces inside link text 189 | MD039: true 190 | 191 | # MD040/fenced-code-language - Fenced code blocks should have a language specified 192 | MD040: true 193 | 194 | # MD041/first-line-heading/first-line-h1 - First line in a file should be a top-level heading 195 | MD041: 196 | # Heading level 197 | level: 1 198 | # RegExp for matching title in front matter 199 | front_matter_title: "^\\s*title\\s*[:=]" 200 | 201 | # MD042/no-empty-links - No empty links 202 | MD042: true 203 | 204 | # MD043/required-headings/required-headers - Required heading structure 205 | # MD043: 206 | # # List of headings 207 | # headings: [] 208 | # # List of headings 209 | # headers: [] 210 | 211 | # MD044/proper-names - Proper names should have the correct capitalization 212 | MD044: 213 | # List of proper names 214 | names: [] 215 | # Include code blocks 216 | code_blocks: true 217 | 218 | # MD045/no-alt-text - Images should have alternate text (alt text) 219 | MD045: true 220 | 221 | # MD046/code-block-style - Code block style 222 | MD046: 223 | # Block style 224 | style: "consistent" 225 | 226 | # MD047/single-trailing-newline - Files should end with a single newline character 227 | MD047: true 228 | 229 | # MD048/code-fence-style - Code fence style 230 | MD048: 231 | # Code fence style 232 | style: "consistent" 233 | 234 | # MD049/emphasis-style - Emphasis style should be consistent 235 | MD049: false 236 | 237 | # MD050/strong-style - Strong style should be consistent 238 | MD050: 239 | # Strong style should be consistent 240 | style: "consistent" 241 | 242 | -------------------------------------------------------------------------------- /docs/run-books/application-team-creates-a-new-application-ring.md: -------------------------------------------------------------------------------- 1 | # Application Team Creates a New Application Ring 2 | 3 | - [Application Team Creates a New Application Ring](#application-team-creates-a-new-application-ring) 4 | - [Overview](#overview) 5 | - [Prerequisites](#prerequisites) 6 | - [1. Admin Access to Application Repositories](#1-admin-access-to-application-repositories) 7 | - [2. Admin Access to Deployment Observability Dashboards](#2-admin-access-to-deployment-observability-dashboards) 8 | - [Steps](#steps) 9 | - [1. Create a new GitOps Branch](#1-create-a-new-gitops-branch) 10 | - [2. Create a new Config Branch](#2-create-a-new-config-branch) 11 | - [3. Merge GitOps PR](#3-merge-gitops-pr) 12 | - [4. Link Environment in Promotion Sequence](#4-link-environment-in-promotion-sequence) 13 | - [5. Update Deployment Observability Dashboards](#5-update-deployment-observability-dashboards) 14 | - [Next Steps](#next-steps) 15 | 16 | ## Overview 17 | 18 | This run book describes how to manage deployment rings for applications. Rings can be used to create separate scopes and stages for deploying applications out to clusters. This creates an additional promotion flow within the platform environments. 19 | 20 | Rings and their promotion flow are unique to each application. In this run book, we will walk through the process for including a new ring for a single application. 21 | 22 | For information on managing environments, see [Platform Team Creates a New Environment](./platform-team-creates-a-new-environment.md). 23 | 24 | ## Prerequisites 25 | 26 | ### 1. Admin Access to Application Repositories 27 | 28 | Each application consists of 3 git repositories. A source repository, a configuration repository, and a GitOps repository. Identify all 3 repositories and make sure you have admin access to all 3. Admin access is required to create and modify GitHub workflows and variables. 29 | 30 | ### 2. Admin Access to Deployment Observability Dashboards 31 | 32 | The deployment observability dashboards for the [Deployment Observability Hub](https://github.com/microsoft/kalypso-observability-hub) show the deployment status for applications across environments, rings, and clusters. In this runbook, we will expand this dashboard to include new rings. 33 | 34 | ![deployment-observability-hub](./images/deployment-observability-hub.png) 35 | 36 | ## Steps 37 | 38 | ### 1. Create a new GitOps Branch 39 | 40 | First, create a branch in the application's GitOps repository. This repo has a branch for every ring and environment combination that holds relevant GitOps manifest files. 41 | 42 | Inside the GitOps repository, create a new git branch based off of the latest ring. In this run book, we will create `dev-red` based off `dev-green`. 43 | 44 | ```sh 45 | # switch to the existing base branch 46 | git checkout dev-green 47 | 48 | # create a new branch based on the existing branch 49 | git checkout -b dev-red 50 | ``` 51 | 52 | Remove all of the generated GitOps manifests files from the previous branch. The exact command will depend on the config repository structure. 53 | 54 | Finally, push the new branch to GitHub. 55 | 56 | ```sh 57 | git add . 58 | git commit -m "Prepare new ring branch" 59 | git push -u origin dev-red 60 | ``` 61 | 62 | Once complete, there should be a new branch in GitHub named `dev-red`. This branch will only contain the `.github/*` files. 63 | 64 | ![new-branch-ring-created](./images/new-branch-ring-created.png) 65 | 66 | ### 2. Create a new Config Branch 67 | 68 | Next, create a branch in the config repository. This repo has a branch for each environment and ring combination that holds configuration values for the environment & ring. 69 | 70 | Inside the config repository, create a new git branch based off the previous ring. In this runbook, we will create `dev-red` based off `dev-green`. 71 | 72 | ```sh 73 | # switch to the existing base branch 74 | git checkout dev-green 75 | 76 | # create a new branch based on the existing branch 77 | git checkout -b dev-red 78 | ``` 79 | 80 | In the new branch, update any values files to fit the new ring. The specific files and values will depend on the application. See [Application Team Manages Application Configuration](./application-team-manages-application-configuration.md) for more information on managing configuration files. 81 | 82 | When you are happy with the values, commit and push the changes to GitHub. 83 | 84 | ```sh 85 | git add . 86 | git commit -m "Prepare dev-red config" 87 | git push -u origin dev-red 88 | ``` 89 | 90 | ### 3. Merge GitOps PR 91 | 92 | After pushing the config branch to GitHub, a PR will automatically be generated against the new GitOps branch for this ring. This PR includes all generated GitOps manifests based on the configuration values in the config branch. 93 | 94 | Merge this PR, and the new ring is ready to use. 95 | 96 | ![alt text](./images/new-ring-pr.png) 97 | 98 | ### 4. Link Environment in Promotion Sequence 99 | 100 | Now that our git branches for the new ring are set up, we need to declare where this ring lives in the promotional sequence. This is defined by GitHub environments inside the application source repository. 101 | 102 | Each GitHub environment optionally contains a variable called `NEXT_ENVIRONMENT` that points to the next ring in the sequence. This creates a linked list that dictates how application changes are promoted through rings. 103 | 104 | We will update `dev-green` to include a new `NEXT_ENVIRONMENT` variable that points to the new `dev-red` ring. 105 | 106 | ![diagram of environment ring sequence](./images/next_ring.drawio.png) 107 | 108 | Update the previous GitHub Environment, `dev-green`, to point to the new `NEXT_ENVIRONMENT`, `dev-red`. 109 | 110 | ![next-new-ring-environment](./images/next-new-ring-environment.png) 111 | 112 | ### 5. Update Deployment Observability Dashboards 113 | 114 | In the Deployment Observability Dashboards, edit the Environment State table Ring column to show the new ring. 115 | 116 | ![deployment-observability-edit](./images/deployment-observability-edit.png) 117 | 118 | Update the value mappings to include a new value and color for the new ring. 119 | 120 | ![deployment-observability-edit-colors](./images/deployment-observability-edit-colors.png) 121 | 122 | Finally, save the dashboard changes. The dashboard JSON can be exported and persisted as configuration for the Observability Hub. 123 | 124 | ![deployment-observability-save](./images/deployment-observability-save.png) 125 | 126 | ## Next Steps 127 | 128 | This run book provides instructions for creating a new ring for an application. Rings allow applications to be promoted within platform environments. To create a new environment, see [Platform Team Creates a New Environment](./platform-team-creates-a-new-environment.md). 129 | 130 | After creating a new ring, only the GitOps manifest files are generated. To actually deploy this ring to a cluster and configure what clusters host applications in this ring, see [Platform Team Onboards a New Cluster](./platform-team-onboards-a-new-cluster.md) and [Platform Team Schedules Applications for Deployment](./platform-team-schedules-applications-for-deployment.md). 131 | 132 | Once the ring exists within the promotional sequence, the application will be promoted through this ring. To learn how to promote an application through rings and environments, see [Application Team Promotes a Change Through Environments](./application-team-promotes-a-change-through-environments.md). 133 | -------------------------------------------------------------------------------- /cicd/setup.md: -------------------------------------------------------------------------------- 1 | # Setup a GitOps CI/CD flow 2 | 3 | ## Have Manifest Templates 4 | 5 | The CD flow generates K8s manifests for each environment. It needs manifests templates for that. This setup expects that the application source repository contains Helm templates for that purpose. For example, like this [Hello World](https://github.com/microsoft/kalypso-app-src/tree/main/helm) application. 6 | 7 | It's not required to use Helm, though. You are free to go with your preferred templating approach, but in this case you'll have to update [generate-manifests.sh](../.github/workflows/templates/utils/generate-manifests.sh) utility to generate manifests in you way, instead of Helm. 8 | 9 | The Helm chart should be put in the `/helm` folder in the source repo. Example - [Helm chart of the Hello World application](https://github.com/microsoft/kalypso-app-src/blob/main/helm/values.yaml). The chart can be created manually or with the help of [Helmify](https://github.com/arttor/helmify). For example, you have a sample of the application manifests, including deployment, service, etc. Helmify can take that sample as an input and generate a Helm chart for you, automatically identifying what parameters it should have. You can use the generated Helm chart as is, or you can manually brush it up to make it cleaner, simpler, nicer. 10 | 11 | ## Bootstrap Config and GitOps repositories 12 | 13 | Run `setup.sh` script: 14 | 15 | Usage: 16 | 17 | ```sh 18 | ./setup.sh -o -r -e 19 | ``` 20 | 21 | Example: 22 | 23 | ```sh 24 | ./setup.sh -o eedorenko -r hello-world -e dev 25 | ``` 26 | 27 | The script requires the following environment variables to be set: 28 | 29 | - TOKEN: GitHub personal access token with the following minimum permissions: 30 | Actions - R/W 31 | Administration - R/W 32 | Commit statuses - R/W 33 | Contents - R/W 34 | Metadata - RO 35 | Pull requests - R/W 36 | Secrets - R/W 37 | Variables - R/W 38 | Workflows - R/W 39 | 40 | - AZURE_CREDENTIALS_SP: service principal azure credentials 41 | 42 | The script will do the following: 43 | 44 | - Create `gitops` repository with the name `/-gitops` (e.g. `eedorenko/hello-world-gitops`). If the repo already exists, it will take it. 45 | - Create a branch in the `gitops` repository with the environment name (e.g. `dev`) 46 | - Add necessary GH actions workflows to the `gitops` repository 47 | - Configure necessary variables and secrets in the `gitops` repository 48 | - Create `configs` repository with the name `/-configs` (e.g. `eedorenko/hello-world-configs`). If the repo already exists, it will take it. 49 | - Create a branch in the `configs` repository with the environment name (e.g. `dev`) 50 | - If the environment branch already exists in the `configs` repository and it contains any folders, the script leaves it as is. Otherwise it creates a folder `rename_me` with empty `values.yaml`. 51 | - Add necessary GH actions workflows to the `configs` repository 52 | - Configure necessary variables and secrets in the `configs` repository 53 | - Create the `source` repository if it doesn't exist. 54 | - Configure necessary variables and secrets in the `source` repository 55 | - Create a PR `GitOps CD setup` to the `source` repo with the necessary GH actions workflows and utility scripts 56 | 57 | Merge the `GitOps CD setup` PR into the `source` repository. 58 | 59 | ## Define application configurations 60 | 61 | The created `configs` repo contains application configuration values of the service for the deployment targets across environments. 62 | It belongs to and it is handled by the app dev team, working on the application. The nature of configs, provided here, is application centric. They determine the application behavior on various deployment targets across different environments/rings (e.g. logging level, number of replicas, feature flags, localizations, etc.). These configs are independent from the infra/platform configs. Learn more about [application and platform configs](https://learn.microsoft.com/azure/azure-arc/kubernetes/conceptual-workload-management#platform-configuration-concepts). 63 | 64 | ### Folder structure 65 | 66 | The folder structure in the `configs` repo is totally custom and may have any desired depth to group deployment targets in a flexible way. The correlation between a deployment target and clusters depends on the environment and app dev team preferences. 67 | 68 | The edge cases: 69 | 70 | - every deployment target represents a single cluster (e.g. `functional-testing`, `performance-testing`) - ok for low environments like `dev` or `qa` where there are a few clusters that are treated as "pets". With this approach every cluster may have unique *application* config values. 71 | - single deployment target represents all thousands of clusters in the environment/ring (`early-adopters`) - ok for high environments like `prod` where all clusters are treated as a "herd" and have identical *application* config values (while having different infra/platform configs) 72 | 73 | Examples: 74 | 75 | - `early-adopters/values.yaml` - all `early-adopters` clusters have same application config values stored in the `values.yaml` 76 | - `external-users/values.yaml`, `external-users/west/values.yaml`, `external-users/west/msu10/values.yaml` - all `external-users` clusters have same common application config values, but in addition to that, the clusters from the west region will apply/override config values specific for them. And the cluster `msu10` has very unique application config values which will be added in addition to the common and west config values. 77 | - `dispatcher/values.yaml`, `dispatcher/dev1/values.yaml`, `dispatcher/dev2/values.yaml` - `dev1` and `dev2` clusters have their own unique values that are applied on top of `dispatcher/values.yaml`. 78 | 79 | The folder structure in the `configs` repo determines the folder structure in the output `GitOps` repo. 80 | 81 | The bootstrap script creates `rename_me` folder in the `configs` repo environment branch with an empty `values.yaml` file in it. Normally, a Helm chart in the source repo contains `values.yaml` file with all possible application configurations that the Helm chart understands. Define what values you want to override at the environment/target level. For example, for the [hello-world](https://github.com/microsoft/kalypso-app-src/blob/main/helm/values.yaml) application we want to provide the following values for `functional-testing` clusters (target) in the `dev` environment. We want to rename `rename_me/values.yaml` to `functional-testing/values.yaml` and put the values in the file: 82 | 83 | ```yaml 84 | app: 85 | name: hello-world-functional 86 | replicas: 3 87 | ``` 88 | 89 | If the `values.yaml` file in the `configs` repo remains empty, the values from the source [values.yaml](https://github.com/microsoft/kalypso-app-src/blob/main/helm/values.yaml) will be taken to generate manifests for the deployment target in this environment. 90 | 91 | ## Add a new environment 92 | 93 | In order to add a new environment to the application you do the following: 94 | 95 | 1. Create branches with the environment name in the `configs` and `gitops` repos 96 | 2. Create /values.yaml files in the environment branch in the `configs` repo with the specific application values (or just leave it empty to take default values from the source repo). 97 | 3. Define an environment GitHub variable `NEXT_ENVIRONMENT` in the previous environment with the value of the new environment name. 98 | For example, we have already configured/bootstrapped environment `d2`. Now we are creating the next environment in the chain `t2`, so the promotion flow should go like `d2`->`t2`. We need to define `NEXT_ENVIRONMENT` variable in the `d2` environment: 99 | 100 | ![next-environment](../docs/images/next-environment.png) 101 | 102 | Steps #1 and #2 can be performed either manually (e.g. creating new `t2` branches from the `d2` branches) or by running the `setup` GitHub Actions workflow again, specifying a new environment name in the parameters. 103 | -------------------------------------------------------------------------------- /cicd/tutorial/deploy.sh: -------------------------------------------------------------------------------- 1 | set -eo pipefail 2 | parent_path=$( cd "$(dirname "${BASH_SOURCE[0]}")" ; pwd -P ) 3 | 4 | while getopts cdl:o:p:t: flag 5 | do 6 | case "${flag}" in 7 | c) ACTION="CREATE";; 8 | d) ACTION="DELETE";; 9 | p) PREFIX=${OPTARG};; 10 | o) ORG=${OPTARG};; 11 | t) export TOKEN=${OPTARG};; 12 | l) LOCATION=${OPTARG};; 13 | esac 14 | done 15 | 16 | # Comment out the following line to opt out telemetry collection 17 | export AZURE_HTTP_USER_AGENT="acce1e78-01E3-4354-9356-B033C290E069" 18 | 19 | gh_prefix="https://github.com" 20 | appsrc_repo_name=$ORG/$PREFIX 21 | appgitops_repo_name=$appsrc_repo_name-gitops 22 | appconfigs_repo_name=$appsrc_repo_name-configs 23 | rg_name=$PREFIX-rg 24 | left_cluster_name=$PREFIX-left-cluster 25 | right_cluster_name=$PREFIX-right-cluster 26 | 27 | check_src_repo() { 28 | gh repo view $appsrc_repo_name >/dev/null 2>&1 29 | if [ $? -eq 0 ]; then 30 | echo "Repository $appsrc_repo_name already exists" 31 | exit 1 32 | else 33 | echo "Creating repository $appsrc_repo_name" 34 | return 1 35 | fi 36 | } 37 | 38 | create_src_repository() { 39 | gh repo create $appsrc_repo_name --public 40 | rm -rf src 41 | git clone https://github.com/microsoft/kalypso-app-src src 42 | pushd src 43 | git remote add mine $gh_prefix/$appsrc_repo_name 44 | rm .github/workflows/cicd.yaml 45 | git add . 46 | git commit -m "Initial commit" 47 | git push mine main 48 | popd 49 | rm -rf src 50 | } 51 | 52 | ensure_src_repository() { 53 | check_src_repo || create_src_repository 54 | 55 | } 56 | 57 | update_configs_repo() { 58 | rm -rf configs 59 | git clone $gh_prefix/$appconfigs_repo_name configs 60 | pushd configs 61 | mv rename_me functional-test 62 | 63 | cat < functional-test/values.yaml 64 | app: 65 | name: hello-world-functional 66 | replicas: 3 67 | namespace: \$ENVIRONMENT 68 | EOF 69 | git add . 70 | git commit -m "Update functional-test" 71 | git config pull.rebase false 72 | git pull --no-edit 73 | git push 74 | popd 75 | rm -rf configs 76 | } 77 | 78 | create_gh_repositories() { 79 | ensure_src_repository 80 | $parent_path/../setup.sh -o $1 -r $2 -e dev 81 | update_configs_repo 82 | $parent_path/../setup.sh -o $1 -r $2 -e stage 83 | gh api --method PUT -H "Accept: application/vnd.github+json" repos/$appsrc_repo_name/environments/dev 84 | gh variable set NEXT_ENVIRONMENT -e dev -b stage -R $appsrc_repo_name 85 | 86 | gh pr merge 1 -s -R $gh_prefix/$appsrc_repo_name 87 | 88 | } 89 | 90 | delete_gh_repositories() { 91 | echo "Deleting GitHub repositories..." 92 | 93 | gh repo delete $gh_prefix/$appsrc_repo_name --yes 2> /dev/null 94 | gh repo delete $gh_prefix/$appgitops_repo_name --yes 2> /dev/null 95 | gh repo delete $gh_prefix/$appconfigs_repo_name --yes 2> /dev/null 96 | 97 | gh api -H "Accept: application/vnd.github+json" -X DELETE "/user/packages/container/$PREFIX%2Fhello-world" > /dev/null 2>&1 98 | 99 | } 100 | 101 | createAzureResources() { 102 | echo "Creating Azure resources..." 103 | az group create -n $rg_name -l $LOCATION 104 | create_flux_cluster_type $left_cluster_name 105 | create_flux_cluster_type $right_cluster_name 106 | } 107 | 108 | create_flux_cluster_type() { 109 | cluster_name=$1 110 | 111 | echo "Creating "$cluster_name" AKS cluster..." 112 | create_AKS_cluster $cluster_name 113 | 114 | az extension add -n k8s-configuration 115 | az extension add -n k8s-extension 116 | 117 | az k8s-configuration flux create \ 118 | --name cluster-config-dev \ 119 | --cluster-name $cluster_name \ 120 | --namespace flux-system \ 121 | --https-user flux \ 122 | --https-key $TOKEN \ 123 | --resource-group $rg_name \ 124 | -u $gh_prefix/$appgitops_repo_name \ 125 | --scope cluster \ 126 | --interval 10s \ 127 | --cluster-type managedClusters \ 128 | --branch dev \ 129 | --kustomization name=cluster-config-dev prune=true sync_interval=10s path=functional-test 130 | 131 | az k8s-configuration flux create \ 132 | --name cluster-config-stage \ 133 | --cluster-name $cluster_name \ 134 | --namespace flux-system \ 135 | --https-user flux \ 136 | --https-key $TOKEN \ 137 | --resource-group $rg_name \ 138 | -u $gh_prefix/$appgitops_repo_name \ 139 | --scope cluster \ 140 | --interval 10s \ 141 | --cluster-type managedClusters \ 142 | --branch stage \ 143 | --kustomization name=cluster-config-stage prune=true sync_interval=10s path=functional-test 144 | 145 | kubectl create ns dev 146 | kubectl create ns stage 147 | 148 | create_platform_config_map $cluster_name dev 149 | create_platform_config_map $cluster_name stage 150 | 151 | } 152 | 153 | create_platform_config_map() { 154 | cluster_name=$1 155 | environment=$2 156 | kubectl create configmap platform-config --from-literal=CLUSTER_NAME=$cluster_name \ 157 | --from-literal=REGION=westus2 \ 158 | --from-literal=ENVIRONMENT=$environment \ 159 | --from-literal=DATABASE_URL=https://database.mysql.com \ 160 | -n $environment 161 | } 162 | 163 | create_AKS_cluster() { 164 | cluster_name=$1 165 | az aks create -g $rg_name -n $cluster_name -l $LOCATION --node-count 1 --generate-ssh-keys 166 | az aks get-credentials -g $rg_name -n $cluster_name 167 | az aks show -g $rg_name -n $cluster_name -o table 168 | } 169 | 170 | 171 | deleteAzureResources() { 172 | echo "Deleting Azure resources..." 173 | az group delete -n $rg_name -y 2> /dev/null 174 | } 175 | 176 | 177 | create() { 178 | echo "---------------------------------" 179 | echo "Starting deployment. Time for a coffee break. It will take a few minutes..." 180 | 181 | create_gh_repositories $ORG $PREFIX 182 | createAzureResources 183 | echo "Deployment is complete!" 184 | echo "---------------------------------" 185 | echo "Created repositories:" 186 | echo " - "$gh_prefix/$appsrc_repo_name 187 | echo " - "$gh_prefix/$appgitops_repo_name 188 | echo " - "$gh_prefix/$appconfigs_repo_name 189 | echo "---------------------------------" 190 | echo "Created AKS clusters in "$rg_name" resource group:" 191 | echo " - $left_cluster_name" 192 | echo " - $right_cluster_name" 193 | echo "---------------------------------" 194 | } 195 | 196 | delete() { 197 | set +e 198 | echo "---------------------------------" 199 | echo "Starting deletion. It will take a few minutes..." 200 | deleteAzureResources 201 | delete_gh_repositories 202 | echo "Deletion is complete!" 203 | echo "---------------------------------" 204 | echo "Deleted repositories:" 205 | echo " - "$gh_prefix/$appsrc_repo_name 206 | echo " - "$gh_prefix/$appgitops_repo_name 207 | echo " - "$gh_prefix/$appconfigs_repo_name 208 | echo "---------------------------------" 209 | echo "Deleted AKS clusters in "$rg_name" resource group:" 210 | echo " - $left_cluster_name" 211 | echo " - $right_cluster_name" 212 | echo "---------------------------------" 213 | } 214 | 215 | print_usage() { 216 | printf "Usage: ./deploy.sh - -p -o -t -l \n" 217 | printf "ACTION: -c for create, -d for delete \n" 218 | printf "\nExamples: ./deploy.sh -c -p hello-world -o eedorenko -t ghp_19LPYNhx4Whcn6l3jzfyBIbE2Es0Kn -l westus2 \n" 219 | exit 1 220 | } 221 | 222 | check_prerequisites() { 223 | type -p gh >/dev/null 224 | type -p git >/dev/null 225 | } 226 | 227 | authenticate_gh() { 228 | echo "$TOKEN" > .githubtoken 229 | gh auth login --with-token < .githubtoken 230 | rm .githubtoken 231 | gh auth setup-git 232 | } 233 | 234 | 235 | print_prerequisites() { 236 | echo "The following tools are required to run this script:" 237 | echo " - gh" 238 | echo " - git" 239 | exit 1 240 | } 241 | 242 | 243 | if [ -z $LOCATION ] || [ -z $PREFIX ] || [ -z $ORG ] || [ -z $TOKEN ]; 244 | then 245 | print_usage 246 | fi 247 | 248 | check_prerequisites || print_prerequisites 249 | 250 | authenticate_gh 251 | 252 | 253 | if [ $ACTION == "CREATE" ]; 254 | then 255 | create 256 | elif [ $ACTION == "DELETE" ]; 257 | then 258 | delete 259 | else 260 | echo "Invalid action: "$ACTION 261 | print_usage 262 | fi 263 | -------------------------------------------------------------------------------- /cicd/setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Usage: setup.sh -o -r -e 4 | # Example: setup.sh -o liupeirong -r ConfiguratiX -e dev 5 | 6 | # The script requires the following environment variables to be set: 7 | # TOKEN: github personal access token with repo access 8 | # AZURE_CREDENTIALS_SP: service principal azure credentials 9 | 10 | set -eo pipefail 11 | parent_path=$( cd "$(dirname "${BASH_SOURCE[0]}")" ; pwd -P ) 12 | 13 | # Find templates directory by walking up the directory tree 14 | find_templates_dir() { 15 | local current_dir="$parent_path" 16 | 17 | while [[ "$current_dir" != "/" ]]; do 18 | local templates_path="$current_dir/.github/workflows/templates" 19 | if [[ -d "$templates_path" ]]; then 20 | echo "$templates_path" 21 | return 0 22 | fi 23 | current_dir="$(dirname "$current_dir")" 24 | done 25 | 26 | echo "Error: Could not find .github/workflows/templates directory" >&2 27 | echo "Searched upward from: $parent_path" >&2 28 | echo "No .github/workflows/templates directory found in any parent directory" >&2 29 | exit 1 30 | } 31 | 32 | templates_dir=$(find_templates_dir) 33 | 34 | while getopts o:r:s:e:d: flag 35 | do 36 | case "${flag}" in 37 | o) ORG=${OPTARG};; 38 | r) REPO=${OPTARG};; 39 | e) ENV=${OPTARG};; 40 | esac 41 | done 42 | 43 | print_usage() { 44 | printf "Usage: setup.sh -o -r -e \n" 45 | printf "\nExample: setup.sh -o eedorenko -r hello-world -e dev \n" 46 | printf "The script requires the following environment variables to be set: \n" 47 | printf " - TOKEN: github personal access token with repo access \n" 48 | printf " - AZURE_CREDENTIALS_SP: service principal azure credentials \n" 49 | 50 | exit 1 51 | } 52 | 53 | 54 | check_gh_username() { 55 | ghusername=$(git config --get user.name) || true 56 | ghuseremail=$(git config --get user.email) || true 57 | 58 | if [ -z "$ghusername" ] || [ -z "$ghuseremail" ]; 59 | then 60 | echo "Please configure your git username and email before running this script" 61 | echo "git config --global user.email \"you@example.com\"" 62 | echo "git config --global user.name \"Your Name\"" 63 | exit 1 64 | fi 65 | } 66 | 67 | 68 | check_prerequisites() { 69 | type -p gh >/dev/null 70 | type -p git >/dev/null 71 | } 72 | 73 | print_prerequisites() { 74 | echo "The following tools are required to run this script:" 75 | echo " - gh" 76 | echo " - git" 77 | exit 1 78 | } 79 | 80 | authenticate_gh() { 81 | gh auth setup-git 82 | } 83 | 84 | 85 | 86 | if [ -z "$ORG" ] || [ -z "$REPO" ] || [ -z "$ENV" ] || [ -z "$TOKEN" ] || [ -z "$AZURE_CREDENTIALS_SP" ]; 87 | then 88 | print_usage 89 | fi 90 | 91 | gh_prefix="https://github.com" 92 | src_repo_name="$ORG/$REPO" 93 | configs_repo_name="${src_repo_name}-configs" 94 | gitops_repo_name="${src_repo_name}-gitops" 95 | 96 | check_repo() { 97 | repo_name=$1 98 | gh repo view $repo_name >/dev/null 2>&1 99 | if [ $? -eq 0 ]; then 100 | echo "Repository $repo_name already exists" 101 | return 0 102 | else 103 | return 1 104 | fi 105 | } 106 | 107 | create_repo() { 108 | repo_name=$1 109 | echo "Repository $repo_name does not exist" 110 | gh repo create $repo_name --private 111 | echo "Created Repository $repo_name" 112 | sleep 3 113 | } 114 | 115 | ensure_gitops_repo() { 116 | rm -rf gitops 117 | check_repo $gitops_repo_name || create_repo $gitops_repo_name 118 | git clone $gh_prefix/$gitops_repo_name gitops 119 | } 120 | 121 | ensure_branch() { 122 | branch_name=$1 123 | if (( $(git branch -a | grep "remotes/origin/$branch_name" | wc -l) == 0)) ; then 124 | echo "Branch $branch_name does not exist" 125 | git checkout -b $branch_name 126 | else 127 | echo "Branch $branch_name already exists" 128 | git checkout $branch_name 129 | fi 130 | } 131 | 132 | configure_gitops_repo() { 133 | echo "Configuring Repository "$gitops_repo_name 134 | 135 | gh variable set SRC_REPO -b $src_repo_name -R $gitops_repo_name 136 | gh secret set CD_BOOTSTRAP_TOKEN -b $TOKEN -R $gitops_repo_name 137 | gh label create promoted --color 0e8a16 --force -R $gitops_repo_name 138 | gh repo edit $gitops_repo_name --delete-branch-on-merge 139 | 140 | pushd gitops 141 | 142 | ensure_branch $ENV 143 | 144 | mkdir -p .github/workflows 145 | cp "$templates_dir/notify-on-pr.yml" .github/workflows/ 146 | 147 | mkdir -p .github/workflows/utils 148 | cp -r "$templates_dir/utils/get-tracking-info.sh" .github/workflows/utils/ 149 | 150 | git add . 151 | git diff-index --quiet HEAD || git commit -m $ENV 152 | git push origin $ENV 153 | 154 | popd 155 | rm -rf gitops 156 | } 157 | 158 | ensure_configs_repo() { 159 | rm -rf configs 160 | check_repo $configs_repo_name || create_repo $configs_repo_name 161 | git clone $gh_prefix/$configs_repo_name configs 162 | } 163 | 164 | ensure_src_repo() { 165 | rm -rf src 166 | check_repo $src_repo_name || create_repo $src_repo_name 167 | git clone $gh_prefix/$src_repo_name src 168 | pushd src 169 | if (( $(git branch -a | grep "remotes/origin/main" | wc -l) == 0)) ; then 170 | echo "# $src_repo_name" >> README.md 171 | git add . 172 | git commit -m 'first commit' 173 | git branch -M main 174 | git push -u origin main 175 | fi 176 | popd 177 | } 178 | 179 | 180 | configure_configs_repo() { 181 | echo "Configuring Repository "$configs_repo_name 182 | 183 | gh variable set MANIFESTS_REPO -b $gitops_repo_name -R $configs_repo_name 184 | gh variable set SRC_REPO -b $src_repo_name -R $configs_repo_name 185 | gh secret set CD_BOOTSTRAP_TOKEN -b $TOKEN -R $configs_repo_name 186 | gh repo edit $configs_repo_name --delete-branch-on-merge 187 | 188 | pushd configs 189 | 190 | ensure_branch $ENV 191 | 192 | if (( $(find . -mindepth 1 -maxdepth 1 -type d ! -path '*.git*' | wc -l) == 0)) ; then 193 | mkdir -p $REPO 194 | touch $REPO/values.yaml 195 | echo "Application config values for a deployment target" > $REPO/Readme.md 196 | fi 197 | 198 | mkdir -p .github/workflows 199 | cp "$templates_dir/notify-on-config-change.yml" .github/workflows/ 200 | 201 | git add . 202 | git diff-index --quiet HEAD || git commit -m $ENV 203 | git push origin $ENV 204 | 205 | popd 206 | rm -rf configs 207 | 208 | } 209 | 210 | setup_gitops_repo() { 211 | ensure_gitops_repo 212 | configure_gitops_repo 213 | } 214 | 215 | setup_configs_repo() { 216 | ensure_configs_repo 217 | configure_configs_repo 218 | } 219 | 220 | 221 | configure_src_repo() { 222 | echo "Configuring Repository "$src_repo_name 223 | 224 | gh variable set MANIFESTS_REPO -b $gitops_repo_name -R $src_repo_name 225 | gh variable set CONFIGS_REPO -b $configs_repo_name -R $src_repo_name 226 | gh secret set CD_BOOTSTRAP_TOKEN -b $TOKEN -R $src_repo_name 227 | gh secret set AZURE_CREDENTIALS_SP -b "$AZURE_CREDENTIALS_SP" -R $src_repo_name 228 | 229 | if gh variable list -R $src_repo_name | grep START_ENVIRONMENT; then 230 | echo "START_ENVIRONMENT already exists" 231 | else 232 | gh variable set START_ENVIRONMENT -b $ENV -R $src_repo_name 233 | fi 234 | 235 | 236 | pushd src 237 | 238 | new_branch_name=feature/cd-setup 239 | ensure_branch $new_branch_name 240 | 241 | mkdir -p .github/workflows 242 | cp "$templates_dir/ci.yml" .github/workflows/ 243 | cp "$templates_dir/deploy.yml" .github/workflows/ 244 | cp "$templates_dir/post-deployment.yml" .github/workflows/ 245 | cp -r "$templates_dir/utils" .github/workflows/ 246 | rm .github/workflows/utils/get-tracking-info.sh 247 | 248 | git add . 249 | if [[ `git status --porcelain | head -1` ]]; then 250 | git commit -m "cd" 251 | git push --set-upstream origin $new_branch_name 252 | 253 | gh pr create --base main --head $new_branch_name --title "GitOps CD setup" --body "GH Actions workflows for CD setup
Utility scripts for GH Actions workflows" 254 | fi 255 | 256 | popd 257 | rm -rf src 258 | 259 | } 260 | 261 | setup_src_repo() { 262 | ensure_src_repo 263 | configure_src_repo 264 | } 265 | 266 | 267 | 268 | check_prerequisites || print_prerequisites 269 | 270 | check_gh_username 271 | authenticate_gh 272 | 273 | setup_gitops_repo 274 | setup_configs_repo 275 | setup_src_repo 276 | -------------------------------------------------------------------------------- /docs/bootstrap/prerequisites.md: -------------------------------------------------------------------------------- 1 | # Prerequisites for Kalypso Bootstrap Script 2 | 3 | This document outlines the prerequisites needed to run the Kalypso Scheduler bootstrap script. 4 | 5 | ## Operating System 6 | 7 | The bootstrap script supports: 8 | 9 | - **macOS** (10.15 Catalina or later) 10 | - **Linux** (Ubuntu 18.04+, RHEL/CentOS 7+, or equivalent) 11 | 12 | Windows users should use WSL2 (Windows Subsystem for Linux). 13 | 14 | ## Required Tools 15 | 16 | ### kubectl 17 | 18 | Kubernetes command-line tool for cluster management. 19 | 20 | **Minimum Version**: 1.20.0 21 | 22 | **Installation**: 23 | 24 | macOS: 25 | 26 | ```bash 27 | brew install kubectl 28 | ``` 29 | 30 | Linux: 31 | 32 | ```bash 33 | curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" 34 | chmod +x kubectl 35 | sudo mv kubectl /usr/local/bin/ 36 | ``` 37 | 38 | **Verify**: 39 | 40 | ```bash 41 | kubectl version --client 42 | ``` 43 | 44 | ### Azure CLI (az) 45 | 46 | Command-line interface for managing Azure resources. 47 | 48 | **Minimum Version**: 2.30.0 49 | 50 | **Installation**: 51 | 52 | macOS: 53 | 54 | ```bash 55 | brew install azure-cli 56 | ``` 57 | 58 | Linux: 59 | 60 | ```bash 61 | curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash 62 | ``` 63 | 64 | **Verify**: 65 | 66 | ```bash 67 | az version 68 | ``` 69 | 70 | ### git 71 | 72 | Version control system for repository management. 73 | 74 | **Minimum Version**: 2.0.0 75 | 76 | **Installation**: 77 | 78 | macOS: 79 | 80 | ```bash 81 | brew install git 82 | ``` 83 | 84 | Ubuntu/Debian: 85 | 86 | ```bash 87 | sudo apt-get install git 88 | ``` 89 | 90 | RHEL/CentOS: 91 | 92 | ```bash 93 | sudo yum install git 94 | ``` 95 | 96 | **Verify**: 97 | 98 | ```bash 99 | git --version 100 | ``` 101 | 102 | ### Helm 103 | 104 | Kubernetes package manager for installing Kalypso. 105 | 106 | **Minimum Version**: 3.0.0 107 | 108 | **Installation**: 109 | 110 | macOS: 111 | 112 | ```bash 113 | brew install helm 114 | ``` 115 | 116 | Linux: 117 | 118 | ```bash 119 | curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash 120 | ``` 121 | 122 | **Verify**: 123 | 124 | ```bash 125 | helm version 126 | ``` 127 | 128 | ### GitHub CLI (gh) 129 | 130 | GitHub command-line tool for managing repositories and secrets. 131 | 132 | **Minimum Version**: 2.0.0 133 | 134 | **Installation**: 135 | 136 | macOS: 137 | 138 | ```bash 139 | brew install gh 140 | ``` 141 | 142 | Linux: 143 | 144 | ```bash 145 | curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg 146 | echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null 147 | sudo apt update 148 | sudo apt install gh 149 | ``` 150 | 151 | **Verify**: 152 | 153 | ```bash 154 | gh --version 155 | ``` 156 | 157 | ### jq 158 | 159 | Command-line JSON processor for parsing API responses and configuration files. 160 | 161 | **Minimum Version**: 1.6 162 | 163 | **Installation**: 164 | 165 | macOS: 166 | 167 | ```bash 168 | brew install jq 169 | ``` 170 | 171 | Ubuntu/Debian: 172 | 173 | ```bash 174 | sudo apt-get install jq 175 | ``` 176 | 177 | RHEL/CentOS: 178 | 179 | ```bash 180 | sudo yum install jq 181 | ``` 182 | 183 | **Verify**: 184 | 185 | ```bash 186 | jq --version 187 | ``` 188 | 189 | ## Optional Tools (Recommended) 190 | 191 | ### yq 192 | 193 | Command-line YAML processor for configuration files. 194 | 195 | **Minimum Version**: 4.0 196 | 197 | **Note**: **Required** if you want to use YAML configuration files with `--config file.yaml`. 198 | 199 | **Installation**: 200 | 201 | macOS: 202 | 203 | ```bash 204 | brew install yq 205 | ``` 206 | 207 | Linux: 208 | 209 | ```bash 210 | wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq 211 | chmod +x /usr/local/bin/yq 212 | ``` 213 | 214 | ## Azure Prerequisites 215 | 216 | ### Azure Account 217 | 218 | You need an Azure account with: 219 | 220 | - Active subscription 221 | - Permissions to create resource groups 222 | - Permissions to create AKS clusters 223 | - Permissions to assign roles (for AKS managed identity) 224 | 225 | ### Azure Authentication 226 | 227 | Before running the script, authenticate with Azure: 228 | 229 | ```bash 230 | az login 231 | ``` 232 | 233 | For non-interactive scenarios (CI/CD), use service principal: 234 | 235 | ```bash 236 | az login --service-principal \ 237 | --username $AZURE_CLIENT_ID \ 238 | --password $AZURE_CLIENT_SECRET \ 239 | --tenant $AZURE_TENANT_ID 240 | ``` 241 | 242 | ### Azure Subscription 243 | 244 | Set your default subscription: 245 | 246 | ```bash 247 | az account set --subscription "your-subscription-id" 248 | ``` 249 | 250 | Or provide it via environment variable: 251 | 252 | ```bash 253 | export AZURE_SUBSCRIPTION_ID="your-subscription-id" 254 | ``` 255 | 256 | ## GitHub Prerequisites 257 | 258 | ### GitHub Account 259 | 260 | You need a GitHub account with: 261 | 262 | - Permissions to create repositories (in your account or organization) 263 | - Repository admin access for existing repositories 264 | - Organization admin access (if using `--github-org`) 265 | 266 | ### GitHub Personal Access Token 267 | 268 | Create a personal access token with these scopes: 269 | 270 | - `repo` - Full control of private repositories 271 | - `workflow` - Update GitHub Action workflows 272 | - `admin:org` - Full control of organizations (if creating in an org) 273 | 274 | **To create a token**: 275 | 276 | 1. Go to GitHub Settings → Developer settings → Personal access tokens 277 | 2. Click "Generate new token" 278 | 3. Select required scopes 279 | 4. Generate and copy the token 280 | 281 | **Provide the token to the script**: 282 | 283 | ```bash 284 | export GITHUB_TOKEN="your-github-token" 285 | ``` 286 | 287 | Or the script will prompt you interactively. 288 | 289 | ## Resource Requirements 290 | 291 | ### For New AKS Cluster 292 | 293 | **Minimum**: 294 | 295 | - 2 CPU cores 296 | - 4 GiB memory per node 297 | - 3 nodes (default) 298 | 299 | **Recommended**: 300 | 301 | - 4 CPU cores 302 | - 8 GiB memory per node 303 | - 3-5 nodes 304 | 305 | **Default Configuration**: 306 | 307 | - VM Size: Standard_DS2_v2 (2 vCPUs, 7 GiB memory) 308 | - Node Count: 3 309 | - Total: 6 vCPUs, 21 GiB memory 310 | 311 | ### Azure Quotas 312 | 313 | Ensure your subscription has sufficient quota for: 314 | 315 | - Compute cores (Standard DSv2 Family or your chosen VM family) 316 | - Public IP addresses 317 | - Load balancers 318 | - Virtual networks 319 | 320 | Check quotas: 321 | 322 | ```bash 323 | az vm list-usage --location eastus --output table 324 | ``` 325 | 326 | ## Network Requirements 327 | 328 | The bootstrap script needs internet access to: 329 | 330 | - Azure APIs (`*.azure.com`) 331 | - GitHub APIs (`api.github.com`, `github.com`) 332 | - Kubernetes APIs (`dl.k8s.io`) 333 | - Helm repositories 334 | - Docker registries (for Kalypso images) 335 | 336 | ## Validation Script 337 | 338 | Run this script to check if all prerequisites are met: 339 | 340 | ```bash 341 | cd scripts/bootstrap 342 | ./bootstrap.sh --help 343 | ``` 344 | 345 | The bootstrap script includes built-in prerequisite checking that will: 346 | 347 | - Verify all required tools are installed 348 | - Check tool versions meet minimum requirements 349 | - Validate Azure authentication 350 | - Validate GitHub authentication 351 | - Display missing tools with installation instructions 352 | 353 | ## Troubleshooting Prerequisites 354 | 355 | ### kubectl not found 356 | 357 | Ensure kubectl is in your PATH: 358 | 359 | ```bash 360 | export PATH=$PATH:/usr/local/bin 361 | ``` 362 | 363 | ### Azure CLI login fails 364 | 365 | Clear cached credentials: 366 | 367 | ```bash 368 | az account clear 369 | az login 370 | ``` 371 | 372 | ### GitHub token invalid 373 | 374 | Verify your token has correct scopes and hasn't expired: 375 | 376 | ```bash 377 | curl -H "Authorization: token $GITHUB_TOKEN" https://api.github.com/user 378 | ``` 379 | 380 | ### Insufficient Azure permissions 381 | 382 | Contact your Azure administrator to grant: 383 | 384 | - Contributor role on the subscription or resource group 385 | - User Access Administrator role (for AKS managed identity) 386 | 387 | ## Next Steps 388 | 389 | Once all prerequisites are satisfied, proceed to: 390 | 391 | - [README.md](README.md) - Main documentation and usage 392 | - [quickstart.md](quickstart.md) - Quick start guide 393 | -------------------------------------------------------------------------------- /docs/run-books/platform-team-creates-a-new-environment.md: -------------------------------------------------------------------------------- 1 | # Platform Team Creates a New Environment 2 | 3 | - [Platform Team Creates a New Environment](#platform-team-creates-a-new-environment) 4 | - [Prerequisites](#prerequisites) 5 | - [1. Admin Access to Platform Repositories](#1-admin-access-to-platform-repositories) 6 | - [Steps](#steps) 7 | - [1. Create a GitOps Branch](#1-create-a-gitops-branch) 8 | - [2. Create a Control Plane Branch](#2-create-a-control-plane-branch) 9 | - [3. Link Environment in Promotion Sequence](#3-link-environment-in-promotion-sequence) 10 | - [4. Declare the Environment in the Control Plane](#4-declare-the-environment-in-the-control-plane) 11 | - [Next Steps](#next-steps) 12 | 13 | ## Prerequisites 14 | 15 | This run book describes how to create a new GitOps environment at the platform level. An example promotional flow through environments might be `dev` -> `test` -> `prod` for workloads in a fleet. This runbook describes the steps necessary for creating these environments. 16 | 17 | ### 1. Admin Access to Platform Repositories 18 | 19 | This run book is intended to be completed by a Platform Engineer with admin access to the Kalypso Control Plane and Platform GitOps repositories. 20 | 21 | Admin access is required for modifying GitHub Environments in the control plane repository. 22 | 23 | For reference, here are the example [Control Plane](https://github.com/microsoft/kalypso-control-plane) and [Platform GitOps](https://github.com/microsoft/kalypso-gitops) repositories from Kalypso. 24 | 25 | Identify the following values for use in this runbook 26 | 27 | | Variable | Description | 28 | | ------------------------ | ----------------------------------------------- | 29 | | `GITOPS_REPO_URL` | The git URL for the Platform GitOps repository. | 30 | | `CONTROL_PLANE_REPO_URL` | The git URL for the Control Plane repository. | 31 | 32 | ## Steps 33 | 34 | This run book will walk through an example of creating a `newenv` environment to the end of the environments chain. 35 | 36 | For creating other environments, replace `newenv` with your environment name. 37 | 38 | ### 1. Create a GitOps Branch 39 | 40 | First, create a branch in the Platform GitOps repository. This repo has a branch for each environment that holds relevant GitOps manifest files. 41 | 42 | Inside the GitOps repository, create a new git branch based off of the latest environment. In this example, that will be `prevenv`. 43 | 44 | ```sh 45 | # switch to the existing base branch 46 | git checkout prevenv 47 | 48 | # create a new branch based on the existing branch 49 | git checkout -b newenv 50 | ``` 51 | 52 | Remove the old tracking information on the new branch. These files will be automatically re-generated. 53 | 54 | ```sh 55 | rm -rf .github/tracking 56 | ``` 57 | 58 | Remove all top-level [`ClusterType`](https://github.com/microsoft/kalypso-scheduler?tab=readme-ov-file#cluster-type) folders. The exact folders may be different based on what `ClusterTypes` were on the base branch. The appropriate folders will be re-generated for the current environment. The goal now is to create an "empty" GitOps environment branch. 59 | 60 | ```sh 61 | # The exact set of cluster types may be different 62 | rm -rf observability-hub/ 63 | rm -rf on-prem/ 64 | rm -rf cloud-uat/ 65 | rm -rf 66 | ``` 67 | 68 | Finally, push this branch up to GitHub. 69 | 70 | ```sh 71 | git add . 72 | git commit -m "Prepare newenv environment branch" 73 | git push -u origin newenv 74 | ``` 75 | 76 | Once complete, there should be a new branch in GitHub named `newenv`. This branch will only contain the `.github/workflows/check-promote.yaml` file and the README.md. 77 | 78 | ![empty-environment-branch](./images/empty-environment-branch.png) 79 | 80 | ### 2. Create a Control Plane Branch 81 | 82 | Next, create a branch in the Control Plane repository. This repo has a branch for each environment that holds [`ClusterTypes`](https://github.com/microsoft/kalypso-scheduler?tab=readme-ov-file#cluster-type), [`ConfigMaps`](https://github.com/microsoft/kalypso-scheduler?tab=readme-ov-file#config), and [`SchedulingPolicies`](https://github.com/microsoft/kalypso-scheduler?tab=readme-ov-file#scheduling-policy) for the environment. 83 | 84 | Inside the Control Plane repository, create a new git branch based off the latest environment. In this runbook, that will be `prevenv`. 85 | 86 | ```sh 87 | # switch to the existing base branch 88 | git checkout prevenv 89 | 90 | # create a new branch based on the existing branch 91 | git checkout -b newenv 92 | ``` 93 | 94 | In the new branch, update `gitops-repo.yaml` to reference the GitOps branch created in [step 1](#1-create-a-gitops-branch). 95 | 96 | ```yaml 97 | apiVersion: scheduler.kalypso.io/v1alpha1 98 | kind: GitOpsRepo 99 | metadata: 100 | name: newenv 101 | spec: 102 | repo: 103 | branch: newenv 104 | path: . 105 | ``` 106 | 107 | It may be appropriate to update any `ClusterType`, `ConfigMap`, or `SchedulingPolicy` objects for the new environment. The exact steps here will vary depending on your use case and what clusters & applications are relevant to this environment. 108 | 109 | > For information on updating `ClusterType` and `SchedulingPolicy` objects, see [Platform Team Schedules Applications for Deployment](./platform-team-schedules-applications-for-deployment.md). 110 | > 111 | > For information on updating `ConfigMap` objects, see [Platform Team Manages Platform Configuration](./platform-team-manages-platform-configuration.md). 112 | 113 | When you are happy with the changes, push the new branch to GitHub. 114 | 115 | ```sh 116 | git add . 117 | git commit -m "Prepare newenv environment" 118 | git push -u origin newenv 119 | ``` 120 | 121 | In our example `newenv` environment, we updated a few `ConfigMap` objects, but did not change any `SchedulingPolicy` or `ClusterType` objects. 122 | 123 | ![update-config](./images/update-config.png) 124 | 125 | ### 3. Link Environment in Promotion Sequence 126 | 127 | Now that our git branches for the new `newenv` environment are set up, we need to declare where this environment lives in the promotion sequence. This is defined using GitHub environments inside the Control Plane repository. 128 | 129 | Each Environment optionally contains a variable called `NEXT_ENVIRONMENT` that points to the next environment in the sequence. This creates a linked list that dictates how Control Plane changes are promoted through environments. 130 | 131 | We will update `prevenv` to include a new `NEXT_ENVIRONMENT` variable that points to the new `newenv` environment. 132 | 133 | ![diagram of enviornment sequence](./images/next_environment.drawio.png) 134 | 135 | First, create a new Environment called `newenv`. 136 | 137 | ![new-environment](./images/new-environment.png) 138 | 139 | Update the previous environment, `prevenv`, to point to the new `NEXT_ENVIRONMENT`, `newenv`. 140 | 141 | ![next-environment](./images/next-environment.png) 142 | 143 | ### 4. Declare the Environment in the Control Plane 144 | 145 | Now, we can declare the environment in the Control Plane for Kalypso to reference. 146 | 147 | On the `main` branch of the Control Plane, create an [`Environment`](https://github.com/microsoft/kalypso-scheduler?tab=readme-ov-file#environment) object inside the `.environments/` folder. 148 | 149 | ```yaml 150 | # newenv.yaml 151 | apiVersion: scheduler.kalypso.io/v1alpha1 152 | kind: Environment 153 | metadata: 154 | name: newenv 155 | spec: 156 | controlPlane: 157 | repo: 158 | branch: newenv 159 | path: . 160 | ``` 161 | 162 | ![environment-definition](./images/environment-definition.png) 163 | 164 | ## Next Steps 165 | 166 | The new environment likely does not have any GitOps manifests inside it. To create these, we will need to declare what applications should be deployed into this environment. To do this, follow [Platform Team Schedules Applications for Deployment](./platform-team-schedules-applications-for-deployment.md). 167 | 168 | It is also likely that applications deployed into this new environment will need changes to their configurations. This will vary by application, but can be achieved through [Application Team Manages Application Configuration](./application-team-manages-application-configuration.md) and [Platform Team Manages Platform Configuration](./platform-team-manages-platform-configuration.md). 169 | 170 | It may also make sense to set up deployment rings for applications in this new environment. See [Application Team Creates a New Application Ring](./application-team-creates-a-new-application-ring.md) for more information. 171 | 172 | Applications can now be deployed through this environment through the standard promotion flow described in [Application Team Promotes a Change Through Environments](./application-team-promotes-a-change-through-environments.md). 173 | -------------------------------------------------------------------------------- /docs/run-books/platform-team-schedules-applications-for-deployment.md: -------------------------------------------------------------------------------- 1 | # Platform Team Schedules Applications for Deployment 2 | 3 | - [Platform Team Schedules Applications for Deployment](#platform-team-schedules-applications-for-deployment) 4 | - [Prerequisites](#prerequisites) 5 | - [1. Access to Platform Control Plane and GitOps Repos](#1-access-to-platform-control-plane-and-gitops-repos) 6 | - [2. Identify a Cluster in the Fleet](#2-identify-a-cluster-in-the-fleet) 7 | - [3. Identify an Application](#3-identify-an-application) 8 | - [Steps](#steps) 9 | - [1. Create or Find a `WorkloadRegistration`](#1-create-or-find-a-workloadregistration) 10 | - [2. Create or Find a `ClusterType`](#2-create-or-find-a-clustertype) 11 | - [3. Schedule an Application for Deployment onto a Cluster](#3-schedule-an-application-for-deployment-onto-a-cluster) 12 | - [4. Merge GitOps PRs](#4-merge-gitops-prs) 13 | - [Next Steps](#next-steps) 14 | 15 | ## Prerequisites 16 | 17 | ### 1. Access to Platform Control Plane and GitOps Repos 18 | 19 | This run book is intended to be completed by the Platform Engineer with contributor access to the Kalypso Control Plane and Platform GitOps repositories. 20 | 21 | For reference, here are the example [Control Plane](https://github.com/microsoft/kalypso-control-plane) and [Platform GitOps](https://github.com/microsoft/kalypso-gitops) repositories from Kalypso. 22 | 23 | ### 2. Identify a Cluster in the Fleet 24 | 25 | The clusters compose the fleet, and the applications are the workloads that will be scheduled to run on the fleet. For instructions on onboarding a cluster, see [Platform Team Onboards a New Cluster](./platform-team-onboards-a-new-cluster.md). 26 | 27 | **It is not strictly required to have a cluster provisioned before scheduling applications onto it.** If the cluster is not onboarded to the fleet, this run book will create the GitOps manifests in GitHub, but there will be no flux reconcilers watching these manifests, so they will not be deployed. 28 | 29 | Collect the following values for use later in this run book. 30 | 31 | | Variable | Description | 32 | | ------------------- | ----------------------------------------------------------------------------------- | 33 | | `CLUSTER_TYPE_NAME` | The logical name of the cluster (e.g. `Mayberry`) or a group of identical clusters | 34 | 35 | ### 3. Identify an Application 36 | 37 | Finally, identify an existing application to schedule for deployment onto the fleet. Before starting this run book, make sure the application is onboarded by following [Application Team Onboards a New Application](./application-team-onboards-a-new-application.md). 38 | 39 | Identify the location of the [Workload](https://github.com/microsoft/kalypso-scheduler?tab=readme-ov-file#workload-registration) declaration file for the application. 40 | 41 | Collect the following values for use later in this run book. 42 | 43 | | Variable | Description | 44 | | --------------------- | ----------------------------------------------------------------------- | 45 | | `WORKLOAD_GIT_URL` | URL of application git repository holding the Workload registration | 46 | | `WORKLOAD_GIT_BRANCH` | git branch holding the Workload registration | 47 | | `WORKLOAD_GIT_PATH` | file path of the Workload registration | 48 | | `APPLICATION_NAME` | the application name inside the workload yaml file | 49 | | `KALYPSO_WORKSPACE` | this is a custom label for grouping workloads within the Kalypso system | 50 | 51 | ![workload](./images/workload.png) 52 | 53 | ## Steps 54 | 55 | ### 1. Create or Find a `WorkloadRegistration` 56 | 57 | This only needs to be done once per application. If this is the first time your application has been scheduled for a GitOps deployment, it will need to be registered with the Control Plane. Otherwise, this should already be done for your application. 58 | 59 | Submit a [`WorkloadRegistration`](https://github.com/microsoft/kalypso-scheduler?tab=readme-ov-file#workload-registration) object on the `main` branch of the Control Plane repository inside the workloads folder. 60 | 61 | ```yaml 62 | apiVersion: scheduler.kalypso.io/v1alpha1 63 | kind: WorkloadRegistration 64 | metadata: 65 | name: 66 | labels: 67 | type: application 68 | spec: 69 | workload: 70 | repo: 71 | branch: 72 | path: 73 | workspace: 74 | ``` 75 | 76 | ![workload-registration](./images/workload-registration.png) 77 | 78 | ### 2. Create or Find a `ClusterType` 79 | 80 | This needs to be done for each environment and cluster type that can host applications. If your cluster and environment already hosts applications, this should be done already. 81 | 82 | Otherwise, you will need to submit a [`ClusterType`](https://github.com/microsoft/kalypso-scheduler?tab=readme-ov-file#cluster-type) to the branch for your cluster inside the `cluster-types/` folder. 83 | 84 | ```yaml 85 | apiVersion: scheduler.kalypso.io/v1alpha1 86 | kind: ClusterType 87 | metadata: 88 | name: 89 | labels: 90 | # These are example labels, you can modify as necessary. 91 | # Labels are free-form and will be used with label-matching to later determine what applications should run on this `ClusterType`. 92 | cloud: "false" 93 | purpose: "factory" 94 | ring: "gray" 95 | spec: 96 | reconciler: arc-flux 97 | namespaceService: default 98 | configType: configmap 99 | ``` 100 | 101 | ![mayberry-cluster](./images/mayberry-cluster.png) 102 | 103 | ### 3. Schedule an Application for Deployment onto a Cluster 104 | 105 | Finally, we need to define a mapping between applications and clusters. This is done by label matching `deploymentTargets` and [`ClusterTypes`](https://github.com/microsoft/kalypso-scheduler?tab=readme-ov-file#cluster-type) in a [`SchedulingPolicy`](https://github.com/microsoft/kalypso-scheduler?tab=readme-ov-file#scheduling-policy) object. 106 | 107 | Create a `SchedulingPolicy` in the `scheduling/` folder of the environment branch. 108 | 109 | > Deployment target labels are defined by the application team in their `workload.yaml` file. This was set up when the application was onboarded with [Application Team Onboards a New Application](./application-team-onboards-a-new-application.md). 110 | > 111 | > The cluster type labels were defined in [step 2](#2-create-or-find-a-clustertype) of this run book. 112 | 113 | ```yaml 114 | apiVersion: scheduler.kalypso.io/v1alpha1 115 | kind: SchedulingPolicy 116 | metadata: 117 | # Feel free to use any name you want 118 | name: example-scheduling-policy 119 | spec: 120 | deploymentTargetSelector: 121 | workspace: 122 | labelSelector: 123 | matchLabels: 124 | # modify these labels to match the desired deployment targets 125 | zone: gray 126 | clusterTypeSelector: 127 | labelSelector: 128 | matchLabels: 129 | # modify these labels to match the desired cluster type 130 | purpose: "factory" 131 | ring: gray 132 | ``` 133 | 134 | > Note: there is no mention of the specific workload or cluster host in this object, but it is the link that makes specific workloads get deployed onto specific clusters because of how applications and clusters are independently labeled. 135 | 136 | ![scheduling-policy](./images/scheduling-policy.png) 137 | 138 | ### 4. Merge GitOps PRs 139 | 140 | Until now, only our application workload scheduling desire has been declared in the Control Plane repository. Nothing has actually been deployed yet. The previous actions will create pull requests automatically in the Platform GitOps repository as appropriate. 141 | 142 | The GitOps pattern uses these final pull requests as a manual approval for actually deploying applications to clusters. 143 | 144 | Navigate to the Platform GitOps Pull Requests view to see any automated PRs that were created. Once these PRs are merged, clusters watching the GitOps repository will automatically pull and install the updates. 145 | 146 | Your specific pull requests will look different based on what applications are scheduled onto which clusters. 147 | 148 | ## Next Steps 149 | 150 | The [Kalypso Observability Hub](https://github.com/microsoft/kalypso-observability-hub?tab=readme-ov-file#deployment-reports) dashboards provide an overview of what applications are deployed into which clusters. 151 | 152 | If there are any issues with scheduled deployments, work with the application experts to troubleshoot. There may be issues with configuration. 153 | 154 | - To fix platform configuration issues, see [Platform Team Manages Platform Configuration](./platform-team-manages-platform-configuration.md). 155 | - To fix application configuration issues, see [Application Team Manages Application Configuration](./application-team-manages-application-configuration.md). 156 | -------------------------------------------------------------------------------- /docs/run-books/application-team-promotes-a-change-through-environments.md: -------------------------------------------------------------------------------- 1 | # Application Team Promotes a Change Through Environments 2 | 3 | - [Application Team Promotes a Change Through Environments](#application-team-promotes-a-change-through-environments) 4 | - [Overview](#overview) 5 | - [Prerequisites](#prerequisites) 6 | - [1. Verify Access to Application Repositories](#1-verify-access-to-application-repositories) 7 | - [Steps](#steps) 8 | - [1. Update the Application Source Code](#1-update-the-application-source-code) 9 | - [2. Continuous Integration](#2-continuous-integration) 10 | - [3. Deploying to `dev`](#3-deploying-to-dev) 11 | - [4. Verify `dev` Deployment](#4-verify-dev-deployment) 12 | - [5. Promote from `dev` to `stage`](#5-promote-from-dev-to-stage) 13 | - [6. Deploying to `stage`](#6-deploying-to-stage) 14 | - [7. Verify `stage` Deployment](#7-verify-stage-deployment) 15 | - [8. Repeat for Higher Environments](#8-repeat-for-higher-environments) 16 | - [Next Steps](#next-steps) 17 | - [Abandoning a Release](#abandoning-a-release) 18 | - [Rolling Back](#rolling-back) 19 | 20 | ## Overview 21 | 22 | This run book describes how to submit an application change and deploy it through environments. In this run book, the example application is deployed first to a `dev` environment and then to `stage` before moving to higher environments. 23 | 24 | This run book does not include instructions on how to manage application or platform configuration. For those details, see [Application Team Manages Application Configuration](./application-team-manages-application-configuration.md) and [Platform Team Manages Platform Configuration](./platform-team-manages-platform-configuration.md). 25 | 26 | A high-level overview of the application promotion flow is shown below. 27 | 28 | ![diagram of application promotion flow](./images/promote-flow.drawio.png) 29 | 30 | 1. Application developer submits a change to the source repository. A pull request is created, reviewed, and merged into the `main` branch. 31 | 2. Continuous integration performs automated code quality checks, and builds artifacts required for deployment. 32 | 3. Continuous deployment generates a pull request automatically against the `dev` environment (the first environment). This pull request is merged manually. 33 | 4. The `dev` clusters see the changes and pull them to run in the cluster using GitOps. 34 | 5. Continuous deployment is triggered for the next environment once the `dev` deployment is confirmed. 35 | 6. Continuous deployment generates another pull request automatically against the next environment, `stage`. This pull request is merged manually. 36 | 7. The `stage` clusters see the changes and pull them to run in the cluster. 37 | 38 | > For more details on the CI/CD flow, see [GitOps CI/CD with GitHub](https://github.com/microsoft/kalypso/blob/main/cicd/readme.md). 39 | 40 | ## Prerequisites 41 | 42 | ### 1. Verify Access to Application Repositories 43 | 44 | Each application consists of 3 git repositories. A source repository, a configuration repository, and a gitops repository. Identify all 3 repositories and make sure you have contributor access to all 3. 45 | 46 | Make sure the **source repository** is cloned locally as this is the repository you will submit a change to. 47 | 48 | ## Steps 49 | 50 | ### 1. Update the Application Source Code 51 | 52 | First, the source code change to be promoted needs to be submitted to the source repository. This will create a new revision of the application that can be promoted across environments and rings. 53 | 54 | The specific steps will vary for each application and change that is made. What is important is that there is a new commit to the `main` branch to trigger the whole promotion flow. 55 | 56 | > Note: Some changes do require updating configuration such as environment variables. This run book focuses on changes to application source. For changes to configuration, you may also need to [Manage Application Configuration](./application-team-manages-application-configuration.md) or [Manage Platform Configuration](./platform-team-manages-platform-configuration.md). 57 | 58 | ![pr-to-src-code](./images/pr-to-src-code.png) 59 | 60 | ### 2. Continuous Integration 61 | 62 | This step should happen automatically when the pull request is merged to the `main` branch and will vary by application. It includes all automated checks and builds for required artifacts that will be deployed through environments (ex: building and pushing a docker image). 63 | 64 | Watch the GitHub Actions tab of your application for the continuous integration workflow. 65 | 66 | ![ci-pipeline](./images/ci-pipeline.png) 67 | 68 | ### 3. Deploying to `dev` 69 | 70 | If continuous integration was successful, another GitHub Actions workflow uses the Configuration Repository to generate an automated pull request against the gitops repository. This pull request includes all updates to Kubernetes manifests that will be deployed to the first environment, `dev`. 71 | 72 | Locate the appropriate pull request in your application's GitOps repository. 73 | 74 | - It will be created shortly after continuous integration runs. 75 | - It will note the latest version of your application. 76 | - It should be marked `promoted` and the version should match. 77 | - It will target a merge into the `dev` branch of the repository. 78 | 79 | Review and merge this pull request to deploy it to `dev`. If you find any issues, or otherwise wish not to deploy it, simply close the pull request without merging. 80 | 81 | > Note: Pull Requests labeled `promoted` indicate changes that are eligible for promotion through rings and environments. 82 | > 83 | > Changes to configuration are not subjects for promotion and are not labeled as such. These PRs are generated from [Application Team Manages Application Configuration](./application-team-manages-application-configuration.md). 84 | > 85 | > ![pr-promoted](./images/pr-promoted.png) 86 | 87 | ![pr-deployment](./images/pr-deployment.png) 88 | 89 | ### 4. Verify `dev` Deployment 90 | 91 | Once the GitOps PR is merged, `dev` clusters will automatically pull the changes in through Flux. Consult the deployment observability dashboards to watch for this change. You should see the version update in the dev environment. 92 | 93 | ![deployment-observability-hw-deployed](./images/deployment-observability-hw-deployed.png) 94 | 95 | ### 5. Promote from `dev` to `stage` 96 | 97 | If the deployment to `dev` was successful, the continuous deployment workflow will automatically trigger for the next environment, `stage`. 98 | 99 | Look for another GitOps PR marked `promoted`. This time, it should target the `stage` environment. 100 | 101 | ![pr-deployment-stage](./images/pr-deployment-stage.png) 102 | 103 | ### 6. Deploying to `stage` 104 | 105 | Review and merge the GitOps PR for `stage`. This will deploy the application to `stage` clusters. 106 | 107 | ### 7. Verify `stage` Deployment 108 | 109 | Monitor the same deployment observability dashboard to watch for this change in the `stage` environment. 110 | 111 | ![deployment-observability-hw-deployed-stage](./images/deployment-observability-hw-deployed-stage.png) 112 | 113 | ### 8. Repeat for Higher Environments 114 | 115 | This run book walks through the first 2 environments: `dev` and `stage`. However, it is likely that there are more environments eventually leading all the way to `prod`. Steps 5, 6, and 7 will repeat for each ring & environment in the promotion chain, and stop at the final environment. 116 | 117 | Application "rings" can also be used to control application release scopes within environments. Applications each define their environments and rings independently, so the exact names may differ by application. 118 | 119 | > For information about managing environments, see [Platform Team Creates a New Environment](./platform-team-creates-a-new-environment.md). 120 | > 121 | > For information about managing application rings, see [Application Team Creates a New Application Ring](./application-team-creates-a-new-application-ring.md). 122 | 123 | ## Next Steps 124 | 125 | It is likely that an application change requires updates to configuration. Configuration changes can be either application or platform level. 126 | 127 | - Application Level: values that do not depend on where the application is deployed (ex: logging verbosity, image tags, etc.) 128 | - Platform Level: values that do depend on where the application is deployed (ex: database endpoints, KeyVault secrets, etc.) 129 | 130 | To update configuration, reference the appropriate run book for the type of configuration that is being changed: [Application Team Manages Application Configuration](./application-team-manages-application-configuration.md) or [Platform Team Manages Platform Configuration](./platform-team-manages-platform-configuration.md). 131 | 132 | ### Abandoning a Release 133 | 134 | If a promoted change needs to be abandoned, simply close and don't merge GitOps PRs. There is no need to see a release through all environments. Subsequent promotions will still function. 135 | 136 | ### Rolling Back 137 | 138 | If an issue was discovered in a deployment, and it needs to be reverted, GitOps makes that easy. 139 | 140 | 1. Locate the GitOps PR for the appropriate deployment & environment. 141 | 2. Create a new GitOps PR that reverts the deployment using the "Revert" button. 142 | 3. Review and merge the resulting PR to roll back the release. 143 | 144 | ![revert-pr](./images/revert-pr.png) 145 | -------------------------------------------------------------------------------- /cicd/tutorial/cicd-tutorial.md: -------------------------------------------------------------------------------- 1 | # Explore CI/CD flow with GitOps 2 | 3 | This tutorial walks you through an example scenario of a CI/CD flow, implemented with the GitOps concept. First, you deploy a sample infrastructure with a few GitHub repositories and AKS clusters. Next, you work through a set of common use cases that reflect daily application development activities such as configuring, building and deploying an application as well as promoting a new application version to the next environment. 4 | 5 | ## Prerequisites 6 | 7 | In order to successfully deploy the sample, you need: 8 | 9 | - An Azure subscription. If you don't already have one, create a [free account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) before you begin. 10 | - [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) 11 | - [GitHub CLI](https://cli.github.com) 12 | - [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) 13 | - GitHub token with the following scopes: 14 | - Classic: `repo`, `workflow`, `write:packages`, `delete:packages`, `read:org`, `delete_repo`. 15 | - Fine-grained: 16 | - `Actions` - R/W 17 | - `Administration` - R/W 18 | - `Commit statuses` - R/W 19 | - `Contents` - R/W 20 | - `Metadata` - RO 21 | - `Pull requests` - R/W 22 | - `Secrets` - R/W 23 | - `Variables` - R/W 24 | - `Workflows` - R/W 25 | 26 | ## Deploy the sample 27 | 28 | To deploy the sample, run the following script: 29 | 30 | ```bash 31 | mkdir kalypso && cd kalypso 32 | curl -fsSL -o deploy.sh https://raw.githubusercontent.com/microsoft/kalypso/main/cicd/tutorial/deploy.sh 33 | chmod 700 deploy.sh 34 | ./deploy.sh -c -p -o -t -l 35 | ``` 36 | 37 | This script may take 10-15 minutes to complete. After it's done, it reports the execution result in the output like this: 38 | 39 | ```output 40 | Deployment is complete! 41 | --------------------------------- 42 | Created repositories: 43 | - https://github.com/eedorenko/hello-world 44 | - https://github.com/eedorenko/hello-world-gitops 45 | - https://github.com/eedorenko/hello-world-configs 46 | --------------------------------- 47 | Created AKS clusters in hello-world-rg resource group: 48 | - hello-world-left-cluster 49 | - hello-world-right-cluster 50 | --------------------------------- 51 | ``` 52 | 53 | > [!NOTE] 54 | > If something goes wrong with the deployment, you can delete the created resources with the following command: 55 | > 56 | > ```bash 57 | > ./deploy.sh -d -p -o -t -l 58 | > ``` 59 | 60 | ## Sample overview 61 | 62 | This deployment script creates an infrastructure, shown in the following diagram: 63 | 64 | ![cicd-tutorial-setup](../../docs/images/cicd-tutorial-setup.png) 65 | 66 | There are a few GitHub repositories: 67 | 68 | - `Application Source` (GITHUB-ORG/PREFIX): Contains a sample application source code, including Docker files, manifest templates and CI/CD workflows. 69 | - `Application Configs` (GITHUB-ORG/PREFIX-configs): Contains application configuration values. 70 | - `Application GitOps` (GITHUB-ORG/PREFIX-gitops): Contains final sample application manifests to be deployed to the deployment targets. 71 | 72 | The script creates `PREFIX-rg` resource group with two Azure Kubernetes Service (AKS) clusters: `PREFIX-left-cluster` and `PREFIX-right-cluster`. 73 | 74 | These clusters have the `GitOps extension` installed and it uses `Flux` to reconcile manifests from the `Application GitOps` repository. 75 | 76 | Both AKS clusters have `dev` and `stage` namespaces to represent `dev` and `stage` environments. 77 | 78 | ## Build 79 | 80 | Right after the deployment, the `Application Source` repository automatically starts `ci` workflow. It builds a Docker image and pushes it to the GitHub packages. At the very end `ci` workflow starts `deploy` workflow, that generates Kubernetes manifests for `dev` environment and creates a PR to the `Application GitOps` repository. 81 | 82 | ![cicd-tutorial-dev-pr](../../docs/images/cicd-tutorial-dev-pr.png) 83 | 84 | Both `ci` and `cd` workflows update Git commit status in the `Application Source` repository, so it reflects the current state of the entire flow: 85 | 86 | ![cicd-build-commit-status](../../docs/images/cicd-build-commit-status.png) 87 | 88 | The application has been built and promoted to `dev` environment. The created PR with the manifests to the `dev` branch in the `Application GitOps` repository represents a fact of promotion. The application has not been deployed yet. 89 | 90 | ## Deploy 91 | 92 | Once the PR with the manifests has been reviewed and merged, Flux controllers, running on the Kubernetes clusters, start to pull the changes and apply them to the cluster. In addition to that, Azure Arc GitOps controllers on the clusters report the state of this process up to Azure. It can be monitored for every cluster in Azure Portal on the `GitOps` tab: 93 | 94 | ![cicd-tutorial-gitops](../../docs/images/cicd-tutorial-gitops.png) 95 | 96 | The `Application Source` repository serves as a main `CI/CD` orchestrator. It starts a GitHub workflow `post-deployment` that polls Azure Resource Graph, waiting until the deployment process on all clusters is finished. 97 | 98 | ![cicd-tutorial-post-deployment](../../docs/images/cicd-tutorial-post-deployment.png) 99 | 100 | If the deployment on both clusters has finished successfully, the whole CD flow proceeds with the promotion to the next environment. If the deployment has failed, the whole process stops. 101 | 102 | Once the deployment is finished, it is reflected in the Git commit status: 103 | 104 | ![cicd-deployed-commit-status](../../docs/images/cicd-deployed-commit-status.png) 105 | 106 | We can manually test the application on `dev` environment on both clusters. To test the application on the `left` cluster run the following command in a terminal: 107 | 108 | ```bash 109 | kubectl port-forward svc/hello-world-functional-service -n dev 8080:8080 --context=-left-cluster 110 | 111 | # output: 112 | # Forwarding from 127.0.0.1:8080 -> 8080 113 | # Forwarding from [::1]:8080 -> 8080 114 | 115 | ``` 116 | 117 | While this command is running, open `localhost:8080` in your browser. You'll see the following greeting page: 118 | 119 | ![cicd-tutorial-greetings](../../docs/images/cicd-tutorial-greetings.png) 120 | 121 | Use different Kubernetes context to test the application on the `right` cluster: 122 | 123 | ```bash 124 | kubectl port-forward svc/hello-world-functional-service -n dev 8080:8080 --context=-right-cluster 125 | 126 | # output: 127 | # Forwarding from 127.0.0.1:8080 -> 8080 128 | # Forwarding from [::1]:8080 -> 8080 129 | 130 | ``` 131 | 132 | ## Promote 133 | 134 | After successful deployment to `dev` environment, the `post-deployment` workflow starts `deploy` flow again to promote the application version to the `stage` environment. The `deploy` workflow takes configuration values from the `stage` branch of the `Application Configs` repository, generates Kubernetes manifests and creates a PR to the `stage` branch of the `Application GitOps` repository. 135 | 136 | This fact is reflected in the `Application Source` repository commit status: 137 | 138 | ![cicd-promoted-stage-commit-status](../../docs/images/cicd-promoted-stage-commit-status.png) 139 | 140 | Once the PR with the manifests is merged, Flux starts applying the changes to the clusters. Again, the `post-deployment` workflow waits until the deployment is finished in the `stage` environment. In this sample setup there is no next environment configured after the `strage` one, so there is no any further promotion. The whole `CI/CD` flow is completed. 141 | 142 | ![cicd-tutorial-done-commit-status](../../docs/images/cicd-tutorial-done-commit-status.png) 143 | 144 | Use the following command to test the application on `stage` environment on the `left` cluster: 145 | 146 | ```bash 147 | kubectl port-forward svc/hello-world-functional-service -n stage 8080:8080 --context=-left-cluster 148 | ``` 149 | 150 | and on the `right` cluster: 151 | 152 | ```bash 153 | kubectl port-forward svc/hello-world-functional-service -n stage 8080:8080 --context=-right-cluster 154 | ``` 155 | 156 | ## Configure 157 | 158 | Let's reconfigure the application on the `dev` environment and decrease a number of replicas to 1. 159 | 160 | Open a terminal and use the following script: 161 | 162 | ```bash 163 | export org= 164 | export prefix= 165 | 166 | # clone the control-plane repo 167 | git clone https://github.com/$org/$prefix-configs configs 168 | cd configs 169 | 170 | # create workload registration file 171 | 172 | cat <functional-test/values.yaml 173 | app: 174 | name: hello-world-functional 175 | replicas: 1 176 | namespace: \$ENVIRONMENT 177 | EOF 178 | 179 | git add . 180 | git commit -m 'scale down' 181 | git push 182 | ``` 183 | 184 | This push to the `Application Configs` repository will start `deploy` workflow in the `Application Source` repository. The `deploy` workflow regenerates Kubernetes manifests for `dev` environment and creates a new PR to the `dev` branch in the `Application GitOps` repository: 185 | 186 | ![cicd-tutorial-config-pr](../../docs/images/cicd-tutorial-config-pr.png) 187 | 188 | Once the PR is merged, Flux reflects this change on the clusters and reduces the number of application pods in `dev` environment. This is an environment specific change, so it's not supposed to be promoted anywhere. 189 | 190 | ## Clean up resources 191 | 192 | When no longer needed, delete the resources that you created. To do so, run the following command: 193 | 194 | ```bash 195 | # In kalypso folder 196 | ./deploy.sh -d -p -o -t -l 197 | ``` 198 | 199 | ## Opt Out of Telemetry 200 | 201 | To opt out of telemetry collection for tutorial usage, comment out the following line in [deploy.sh](./deploy.sh). 202 | 203 | ```bash 204 | export AZURE_HTTP_USER_AGENT=... 205 | ``` 206 | --------------------------------------------------------------------------------