├── .github ├── ISSUE_TEMPLATE │ ├── BUG_REPORT.md │ └── FEATURE_REQUEST.md ├── PULL_REQUEST_TEMPLATE.md └── workflows │ ├── docker-publish-on-comment.yml │ ├── docker-publish.yml │ ├── lint.yml │ └── stale.yml ├── .gitignore ├── .golangci.yml ├── .markdownlint.json ├── CHANGELOG.md ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── Dockerfile ├── LICENSE ├── MAINTAINERS.md ├── Makefile ├── README.md ├── SECURITY.md ├── cmd └── telefonistka │ ├── bump-version-overwrite.go │ ├── bump-version-regex.go │ ├── bump-version-yaml.go │ ├── bump-version-yaml_test.go │ ├── event.go │ ├── root.go │ └── server.go ├── docs ├── installation.md ├── modeling_environments_in_gitops_repo.md ├── observability.md ├── version_bumping.md └── webhook_multiplexing.md ├── go.mod ├── go.sum ├── internal └── pkg │ ├── argocd │ ├── argocd.go │ ├── argocd_copied_from_upstream.go │ ├── argocd_test.go │ ├── diff │ │ ├── README.md │ │ ├── diff.go │ │ ├── diff_test.go │ │ └── testdata │ │ │ ├── allnew.txt │ │ │ ├── allold.txt │ │ │ ├── basic.txt │ │ │ ├── dups.txt │ │ │ ├── end.txt │ │ │ ├── eof.txt │ │ │ ├── eof1.txt │ │ │ ├── eof2.txt │ │ │ ├── long.txt │ │ │ ├── same.txt │ │ │ ├── start.txt │ │ │ └── triv.txt │ └── testdata │ │ ├── TestDiffLiveVsTargetObject │ │ ├── 1.live │ │ ├── 1.target │ │ └── 1.want │ │ ├── TestRenderDiff.live │ │ ├── TestRenderDiff.md │ │ └── TestRenderDiff.target │ ├── configuration │ ├── config.go │ ├── config_test.go │ └── tests │ │ └── testConfigurationParsing.yaml │ ├── githubapi │ ├── .tmpMJcSWN │ ├── clients.go │ ├── drift_detection.go │ ├── drift_detection_test.go │ ├── github.go │ ├── github_graphql.go │ ├── github_test.go │ ├── pr_metrics.go │ ├── pr_metrics_test.go │ ├── promotion.go │ ├── promotion_test.go │ ├── testdata │ │ ├── custom_commit_status_invalid_template.gotmpl │ │ ├── custom_commit_status_valid_template.gotmpl │ │ ├── diff_comment_data_test.json │ │ ├── pr_body.golden.md │ │ └── pr_body_multi_component.golden.md │ ├── webhook_proxy.go │ └── webhook_proxy_test.go │ ├── mocks │ ├── .gitignore │ └── mocks.go │ ├── prometheus │ ├── prometheus.go │ └── prometheus_test.go │ └── testutils │ └── testutils.go ├── main.go ├── mirrord.json ├── renovate.json └── templates ├── argoCD-diff-pr-comment-concise.gotmpl ├── argoCD-diff-pr-comment.gotmpl ├── auto-merge-comment.gotmpl ├── drift-pr-comment.gotmpl └── dry-run-pr-comment.gotmpl /.github/ISSUE_TEMPLATE/BUG_REPORT.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug Report 3 | about: Report a bug to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | ## Description 11 | 12 | Please provide a description of the problem. 13 | 14 | ## Expected Behaviour 15 | 16 | Please describe what you expected would happen. 17 | 18 | ## Actual Behaviour 19 | 20 | Please describe what happened instead. 21 | 22 | ## Affected Version 23 | 24 | Please provide the version number where this issue was encountered. 25 | 26 | ## Steps to Reproduce 27 | 28 | 1. First step 29 | 1. Second step 30 | 1. etc. 31 | 32 | ## Checklist 33 | 34 | 35 | - [ ] I have read the [contributing guidelines](https://github.com/wayfair-incubator/telefonistka/blob/main/CONTRIBUTING.md) 36 | - [ ] I have verified this does not duplicate an existing issue 37 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/FEATURE_REQUEST.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature Request 3 | about: Suggest a feature for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | ## Problem Statement 11 | 12 | Please describe the problem to be addressed by the proposed feature. 13 | 14 | ## Proposed Solution 15 | 16 | Please describe what you envision the solution to this problem would look like. 17 | 18 | ## Alternatives Considered 19 | 20 | Please briefly describe which alternatives, if any, have been considered, including merits of alternate approaches and 21 | tradeoffs being made. 22 | 23 | ## Additional Context 24 | 25 | Please provide any other information that may be relevant. 26 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | ## Description 2 | 3 | Please provide a meaningful description of what this change will do, or is for. Bonus points for including links to related issues, other PRs, or technical references. 4 | 5 | Note that by _not_ including a description, you are asking reviewers to do extra work to understand the context of this change, which may lead to your PR taking much longer to review, or result in it not being reviewed at all. 6 | 7 | ## Type of Change 8 | 9 | - [ ] Bug Fix 10 | - [ ] New Feature 11 | - [ ] Breaking Change 12 | - [ ] Refactor 13 | - [ ] Documentation 14 | - [ ] Other (please describe) 15 | 16 | ## Checklist 17 | 18 | 19 | - [ ] I have read the [contributing guidelines](https://github.com/wayfair-incubator/telefonistka/blob/main/CONTRIBUTING.md) 20 | - [ ] Existing issues have been referenced (where applicable) 21 | - [ ] I have verified this change is not present in other open pull requests 22 | - [ ] Functionality is documented 23 | - [ ] All code style checks pass 24 | - [ ] New code contribution is covered by automated tests 25 | - [ ] All new and existing tests pass 26 | -------------------------------------------------------------------------------- /.github/workflows/docker-publish-on-comment.yml: -------------------------------------------------------------------------------- 1 | name: oci-image-publish-on-comment 2 | 3 | # This workflow uses actions that are not certified by GitHub. 4 | # They are provided by a third-party and are governed by 5 | # separate terms of service, privacy policy, and support 6 | # documentation. 7 | 8 | on: 9 | issue_comment: 10 | types: [created] 11 | 12 | 13 | env: 14 | DOCKERHUB_USERNAME: ${{ vars.DOCKERHUB_USERNAME }} 15 | IMAGE_NAME: ${{ vars.IMAGE_NAME }} 16 | REGISTRY: ${{ vars.REGISTRY }} 17 | 18 | jobs: 19 | build: 20 | runs-on: ubuntu-latest 21 | if: github.event.issue.pull_request && contains(github.event.comment.body, '/publish') && github.event.comment.user.login == 'Oded-B' 22 | permissions: 23 | contents: read 24 | packages: write 25 | # This is used to complete the identity challenge 26 | # with sigstore/fulcio when running outside of PRs. 27 | id-token: write 28 | statuses: write 29 | pull-requests: write 30 | issues: write 31 | 32 | steps: 33 | - name: Get PR branch 34 | uses: xt0rted/pull-request-comment-branch@v3 35 | id: comment-branch 36 | - name: Checkout repository 37 | uses: actions/checkout@v4 38 | with: 39 | ref: ${{ steps.comment-branch.outputs.head_ref }} 40 | - name: Set latest commit status as pending 41 | uses: myrotvorets/set-commit-status-action@master 42 | with: 43 | sha: ${{ steps.comment-branch.outputs.head_sha }} 44 | token: ${{ secrets.GITHUB_TOKEN }} 45 | status: pending 46 | 47 | # Workaround: https://github.com/docker/build-push-action/issues/461 48 | - name: Setup Docker buildx 49 | uses: docker/setup-buildx-action@6524bf65af31da8d45b59e8c27de4bd072b392f5 50 | 51 | # Login against a Docker registry except on PR 52 | # https://github.com/docker/login-action 53 | - name: Log into GH registry (ghcr.io) 54 | uses: docker/login-action@7ca345011ac4304463197fac0e56eab1bc7e6af0 55 | with: 56 | registry: ghcr.io 57 | username: ${{ github.actor }} 58 | password: ${{ secrets.GITHUB_TOKEN }} 59 | 60 | - name: Log into Docker Hub registry 61 | if: env.DOCKERHUB_USERNAME != '' 62 | uses: docker/login-action@7ca345011ac4304463197fac0e56eab1bc7e6af0 63 | with: 64 | username: ${{ env.DOCKERHUB_USERNAME }} 65 | password: ${{ secrets.DOCKERHUB_TOKEN }} 66 | 67 | # Extract metadata (tags, labels) for Docker 68 | # https://github.com/docker/metadata-action 69 | # 1st image name is for GH package repo 70 | # 2nd image name is for DockerHub image 71 | - name: Extract Docker metadata 72 | id: meta 73 | uses: docker/metadata-action@906ecf0fc0a80f9110f79d9e6c04b1080f4a2621 74 | with: 75 | context: git 76 | images: | 77 | ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} 78 | tags: | 79 | type=ref,event=branch 80 | type=ref,event=pr 81 | type=sha 82 | 83 | 84 | # Build and push Docker image with Buildx (don't push on PR) 85 | # https://github.com/docker/build-push-action 86 | - name: Build and push Docker image 87 | id: build-and-push 88 | uses: docker/build-push-action@7e094594beda23fc8f21fa31049f4b203e51096b 89 | with: 90 | context: . 91 | push: true 92 | tags: ${{ steps.meta.outputs.tags }} 93 | labels: ${{ steps.meta.outputs.labels }} 94 | cache-from: type=gha 95 | cache-to: type=gha,mode=max 96 | 97 | - name: Extract Docker metadata - alpine 98 | id: meta-alpine 99 | uses: docker/metadata-action@906ecf0fc0a80f9110f79d9e6c04b1080f4a2621 100 | with: 101 | context: git 102 | images: | 103 | ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} 104 | tags: | 105 | type=ref,event=branch 106 | type=ref,event=pr 107 | type=sha 108 | flavor: prefix=alpine-,onlatest=true 109 | - name: Build and push Docker image - alpine 110 | id: build-and-push-alpine 111 | uses: docker/build-push-action@7e094594beda23fc8f21fa31049f4b203e51096b 112 | with: 113 | context: . 114 | target: alpine-release 115 | push: true 116 | tags: ${{ steps.meta-alpine.outputs.tags }} 117 | labels: ${{ steps.meta-alpine.outputs.labels }} 118 | cache-from: type=gha 119 | cache-to: type=gha,mode=max 120 | - name: Set latest commit status as ${{ job.status }} 121 | uses: myrotvorets/set-commit-status-action@master 122 | if: always() 123 | with: 124 | sha: ${{ steps.comment-branch.outputs.head_sha }} 125 | token: ${{ secrets.GITHUB_TOKEN }} 126 | status: ${{ job.status }} 127 | - name: Add comment to PR 128 | uses: actions/github-script@v7 129 | if: always() 130 | with: 131 | script: | 132 | const name = '${{ github.workflow }}'; 133 | const url = '${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}'; 134 | const success = '${{ job.status }}' === 'success'; 135 | const body = `${name}: ${success ? 'succeeded ✅' : 'failed ❌'}\n${url}\n${{ steps.meta.outputs.tags }}`; 136 | 137 | await github.rest.issues.createComment({ 138 | issue_number: context.issue.number, 139 | owner: context.repo.owner, 140 | repo: context.repo.repo, 141 | body: body 142 | }) 143 | -------------------------------------------------------------------------------- /.github/workflows/docker-publish.yml: -------------------------------------------------------------------------------- 1 | name: Docker 2 | 3 | # This workflow uses actions that are not certified by GitHub. 4 | # They are provided by a third-party and are governed by 5 | # separate terms of service, privacy policy, and support 6 | # documentation. 7 | 8 | on: 9 | # push: 10 | # branches: [ "main" ] 11 | # Publish semver tags as releases. 12 | # tags: [ 'v*.*.*' ] 13 | release: 14 | types: [published] 15 | pull_request: 16 | branches: [ "main" ] 17 | 18 | env: 19 | DOCKERHUB_USERNAME: ${{ vars.DOCKERHUB_USERNAME }} 20 | IMAGE_NAME: ${{ vars.IMAGE_NAME }} 21 | REGISTRY: ${{ vars.REGISTRY }} 22 | 23 | jobs: 24 | build: 25 | 26 | runs-on: ubuntu-latest 27 | permissions: 28 | contents: read 29 | packages: write 30 | # This is used to complete the identity challenge 31 | # with sigstore/fulcio when running outside of PRs. 32 | id-token: write 33 | 34 | steps: 35 | - name: Checkout repository 36 | uses: actions/checkout@v4 37 | 38 | # Install the cosign tool except on PR 39 | # https://github.com/sigstore/cosign-installer 40 | # TODO enable once the repo goes public 41 | # - name: Install cosign 42 | # if: github.event_name != 'pull_request' 43 | # uses: sigstore/cosign-installer@f3c664df7af409cb4873aa5068053ba9d61a57b6 #v2.6.0 44 | # with: 45 | # cosign-release: 'v1.11.0' 46 | 47 | 48 | # Workaround: https://github.com/docker/build-push-action/issues/461 49 | - name: Setup Docker buildx 50 | uses: docker/setup-buildx-action@6524bf65af31da8d45b59e8c27de4bd072b392f5 51 | 52 | # Login against a Docker registry except on PR 53 | # https://github.com/docker/login-action 54 | - name: Log into GH registry (ghcr.io) 55 | if: github.event_name != 'pull_request' 56 | uses: docker/login-action@7ca345011ac4304463197fac0e56eab1bc7e6af0 57 | with: 58 | registry: ghcr.io 59 | username: ${{ github.actor }} 60 | password: ${{ secrets.GITHUB_TOKEN }} 61 | 62 | - name: Log into Docker Hub registry 63 | if: github.event_name != 'pull_request' && env.DOCKERHUB_USERNAME != '' 64 | uses: docker/login-action@7ca345011ac4304463197fac0e56eab1bc7e6af0 65 | with: 66 | username: ${{ env.DOCKERHUB_USERNAME }} 67 | password: ${{ secrets.DOCKERHUB_TOKEN }} 68 | 69 | # Extract metadata (tags, labels) for Docker 70 | # https://github.com/docker/metadata-action 71 | # 1st image name is for GH package repo 72 | # 2nd image name is for DockerHub image 73 | - name: Extract Docker metadata 74 | id: meta 75 | uses: docker/metadata-action@906ecf0fc0a80f9110f79d9e6c04b1080f4a2621 76 | with: 77 | images: | 78 | ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} 79 | 80 | 81 | # Build and push Docker image with Buildx (don't push on PR) 82 | # https://github.com/docker/build-push-action 83 | - name: Build and push Docker image 84 | id: build-and-push 85 | uses: docker/build-push-action@7e094594beda23fc8f21fa31049f4b203e51096b 86 | with: 87 | context: . 88 | push: ${{ github.event_name != 'pull_request' }} 89 | tags: ${{ steps.meta.outputs.tags }} 90 | labels: ${{ steps.meta.outputs.labels }} 91 | cache-from: type=gha 92 | cache-to: type=gha,mode=max 93 | 94 | # Extract metadata (tags, labels) for Docker 95 | # https://github.com/docker/metadata-action 96 | # 1st image name is for GH package repo 97 | # 2nd image name is for DockerHub image 98 | - name: Extract Docker metadata - alpine 99 | id: meta-alpine 100 | uses: docker/metadata-action@906ecf0fc0a80f9110f79d9e6c04b1080f4a2621 101 | with: 102 | images: | 103 | ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} 104 | flavor: prefix=alpine-,onlatest=true 105 | 106 | # Build and push Docker image with Buildx (don't push on PR) 107 | # https://github.com/docker/build-push-action 108 | - name: Build and push Docker image - alpine 109 | id: build-and-push-alpine 110 | uses: docker/build-push-action@7e094594beda23fc8f21fa31049f4b203e51096b 111 | with: 112 | context: . 113 | target: alpine-release 114 | push: ${{ github.event_name != 'pull_request' }} 115 | tags: ${{ steps.meta-alpine.outputs.tags }} 116 | labels: ${{ steps.meta-alpine.outputs.labels }} 117 | cache-from: type=gha 118 | cache-to: type=gha,mode=max 119 | 120 | 121 | # Sign the resulting Docker image digest except on PRs. 122 | # This will only write to the public Rekor transparency log when the Docker 123 | # repository is public to avoid leaking data. If you would like to publish 124 | # transparency data even for private images, pass --force to cosign below. 125 | # https://github.com/sigstore/cosign 126 | # TODO enable once the repo goes public 127 | # - name: Sign the published Docker image 128 | # if: ${{ github.event_name != 'pull_request' }} 129 | # env: 130 | # COSIGN_EXPERIMENTAL: "true" 131 | # This step uses the identity token to provision an ephemeral certificate 132 | # against the sigstore community Fulcio instance. 133 | # run: echo "${{ steps.meta.outputs.tags }}" | xargs -I {} cosign sign {}@${{ steps.build-and-push.outputs.digest }} 134 | 135 | -------------------------------------------------------------------------------- /.github/workflows/lint.yml: -------------------------------------------------------------------------------- 1 | name: lint 2 | 3 | on: # Rebuild any PRs and main branch changes 4 | push: 5 | branches: 6 | - main 7 | pull_request: 8 | branches: 9 | - main 10 | 11 | jobs: 12 | markdown: 13 | runs-on: ubuntu-latest 14 | steps: 15 | - uses: actions/checkout@v4 16 | - name: ⬇️ lint markdown files # Lints all markdown (.md) files 17 | uses: avto-dev/markdown-lint@v1 18 | with: 19 | config: '.markdownlint.json' 20 | args: '**/*.md .github/**/*.md' 21 | renovate: 22 | runs-on: ubuntu-latest 23 | steps: 24 | - uses: actions/checkout@v4 25 | - name: 🧼 lint renovate config # Validates changes to renovate.json config file 26 | uses: suzuki-shunsuke/github-action-renovate-config-validator@v1.1.1 27 | with: 28 | config_file_path: 'renovate.json' 29 | golangci: 30 | name: golangci-lint 31 | runs-on: ubuntu-latest 32 | steps: 33 | - uses: actions/setup-go@v5 34 | with: 35 | go-version: 1.21 36 | - uses: actions/checkout@v4 37 | - run: make get-deps 38 | - name: golangci-lint 39 | uses: golangci/golangci-lint-action@v6 40 | with: 41 | # Optional: version of golangci-lint to use in form of v1.2 or v1.2.3 or `latest` to use the latest version 42 | version: v1.60.3 43 | 44 | # Optional: working directory, useful for monorepos 45 | # working-directory: somedir 46 | 47 | # Optional: golangci-lint command line arguments. 48 | # args: --issues-exit-code=0 49 | 50 | # Optional: show only new issues if it's a pull request. The default value is `false`. 51 | # only-new-issues: true 52 | 53 | # Optional: if set to true then the all caching functionality will be complete disabled, 54 | # takes precedence over all other caching options. 55 | # skip-cache: true 56 | 57 | # Optional: if set to true then the action don't cache or restore ~/go/pkg. 58 | # skip-pkg-cache: true 59 | 60 | # Optional: if set to true then the action don't cache or restore ~/.cache/go-build. 61 | # skip-build-cache: true 62 | 63 | -------------------------------------------------------------------------------- /.github/workflows/stale.yml: -------------------------------------------------------------------------------- 1 | name: stale 2 | 3 | on: 4 | schedule: 5 | - cron: "0 0 * * 0" 6 | 7 | jobs: 8 | stale: 9 | runs-on: ubuntu-latest 10 | permissions: 11 | issues: write 12 | pull-requests: write 13 | steps: 14 | - name: 📆 mark stale PRs # Automatically marks inactive PRs as stale 15 | uses: actions/stale@v9 16 | with: 17 | repo-token: ${{ secrets.GITHUB_TOKEN }} 18 | days-before-stale: 60 19 | stale-issue-label: 'stale' 20 | stale-pr-label: 'stale' 21 | stale-issue-message: 'Automatically marking issue as stale due to lack of activity' 22 | stale-pr-message: 'Automatically marking pull request as stale due to lack of activity' 23 | days-before-close: 7 24 | close-issue-message: 'Automatically closing this issue as stale' 25 | close-pr-message: 'Automatically closing this pull request as stale' 26 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /telefonistka 2 | vendor/ 3 | internal/pkg/mocks/argocd_settings.go 4 | internal/pkg/mocks/argocd_project.go 5 | -------------------------------------------------------------------------------- /.golangci.yml: -------------------------------------------------------------------------------- 1 | linters: 2 | enable: 3 | - asasalint 4 | - asciicheck 5 | - bidichk 6 | - bodyclose 7 | - containedctx 8 | - decorder 9 | - dogsled 10 | - durationcheck 11 | - errcheck 12 | - errname 13 | - exportloopref 14 | - gci 15 | - gochecknoinits 16 | - gofmt 17 | - gofumpt 18 | - goimports 19 | - goprintffuncname 20 | - gosec 21 | - gosimple 22 | - govet 23 | - grouper 24 | - importas 25 | - ineffassign 26 | - makezero 27 | - misspell 28 | - noctx 29 | - nolintlint 30 | - nosprintfhostport 31 | - paralleltest 32 | - staticcheck 33 | - tenv 34 | - thelper 35 | - tparallel 36 | - typecheck 37 | - unconvert 38 | - unused 39 | - whitespace 40 | 41 | run: 42 | timeout: 10m 43 | -------------------------------------------------------------------------------- /.markdownlint.json: -------------------------------------------------------------------------------- 1 | { 2 | "default": true, 3 | "MD013": { 4 | "line_length": 10000, 5 | "headings": false, 6 | "code_blocks": false, 7 | "tables": false 8 | }, 9 | "MD024": { 10 | "siblings_only": true 11 | }, 12 | "MD025": { 13 | "front_matter_title": "" 14 | }, 15 | "MD041": false, 16 | "MD034": false 17 | } 18 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | All notable changes to this project will be documented in this file. 4 | 5 | The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), 6 | and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 7 | 8 | ## [Unreleased] 9 | 10 | ### TBA 11 | 12 | ## [0.0.1] - 2023-01-26 13 | 14 | ### Added 15 | 16 | - Initial code commit 17 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | We as members, contributors, and leaders pledge to make participation in our 6 | community a harassment-free experience for everyone, regardless of age, body 7 | size, visible or invisible disability, ethnicity, sex characteristics, gender 8 | identity and expression, level of experience, education, socio-economic status, 9 | nationality, personal appearance, race, caste, color, religion, or sexual identity 10 | and orientation. 11 | 12 | We pledge to act and interact in ways that contribute to an open, welcoming, 13 | diverse, inclusive, and healthy community. 14 | 15 | ## Our Standards 16 | 17 | Examples of behavior that contributes to a positive environment for our 18 | community include: 19 | 20 | * Demonstrating empathy and kindness toward other people 21 | * Being respectful of differing opinions, viewpoints, and experiences 22 | * Giving and gracefully accepting constructive feedback 23 | * Accepting responsibility and apologizing to those affected by our mistakes, 24 | and learning from the experience 25 | * Focusing on what is best not just for us as individuals, but for the 26 | overall community 27 | 28 | Examples of unacceptable behavior include: 29 | 30 | * The use of sexualized language or imagery, and sexual attention or 31 | advances of any kind 32 | * Trolling, insulting or derogatory comments, and personal or political attacks 33 | * Public or private harassment 34 | * Publishing others' private information, such as a physical or email 35 | address, without their explicit permission 36 | * Other conduct which could reasonably be considered inappropriate in a 37 | professional setting 38 | 39 | ## Enforcement Responsibilities 40 | 41 | Community leaders are responsible for clarifying and enforcing our standards of 42 | acceptable behavior and will take appropriate and fair corrective action in 43 | response to any behavior that they deem inappropriate, threatening, offensive, 44 | or harmful. 45 | 46 | Community leaders have the right and responsibility to remove, edit, or reject 47 | comments, commits, code, wiki edits, issues, and other contributions that are 48 | not aligned to this Code of Conduct, and will communicate reasons for moderation 49 | decisions when appropriate. 50 | 51 | ## Scope 52 | 53 | This Code of Conduct applies within all community spaces, and also applies when 54 | an individual is officially representing the community in public spaces. 55 | Examples of representing our community include using an official e-mail address, 56 | posting via an official social media account, or acting as an appointed 57 | representative at an online or offline event. 58 | 59 | ## Enforcement 60 | 61 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 62 | reported to the [community leaders](MAINTAINERS.md) responsible for enforcement. 63 | All complaints will be reviewed and investigated promptly and fairly. 64 | 65 | All community leaders are obligated to respect the privacy and security of the 66 | reporter of any incident. 67 | 68 | ## Enforcement Guidelines 69 | 70 | Community leaders will follow these Community Impact Guidelines in determining 71 | the consequences for any action they deem in violation of this Code of Conduct: 72 | 73 | ### 1. Correction 74 | 75 | **Community Impact**: Use of inappropriate language or other behavior deemed 76 | unprofessional or unwelcome in the community. 77 | 78 | **Consequence**: A private, written warning from community leaders, providing 79 | clarity around the nature of the violation and an explanation of why the 80 | behavior was inappropriate. A public apology may be requested. 81 | 82 | ### 2. Warning 83 | 84 | **Community Impact**: A violation through a single incident or series 85 | of actions. 86 | 87 | **Consequence**: A warning with consequences for continued behavior. No 88 | interaction with the people involved, including unsolicited interaction with 89 | those enforcing the Code of Conduct, for a specified period of time. This 90 | includes avoiding interactions in community spaces as well as external channels 91 | like social media. Violating these terms may lead to a temporary or 92 | permanent ban. 93 | 94 | ### 3. Temporary Ban 95 | 96 | **Community Impact**: A serious violation of community standards, including 97 | sustained inappropriate behavior. 98 | 99 | **Consequence**: A temporary ban from any sort of interaction or public 100 | communication with the community for a specified period of time. No public or 101 | private interaction with the people involved, including unsolicited interaction 102 | with those enforcing the Code of Conduct, is allowed during this period. 103 | Violating these terms may lead to a permanent ban. 104 | 105 | ### 4. Permanent Ban 106 | 107 | **Community Impact**: Demonstrating a pattern of violation of community 108 | standards, including sustained inappropriate behavior, harassment of an 109 | individual, or aggression toward or disparagement of classes of individuals. 110 | 111 | **Consequence**: A permanent ban from any sort of public interaction within 112 | the community. 113 | 114 | ## Attribution 115 | 116 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], 117 | version 2.0, available at 118 | [https://www.contributor-covenant.org/version/2/0/code_of_conduct.html][v2.0]. 119 | 120 | Community Impact Guidelines were inspired by 121 | [Mozilla's code of conduct enforcement ladder][Mozilla CoC]. 122 | 123 | For answers to common questions about this code of conduct, see the FAQ at 124 | [https://www.contributor-covenant.org/faq][FAQ]. Translations are available 125 | at [https://www.contributor-covenant.org/translations][translations]. 126 | 127 | [homepage]: https://www.contributor-covenant.org 128 | [v2.0]: https://www.contributor-covenant.org/version/2/0/code_of_conduct.html 129 | [Mozilla CoC]: https://github.com/mozilla/diversity 130 | [FAQ]: https://www.contributor-covenant.org/faq 131 | [translations]: https://www.contributor-covenant.org/translations 132 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # How to Contribute 2 | 3 | Thanks for your interest in contributing to Telefonistka! Here are a few general guidelines on contributing and 4 | reporting bugs that we ask you to review. Following these guidelines helps to communicate that you respect the time of 5 | the contributors managing and developing this open source project. In return, they should reciprocate that respect in 6 | addressing your issue, assessing changes, and helping you finalize your pull requests. In that spirit of mutual respect, 7 | we endeavour to review incoming issues and pull requests within 10 days, and will close any lingering issues or pull 8 | requests after 60 days of inactivity. 9 | 10 | Please note that all of your interactions in the project are subject to our [Code of Conduct](CODE_OF_CONDUCT.md). This 11 | includes creation of issues or pull requests, commenting on issues or pull requests, and extends to all interactions in 12 | any real-time space (eg. Slack, Discord, etc). 13 | 14 | ## Reporting Issues 15 | 16 | Before reporting a new issue, please ensure that the issue was not already reported or fixed by searching through our 17 | [issues list](https://github.com/wayfair-incubator/telefonistka/issues). 18 | 19 | When creating a new issue, please be sure to include a **title and clear description**, as much relevant information as 20 | possible, and, if possible, a test case. 21 | 22 | **If you discover a security bug, please do not report it through GitHub. Instead, please see security procedures in 23 | [SECURITY.md](SECURITY.md).** 24 | 25 | ## Sending Pull Requests 26 | 27 | Before sending a new pull request, take a look at existing pull requests and issues to see if the proposed change or fix 28 | has been discussed in the past, or if the change was already implemented but not yet released. 29 | 30 | We expect new pull requests to include tests for any affected behavior, and, as we follow semantic versioning, we may 31 | reserve breaking changes until the next major version release. 32 | 33 | ## Other Ways to Contribute 34 | 35 | We welcome anyone that wants to contribute to Telefonistka to triage and reply to open issues to help troubleshoot 36 | and fix existing bugs. Here is what you can do: 37 | 38 | - Help ensure that existing issues follows the recommendations from the _[Reporting Issues](#reporting-issues)_ section, 39 | providing feedback to the issue's author on what might be missing. 40 | - Review and update the existing content of our [Wiki](https://github.com/wayfair-incubator/telefonistka/wiki) with up-to-date 41 | instructions and code samples. 42 | - Review existing pull requests, and testing patches against real existing applications that use Telefonistka. 43 | - Write a test, or add a missing test case to an existing test. 44 | 45 | Thanks again for your interest on contributing to Telefonistka! 46 | 47 | :heart: 48 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | 2 | FROM golang:1.23.4 as test 3 | ARG GOPROXY 4 | ENV GOPATH=/go 5 | ENV PATH="$PATH:$GOPATH/bin" 6 | WORKDIR /go/src/github.com/wayfair-incubator/telefonistka 7 | COPY . ./ 8 | RUN make test 9 | 10 | FROM test as build 11 | ARG GOPROXY 12 | ENV GOPATH=/go 13 | ENV PATH="$PATH:$GOPATH/bin" 14 | WORKDIR /go/src/github.com/wayfair-incubator/telefonistka 15 | COPY . ./ 16 | RUN make build 17 | 18 | 19 | FROM alpine:latest as alpine-release 20 | WORKDIR /telefonistka 21 | COPY --from=build /go/src/github.com/wayfair-incubator/telefonistka/telefonistka /telefonistka/bin/telefonistka 22 | COPY templates/ /telefonistka/templates/ 23 | # This next line is hack to overcome GH actions lack of support for docker workdir override https://github.com/actions/runner/issues/878 24 | COPY templates/ /github/workspace/templates/ 25 | USER 1001 26 | ENTRYPOINT ["/telefonistka/bin/telefonistka"] 27 | CMD ["server"] 28 | 29 | 30 | 31 | FROM scratch 32 | ENV wf_version="0.0.5" 33 | ENV wf_description="K8s team GitOps prmoter webhook server" 34 | WORKDIR /telefonistka 35 | COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ 36 | COPY --from=build /go/src/github.com/wayfair-incubator/telefonistka/telefonistka /telefonistka/bin/telefonistka 37 | COPY templates/ /telefonistka/templates/ 38 | # This next line is hack to overcome GH actions lack of support for docker workdir override https://github.com/actions/runner/issues/878 39 | COPY templates/ /github/workspace/templates/ 40 | USER 1001 41 | ENTRYPOINT ["/telefonistka/bin/telefonistka"] 42 | CMD ["server"] 43 | 44 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Wayfair Tech – Incubator 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /MAINTAINERS.md: -------------------------------------------------------------------------------- 1 | # Maintainers 2 | 3 | - [Oded Ben Ozer](https://github.com/Oded-B) 4 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # 2 | # Makefile 3 | # 4 | # Simple makefile to build binary. 5 | # 6 | # @author Kubernetes Team 7 | # @copyright 2019 Wayfair, LLC. -- All rights reserved. 8 | 9 | VENDOR_DIR = vendor 10 | 11 | .PHONY: get-deps 12 | get-deps: $(VENDOR_DIR) 13 | 14 | $(VENDOR_DIR): 15 | go generate $$(go list ./internal/pkg/mocks/...) 16 | GO111MODULE=on go mod vendor 17 | 18 | .PHONY: build 19 | build: $(VENDOR_DIR) 20 | GOOS=linux CGO_ENABLED=0 go build -a -ldflags '-extldflags "-static"' -o telefonistka . 21 | 22 | .PHONY: clean 23 | clean: 24 | rm -f telefonistka 25 | 26 | .PHONY: test 27 | test: $(VENDOR_DIR) 28 | TEMPLATES_PATH=../../../templates/ go test -v -timeout 30s ./... 29 | 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | > [!IMPORTANT] 2 | > Development of Telefonistka will continue under the commercetools GitHub org at https://github.com/commercetools/telefonistka 3 | > Most of Telefonistka maintainers work in commercetools and I am no longer in contact with Wayfair GH org administrator so I feel this will better serve our small community. 4 | > Just to be clear - the project is currently active, averaging > 3 commits on most work weeks (not including dependency updates!), this move is just about ensuring the continuity of the project. 5 | 6 | 7 |

8 | 9 | #

Telefonistka

10 | 11 | 12 | 13 | Telefonistka is a Github webhook server/Bot that facilitates change promotion across environments/failure domains in Infrastructure as Code(IaC) GitOps repos. 14 | 15 | It assumes the [repeatable part of your infrastucture is modeled in folders](#modeling-environmentsfailure-domains-in-an-iac-gitops-repo) 16 | 17 | Based on configuration in the IaC repo, the bot will open pull requests that sync components from "sourcePaths" to "targetPaths". 18 | 19 | Providing reasonably flexible control over what is promoted to where and in what order. 20 | 21 | A 10 minutes ArgoCon EU 2023 session describing the project: 22 | 23 | [![ArgoCon EU 2023 session](https://img.youtube.com/vi/oiSsSiROj10/0.jpg)](https://www.youtube.com/watch?v=oiSsSiROj10) 24 | 25 | ## Modeling environments/failure-domains in an IaC GitOps repo 26 | 27 | RY is the new DRY! 28 | 29 | In GitOps IaC implementations, different environments(`dev`/`prod`/...) and failure domains(`us-east-1`/`us-west-1`/...) must be represented in distinct files, folders, Git branches or even repositories to allow gradual and controlled rollout of changes across said environments/failure domains. 30 | 31 | At Wayfair's Kubernetes team we choose the "folders" approach, more about other choices [here](docs/modeling_environments_in_gitops_repo.md). 32 | 33 | Specifically, we choose the following scheme to represent all the Infrastructure components running in our Kubernetes clusters: 34 | `clusters`/`[environment]`/`[cloud region]`/`[cluster identifier]`/`[component name]` 35 | 36 | for example: 37 | 38 | ```text 39 | clusters/staging/us-central1/c2/prometheus/ 40 | clusters/staging/us-central1/c2/nginx-ingress/ 41 | clusters/prod/us-central1/c2/prometheus/ 42 | clusters/prod/us-central1/c2/nginx-ingress/ 43 | clusters/prod/europe-west4/c2/prometheus/ 44 | clusters/prod/europe-west4/c2/nginx-ingress/ 45 | ``` 46 | 47 | While this approach provides multiple benefits it does mean the user is expected to make changes in multiple files and folders in order to apply a single change to multiple environments/FDs. 48 | 49 | Manually syncing those files is time consuming, error prone and generally not fun. And in the long run, undesired drift between those environments/FDs is almost guaranteed to accumulate as humans do that thing where they fail to be perfect at what they do. 50 | 51 | This is where Telefonistka comes in. 52 | 53 | Telefonistka will automagically create pull requests that "sync" our changes to the right folder or folders, enabling the usage of the familiar PR functionality to control promotions while avoiding the toil related to manually syncing directories and checking for environments/FDs drift. 54 | 55 | ## Notable Features 56 | 57 | ### IaC stack agnostic 58 | 59 | Terraform, Helmfile, ArgoCD whatever, as long as environments and sites are modeled as folders and components are copied between environments "as is". 60 | 61 | ### Unopinionated directory structure 62 | 63 | The [in-configuration file](docs/installation.md#repo-configuration) is flexible and even has some regex support. 64 | 65 | The project goal is support any reasonable setup and we'll try to address unsupported setups. 66 | 67 | ### Multi stage promotion schemes 68 | 69 | ```text 70 | lab -> staging -> production 71 | ``` 72 | 73 | or 74 | 75 | ```text 76 | dev -> production-us-east-1 -> production-us-east-3 -> production-eu-east-1 77 | ``` 78 | 79 | Fan out, like: 80 | 81 | ```text 82 | lab -> staging1 --> 83 | staging2 --> production 84 | staging3 --> 85 | ``` 86 | 87 | Telefonistka annotates the PR with the historic "flow" of the promotion: 88 | 89 | 90 | 91 | 92 | 93 | ### Control granularity of promotion PRs 94 | 95 | Allows separating promotions into a separate PRs per environment/failure domain or group some/all of them. 96 | 97 | e.g. "Sync all dev clusters in one PR but open a dedicated PR for every production cluster" 98 | 99 | Also allows automatic merging of PRs based on the promotion policy. 100 | 101 | e.g. "Automatically merge PRs that promote to multiple `lab` environments" 102 | 103 | ### Optional per-component allow/block override list 104 | 105 | Allows overriding the general(per-repo) promotion policy on a per component level. 106 | 107 | e.g. "This component should not be deployed to production" or "Promote this only to the us-east-4 region" 108 | 109 | ### Drift detection and warning 110 | 111 | Warns user on [drift between environment/failure domains](docs/modeling_environments_in_gitops_repo.md#terminology) on open PRs ("Staging and Production are not synced, these are the differences") 112 | This is how this warning looks in the PR: 113 | 114 | 115 | 116 | 117 | 118 | ### ArgoCD integration 119 | 120 | Telefonistka can compare manifests in PR branches to live objects in the clusters and comment on the difference in PRs 121 | 122 | 123 | image 124 | 125 | 126 | ### Artifact version bumping from CLI 127 | 128 | If your IaC repo deploys software you maintain internally you probably want to automate artifact version bumping. 129 | Telefonistka can automate opening the IaC repo PR for the version change from the Code repo pipeline: 130 | 131 | ```shell 132 | telefonistka bump-overwrite \ 133 | --target-repo Oded-B/telefonistka-example \ 134 | --target-file workspace/nginx/values-version.yaml \ 135 | --file <(echo -e "image:\n tag: v3.4.9") \ 136 | ``` 137 | 138 | It currently supports full file overwrite, regex and yaml based replacement. 139 | See [here](docs/version_bumping.md) for more details 140 | 141 | ### GitHub Push events fanout/multiplexing 142 | 143 | Some GitOps operators can listen for GitHub webhooks to ensure short delays in the reconciliation loop. 144 | 145 | But in some scenarios the number of needed webhooks endpoint exceed the maximum supported by GitHub(think 10 cluster each with in-cluster ArgoCD server and ArgoCD applicationSet controller). 146 | 147 | Telefonistka can forward these HTTP requests to multiple endpoint and can even filter or dynamically choose the endpoint URL based on the file changed in the Commit. 148 | 149 | This example configuration includes regex bases endpoint URL generation: 150 | 151 | ```yaml 152 | webhookEndpointRegexs: 153 | - expression: "^workspace/[^/]*/.*" 154 | replacements: 155 | - "https://kube-argocd-c1.service.lab.example.com/api/webhoook" 156 | - "https://kube-argocd-applicationset-c1.service.lab.example.com/api/webhoook" 157 | - "https://example.com" 158 | - expression: "^clusters/([^/]*)/([^/]*)/([^/]*)/.*" 159 | replacements: 160 | - "https://kube-argocd-${3}.${1}.service.{2}.example.com/api/webhoook" 161 | - "https://kube-argocd-applicationset-${2}.service.${1}.example.com/api/webhoook" 162 | 163 | ``` 164 | 165 | see [here](docs/webhook_multiplexing.md) for more details 166 | 167 | ## Installation and Configuration 168 | 169 | See [here](docs/installation.md) 170 | 171 | ## Observability 172 | 173 | See [here](docs/observability.md) 174 | 175 | ## Development 176 | 177 | ### Local Testing 178 | 179 | Telefonistka have 3 major methods to interact with the world: 180 | 181 | * Receive event webhooks from GitHub 182 | * Send API calls to GitHub REST and GraphQL APIs(requires network access and credentials) 183 | * Send API calls to ArgoCD API(requires network access and credentials) 184 | 185 | Supporting all those requirements in a local environment might require lots of setup. 186 | Assuming you have a working lab environment, the easiest way to locally test Telefonistka might be with tools like [mirrord](https://mirrord.dev/) or [telepresence](https://www.telepresence.io/) 187 | 188 | A [mirrord.json](mirrord.json) is supplied as reference. 189 | 190 | This is how I compile and trigger mirrord execution 191 | 192 | ```sh 193 | go build . && mirrord exec -f mirrord.json ./telefonistka server 194 | ``` 195 | 196 | Alternatively, you can use `ngrok` or similar services to route webhook to a local instance, but you still need to provide credentials to all outbound API calls. 197 | 198 | * use Ngrok ( `ngrok http 8080` ) to expose the local instance 199 | * See the URLs in ngrok command output. 200 | * Add a webhook to repo setting (don't forget the `/webhook` path in the URL). 201 | * Content type needs to be `application/json`, **currently** only PR events are needed 202 | 203 | ### Building Container Image From Forks 204 | 205 | To publish container images from a forked repo set the `IMAGE_NAME` and `REGISTRY` GitHub Action Repository variables to use GitHub packages. 206 | `REGISTRY` should be `ghcr.io` and `IMAGE_NAME` should match the repository slug, like so: 207 | like so: 208 | 209 | image 210 | 211 | 212 | ## Roadmap 213 | 214 | See the [open issues](https://github.com/wayfair-incubator/telefonistka/issues) for a list of proposed features (and known issues). 215 | 216 | ## Contributing 217 | 218 | Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are **greatly appreciated**. For detailed contributing guidelines, please see [CONTRIBUTING.md](CONTRIBUTING.md) 219 | 220 | ## License 221 | 222 | Distributed under the MIT License. See [LICENSE](LICENSE) for more information. 223 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | # Security Policies and Procedures 2 | 3 | This document outlines security procedures and general policies for the 4 | Telefonistka project. 5 | 6 | - [Reporting a Bug](#reporting-a-bug) 7 | - [Disclosure Policy](#disclosure-policy) 8 | - [Comments on this Policy](#comments-on-this-policy) 9 | 10 | ## Reporting a Bug 11 | 12 | The Telefonistka team and community take all security bugs in 13 | Telefonistka seriously. Thank you for improving the security of 14 | Telefonistka. We appreciate your efforts and responsible disclosure and 15 | will make every effort to acknowledge your contributions. 16 | 17 | Report security bugs by emailing `OpenSource@wayfair.com`. 18 | 19 | The lead maintainer will acknowledge your email within 48 hours, and will send a 20 | more detailed response within 48 hours indicating the next steps in handling 21 | your report. After the initial reply to your report, the security team will 22 | endeavor to keep you informed of the progress towards a fix and full 23 | announcement, and may ask for additional information or guidance. 24 | 25 | ## Disclosure Policy 26 | 27 | When the security team receives a security bug report, they will assign it to a 28 | primary handler. This person will coordinate the fix and release process, 29 | involving the following steps: 30 | 31 | - Confirm the problem and determine the affected versions. 32 | - Audit code to find any potential similar problems. 33 | - Prepare fixes for all releases still under maintenance. These fixes will be 34 | released as quickly as possible. 35 | 36 | ## Comments on this Policy 37 | 38 | If you have suggestions on how this process could be improved please submit a 39 | pull request. 40 | -------------------------------------------------------------------------------- /cmd/telefonistka/bump-version-overwrite.go: -------------------------------------------------------------------------------- 1 | package telefonistka 2 | 3 | import ( 4 | "context" 5 | "os" 6 | "strings" 7 | 8 | lru "github.com/hashicorp/golang-lru/v2" 9 | "github.com/hexops/gotextdiff" 10 | "github.com/hexops/gotextdiff/myers" 11 | "github.com/hexops/gotextdiff/span" 12 | log "github.com/sirupsen/logrus" 13 | "github.com/spf13/cobra" 14 | "github.com/wayfair-incubator/telefonistka/internal/pkg/githubapi" 15 | ) 16 | 17 | // This is still(https://github.com/spf13/cobra/issues/1862) the documented way to use cobra 18 | func init() { //nolint:gochecknoinits 19 | var targetRepo string 20 | var targetFile string 21 | var file string 22 | var githubHost string 23 | var triggeringRepo string 24 | var triggeringRepoSHA string 25 | var triggeringActor string 26 | var autoMerge bool 27 | eventCmd := &cobra.Command{ 28 | Use: "bump-overwrite", 29 | Short: "Bump artifact version based on provided file content.", 30 | Long: "Bump artifact version based on provided file content.\nThis open a pull request in the target repo.", 31 | Args: cobra.ExactArgs(0), 32 | Run: func(cmd *cobra.Command, args []string) { 33 | bumpVersionOverwrite(targetRepo, targetFile, file, githubHost, triggeringRepo, triggeringRepoSHA, triggeringActor, autoMerge) 34 | }, 35 | } 36 | eventCmd.Flags().StringVarP(&targetRepo, "target-repo", "t", getEnv("TARGET_REPO", ""), "Target Git repository slug(e.g. org-name/repo-name), defaults to TARGET_REPO env var.") 37 | eventCmd.Flags().StringVarP(&targetFile, "target-file", "f", getEnv("TARGET_FILE", ""), "Target file path(from repo root), defaults to TARGET_FILE env var.") 38 | eventCmd.Flags().StringVarP(&file, "file", "c", "", "File that holds the content the target file will be overwritten with, like \"version.yaml\" or '<(echo -e \"image:\\n tag: ${VERSION}\")'.") 39 | eventCmd.Flags().StringVarP(&githubHost, "github-host", "g", "", "GitHub instance HOSTNAME, defaults to \"github.com\". This is used for GitHub Enterprise Server instances.") 40 | eventCmd.Flags().StringVarP(&triggeringRepo, "triggering-repo", "p", getEnv("GITHUB_REPOSITORY", ""), "Github repo triggering the version bump(e.g. `octocat/Hello-World`) defaults to GITHUB_REPOSITORY env var.") 41 | eventCmd.Flags().StringVarP(&triggeringRepoSHA, "triggering-repo-sha", "s", getEnv("GITHUB_SHA", ""), "Git SHA of triggering repo, defaults to GITHUB_SHA env var.") 42 | eventCmd.Flags().StringVarP(&triggeringActor, "triggering-actor", "a", getEnv("GITHUB_ACTOR", ""), "GitHub user of the person/bot who triggered the bump, defaults to GITHUB_ACTOR env var.") 43 | eventCmd.Flags().BoolVar(&autoMerge, "auto-merge", false, "Automatically merges the created PR, defaults to false.") 44 | rootCmd.AddCommand(eventCmd) 45 | } 46 | 47 | func bumpVersionOverwrite(targetRepo string, targetFile string, file string, githubHost string, triggeringRepo string, triggeringRepoSHA string, triggeringActor string, autoMerge bool) { 48 | b, err := os.ReadFile(file) 49 | if err != nil { 50 | log.Errorf("Failed to read file %s, %v", file, err) 51 | os.Exit(1) 52 | } 53 | newFileContent := string(b) 54 | 55 | ctx := context.Background() 56 | var githubRestAltURL string 57 | 58 | if githubHost != "" { 59 | githubRestAltURL = "https://" + githubHost + "/api/v3" 60 | log.Infof("Github REST API endpoint is configured to %s", githubRestAltURL) 61 | } 62 | var mainGithubClientPair githubapi.GhClientPair 63 | mainGhClientCache, _ := lru.New[string, githubapi.GhClientPair](128) 64 | 65 | mainGithubClientPair.GetAndCache(mainGhClientCache, "GITHUB_APP_ID", "GITHUB_APP_PRIVATE_KEY_PATH", "GITHUB_OAUTH_TOKEN", strings.Split(targetRepo, "/")[0], ctx) 66 | 67 | var ghPrClientDetails githubapi.GhPrClientDetails 68 | 69 | ghPrClientDetails.GhClientPair = &mainGithubClientPair 70 | ghPrClientDetails.Ctx = ctx 71 | ghPrClientDetails.Owner = strings.Split(targetRepo, "/")[0] 72 | ghPrClientDetails.Repo = strings.Split(targetRepo, "/")[1] 73 | ghPrClientDetails.PrLogger = log.WithFields(log.Fields{}) // TODO what fields should be here? 74 | 75 | defaultBranch, _ := ghPrClientDetails.GetDefaultBranch() 76 | initialFileContent, statusCode, err := githubapi.GetFileContent(ghPrClientDetails, defaultBranch, targetFile) 77 | if statusCode == 404 { 78 | ghPrClientDetails.PrLogger.Infof("File %s was not found\n", targetFile) 79 | } else if err != nil { 80 | ghPrClientDetails.PrLogger.Errorf("Fail to fetch file content:%s\n", err) 81 | os.Exit(1) 82 | } 83 | 84 | edits := myers.ComputeEdits(span.URIFromPath(""), initialFileContent, newFileContent) 85 | ghPrClientDetails.PrLogger.Infof("Diff:\n%s", gotextdiff.ToUnified("Before", "After", initialFileContent, edits)) 86 | 87 | err = githubapi.BumpVersion(ghPrClientDetails, "main", targetFile, newFileContent, triggeringRepo, triggeringRepoSHA, triggeringActor, autoMerge) 88 | if err != nil { 89 | log.Errorf("Failed to bump version: %v", err) 90 | os.Exit(1) 91 | } 92 | } 93 | -------------------------------------------------------------------------------- /cmd/telefonistka/bump-version-regex.go: -------------------------------------------------------------------------------- 1 | package telefonistka 2 | 3 | import ( 4 | "context" 5 | "os" 6 | "regexp" 7 | "strings" 8 | 9 | lru "github.com/hashicorp/golang-lru/v2" 10 | "github.com/hexops/gotextdiff" 11 | "github.com/hexops/gotextdiff/myers" 12 | "github.com/hexops/gotextdiff/span" 13 | log "github.com/sirupsen/logrus" 14 | "github.com/spf13/cobra" 15 | "github.com/wayfair-incubator/telefonistka/internal/pkg/githubapi" 16 | ) 17 | 18 | // This is still(https://github.com/spf13/cobra/issues/1862) the documented way to use cobra 19 | func init() { //nolint:gochecknoinits 20 | var targetRepo string 21 | var targetFile string 22 | var regex string 23 | var replacement string 24 | var githubHost string 25 | var triggeringRepo string 26 | var triggeringRepoSHA string 27 | var triggeringActor string 28 | var autoMerge bool 29 | eventCmd := &cobra.Command{ 30 | Use: "bump-regex", 31 | Short: "Bump artifact version in a file using regex", 32 | Long: "Bump artifact version in a file using regex.\nThis open a pull request in the target repo.\n", 33 | Args: cobra.ExactArgs(0), 34 | Run: func(cmd *cobra.Command, args []string) { 35 | bumpVersionRegex(targetRepo, targetFile, regex, replacement, githubHost, triggeringRepo, triggeringRepoSHA, triggeringActor, autoMerge) 36 | }, 37 | } 38 | eventCmd.Flags().StringVarP(&targetRepo, "target-repo", "t", getEnv("TARGET_REPO", ""), "Target Git repository slug(e.g. org-name/repo-name), defaults to TARGET_REPO env var.") 39 | eventCmd.Flags().StringVarP(&targetFile, "target-file", "f", getEnv("TARGET_FILE", ""), "Target file path(from repo root), defaults to TARGET_FILE env var.") 40 | eventCmd.Flags().StringVarP(®ex, "regex-string", "r", "", "Regex used to replace artifact version, e.g. 'tag:\\s*(\\S*)'.") 41 | eventCmd.Flags().StringVarP(&replacement, "replacement-string", "n", "", "Replacement string that includes the version of new artifact, e.g. 'tag: v2.7.1'.") 42 | eventCmd.Flags().StringVarP(&githubHost, "github-host", "g", "", "GitHub instance HOSTNAME, defaults to \"github.com\". This is used for GitHub Enterprise Server instances.") 43 | eventCmd.Flags().StringVarP(&triggeringRepo, "triggering-repo", "p", getEnv("GITHUB_REPOSITORY", ""), "Github repo triggering the version bump(e.g. `octocat/Hello-World`) defaults to GITHUB_REPOSITORY env var.") 44 | eventCmd.Flags().StringVarP(&triggeringRepoSHA, "triggering-repo-sha", "s", getEnv("GITHUB_SHA", ""), "Git SHA of triggering repo, defaults to GITHUB_SHA env var.") 45 | eventCmd.Flags().StringVarP(&triggeringActor, "triggering-actor", "a", getEnv("GITHUB_ACTOR", ""), "GitHub user of the person/bot who triggered the bump, defaults to GITHUB_ACTOR env var.") 46 | eventCmd.Flags().BoolVar(&autoMerge, "auto-merge", false, "Automatically merges the created PR, defaults to false.") 47 | rootCmd.AddCommand(eventCmd) 48 | } 49 | 50 | func bumpVersionRegex(targetRepo string, targetFile string, regex string, replacement string, githubHost string, triggeringRepo string, triggeringRepoSHA string, triggeringActor string, autoMerge bool) { 51 | ctx := context.Background() 52 | var githubRestAltURL string 53 | 54 | if githubHost != "" { 55 | githubRestAltURL = "https://" + githubHost + "/api/v3" 56 | log.Infof("Github REST API endpoint is configured to %s", githubRestAltURL) 57 | } 58 | var mainGithubClientPair githubapi.GhClientPair 59 | mainGhClientCache, _ := lru.New[string, githubapi.GhClientPair](128) 60 | 61 | mainGithubClientPair.GetAndCache(mainGhClientCache, "GITHUB_APP_ID", "GITHUB_APP_PRIVATE_KEY_PATH", "GITHUB_OAUTH_TOKEN", strings.Split(targetRepo, "/")[0], ctx) 62 | 63 | var ghPrClientDetails githubapi.GhPrClientDetails 64 | 65 | ghPrClientDetails.GhClientPair = &mainGithubClientPair 66 | ghPrClientDetails.Ctx = ctx 67 | ghPrClientDetails.Owner = strings.Split(targetRepo, "/")[0] 68 | ghPrClientDetails.Repo = strings.Split(targetRepo, "/")[1] 69 | ghPrClientDetails.PrLogger = log.WithFields(log.Fields{}) // TODO what fields should be here? 70 | 71 | r := regexp.MustCompile(regex) 72 | defaultBranch, _ := ghPrClientDetails.GetDefaultBranch() 73 | 74 | initialFileContent, _, err := githubapi.GetFileContent(ghPrClientDetails, defaultBranch, targetFile) 75 | if err != nil { 76 | ghPrClientDetails.PrLogger.Errorf("Fail to fetch file content:%s\n", err) 77 | os.Exit(1) 78 | } 79 | newFileContent := r.ReplaceAllString(initialFileContent, replacement) 80 | 81 | edits := myers.ComputeEdits(span.URIFromPath(""), initialFileContent, newFileContent) 82 | ghPrClientDetails.PrLogger.Infof("Diff:\n%s", gotextdiff.ToUnified("Before", "After", initialFileContent, edits)) 83 | 84 | err = githubapi.BumpVersion(ghPrClientDetails, "main", targetFile, newFileContent, triggeringRepo, triggeringRepoSHA, triggeringActor, autoMerge) 85 | if err != nil { 86 | log.Errorf("Failed to bump version: %v", err) 87 | os.Exit(1) 88 | } 89 | } 90 | -------------------------------------------------------------------------------- /cmd/telefonistka/bump-version-yaml.go: -------------------------------------------------------------------------------- 1 | package telefonistka 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "os" 7 | "strings" 8 | 9 | lru "github.com/hashicorp/golang-lru/v2" 10 | "github.com/hexops/gotextdiff" 11 | "github.com/hexops/gotextdiff/myers" 12 | "github.com/hexops/gotextdiff/span" 13 | "github.com/mikefarah/yq/v4/pkg/yqlib" 14 | log "github.com/sirupsen/logrus" 15 | "github.com/spf13/cobra" 16 | "github.com/wayfair-incubator/telefonistka/internal/pkg/githubapi" 17 | ) 18 | 19 | // This is still(https://github.com/spf13/cobra/issues/1862) the documented way to use cobra 20 | func init() { //nolint:gochecknoinits 21 | var targetRepo string 22 | var targetFile string 23 | var address string 24 | var replacement string 25 | var githubHost string 26 | var triggeringRepo string 27 | var triggeringRepoSHA string 28 | var triggeringActor string 29 | var autoMerge bool 30 | eventCmd := &cobra.Command{ 31 | Use: "bump-yaml", 32 | Short: "Bump artifact version in a file using yaml selector", 33 | Long: `Bump artifact version in a file using yaml selector. 34 | This will open a pull request in the target repo. 35 | This command uses yq selector to find the yaml value to replace. 36 | `, 37 | Args: cobra.ExactArgs(0), 38 | Run: func(cmd *cobra.Command, args []string) { 39 | bumpVersionYaml(targetRepo, targetFile, address, replacement, githubHost, triggeringRepo, triggeringRepoSHA, triggeringActor, autoMerge) 40 | }, 41 | } 42 | eventCmd.Flags().StringVarP(&targetRepo, "target-repo", "t", getEnv("TARGET_REPO", ""), "Target Git repository slug(e.g. org-name/repo-name), defaults to TARGET_REPO env var.") 43 | eventCmd.Flags().StringVarP(&targetFile, "target-file", "f", getEnv("TARGET_FILE", ""), "Target file path(from repo root), defaults to TARGET_FILE env var.") 44 | eventCmd.Flags().StringVar(&address, "address", "", "Yaml value address described as a yq selector, e.g. '.db.[] | select(.name == \"postgres\").image.tag'.") 45 | eventCmd.Flags().StringVarP(&replacement, "replacement-string", "n", "", "Replacement string that includes the version value of new artifact, e.g. 'v2.7.1'.") 46 | eventCmd.Flags().StringVarP(&githubHost, "github-host", "g", "", "GitHub instance HOSTNAME, defaults to \"github.com\". This is used for GitHub Enterprise Server instances.") 47 | eventCmd.Flags().StringVarP(&triggeringRepo, "triggering-repo", "p", getEnv("GITHUB_REPOSITORY", ""), "Github repo triggering the version bump(e.g. `octocat/Hello-World`) defaults to GITHUB_REPOSITORY env var.") 48 | eventCmd.Flags().StringVarP(&triggeringRepoSHA, "triggering-repo-sha", "s", getEnv("GITHUB_SHA", ""), "Git SHA of triggering repo, defaults to GITHUB_SHA env var.") 49 | eventCmd.Flags().StringVarP(&triggeringActor, "triggering-actor", "a", getEnv("GITHUB_ACTOR", ""), "GitHub user of the person/bot who triggered the bump, defaults to GITHUB_ACTOR env var.") 50 | eventCmd.Flags().BoolVar(&autoMerge, "auto-merge", false, "Automatically merges the created PR, defaults to false.") 51 | rootCmd.AddCommand(eventCmd) 52 | } 53 | 54 | func bumpVersionYaml(targetRepo string, targetFile string, address string, value string, githubHost string, triggeringRepo string, triggeringRepoSHA string, triggeringActor string, autoMerge bool) { 55 | ctx := context.Background() 56 | var githubRestAltURL string 57 | 58 | if githubHost != "" { 59 | githubRestAltURL = "https://" + githubHost + "/api/v3" 60 | log.Infof("Github REST API endpoint is configured to %s", githubRestAltURL) 61 | } 62 | var mainGithubClientPair githubapi.GhClientPair 63 | mainGhClientCache, _ := lru.New[string, githubapi.GhClientPair](128) 64 | 65 | mainGithubClientPair.GetAndCache(mainGhClientCache, "GITHUB_APP_ID", "GITHUB_APP_PRIVATE_KEY_PATH", "GITHUB_OAUTH_TOKEN", strings.Split(targetRepo, "/")[0], ctx) 66 | 67 | var ghPrClientDetails githubapi.GhPrClientDetails 68 | 69 | ghPrClientDetails.GhClientPair = &mainGithubClientPair 70 | ghPrClientDetails.Ctx = ctx 71 | ghPrClientDetails.Owner = strings.Split(targetRepo, "/")[0] 72 | ghPrClientDetails.Repo = strings.Split(targetRepo, "/")[1] 73 | ghPrClientDetails.PrLogger = log.WithFields(log.Fields{}) // TODO what fields should be here? 74 | 75 | defaultBranch, _ := ghPrClientDetails.GetDefaultBranch() 76 | 77 | initialFileContent, _, err := githubapi.GetFileContent(ghPrClientDetails, defaultBranch, targetFile) 78 | if err != nil { 79 | ghPrClientDetails.PrLogger.Errorf("Fail to fetch file content:%s\n", err) 80 | os.Exit(1) 81 | } 82 | newFileContent, err := updateYaml(initialFileContent, address, value) 83 | if err != nil { 84 | ghPrClientDetails.PrLogger.Errorf("Fail to update yaml:%s\n", err) 85 | os.Exit(1) 86 | } 87 | 88 | edits := myers.ComputeEdits(span.URIFromPath(""), initialFileContent, newFileContent) 89 | ghPrClientDetails.PrLogger.Infof("Diff:\n%s", gotextdiff.ToUnified("Before", "After", initialFileContent, edits)) 90 | 91 | err = githubapi.BumpVersion(ghPrClientDetails, "main", targetFile, newFileContent, triggeringRepo, triggeringRepoSHA, triggeringActor, autoMerge) 92 | if err != nil { 93 | log.Errorf("Failed to bump version: %v", err) 94 | os.Exit(1) 95 | } 96 | } 97 | 98 | func updateYaml(yamlContent string, address string, value string) (string, error) { 99 | yqExpression := fmt.Sprintf("(%s)=\"%s\"", address, value) 100 | 101 | preferences := yqlib.NewDefaultYamlPreferences() 102 | evaluate, err := yqlib.NewStringEvaluator().Evaluate(yqExpression, yamlContent, yqlib.NewYamlEncoder(preferences), yqlib.NewYamlDecoder(preferences)) 103 | if err != nil { 104 | return "", err 105 | } 106 | return evaluate, nil 107 | } 108 | -------------------------------------------------------------------------------- /cmd/telefonistka/bump-version-yaml_test.go: -------------------------------------------------------------------------------- 1 | package telefonistka 2 | 3 | import ( 4 | "testing" 5 | ) 6 | 7 | func TestUpdateYaml(t *testing.T) { 8 | t.Parallel() 9 | 10 | tests := []struct { 11 | name string 12 | yamlContent string 13 | address string 14 | value string 15 | want string 16 | }{ 17 | { 18 | name: "Test simple", 19 | yamlContent: ` 20 | tag: "16.1" 21 | `, 22 | address: `.tag`, 23 | value: "16.2", 24 | want: ` 25 | tag: "16.2" 26 | `, 27 | }, 28 | { 29 | name: "Test nested", 30 | yamlContent: ` 31 | image: 32 | repository: "postgres" 33 | tag: "16.1" 34 | `, 35 | address: `.image.tag`, 36 | value: "16.2", 37 | want: ` 38 | image: 39 | repository: "postgres" 40 | tag: "16.2" 41 | `, 42 | }, 43 | { 44 | name: "Test nested select", 45 | yamlContent: ` 46 | db: 47 | - name: "postgres" 48 | image: 49 | repository: "postgres" 50 | tag: "16.1" 51 | `, 52 | address: `.db.[] | select(.name == "postgres").image.tag`, 53 | value: "16.2", 54 | want: ` 55 | db: 56 | - name: "postgres" 57 | image: 58 | repository: "postgres" 59 | tag: "16.2" 60 | `, 61 | }, 62 | { 63 | name: "Test add missing", 64 | yamlContent: ` 65 | image: 66 | repository: "postgres" 67 | `, 68 | address: `.image.tag`, 69 | value: "16.2", 70 | want: ` 71 | image: 72 | repository: "postgres" 73 | tag: "16.2" 74 | `, 75 | }, 76 | } 77 | 78 | for _, tt := range tests { 79 | tt := tt 80 | t.Run(tt.name, func(t *testing.T) { 81 | t.Parallel() 82 | got, err := updateYaml(tt.yamlContent, tt.address, tt.value) 83 | if err != nil { 84 | t.Errorf("updateYaml() error = %v", err) 85 | return 86 | } 87 | if got != tt.want { 88 | t.Errorf("updateYaml() got = %v, want %v", got, tt.want) 89 | } 90 | }) 91 | } 92 | } 93 | -------------------------------------------------------------------------------- /cmd/telefonistka/event.go: -------------------------------------------------------------------------------- 1 | package telefonistka 2 | 3 | import ( 4 | "os" 5 | 6 | lru "github.com/hashicorp/golang-lru/v2" 7 | "github.com/spf13/cobra" 8 | "github.com/wayfair-incubator/telefonistka/internal/pkg/githubapi" 9 | ) 10 | 11 | // This is still(https://github.com/spf13/cobra/issues/1862) the documented way to use cobra 12 | func init() { //nolint:gochecknoinits 13 | var eventType string 14 | var eventFilePath string 15 | eventCmd := &cobra.Command{ 16 | Use: "event", 17 | Short: "Handles a GitHub event based on event JSON file", 18 | Long: "Handles a GitHub event based on event JSON file.\nThis operation mode was was built with GitHub Actions in mind", 19 | Args: cobra.ExactArgs(0), 20 | Run: func(cmd *cobra.Command, args []string) { 21 | event(eventType, eventFilePath) 22 | }, 23 | } 24 | eventCmd.Flags().StringVarP(&eventType, "type", "t", getEnv("GITHUB_EVENT_NAME", ""), "Event type, defaults to GITHUB_EVENT_NAME env var") 25 | eventCmd.Flags().StringVarP(&eventFilePath, "file", "f", getEnv("GITHUB_EVENT_PATH", ""), "File path for event JSON, defaults to GITHUB_EVENT_PATH env var") 26 | rootCmd.AddCommand(eventCmd) 27 | } 28 | 29 | func event(eventType string, eventFilePath string) { 30 | mainGhClientCache, _ := lru.New[string, githubapi.GhClientPair](128) 31 | prApproverGhClientCache, _ := lru.New[string, githubapi.GhClientPair](128) 32 | githubapi.ReciveEventFile(eventFilePath, eventType, mainGhClientCache, prApproverGhClientCache) 33 | } 34 | 35 | func getEnv(key, fallback string) string { 36 | if value, ok := os.LookupEnv(key); ok { 37 | return value 38 | } 39 | return fallback 40 | } 41 | -------------------------------------------------------------------------------- /cmd/telefonistka/root.go: -------------------------------------------------------------------------------- 1 | package telefonistka 2 | 3 | import ( 4 | "fmt" 5 | "os" 6 | 7 | log "github.com/sirupsen/logrus" 8 | "github.com/spf13/cobra" 9 | ) 10 | 11 | var rootCmd = &cobra.Command{ 12 | Use: "telefonistka", 13 | Version: "0.0.0", 14 | Short: "telefonistka - Safe and Controlled GitOps Promotion Across Environments/Failure-Domains", 15 | Long: `Telefonistka is a Github webhook server/CLI tool that facilitates change promotion across environments/failure domains in Infrastructure as Code GitOps repos 16 | 17 | see https://github.com/wayfair-incubator/telefonistka`, 18 | Run: func(cmd *cobra.Command, args []string) { 19 | }, 20 | } 21 | 22 | func Execute() { 23 | switch getEnv("LOG_LEVEL", "info") { 24 | case "debug": 25 | log.SetLevel(log.DebugLevel) 26 | log.SetReportCaller(true) 27 | case "info": 28 | log.SetLevel(log.InfoLevel) 29 | case "warn": 30 | log.SetLevel(log.WarnLevel) 31 | case "error": 32 | log.SetLevel(log.ErrorLevel) 33 | case "fatal": 34 | log.SetLevel(log.FatalLevel) 35 | case "panic": 36 | log.SetLevel(log.PanicLevel) 37 | } 38 | 39 | log.SetFormatter(&log.TextFormatter{ 40 | DisableColors: false, 41 | // ForceColors: true, 42 | FullTimestamp: true, 43 | }) // TimestampFormat 44 | if err := rootCmd.Execute(); err != nil { 45 | fmt.Fprintf(os.Stderr, "Whoops. There was an error while executing your CLI '%s'", err) 46 | os.Exit(1) 47 | } 48 | } 49 | -------------------------------------------------------------------------------- /cmd/telefonistka/server.go: -------------------------------------------------------------------------------- 1 | package telefonistka 2 | 3 | import ( 4 | "net/http" 5 | "os" 6 | "time" 7 | 8 | "github.com/alexliesenfeld/health" 9 | lru "github.com/hashicorp/golang-lru/v2" 10 | "github.com/prometheus/client_golang/prometheus/promhttp" 11 | log "github.com/sirupsen/logrus" 12 | "github.com/spf13/cobra" 13 | "github.com/wayfair-incubator/telefonistka/internal/pkg/githubapi" 14 | ) 15 | 16 | func getCrucialEnv(key string) string { 17 | if value, ok := os.LookupEnv(key); ok { 18 | return value 19 | } 20 | log.Fatalf("%s environment variable is required", key) 21 | os.Exit(3) 22 | return "" 23 | } 24 | 25 | var serveCmd = &cobra.Command{ 26 | Use: "server", 27 | Short: "Runs the web server that listens to GitHub webhooks", 28 | Args: cobra.ExactArgs(0), 29 | Run: func(cmd *cobra.Command, args []string) { 30 | serve() 31 | }, 32 | } 33 | 34 | // This is still(https://github.com/spf13/cobra/issues/1862) the documented way to use cobra 35 | func init() { //nolint:gochecknoinits 36 | rootCmd.AddCommand(serveCmd) 37 | } 38 | 39 | func handleWebhook(githubWebhookSecret []byte, mainGhClientCache *lru.Cache[string, githubapi.GhClientPair], prApproverGhClientCache *lru.Cache[string, githubapi.GhClientPair]) func(http.ResponseWriter, *http.Request) { 40 | return func(w http.ResponseWriter, r *http.Request) { 41 | err := githubapi.ReciveWebhook(r, mainGhClientCache, prApproverGhClientCache, githubWebhookSecret) 42 | if err != nil { 43 | log.Errorf("error handling webhook: %v", err) 44 | http.Error(w, "Internal server error", http.StatusInternalServerError) 45 | return 46 | } 47 | w.WriteHeader(http.StatusOK) 48 | } 49 | } 50 | 51 | func serve() { 52 | githubWebhookSecret := []byte(getCrucialEnv("GITHUB_WEBHOOK_SECRET")) 53 | livenessChecker := health.NewChecker() // No checks for the moment, other then the http server availability 54 | readinessChecker := health.NewChecker() 55 | 56 | // mainGhClientCache := map[string]githubapi.GhClientPair{} //GH apps use a per-account/org client 57 | mainGhClientCache, _ := lru.New[string, githubapi.GhClientPair](128) 58 | prApproverGhClientCache, _ := lru.New[string, githubapi.GhClientPair](128) 59 | 60 | go githubapi.MainGhMetricsLoop(mainGhClientCache) 61 | 62 | mux := http.NewServeMux() 63 | mux.HandleFunc("/webhook", handleWebhook(githubWebhookSecret, mainGhClientCache, prApproverGhClientCache)) 64 | mux.Handle("/metrics", promhttp.Handler()) 65 | mux.Handle("/live", health.NewHandler(livenessChecker)) 66 | mux.Handle("/ready", health.NewHandler(readinessChecker)) 67 | 68 | srv := &http.Server{ 69 | Handler: mux, 70 | Addr: ":8080", 71 | ReadTimeout: 10 * time.Second, 72 | WriteTimeout: 10 * time.Second, 73 | } 74 | 75 | log.Infoln("server started") 76 | log.Fatal(srv.ListenAndServe()) 77 | } 78 | -------------------------------------------------------------------------------- /docs/modeling_environments_in_gitops_repo.md: -------------------------------------------------------------------------------- 1 | # Modeling Distinct Environments/Failure Domains In Gitops Repo 2 | 3 | This document give a short overview on the available methods to model multiple environments/failure domains in a GitOps IaC repo. 4 | More thorough articles can be found in the [external resources section](#external-resources). 5 | 6 | ## Terminology 7 | 8 | `Environment(env)`: a distinct part of your Infrastructure that is used to run your services and cloud resources to support some goal. For example, `Production` is used to serve actual customers, `Staging` could be used to test how new version of services interact with the rest of the platform, and `Lab` is where new and untested changes are initial deployed to. 9 | Having a production and at least one non-production environment is practically assumed. 10 | 11 | `Failure domain(FD)`: These represent a repeatable part of your infrastructure that you *choose* to deploy in gradual steps to control the "blast radius" of a bad deploy/configuration. A classic example could be cloud regions like `us-east-1` or `us-west-1` for a company that runs a multi-region setup. 12 | Smaller or younger companies might not have such distinct failure domains as the extra "safety" they provide might not be worth the investment. 13 | 14 | In some cases I will use the term `envs` to refer to both Environments and Failure domains as from the perspective of the IaC tool, the GitOps pipeline/controller and Telefonistka they are the same. 15 | 16 | `Drift`: in this context, drift describes an unwanted/unintended difference between environment/FDs that is present in the Git state, for example a change that was made to the `Staging` environment but wasn't promoted to `Prod` 17 | 18 | ## Available Methods 19 | 20 | ### Single instance 21 | 22 | All envs are controlled from a single file/folders, a single git commit change them all *at once*. 23 | Even if you have per env/FD parameter override files(e.g. Helm value files/Terraform `.tfvars`), any change to the shared code(or a reference to a versioned artifact hosting the code) will be applied the all envs at once(GitOps!), somewhat negating the benefits of maintaining multiple envs. 24 | 25 | ### Git Branch per Env/FD 26 | 27 | This allows using git native tools for promoting changes(`git merge`) and inspecting drift(`git diff`) but it quickly becomes cumbersome as the number of distinct environment/FDs grows. Additionally, syncing all your infrastructure from the main branch keeps the GitOps side of things more intuitive and make the promotion side more observable. 28 | 29 | ### Directory per Env/FD 30 | 31 | This is our chosen approach and what Telefonistka currently supports. 32 | 33 | See [section in README.md](../README.md#modeling-environmentsfailure-domains-in-an-iac-gitops-repo) 34 | 35 | ### Git Repo per Env/FD 36 | 37 | This is the most complex but flexible solution, providing the strongest isolation in permission and policy enforcement. 38 | This feels a bit too much considering the added complexity, especially if the number of envs is high or dynamic. 39 | Telefonistka doesn't support this model currently. 40 | 41 | ## External resources 42 | 43 | [Stop Using Branches for Deploying to Different GitOps Environments](https://codefresh.io/blog/stop-using-branches-deploying-different-gitops-environments/) 44 | 45 | [How to Model Your Gitops Environments and Promote Releases between Them](https://codefresh.io/blog/how-to-model-your-gitops-environments-and-promote-releases-between-them/) 46 | 47 | [Promoting changes and releases with GitOps](https://www.sokube.io/en/blog/promoting-changes-and-releases-with-gitops) 48 | -------------------------------------------------------------------------------- /docs/observability.md: -------------------------------------------------------------------------------- 1 | ## Metrics 2 | 3 | |name|type|description|labels| 4 | |---|---|---|---| 5 | |telefonistka_github_github_operations_total|counter|"The total number of Github API operations|`api_group`, `api_path`, `repo_slug`, `status`, `method`| 6 | |telefonistka_github_github_rest_api_client_rate_remaining|gauge|The number of remaining requests the client can make this hour|| 7 | |telefonistka_github_github_rest_api_client_rate_limit|gauge|The number of requests per hour the client is currently limited to|| 8 | |telefonistka_webhook_server_webhook_hits_total|counter|The total number of validated webhook hits|`parsing`| 9 | |telefonistka_github_open_prs|gauge|The number of open PRs|`repo_slug`| 10 | |telefonistka_github_open_promotion_prs|gauge|The number of open promotion PRs|`repo_slug`| 11 | |telefonistka_github_open_prs_with_pending_telefonistka_checks|gauge|The number of open PRs with pending Telefonistka checks(excluding PRs with very recent commits)|`repo_slug`| 12 | |telefonistka_github_commit_status_updates_total|counter|The total number of commit status updates, and their status (success/pending/failure)|`repo_slug`, `status`| 13 | 14 | > [!NOTE] 15 | > telefonistka_github_*_prs metrics are only supported on installtions that uses GitHub App authentication as it provides an easy way to query the relevant GH repos. 16 | 17 | Example metrics snippet: 18 | 19 | ```text 20 | # HELP telefonistka_github_github_operations_total The total number of Github operations 21 | # TYPE telefonistka_github_github_operations_total counter 22 | telefonistka_github_github_operations_total{api_group="repos",api_path="",method="GET",repo_slug="Oded-B/telefonistka-example",status="200"} 8 23 | telefonistka_github_github_operations_total{api_group="repos",api_path="contents",method="GET",repo_slug="Oded-B/telefonistka-example",status="200"} 76 24 | telefonistka_github_github_operations_total{api_group="repos",api_path="contents",method="GET",repo_slug="Oded-B/telefonistka-example",status="404"} 13 25 | telefonistka_github_github_operations_total{api_group="repos",api_path="issues",method="POST",repo_slug="Oded-B/telefonistka-example",status="201"} 3 26 | telefonistka_github_github_operations_total{api_group="repos",api_path="pulls",method="GET",repo_slug="Oded-B/telefonistka-example",status="200"} 8 27 | # HELP telefonistka_github_github_rest_api_client_rate_limit The number of requests per hour the client is currently limited to 28 | # TYPE telefonistka_github_github_rest_api_client_rate_limit gauge 29 | telefonistka_github_github_rest_api_client_rate_limit 100000 30 | # HELP telefonistka_github_github_rest_api_client_rate_remaining The number of remaining requests the client can make this hour 31 | # TYPE telefonistka_github_github_rest_api_client_rate_remaining gauge 32 | telefonistka_github_github_rest_api_client_rate_remaining 99668 33 | # HELP telefonistka_webhook_server_webhook_hits_total The total number of validated webhook hits 34 | # TYPE telefonistka_webhook_server_webhook_hits_total counter 35 | telefonistka_webhook_server_webhook_hits_total{parsing="successful"} 8 36 | # HELP telefonistka_github_commit_status_updates_total The total number of commit status updates, and their status (success/pending/failure) 37 | # TYPE telefonistka_github_commit_status_updates_total counter 38 | telefonistka_github_commit_status_updates_total{repo_slug="foo/bar2",status="error"} 1 39 | telefonistka_github_commit_status_updates_total{repo_slug="foo/bar2",status="pending"} 1 40 | # HELP telefonistka_github_open_promotion_prs The total number of open PRs with promotion label 41 | # TYPE telefonistka_github_open_promotion_prs gauge 42 | telefonistka_github_open_promotion_prs{repo_slug="foo/bar1"} 0 43 | telefonistka_github_open_promotion_prs{repo_slug="foo/bar2"} 10 44 | # HELP telefonistka_github_open_prs The total number of open PRs 45 | # TYPE telefonistka_github_open_prs gauge 46 | telefonistka_github_open_prs{repo_slug="foo/bar1"} 0 47 | telefonistka_github_open_prs{repo_slug="foo/bar2"} 21 48 | # HELP telefonistka_github_open_prs_with_pending_telefonistka_checks The total number of open PRs with pending Telefonistka checks(excluding PRs with very recent commits) 49 | # TYPE telefonistka_github_open_prs_with_pending_telefonistka_checks gauge 50 | telefonistka_github_open_prs_with_pending_telefonistka_checks{repo_slug="foo/bar1"} 0 51 | telefonistka_github_open_prs_with_pending_telefonistka_checks{repo_slug="foo/bar2"} 0 52 | ``` 53 | -------------------------------------------------------------------------------- /docs/version_bumping.md: -------------------------------------------------------------------------------- 1 | # Version Bumping 2 | 3 | If your IaC repo deploys software you maintain internally you probably want to automate artifact version bumping. 4 | Telefonistka can automate opening the IaC repo PR for the version change from the Code repo pipeline. 5 | 6 | Currently, three modes of operation are supported: 7 | 8 | ## Whole file overwrite 9 | 10 | ```shell 11 | Bump artifact version based on provided file content. 12 | This open a pull request in the target repo. 13 | 14 | Usage: 15 | telefonistka bump-overwrite [flags] 16 | 17 | Flags: 18 | --auto-merge Automatically merges the created PR, defaults to false. 19 | -c, --file string File that holds the content the target file will be overwritten with, like "version.yaml" or '<(echo -e "image:\n tag: ${VERSION}")'. 20 | -g, --github-host string GitHub instance HOSTNAME, defaults to "github.com". This is used for GitHub Enterprise Server instances. 21 | -h, --help help for bump-overwrite. 22 | -f, --target-file string Target file path(from repo root), defaults to TARGET_FILE env var. 23 | -t, --target-repo string Target Git repository slug(e.g. org-name/repo-name), defaults to TARGET_REPO env var. 24 | -a, --triggering-actor string GitHub user of the person/bot who triggered the bump, defaults to GITHUB_ACTOR env var. 25 | -p, --triggering-repo octocat/Hello-World Github repo triggering the version bump(e.g. octocat/Hello-World) defaults to GITHUB_REPOSITORY env var. 26 | -s, --triggering-repo-sha string Git SHA of triggering repo, defaults to GITHUB_SHA env var. 27 | ``` 28 | 29 | notes: 30 | 31 | * This can create new files in the target repo. 32 | * This was intended for cases where the IaC configuration allows adding additional minimal parameter/values file that only includes version information. 33 | 34 | ## Regex based search and replace 35 | 36 | ```shell 37 | Bump artifact version in a file using regex. 38 | This open a pull request in the target repo. 39 | 40 | Usage: 41 | telefonistka bump-regex [flags] 42 | 43 | Flags: 44 | --auto-merge Automatically merges the created PR, defaults to false. 45 | -g, --github-host string GitHub instance HOSTNAME, defaults to "github.com". This is used for GitHub Enterprise Server instances. 46 | -h, --help help for bump-regex. 47 | -r, --regex-string string Regex used to replace artifact version, e.g. 'tag:\s*(\S*)', 48 | -n, --replacement-string string Replacement string that includes the version of new artifact, e.g. 'tag: v2.7.1'. 49 | -f, --target-file string Target file path(from repo root), defaults to TARGET_FILE env var. 50 | -t, --target-repo string Target Git repository slug(e.g. org-name/repo-name), defaults to TARGET_REPO env var. 51 | -a, --triggering-actor string GitHub user of the person/bot who triggered the bump, defaults to GITHUB_ACTOR env var. 52 | -p, --triggering-repo octocat/Hello-World Github repo triggering the version bump(e.g. octocat/Hello-World) defaults to GITHUB_REPOSITORY env var. 53 | -s, --triggering-repo-sha string Git SHA of triggering repo, defaults to GITHUB_SHA env var. 54 | ``` 55 | 56 | notes: 57 | 58 | * This assumes the target file already exist in the target repo. 59 | 60 | ## YAML based value replace 61 | 62 | ```shell 63 | Bump artifact version in a file using yaml selector. 64 | This will open a pull request in the target repo. 65 | This command uses yq selector to find the yaml value to replace. 66 | 67 | Usage: 68 | telefonistka bump-yaml [flags] 69 | 70 | Flags: 71 | --address string Yaml value address described as a yq selector, e.g. '.db.[] | select(.name == "postgres").image.tag'. 72 | --auto-merge Automatically merges the created PR, defaults to false. 73 | -g, --github-host string GitHub instance HOSTNAME, defaults to "github.com". This is used for GitHub Enterprise Server instances. 74 | -h, --help help for bump-yaml 75 | -n, --replacement-string string Replacement string that includes the version value of new artifact, e.g. 'v2.7.1'. 76 | -f, --target-file string Target file path(from repo root), defaults to TARGET_FILE env var. 77 | -t, --target-repo string Target Git repository slug(e.g. org-name/repo-name), defaults to TARGET_REPO env var. 78 | -a, --triggering-actor string GitHub user of the person/bot who triggered the bump, defaults to GITHUB_ACTOR env var. 79 | -p, --triggering-repo octocat/Hello-World Github repo triggering the version bump(e.g. octocat/Hello-World) defaults to GITHUB_REPOSITORY env var. 80 | -s, --triggering-repo-sha string Git SHA of triggering repo, defaults to GITHUB_SHA env var. 81 | ``` 82 | 83 | notes: 84 | 85 | * This assumes the target file already exist in the target repo. 86 | -------------------------------------------------------------------------------- /docs/webhook_multiplexing.md: -------------------------------------------------------------------------------- 1 | # GitHub Push events fanout/multiplexing 2 | 3 | GitOps operators like ArgoCD can listen for GitHub webhooks to ensure short delays in their reconciliation loop. 4 | 5 | But in some scenarios the number of needed webhooks endpoint exceed the maximum supported by GitHub(think 10 cluster each with in-cluster ArgoCD server and ArgoCD applicationSet controller). 6 | Additionally, configuring said webhooks manually is time consuming and error prone. 7 | 8 | Telefonistka can forward these HTTP requests to multiple endpoint and can even filter or dynamically choose the endpoint URL based on the file changed in the Commit. 9 | Assuming Telefonistka is deployed as a GitHub Application, this also ease the setup process as the webhook setting(event types, URL, secret) is already a part of the application configuration. 10 | 11 | This configuration example will forward github push events that include changes in `workspace/` dir to the lab ArgoCD server and  applicationset controllers webhook servers and will forward event  that touches `clusters/`to URLs generated with regex, base of first 3 directory elements after `clusters/` 12 | 13 | ```yaml 14 | webhookEndpointRegexs: 15 | - expression: "^workspace/[^/]*/.*" 16 | replacements: 17 | - "https://kube-argocd-c1.service.lab.example.com/api/webhook" 18 | - "https://kube-argocd-applicationset-c1.service.lab.example.com/api/webhook" 19 | - "https://example.com" 20 | - expression: "^clusters/([^/]*)/([^/]*)/([^/]*)/.*" 21 | replacements: 22 | - "https://kube-argocd-${3}.${1}.service.{2}.example.com/api/webhook" 23 | - "https://kube-argocd-applicationset-${2}.service.${1}.example.com/api/webhook" 24 | 25 | ``` 26 | 27 | Telefonistka checks the regex per each file affected by a commit, but stops after the first expression match(per file). 28 | 29 | So ordering of the `webhookEndpointRegexs` elements is significant. 30 | 31 | This simpeller configuration will and push event to 7 hardcoded servers 32 | 33 | ```yaml 34 | webhookEndpointRegexs: 35 | - expression: "^.*$" 36 | replacements: 37 | - "https://argocd-server1.example.com/api/webhook" 38 | - "https://argocd-server2.example.com/api/webhook" 39 | - "https://argocd-server3.example.com/api/webhook" 40 | - "https://argocd-server4.example.com/api/webhook" 41 | - "https://argocd-server5.example.com/api/webhook" 42 | - "https://argocd-server6.example.com/api/webhook" 43 | - "https://argocd-server6.example.com/api/webhook" 44 | ``` 45 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/wayfair-incubator/telefonistka 2 | 3 | go 1.23 4 | 5 | toolchain go1.23.4 6 | 7 | require github.com/alexliesenfeld/health v0.8.0 8 | 9 | require ( 10 | github.com/argoproj/argo-cd/v2 v2.11.7 11 | github.com/argoproj/gitops-engine v0.7.1-0.20240715141605-18ba62e1f1fb 12 | github.com/bradleyfalzon/ghinstallation/v2 v2.10.0 13 | github.com/cenkalti/backoff/v4 v4.2.1 14 | github.com/cenkalti/backoff/v5 v5.0.0 15 | github.com/go-test/deep v1.1.0 16 | github.com/golang/mock v1.6.0 17 | github.com/google/go-github/v62 v62.0.0 18 | github.com/google/go-github/v67 v67.0.0 19 | github.com/hashicorp/golang-lru/v2 v2.0.7 20 | github.com/hexops/gotextdiff v1.0.3 21 | github.com/migueleliasweb/go-github-mock v1.1.0 22 | github.com/mikefarah/yq/v4 v4.43.1 23 | github.com/prometheus/client_golang v1.19.0 24 | github.com/shurcooL/githubv4 v0.0.0-20240727222349-48295856cce7 25 | github.com/sirupsen/logrus v1.9.3 26 | github.com/spf13/cobra v1.8.0 27 | github.com/stretchr/testify v1.9.0 28 | golang.org/x/exp v0.0.0-20241217172543-b2144cdd0a67 29 | golang.org/x/oauth2 v0.19.0 30 | golang.org/x/tools v0.28.0 31 | google.golang.org/grpc v1.63.2 32 | gopkg.in/yaml.v2 v2.4.0 33 | k8s.io/api v0.26.11 34 | k8s.io/apimachinery v0.26.11 35 | ) 36 | 37 | require ( 38 | cloud.google.com/go/compute v1.25.1 // indirect 39 | cloud.google.com/go/compute/metadata v0.2.3 // indirect 40 | dario.cat/mergo v1.0.0 // indirect 41 | github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect 42 | github.com/MakeNowJust/heredoc v1.0.0 // indirect 43 | github.com/Masterminds/goutils v1.1.1 // indirect 44 | github.com/Masterminds/semver/v3 v3.2.1 // indirect 45 | github.com/Masterminds/sprig/v3 v3.2.3 // indirect 46 | github.com/Microsoft/go-winio v0.6.1 // indirect 47 | github.com/ProtonMail/go-crypto v1.0.0 // indirect 48 | github.com/a8m/envsubst v1.4.2 // indirect 49 | github.com/alecthomas/participle/v2 v2.1.1 // indirect 50 | github.com/argoproj/pkg v0.13.7-0.20230626144333-d56162821bd1 // indirect 51 | github.com/beorn7/perks v1.0.1 // indirect 52 | github.com/blang/semver/v4 v4.0.0 // indirect 53 | github.com/bmatcuk/doublestar/v4 v4.6.1 // indirect 54 | github.com/bombsimon/logrusr/v2 v2.0.1 // indirect 55 | github.com/cespare/xxhash/v2 v2.3.0 // indirect 56 | github.com/chai2010/gettext-go v1.0.2 // indirect 57 | github.com/cloudflare/circl v1.3.7 // indirect 58 | github.com/coreos/go-oidc/v3 v3.6.0 // indirect 59 | github.com/cyphar/filepath-securejoin v0.2.4 // indirect 60 | github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect 61 | github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect 62 | github.com/dimchansky/utfbom v1.1.1 // indirect 63 | github.com/docker/distribution v2.8.2+incompatible // indirect 64 | github.com/elliotchance/orderedmap v1.5.1 // indirect 65 | github.com/emicklei/go-restful/v3 v3.12.0 // indirect 66 | github.com/emirpasic/gods v1.18.1 // indirect 67 | github.com/evanphx/json-patch v5.9.0+incompatible // indirect 68 | github.com/evanphx/json-patch/v5 v5.6.0 // indirect 69 | github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f // indirect 70 | github.com/fatih/camelcase v1.0.0 // indirect 71 | github.com/fatih/color v1.16.0 // indirect 72 | github.com/felixge/httpsnoop v1.0.4 // indirect 73 | github.com/fvbommel/sortorder v1.1.0 // indirect 74 | github.com/go-errors/errors v1.5.1 // indirect 75 | github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect 76 | github.com/go-git/go-billy/v5 v5.5.0 // indirect 77 | github.com/go-git/go-git/v5 v5.12.0 // indirect 78 | github.com/go-jose/go-jose/v3 v3.0.3 // indirect 79 | github.com/go-logr/logr v1.4.1 // indirect 80 | github.com/go-logr/stdr v1.2.2 // indirect 81 | github.com/go-openapi/jsonpointer v0.21.0 // indirect 82 | github.com/go-openapi/jsonreference v0.21.0 // indirect 83 | github.com/go-openapi/swag v0.23.0 // indirect 84 | github.com/go-redis/cache/v9 v9.0.0 // indirect 85 | github.com/gobwas/glob v0.2.3 // indirect 86 | github.com/goccy/go-json v0.10.2 // indirect 87 | github.com/goccy/go-yaml v1.11.3 // indirect 88 | github.com/gogo/protobuf v1.3.2 // indirect 89 | github.com/golang-jwt/jwt/v4 v4.5.0 // indirect 90 | github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect 91 | github.com/golang/protobuf v1.5.4 // indirect 92 | github.com/google/btree v1.1.2 // indirect 93 | github.com/google/gnostic v0.7.0 // indirect 94 | github.com/google/gnostic-models v0.6.9-0.20230804172637-c7be7c783f49 // indirect 95 | github.com/google/go-cmp v0.6.0 // indirect 96 | github.com/google/go-github/v56 v56.0.0 // indirect 97 | github.com/google/go-github/v60 v60.0.0 // indirect 98 | github.com/google/go-github/v64 v64.0.0 // indirect 99 | github.com/google/go-querystring v1.1.0 // indirect 100 | github.com/google/gofuzz v1.2.0 // indirect 101 | github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect 102 | github.com/google/uuid v1.6.0 // indirect 103 | github.com/gorilla/mux v1.8.0 // indirect 104 | github.com/gosimple/slug v1.13.1 // indirect 105 | github.com/gosimple/unidecode v1.0.1 // indirect 106 | github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 // indirect 107 | github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 // indirect 108 | github.com/grpc-ecosystem/grpc-gateway v1.16.0 // indirect 109 | github.com/hashicorp/go-cleanhttp v0.5.2 // indirect 110 | github.com/hashicorp/go-retryablehttp v0.7.4 // indirect 111 | github.com/huandu/xstrings v1.3.3 // indirect 112 | github.com/imdario/mergo v0.3.16 // indirect 113 | github.com/inconshreveable/mousetrap v1.1.0 // indirect 114 | github.com/itchyny/gojq v0.12.13 // indirect 115 | github.com/itchyny/timefmt-go v0.1.5 // indirect 116 | github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect 117 | github.com/jinzhu/copier v0.4.0 // indirect 118 | github.com/jonboulle/clockwork v0.4.0 // indirect 119 | github.com/josharian/intern v1.0.0 // indirect 120 | github.com/json-iterator/go v1.1.12 // indirect 121 | github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect 122 | github.com/kevinburke/ssh_config v1.2.0 // indirect 123 | github.com/klauspost/compress v1.17.8 // indirect 124 | github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect 125 | github.com/magiconair/properties v1.8.7 // indirect 126 | github.com/mailru/easyjson v0.7.7 // indirect 127 | github.com/mattn/go-colorable v0.1.13 // indirect 128 | github.com/mattn/go-isatty v0.0.20 // indirect 129 | github.com/mitchellh/copystructure v1.0.0 // indirect 130 | github.com/mitchellh/go-wordwrap v1.0.1 // indirect 131 | github.com/mitchellh/reflectwalk v1.0.0 // indirect 132 | github.com/moby/spdystream v0.2.0 // indirect 133 | github.com/moby/term v0.5.0 // indirect 134 | github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect 135 | github.com/modern-go/reflect2 v1.0.2 // indirect 136 | github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect 137 | github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect 138 | github.com/onsi/ginkgo/v2 v2.15.0 // indirect 139 | github.com/onsi/gomega v1.31.0 // indirect 140 | github.com/opencontainers/go-digest v1.0.0 // indirect 141 | github.com/opencontainers/image-spec v1.1.0 // indirect 142 | github.com/patrickmn/go-cache v2.1.0+incompatible // indirect 143 | github.com/pelletier/go-toml/v2 v2.2.0 // indirect 144 | github.com/peterbourgon/diskv v2.0.1+incompatible // indirect 145 | github.com/pjbgf/sha1cd v0.3.0 // indirect 146 | github.com/pkg/errors v0.9.1 // indirect 147 | github.com/pmezard/go-difflib v1.0.0 // indirect 148 | github.com/prometheus/client_model v0.6.1 // indirect 149 | github.com/prometheus/common v0.52.3 // indirect 150 | github.com/prometheus/procfs v0.13.0 // indirect 151 | github.com/r3labs/diff v1.1.0 // indirect 152 | github.com/redis/go-redis/v9 v9.5.1 // indirect 153 | github.com/robfig/cron/v3 v3.0.1 // indirect 154 | github.com/russross/blackfriday/v2 v2.1.0 // indirect 155 | github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect 156 | github.com/shopspring/decimal v1.2.0 // indirect 157 | github.com/shurcooL/graphql v0.0.0-20230722043721-ed46e5a46466 // indirect 158 | github.com/skeema/knownhosts v1.2.2 // indirect 159 | github.com/spf13/cast v1.6.0 // indirect 160 | github.com/spf13/pflag v1.0.5 // indirect 161 | github.com/valyala/bytebufferpool v1.0.0 // indirect 162 | github.com/valyala/fasttemplate v1.2.2 // indirect 163 | github.com/vmihailenco/go-tinylfu v0.2.2 // indirect 164 | github.com/vmihailenco/msgpack/v5 v5.4.1 // indirect 165 | github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect 166 | github.com/xanzy/ssh-agent v0.3.3 // indirect 167 | github.com/xlab/treeprint v1.2.0 // indirect 168 | github.com/yuin/gopher-lua v1.1.1 // indirect 169 | go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.50.0 // indirect 170 | go.opentelemetry.io/otel v1.25.0 // indirect 171 | go.opentelemetry.io/otel/metric v1.25.0 // indirect 172 | go.opentelemetry.io/otel/trace v1.25.0 // indirect 173 | go.starlark.net v0.0.0-20240408152805-3f0a3703c02a // indirect 174 | golang.org/x/crypto v0.30.0 // indirect 175 | golang.org/x/mod v0.22.0 // indirect 176 | golang.org/x/net v0.32.0 // indirect 177 | golang.org/x/sync v0.10.0 // indirect 178 | golang.org/x/sys v0.28.0 // indirect 179 | golang.org/x/term v0.27.0 // indirect 180 | golang.org/x/text v0.21.0 // indirect 181 | golang.org/x/time v0.5.0 // indirect 182 | golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect 183 | google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda // indirect 184 | google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda // indirect 185 | google.golang.org/genproto/googleapis/rpc v0.0.0-20240401170217-c3f982113cda // indirect 186 | google.golang.org/protobuf v1.33.0 // indirect 187 | gopkg.in/inf.v0 v0.9.1 // indirect 188 | gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473 // indirect 189 | gopkg.in/warnings.v0 v0.1.2 // indirect 190 | gopkg.in/yaml.v3 v3.0.1 // indirect 191 | k8s.io/apiextensions-apiserver v0.26.10 // indirect 192 | k8s.io/apiserver v0.26.11 // indirect 193 | k8s.io/cli-runtime v0.26.11 // indirect 194 | k8s.io/client-go v0.26.11 // indirect 195 | k8s.io/component-base v0.26.11 // indirect 196 | k8s.io/component-helpers v0.26.11 // indirect 197 | k8s.io/klog/v2 v2.120.1 // indirect 198 | k8s.io/kube-aggregator v0.26.4 // indirect 199 | k8s.io/kube-openapi v0.0.0-20240403164606-bc84c2ddaf99 // indirect 200 | k8s.io/kubectl v0.26.4 // indirect 201 | k8s.io/kubernetes v1.26.11 // indirect 202 | k8s.io/utils v0.0.0-20240310230437-4693a0247e57 // indirect 203 | layeh.com/gopher-json v0.0.0-20190114024228-97fed8db8427 // indirect 204 | oras.land/oras-go/v2 v2.5.0 // indirect 205 | sigs.k8s.io/controller-runtime v0.14.7 // indirect 206 | sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect 207 | sigs.k8s.io/kustomize/api v0.17.1 // indirect 208 | sigs.k8s.io/kustomize/kyaml v0.17.0 // indirect 209 | sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect 210 | sigs.k8s.io/yaml v1.4.0 // indirect 211 | ) 212 | 213 | replace sigs.k8s.io/kustomize/api => sigs.k8s.io/kustomize/api v0.12.1 214 | 215 | replace sigs.k8s.io/kustomize/kyaml => sigs.k8s.io/kustomize/kyaml v0.13.9 216 | -------------------------------------------------------------------------------- /internal/pkg/argocd/argocd_copied_from_upstream.go: -------------------------------------------------------------------------------- 1 | package argocd 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | 7 | "github.com/argoproj/argo-cd/v2/controller" 8 | "github.com/argoproj/argo-cd/v2/pkg/apiclient/application" 9 | "github.com/argoproj/argo-cd/v2/pkg/apiclient/settings" 10 | argoappv1 "github.com/argoproj/argo-cd/v2/pkg/apis/application/v1alpha1" 11 | repoapiclient "github.com/argoproj/argo-cd/v2/reposerver/apiclient" 12 | "github.com/argoproj/argo-cd/v2/util/argo" 13 | "github.com/argoproj/gitops-engine/pkg/sync/hook" 14 | "github.com/argoproj/gitops-engine/pkg/sync/ignore" 15 | "github.com/argoproj/gitops-engine/pkg/utils/kube" 16 | "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" 17 | "k8s.io/apimachinery/pkg/runtime/schema" 18 | ) 19 | 20 | // DifferenceOption struct to store diff options 21 | type DifferenceOption struct { 22 | revision string 23 | res *repoapiclient.ManifestResponse 24 | } 25 | 26 | type resourceInfoProvider struct { 27 | namespacedByGk map[schema.GroupKind]bool 28 | } 29 | 30 | type objKeyLiveTarget struct { 31 | key kube.ResourceKey 32 | live *unstructured.Unstructured 33 | target *unstructured.Unstructured 34 | } 35 | 36 | // Infer if obj is namespaced or not from corresponding live objects list. If corresponding live object has namespace then target object is also namespaced. 37 | // If live object is missing then it does not matter if target is namespaced or not. 38 | func (p *resourceInfoProvider) IsNamespaced(gk schema.GroupKind) (bool, error) { 39 | return p.namespacedByGk[gk], nil 40 | } 41 | 42 | // This function creates a map of objects by key(object name/kind/ns) from the rendered manifests. 43 | // That map is used to compare the objects in the application with the objects in the cluster. 44 | // copied from https://github.com/argoproj/argo-cd/blob/4f6a8dce80f0accef7ed3b5510e178a6b398b331/cmd/argocd/commands/app.go#L1091-L1109 45 | func groupObjsByKey(localObs []*unstructured.Unstructured, liveObjs []*unstructured.Unstructured, appNamespace string) (map[kube.ResourceKey]*unstructured.Unstructured, error) { 46 | namespacedByGk := make(map[schema.GroupKind]bool) 47 | for i := range liveObjs { 48 | if liveObjs[i] != nil { 49 | key := kube.GetResourceKey(liveObjs[i]) 50 | namespacedByGk[schema.GroupKind{Group: key.Group, Kind: key.Kind}] = key.Namespace != "" 51 | } 52 | } 53 | localObs, _, err := controller.DeduplicateTargetObjects(appNamespace, localObs, &resourceInfoProvider{namespacedByGk: namespacedByGk}) 54 | if err != nil { 55 | return nil, fmt.Errorf("Failed to DeDuplicate target objects: %v", err) 56 | } 57 | objByKey := make(map[kube.ResourceKey]*unstructured.Unstructured) 58 | for i := range localObs { 59 | obj := localObs[i] 60 | if !(hook.IsHook(obj) || ignore.Ignore(obj)) { 61 | objByKey[kube.GetResourceKey(obj)] = obj 62 | } 63 | } 64 | return objByKey, nil 65 | } 66 | 67 | // This function create a slice of objects to be "diff'ed", each element contains the key, live(in-cluster API state) and target(rended manifest from git) object. 68 | // Copied from https://github.com/argoproj/argo-cd/blob/4f6a8dce80f0accef7ed3b5510e178a6b398b331/cmd/argocd/commands/app.go#L1341-L1372 69 | func groupObjsForDiff(resources *application.ManagedResourcesResponse, objs map[kube.ResourceKey]*unstructured.Unstructured, items []objKeyLiveTarget, argoSettings *settings.Settings, appName, namespace string) ([]objKeyLiveTarget, error) { 70 | resourceTracking := argo.NewResourceTracking() 71 | for _, res := range resources.Items { 72 | live := &unstructured.Unstructured{} 73 | err := json.Unmarshal([]byte(res.NormalizedLiveState), &live) 74 | if err != nil { 75 | return nil, fmt.Errorf("Failed to unmarshal live object(%v): %v", res.Name, err) 76 | } 77 | 78 | key := kube.ResourceKey{Name: res.Name, Namespace: res.Namespace, Group: res.Group, Kind: res.Kind} 79 | if key.Kind == kube.SecretKind && key.Group == "" { 80 | // Don't bother comparing secrets, argo-cd doesn't have access to k8s secret data 81 | delete(objs, key) 82 | continue 83 | } 84 | if local, ok := objs[key]; ok || live != nil { 85 | if local != nil && !kube.IsCRD(local) { 86 | err = resourceTracking.SetAppInstance(local, argoSettings.AppLabelKey, appName, namespace, argoappv1.TrackingMethod(argoSettings.GetTrackingMethod())) 87 | if err != nil { 88 | return nil, fmt.Errorf("Failed to set app instance label: %v", err) 89 | } 90 | } 91 | 92 | items = append(items, objKeyLiveTarget{key, live, local}) 93 | delete(objs, key) 94 | } 95 | } 96 | for key, local := range objs { 97 | if key.Kind == kube.SecretKind && key.Group == "" { 98 | // Don't bother comparing secrets, argo-cd doesn't have access to k8s secret data 99 | delete(objs, key) 100 | continue 101 | } 102 | items = append(items, objKeyLiveTarget{key, nil, local}) 103 | } 104 | return items, nil 105 | } 106 | -------------------------------------------------------------------------------- /internal/pkg/argocd/argocd_test.go: -------------------------------------------------------------------------------- 1 | package argocd 2 | 3 | import ( 4 | "bytes" 5 | "context" 6 | "fmt" 7 | "log" 8 | "os" 9 | "strings" 10 | "testing" 11 | "text/template" 12 | "time" 13 | 14 | "github.com/argoproj/argo-cd/v2/pkg/apiclient/application" 15 | "github.com/argoproj/argo-cd/v2/pkg/apiclient/project" 16 | "github.com/argoproj/argo-cd/v2/pkg/apiclient/settings" 17 | argoappv1 "github.com/argoproj/argo-cd/v2/pkg/apis/application/v1alpha1" 18 | reposerverApiClient "github.com/argoproj/argo-cd/v2/reposerver/apiclient" 19 | "github.com/golang/mock/gomock" 20 | "github.com/stretchr/testify/assert" 21 | "github.com/wayfair-incubator/telefonistka/internal/pkg/mocks" 22 | "github.com/wayfair-incubator/telefonistka/internal/pkg/testutils" 23 | metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" 24 | "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" 25 | ) 26 | 27 | func readLiveTarget(t *testing.T) (live, target *unstructured.Unstructured, expected string) { 28 | t.Helper() 29 | live = readManifest(t, "testdata/"+t.Name()+".live") 30 | target = readManifest(t, "testdata/"+t.Name()+".target") 31 | expected = readFileString(t, "testdata/"+t.Name()+".want") 32 | return live, target, expected 33 | } 34 | 35 | func readFileString(t *testing.T, path string) string { 36 | t.Helper() 37 | b, err := os.ReadFile(path) 38 | if err != nil { 39 | t.Fatal(err) 40 | } 41 | return string(b) 42 | } 43 | 44 | func readManifest(t *testing.T, path string) *unstructured.Unstructured { 45 | t.Helper() 46 | 47 | s := readFileString(t, path) 48 | obj, err := argoappv1.UnmarshalToUnstructured(s) 49 | if err != nil { 50 | t.Fatalf("unmarshal %v: %v", path, err) 51 | } 52 | return obj 53 | } 54 | 55 | func TestDiffLiveVsTargetObject(t *testing.T) { 56 | t.Parallel() 57 | tests := []struct { 58 | name string 59 | }{ 60 | {"1"}, 61 | } 62 | for _, test := range tests { 63 | t.Run(test.name, func(t *testing.T) { 64 | t.Parallel() 65 | live, target, want := readLiveTarget(t) 66 | got, err := diffLiveVsTargetObject(live, target) 67 | if err != nil { 68 | t.Errorf("unexpected error: %v", err) 69 | } 70 | 71 | if got != want { 72 | t.Errorf("got %q, want %q", got, want) 73 | } 74 | }) 75 | } 76 | } 77 | 78 | func TestRenderDiff(t *testing.T) { 79 | t.Parallel() 80 | live := readManifest(t, "testdata/TestRenderDiff.live") 81 | target := readManifest(t, "testdata/TestRenderDiff.target") 82 | want := readFileString(t, "testdata/TestRenderDiff.md") 83 | data, err := diffLiveVsTargetObject(live, target) 84 | if err != nil { 85 | t.Errorf("unexpected error: %v", err) 86 | } 87 | 88 | // backticks are tricky https://github.com/golang/go/issues/24475 89 | r := strings.NewReplacer("¬", "`") 90 | tmpl := r.Replace("¬¬¬diff\n{{.}}¬¬¬\n") 91 | 92 | rendered := renderTemplate(t, tmpl, data) 93 | 94 | if got, want := rendered.String(), want; got != want { 95 | t.Errorf("got %q, want %q", got, want) 96 | } 97 | } 98 | 99 | func renderTemplate(t *testing.T, tpl string, data any) *bytes.Buffer { 100 | t.Helper() 101 | buf := bytes.NewBuffer(nil) 102 | tmpl := template.New("") 103 | tmpl = template.Must(tmpl.Parse(tpl)) 104 | if err := tmpl.Execute(buf, data); err != nil { 105 | t.Fatalf("unexpected error: %v", err) 106 | } 107 | return buf 108 | } 109 | 110 | func TestFindArgocdAppBySHA1Label(t *testing.T) { 111 | // Here the filtering is done on the ArgoCD server side, so we are just testing the function returns a app 112 | t.Parallel() 113 | ctx := context.Background() 114 | ctrl := gomock.NewController(t) 115 | defer ctrl.Finish() 116 | mockApplicationClient := mocks.NewMockApplicationServiceClient(ctrl) 117 | expectedResponse := &argoappv1.ApplicationList{ 118 | Items: []argoappv1.Application{ 119 | { 120 | ObjectMeta: metav1.ObjectMeta{ 121 | Labels: map[string]string{ 122 | "telefonistka.io/component-path-sha1": "111111", 123 | }, 124 | Name: "right-app", 125 | }, 126 | }, 127 | }, 128 | } 129 | 130 | mockApplicationClient.EXPECT().List(gomock.Any(), gomock.Any()).Return(expectedResponse, nil) 131 | 132 | app, err := findArgocdAppBySHA1Label(ctx, "random/path", "some-repo", mockApplicationClient) 133 | if err != nil { 134 | t.Errorf("Error: %v", err) 135 | } 136 | if app.Name != "right-app" { 137 | t.Errorf("App name is not right-app") 138 | } 139 | } 140 | 141 | func TestFindArgocdAppByPathAnnotation(t *testing.T) { 142 | t.Parallel() 143 | ctx := context.Background() 144 | ctrl := gomock.NewController(t) 145 | defer ctrl.Finish() 146 | mockApplicationClient := mocks.NewMockApplicationServiceClient(ctrl) 147 | expectedResponse := &argoappv1.ApplicationList{ 148 | Items: []argoappv1.Application{ 149 | { 150 | ObjectMeta: metav1.ObjectMeta{ 151 | Annotations: map[string]string{ 152 | "argocd.argoproj.io/manifest-generate-paths": "wrong/path/", 153 | }, 154 | Name: "wrong-app", 155 | }, 156 | }, 157 | { 158 | ObjectMeta: metav1.ObjectMeta{ 159 | Annotations: map[string]string{ 160 | "argocd.argoproj.io/manifest-generate-paths": "right/path/", 161 | }, 162 | Name: "right-app", 163 | }, 164 | }, 165 | }, 166 | } 167 | 168 | mockApplicationClient.EXPECT().List(gomock.Any(), gomock.Any()).Return(expectedResponse, nil) 169 | 170 | apps, err := findArgocdAppByManifestPathAnnotation(ctx, "right/path", "some-repo", mockApplicationClient) 171 | if err != nil { 172 | t.Errorf("Error: %v", err) 173 | } 174 | t.Logf("apps: %v", apps) 175 | } 176 | 177 | // Here I'm testing a ";" delimted path annotation 178 | func TestFindArgocdAppByPathAnnotationSemiColon(t *testing.T) { 179 | t.Parallel() 180 | ctx := context.Background() 181 | ctrl := gomock.NewController(t) 182 | defer ctrl.Finish() 183 | mockApplicationClient := mocks.NewMockApplicationServiceClient(ctrl) 184 | expectedResponse := &argoappv1.ApplicationList{ 185 | Items: []argoappv1.Application{ 186 | { 187 | ObjectMeta: metav1.ObjectMeta{ 188 | Annotations: map[string]string{ 189 | "argocd.argoproj.io/manifest-generate-paths": "wrong/path/;wrong/path2/", 190 | }, 191 | Name: "wrong-app", 192 | }, 193 | }, 194 | { // This is the app we want to find - it has the right path as one of the elements in the annotation 195 | ObjectMeta: metav1.ObjectMeta{ 196 | Annotations: map[string]string{ 197 | "argocd.argoproj.io/manifest-generate-paths": "wrong/path/;right/path/", 198 | }, 199 | Name: "right-app", 200 | }, 201 | }, 202 | }, 203 | } 204 | 205 | mockApplicationClient.EXPECT().List(gomock.Any(), gomock.Any()).Return(expectedResponse, nil) 206 | 207 | app, err := findArgocdAppByManifestPathAnnotation(ctx, "right/path", "some-repo", mockApplicationClient) 208 | if err != nil { 209 | t.Errorf("Error: %v", err) 210 | } 211 | if app.Name != "right-app" { 212 | t.Errorf("App name is not right-app") 213 | } 214 | } 215 | 216 | // Here I'm testing a "." path annotation - this is a special case where the path is relative to the repo root specified in the application .spec 217 | func TestFindArgocdAppByPathAnnotationRelative(t *testing.T) { 218 | t.Parallel() 219 | ctx := context.Background() 220 | ctrl := gomock.NewController(t) 221 | defer ctrl.Finish() 222 | mockApplicationClient := mocks.NewMockApplicationServiceClient(ctrl) 223 | expectedResponse := &argoappv1.ApplicationList{ 224 | Items: []argoappv1.Application{ 225 | { 226 | ObjectMeta: metav1.ObjectMeta{ 227 | Annotations: map[string]string{ 228 | "argocd.argoproj.io/manifest-generate-paths": ".", 229 | }, 230 | Name: "right-app", 231 | }, 232 | Spec: argoappv1.ApplicationSpec{ 233 | Source: &argoappv1.ApplicationSource{ 234 | RepoURL: "", 235 | Path: "right/path", 236 | }, 237 | }, 238 | }, 239 | }, 240 | } 241 | 242 | mockApplicationClient.EXPECT().List(gomock.Any(), gomock.Any()).Return(expectedResponse, nil) 243 | app, err := findArgocdAppByManifestPathAnnotation(ctx, "right/path", "some-repo", mockApplicationClient) 244 | if err != nil { 245 | t.Errorf("Error: %v", err) 246 | } else if app.Name != "right-app" { 247 | t.Errorf("App name is not right-app") 248 | } 249 | } 250 | 251 | // Here I'm testing a "." path annotation - this is a special case where the path is relative to the repo root specified in the application .spec 252 | func TestFindArgocdAppByPathAnnotationRelative2(t *testing.T) { 253 | t.Parallel() 254 | ctx := context.Background() 255 | ctrl := gomock.NewController(t) 256 | defer ctrl.Finish() 257 | mockApplicationClient := mocks.NewMockApplicationServiceClient(ctrl) 258 | expectedResponse := &argoappv1.ApplicationList{ 259 | Items: []argoappv1.Application{ 260 | { 261 | ObjectMeta: metav1.ObjectMeta{ 262 | Annotations: map[string]string{ 263 | "argocd.argoproj.io/manifest-generate-paths": "./path", 264 | }, 265 | Name: "right-app", 266 | }, 267 | Spec: argoappv1.ApplicationSpec{ 268 | Source: &argoappv1.ApplicationSource{ 269 | RepoURL: "", 270 | Path: "right/", 271 | }, 272 | }, 273 | }, 274 | }, 275 | } 276 | 277 | mockApplicationClient.EXPECT().List(gomock.Any(), gomock.Any()).Return(expectedResponse, nil) 278 | app, err := findArgocdAppByManifestPathAnnotation(ctx, "right/path", "some-repo", mockApplicationClient) 279 | if err != nil { 280 | t.Errorf("Error: %v", err) 281 | } else if app.Name != "right-app" { 282 | t.Errorf("App name is not right-app") 283 | } 284 | } 285 | 286 | func TestFindArgocdAppByPathAnnotationNotFound(t *testing.T) { 287 | t.Parallel() 288 | defer testutils.Quiet()() 289 | ctx := context.Background() 290 | ctrl := gomock.NewController(t) 291 | defer ctrl.Finish() 292 | mockApplicationClient := mocks.NewMockApplicationServiceClient(ctrl) 293 | expectedResponse := &argoappv1.ApplicationList{ 294 | Items: []argoappv1.Application{ 295 | { 296 | ObjectMeta: metav1.ObjectMeta{ 297 | Annotations: map[string]string{ 298 | "argocd.argoproj.io/manifest-generate-paths": "non-existing-path", 299 | }, 300 | Name: "non-existing-app", 301 | }, 302 | Spec: argoappv1.ApplicationSpec{ 303 | Source: &argoappv1.ApplicationSource{ 304 | RepoURL: "", 305 | Path: "non-existing/", 306 | }, 307 | }, 308 | }, 309 | }, 310 | } 311 | 312 | mockApplicationClient.EXPECT().List(gomock.Any(), gomock.Any()).Return(expectedResponse, nil) 313 | app, err := findArgocdAppByManifestPathAnnotation(ctx, "non-existing/path", "some-repo", mockApplicationClient) 314 | if err != nil { 315 | t.Errorf("Error: %v", err) 316 | } 317 | if app != nil { 318 | log.Fatal("expected the application to be nil") 319 | } 320 | } 321 | 322 | func TestFetchArgoDiffConcurrently(t *testing.T) { 323 | t.Parallel() 324 | // MockApplicationServiceClient 325 | mockCtrl := gomock.NewController(t) 326 | defer mockCtrl.Finish() 327 | 328 | // mock the argoClients 329 | mockAppServiceClient := mocks.NewMockApplicationServiceClient(mockCtrl) 330 | mockSettingsServiceClient := mocks.NewMockSettingsServiceClient(mockCtrl) 331 | mockProjectServiceClient := mocks.NewMockProjectServiceClient(mockCtrl) 332 | // fake InitArgoClients 333 | 334 | argoClients := argoCdClients{ 335 | app: mockAppServiceClient, 336 | setting: mockSettingsServiceClient, 337 | project: mockProjectServiceClient, 338 | } 339 | // slowReply simulates a slow reply from the server 340 | slowReply := func(ctx context.Context, in any, opts ...any) { 341 | time.Sleep(time.Second) 342 | } 343 | 344 | // makeComponents for test 345 | makeComponents := func(num int) map[string]bool { 346 | components := make(map[string]bool, num) 347 | for i := 0; i < num; i++ { 348 | components[fmt.Sprintf("component/to/diff/%d", i)] = true 349 | } 350 | return components 351 | } 352 | 353 | mockSettingsServiceClient.EXPECT(). 354 | Get(gomock.Any(), gomock.Any()). 355 | Return(&settings.Settings{ 356 | URL: "https://test-argocd.test.test", 357 | }, nil) 358 | // mock the List method 359 | mockAppServiceClient.EXPECT(). 360 | List(gomock.Any(), gomock.Any(), gomock.Any()). 361 | Return(&argoappv1.ApplicationList{ 362 | Items: []argoappv1.Application{ 363 | { 364 | TypeMeta: metav1.TypeMeta{}, 365 | ObjectMeta: metav1.ObjectMeta{}, 366 | Spec: argoappv1.ApplicationSpec{}, 367 | Status: argoappv1.ApplicationStatus{}, 368 | Operation: &argoappv1.Operation{}, 369 | }, 370 | }, 371 | }, nil). 372 | AnyTimes(). 373 | Do(slowReply) // simulate slow reply 374 | 375 | // mock the Get method 376 | mockAppServiceClient.EXPECT(). 377 | Get(gomock.Any(), gomock.Any()). 378 | Return(&argoappv1.Application{ 379 | TypeMeta: metav1.TypeMeta{}, 380 | ObjectMeta: metav1.ObjectMeta{ 381 | Name: "test-app", 382 | }, 383 | Spec: argoappv1.ApplicationSpec{ 384 | Source: &argoappv1.ApplicationSource{ 385 | TargetRevision: "test-revision", 386 | }, 387 | SyncPolicy: &argoappv1.SyncPolicy{ 388 | Automated: &argoappv1.SyncPolicyAutomated{}, 389 | }, 390 | }, 391 | Status: argoappv1.ApplicationStatus{}, 392 | Operation: &argoappv1.Operation{}, 393 | }, nil). 394 | AnyTimes() 395 | 396 | // mock managedResource 397 | mockAppServiceClient.EXPECT(). 398 | ManagedResources(gomock.Any(), gomock.Any()). 399 | Return(&application.ManagedResourcesResponse{}, nil). 400 | AnyTimes() 401 | 402 | // mock the GetManifests method 403 | mockAppServiceClient.EXPECT(). 404 | GetManifests(gomock.Any(), gomock.Any()). 405 | Return(&reposerverApiClient.ManifestResponse{}, nil). 406 | AnyTimes() 407 | 408 | // mock the GetDetailedProject method 409 | mockProjectServiceClient.EXPECT(). 410 | GetDetailedProject(gomock.Any(), gomock.Any()). 411 | Return(&project.DetailedProjectsResponse{}, nil). 412 | AnyTimes() 413 | 414 | const numComponents = 5 415 | // start timer 416 | start := time.Now() 417 | 418 | // TODO: Test all the return values, for now we will just ignore the linter. 419 | _, _, diffResults, _ := GenerateDiffOfChangedComponents( //nolint:dogsled 420 | context.TODO(), 421 | makeComponents(numComponents), 422 | "test-pr-branch", 423 | "test-repo", 424 | true, 425 | false, 426 | argoClients, 427 | ) 428 | 429 | // stop timer 430 | elapsed := time.Since(start) 431 | assert.Equal(t, numComponents, len(diffResults)) 432 | // assert that the entire run takes less than numComponents * 1 second 433 | assert.Less(t, elapsed, time.Duration(numComponents)*time.Second) 434 | } 435 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/README.md: -------------------------------------------------------------------------------- 1 | This package has been pulled from the internal 2 | [diff](https://github.com/golang/go/tree/master/src/internal/diff) package. 3 | 4 | Minor changes were done to allow a custom number of context lines, import a 5 | public `txtar` package and adhere to the local linter settings. 6 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/diff.go: -------------------------------------------------------------------------------- 1 | // Copyright 2022 The Go Authors. All rights reserved. 2 | // Use of this source code is governed by a BSD-style 3 | // license that can be found in the LICENSE file.package argocd 4 | 5 | package diff 6 | 7 | import ( 8 | "bytes" 9 | "fmt" 10 | "sort" 11 | "strings" 12 | ) 13 | 14 | // A pair is a pair of values tracked for both the x and y side of a diff. 15 | // It is typically a pair of line indexes. 16 | type pair struct{ x, y int } 17 | 18 | // Diff returns an anchored diff of the two texts old and new 19 | // in the “unified diff” format. If old and new are identical, 20 | // Diff returns a nil slice (no output). 21 | // 22 | // Unix diff implementations typically look for a diff with 23 | // the smallest number of lines inserted and removed, 24 | // which can in the worst case take time quadratic in the 25 | // number of lines in the texts. As a result, many implementations 26 | // either can be made to run for a long time or cut off the search 27 | // after a predetermined amount of work. 28 | // 29 | // In contrast, this implementation looks for a diff with the 30 | // smallest number of “unique” lines inserted and removed, 31 | // where unique means a line that appears just once in both old and new. 32 | // We call this an “anchored diff” because the unique lines anchor 33 | // the chosen matching regions. An anchored diff is usually clearer 34 | // than a standard diff, because the algorithm does not try to 35 | // reuse unrelated blank lines or closing braces. 36 | // The algorithm also guarantees to run in O(n log n) time 37 | // instead of the standard O(n²) time. 38 | // 39 | // Some systems call this approach a “patience diff,” named for 40 | // the “patience sorting” algorithm, itself named for a solitaire card game. 41 | // We avoid that name for two reasons. First, the name has been used 42 | // for a few different variants of the algorithm, so it is imprecise. 43 | // Second, the name is frequently interpreted as meaning that you have 44 | // to wait longer (to be patient) for the diff, meaning that it is a slower algorithm, 45 | // when in fact the algorithm is faster than the standard one. 46 | func Diff(C int, oldName string, old []byte, newName string, new []byte) []byte { 47 | if bytes.Equal(old, new) { 48 | return nil 49 | } 50 | x := lines(old) 51 | y := lines(new) 52 | 53 | // Print diff header. 54 | var out bytes.Buffer 55 | fmt.Fprintf(&out, "diff %s %s\n", oldName, newName) 56 | fmt.Fprintf(&out, "--- %s\n", oldName) 57 | fmt.Fprintf(&out, "+++ %s\n", newName) 58 | 59 | // Loop over matches to consider, 60 | // expanding each match to include surrounding lines, 61 | // and then printing diff chunks. 62 | // To avoid setup/teardown cases outside the loop, 63 | // tgs returns a leading {0,0} and trailing {len(x), len(y)} pair 64 | // in the sequence of matches. 65 | var ( 66 | done pair // printed up to x[:done.x] and y[:done.y] 67 | chunk pair // start lines of current chunk 68 | count pair // number of lines from each side in current chunk 69 | ctext []string // lines for current chunk 70 | ) 71 | for _, m := range tgs(x, y) { 72 | if m.x < done.x { 73 | // Already handled scanning forward from earlier match. 74 | continue 75 | } 76 | 77 | // Expand matching lines as far as possible, 78 | // establishing that x[start.x:end.x] == y[start.y:end.y]. 79 | // Note that on the first (or last) iteration we may (or definitely do) 80 | // have an empty match: start.x==end.x and start.y==end.y. 81 | start := m 82 | for start.x > done.x && start.y > done.y && x[start.x-1] == y[start.y-1] { 83 | start.x-- 84 | start.y-- 85 | } 86 | end := m 87 | for end.x < len(x) && end.y < len(y) && x[end.x] == y[end.y] { 88 | end.x++ 89 | end.y++ 90 | } 91 | 92 | // Emit the mismatched lines before start into this chunk. 93 | // (No effect on first sentinel iteration, when start = {0,0}.) 94 | for _, s := range x[done.x:start.x] { 95 | ctext = append(ctext, "-"+s) 96 | count.x++ 97 | } 98 | for _, s := range y[done.y:start.y] { 99 | ctext = append(ctext, "+"+s) 100 | count.y++ 101 | } 102 | 103 | // If we're not at EOF and have too few common lines, 104 | // the chunk includes all the common lines and continues. 105 | if (end.x < len(x) || end.y < len(y)) && 106 | (end.x-start.x < C || (len(ctext) > 0 && end.x-start.x < 2*C)) { 107 | for _, s := range x[start.x:end.x] { 108 | ctext = append(ctext, " "+s) 109 | count.x++ 110 | count.y++ 111 | } 112 | done = end 113 | continue 114 | } 115 | 116 | // End chunk with common lines for context. 117 | if len(ctext) > 0 { 118 | n := end.x - start.x 119 | if n > C { 120 | n = C 121 | } 122 | for _, s := range x[start.x : start.x+n] { 123 | ctext = append(ctext, " "+s) 124 | count.x++ 125 | count.y++ 126 | } 127 | done = pair{start.x + n, start.y + n} 128 | 129 | // Format and emit chunk. 130 | // Convert line numbers to 1-indexed. 131 | // Special case: empty file shows up as 0,0 not 1,0. 132 | if count.x > 0 { 133 | chunk.x++ 134 | } 135 | if count.y > 0 { 136 | chunk.y++ 137 | } 138 | fmt.Fprintf(&out, "@@ -%d,%d +%d,%d @@\n", chunk.x, count.x, chunk.y, count.y) 139 | for _, s := range ctext { 140 | out.WriteString(s) 141 | } 142 | count.x = 0 143 | count.y = 0 144 | ctext = ctext[:0] 145 | } 146 | 147 | // If we reached EOF, we're done. 148 | if end.x >= len(x) && end.y >= len(y) { 149 | break 150 | } 151 | 152 | // Otherwise start a new chunk. 153 | chunk = pair{end.x - C, end.y - C} 154 | for _, s := range x[chunk.x:end.x] { 155 | ctext = append(ctext, " "+s) 156 | count.x++ 157 | count.y++ 158 | } 159 | done = end 160 | } 161 | 162 | return out.Bytes() 163 | } 164 | 165 | // lines returns the lines in the file x, including newlines. 166 | // If the file does not end in a newline, one is supplied 167 | // along with a warning about the missing newline. 168 | func lines(x []byte) []string { 169 | l := strings.SplitAfter(string(x), "\n") 170 | if l[len(l)-1] == "" { 171 | l = l[:len(l)-1] 172 | } else { 173 | // Treat last line as having a message about the missing newline attached, 174 | // using the same text as BSD/GNU diff (including the leading backslash). 175 | l[len(l)-1] += "\n\\ No newline at end of file\n" 176 | } 177 | return l 178 | } 179 | 180 | // tgs returns the pairs of indexes of the longest common subsequence 181 | // of unique lines in x and y, where a unique line is one that appears 182 | // once in x and once in y. 183 | // 184 | // The longest common subsequence algorithm is as described in 185 | // Thomas G. Szymanski, “A Special Case of the Maximal Common 186 | // Subsequence Problem,” Princeton TR #170 (January 1975), 187 | // available at https://research.swtch.com/tgs170.pdf. 188 | func tgs(x, y []string) []pair { 189 | // Count the number of times each string appears in a and b. 190 | // We only care about 0, 1, many, counted as 0, -1, -2 191 | // for the x side and 0, -4, -8 for the y side. 192 | // Using negative numbers now lets us distinguish positive line numbers later. 193 | m := make(map[string]int) 194 | for _, s := range x { 195 | if c := m[s]; c > -2 { 196 | m[s] = c - 1 197 | } 198 | } 199 | for _, s := range y { 200 | if c := m[s]; c > -8 { 201 | m[s] = c - 4 202 | } 203 | } 204 | 205 | // Now unique strings can be identified by m[s] = -1+-4. 206 | // 207 | // Gather the indexes of those strings in x and y, building: 208 | // xi[i] = increasing indexes of unique strings in x. 209 | // yi[i] = increasing indexes of unique strings in y. 210 | // inv[i] = index j such that x[xi[i]] = y[yi[j]]. 211 | var xi, yi, inv []int 212 | for i, s := range y { 213 | if m[s] == -1+-4 { 214 | m[s] = len(yi) 215 | yi = append(yi, i) 216 | } 217 | } 218 | for i, s := range x { 219 | if j, ok := m[s]; ok && j >= 0 { 220 | xi = append(xi, i) 221 | inv = append(inv, j) 222 | } 223 | } 224 | 225 | // Apply Algorithm A from Szymanski's paper. 226 | // In those terms, A = J = inv and B = [0, n). 227 | // We add sentinel pairs {0,0}, and {len(x),len(y)} 228 | // to the returned sequence, to help the processing loop. 229 | J := inv 230 | n := len(xi) 231 | T := make([]int, n) 232 | L := make([]int, n) 233 | for i := range T { 234 | T[i] = n + 1 235 | } 236 | for i := 0; i < n; i++ { 237 | k := sort.Search(n, func(k int) bool { 238 | return T[k] >= J[i] 239 | }) 240 | T[k] = J[i] 241 | L[i] = k + 1 242 | } 243 | k := 0 244 | for _, v := range L { 245 | if k < v { 246 | k = v 247 | } 248 | } 249 | seq := make([]pair, 2+k) 250 | seq[1+k] = pair{len(x), len(y)} // sentinel at end 251 | lastj := n 252 | for i := n - 1; i >= 0; i-- { 253 | if L[i] == k && J[i] < lastj { 254 | seq[k] = pair{xi[i], yi[J[i]]} 255 | k-- 256 | } 257 | } 258 | seq[0] = pair{0, 0} // sentinel at start 259 | return seq 260 | } 261 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/diff_test.go: -------------------------------------------------------------------------------- 1 | // Copyright 2022 The Go Authors. All rights reserved. 2 | // Use of this source code is governed by a BSD-style 3 | // license that can be found in the LICENSE file. 4 | 5 | package diff 6 | 7 | import ( 8 | "bytes" 9 | "path/filepath" 10 | "testing" 11 | 12 | "golang.org/x/tools/txtar" 13 | ) 14 | 15 | func clean(text []byte) []byte { 16 | text = bytes.ReplaceAll(text, []byte("$\n"), []byte("\n")) 17 | text = bytes.TrimSuffix(text, []byte("^D\n")) 18 | return text 19 | } 20 | 21 | func TestDiff(t *testing.T) { 22 | t.Parallel() 23 | files, _ := filepath.Glob("testdata/*.txt") 24 | if len(files) == 0 { 25 | t.Fatalf("no testdata") 26 | } 27 | 28 | for _, file := range files { 29 | t.Run(filepath.Base(file), func(t *testing.T) { 30 | t.Parallel() 31 | a, err := txtar.ParseFile(file) 32 | if err != nil { 33 | t.Fatal(err) 34 | } 35 | if len(a.Files) != 3 || a.Files[2].Name != "diff" { 36 | t.Fatalf("%s: want three files, third named \"diff\"", file) 37 | } 38 | diffs := Diff(3, a.Files[0].Name, clean(a.Files[0].Data), a.Files[1].Name, clean(a.Files[1].Data)) 39 | want := clean(a.Files[2].Data) 40 | if !bytes.Equal(diffs, want) { 41 | t.Fatalf("%s: have:\n%s\nwant:\n%s\n%s", file, 42 | diffs, want, Diff(3, "have", diffs, "want", want)) 43 | } 44 | }) 45 | } 46 | } 47 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/testdata/allnew.txt: -------------------------------------------------------------------------------- 1 | -- old -- 2 | -- new -- 3 | a 4 | b 5 | c 6 | -- diff -- 7 | diff old new 8 | --- old 9 | +++ new 10 | @@ -0,0 +1,3 @@ 11 | +a 12 | +b 13 | +c 14 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/testdata/allold.txt: -------------------------------------------------------------------------------- 1 | -- old -- 2 | a 3 | b 4 | c 5 | -- new -- 6 | -- diff -- 7 | diff old new 8 | --- old 9 | +++ new 10 | @@ -1,3 +0,0 @@ 11 | -a 12 | -b 13 | -c 14 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/testdata/basic.txt: -------------------------------------------------------------------------------- 1 | Example from Hunt and McIlroy, “An Algorithm for Differential File Comparison.” 2 | https://www.cs.dartmouth.edu/~doug/diff.pdf 3 | 4 | -- old -- 5 | a 6 | b 7 | c 8 | d 9 | e 10 | f 11 | g 12 | -- new -- 13 | w 14 | a 15 | b 16 | x 17 | y 18 | z 19 | e 20 | -- diff -- 21 | diff old new 22 | --- old 23 | +++ new 24 | @@ -1,7 +1,7 @@ 25 | +w 26 | a 27 | b 28 | -c 29 | -d 30 | +x 31 | +y 32 | +z 33 | e 34 | -f 35 | -g 36 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/testdata/dups.txt: -------------------------------------------------------------------------------- 1 | -- old -- 2 | a 3 | 4 | b 5 | 6 | c 7 | 8 | d 9 | 10 | e 11 | 12 | f 13 | -- new -- 14 | a 15 | 16 | B 17 | 18 | C 19 | 20 | d 21 | 22 | e 23 | 24 | f 25 | -- diff -- 26 | diff old new 27 | --- old 28 | +++ new 29 | @@ -1,8 +1,8 @@ 30 | a 31 | $ 32 | -b 33 | - 34 | -c 35 | +B 36 | + 37 | +C 38 | $ 39 | d 40 | $ 41 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/testdata/end.txt: -------------------------------------------------------------------------------- 1 | -- old -- 2 | 1 3 | 2 4 | 3 5 | 4 6 | 5 7 | 6 8 | 7 9 | eight 10 | nine 11 | ten 12 | eleven 13 | -- new -- 14 | 1 15 | 2 16 | 3 17 | 4 18 | 5 19 | 6 20 | 7 21 | 8 22 | 9 23 | 10 24 | -- diff -- 25 | diff old new 26 | --- old 27 | +++ new 28 | @@ -5,7 +5,6 @@ 29 | 5 30 | 6 31 | 7 32 | -eight 33 | -nine 34 | -ten 35 | -eleven 36 | +8 37 | +9 38 | +10 39 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/testdata/eof.txt: -------------------------------------------------------------------------------- 1 | -- old -- 2 | a 3 | b 4 | c^D 5 | -- new -- 6 | a 7 | b 8 | c^D 9 | -- diff -- 10 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/testdata/eof1.txt: -------------------------------------------------------------------------------- 1 | -- old -- 2 | a 3 | b 4 | c 5 | -- new -- 6 | a 7 | b 8 | c^D 9 | -- diff -- 10 | diff old new 11 | --- old 12 | +++ new 13 | @@ -1,3 +1,3 @@ 14 | a 15 | b 16 | -c 17 | +c 18 | \ No newline at end of file 19 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/testdata/eof2.txt: -------------------------------------------------------------------------------- 1 | -- old -- 2 | a 3 | b 4 | c^D 5 | -- new -- 6 | a 7 | b 8 | c 9 | -- diff -- 10 | diff old new 11 | --- old 12 | +++ new 13 | @@ -1,3 +1,3 @@ 14 | a 15 | b 16 | -c 17 | \ No newline at end of file 18 | +c 19 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/testdata/long.txt: -------------------------------------------------------------------------------- 1 | -- old -- 2 | 1 3 | 2 4 | 3 5 | 4 6 | 5 7 | 6 8 | 7 9 | 8 10 | 9 11 | 10 12 | 11 13 | 12 14 | 13 15 | 14 16 | 14½ 17 | 15 18 | 16 19 | 17 20 | 18 21 | 19 22 | 20 23 | -- new -- 24 | 1 25 | 2 26 | 3 27 | 4 28 | 5 29 | 6 30 | 8 31 | 9 32 | 10 33 | 11 34 | 12 35 | 13 36 | 14 37 | 17 38 | 18 39 | 19 40 | 20 41 | -- diff -- 42 | diff old new 43 | --- old 44 | +++ new 45 | @@ -4,7 +4,6 @@ 46 | 4 47 | 5 48 | 6 49 | -7 50 | 8 51 | 9 52 | 10 53 | @@ -12,9 +11,6 @@ 54 | 12 55 | 13 56 | 14 57 | -14½ 58 | -15 59 | -16 60 | 17 61 | 18 62 | 19 63 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/testdata/same.txt: -------------------------------------------------------------------------------- 1 | -- old -- 2 | hello world 3 | -- new -- 4 | hello world 5 | -- diff -- 6 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/testdata/start.txt: -------------------------------------------------------------------------------- 1 | -- old -- 2 | e 3 | pi 4 | 4 5 | 5 6 | 6 7 | 7 8 | 8 9 | 9 10 | 10 11 | -- new -- 12 | 1 13 | 2 14 | 3 15 | 4 16 | 5 17 | 6 18 | 7 19 | 8 20 | 9 21 | 10 22 | -- diff -- 23 | diff old new 24 | --- old 25 | +++ new 26 | @@ -1,5 +1,6 @@ 27 | -e 28 | -pi 29 | +1 30 | +2 31 | +3 32 | 4 33 | 5 34 | 6 35 | -------------------------------------------------------------------------------- /internal/pkg/argocd/diff/testdata/triv.txt: -------------------------------------------------------------------------------- 1 | Another example from Hunt and McIlroy, 2 | “An Algorithm for Differential File Comparison.” 3 | https://www.cs.dartmouth.edu/~doug/diff.pdf 4 | 5 | Anchored diff gives up on finding anything, 6 | since there are no unique lines. 7 | 8 | -- old -- 9 | a 10 | b 11 | c 12 | a 13 | b 14 | b 15 | a 16 | -- new -- 17 | c 18 | a 19 | b 20 | a 21 | b 22 | c 23 | -- diff -- 24 | diff old new 25 | --- old 26 | +++ new 27 | @@ -1,7 +1,6 @@ 28 | -a 29 | -b 30 | -c 31 | -a 32 | -b 33 | -b 34 | -a 35 | +c 36 | +a 37 | +b 38 | +a 39 | +b 40 | +c 41 | -------------------------------------------------------------------------------- /internal/pkg/argocd/testdata/TestDiffLiveVsTargetObject/1.live: -------------------------------------------------------------------------------- 1 | { 2 | "apiVersion": "commercetools.io/v1alpha1", 3 | "kind": "Bar", 4 | "metadata": { 5 | "generation": 46, 6 | "labels": { 7 | "argocd.argoproj.io/instance": "foobar-plg-gcp-eu-west1-v1" 8 | }, 9 | "managedFields": [ 10 | { 11 | "apiVersion": "commercetools.io/v1alpha1", 12 | "fieldsType": "FieldsV1", 13 | "fieldsV1": { 14 | "f:metadata": { 15 | "f:labels": { 16 | "f:argocd.argoproj.io/instance": {} 17 | } 18 | }, 19 | "f:spec": { 20 | "f:deploymentName": {}, 21 | "f:replicas": {} 22 | } 23 | }, 24 | "manager": "argocd-controller", 25 | "operation": "Apply", 26 | "time": "2024-08-30T15:08:52Z" 27 | } 28 | ], 29 | "name": "example-baz-bar", 30 | "namespace": "gitops-demo-ns2", 31 | "resourceVersion": "2168525640", 32 | "uid": "f845fd72-d6d9-48f2-b0f2-2def6807deb8" 33 | }, 34 | "spec": { 35 | "deploymentName": "example-baz-bar", 36 | "replicas": 63 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /internal/pkg/argocd/testdata/TestDiffLiveVsTargetObject/1.target: -------------------------------------------------------------------------------- 1 | { 2 | "apiVersion": "commercetools.io/v1alpha1", 3 | "kind": "Bar", 4 | "metadata": { 5 | "generation": 46, 6 | "labels": { 7 | "argocd.argoproj.io/instance": "foobar-plg-gcp-eu-west1-v1" 8 | }, 9 | "managedFields": [ 10 | { 11 | "apiVersion": "commercetools.io/v1alpha1", 12 | "fieldsType": "FieldsV1", 13 | "fieldsV1": { 14 | "f:metadata": { 15 | "f:labels": { 16 | "f:argocd.argoproj.io/instance": {} 17 | } 18 | }, 19 | "f:spec": { 20 | "f:deploymentName": {}, 21 | "f:replicas": {} 22 | } 23 | }, 24 | "manager": "argocd-controller", 25 | "operation": "Apply", 26 | "time": "2024-08-30T15:08:52Z" 27 | } 28 | ], 29 | "name": "example-baz-bar", 30 | "namespace": "gitops-demo-ns2", 31 | "resourceVersion": "2168525640", 32 | "uid": "f845fd72-d6d9-48f2-b0f2-2def6807deb8" 33 | }, 34 | "spec": { 35 | "deploymentName": "example-baz-bar", 36 | "replicas": 42 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /internal/pkg/argocd/testdata/TestDiffLiveVsTargetObject/1.want: -------------------------------------------------------------------------------- 1 | diff live target 2 | --- live 3 | +++ target 4 | @@ -17,11 +17,11 @@ 5 | f:replicas: {} 6 | manager: argocd-controller 7 | operation: Apply 8 | time: "2024-08-30T15:08:52Z" 9 | name: example-baz-bar 10 | namespace: gitops-demo-ns2 11 | resourceVersion: "2168525640" 12 | uid: f845fd72-d6d9-48f2-b0f2-2def6807deb8 13 | spec: 14 | deploymentName: example-baz-bar 15 | - replicas: 63 16 | + replicas: 42 17 | -------------------------------------------------------------------------------- /internal/pkg/argocd/testdata/TestRenderDiff.live: -------------------------------------------------------------------------------- 1 | { 2 | "apiVersion": "commercetools.io/v1alpha1", 3 | "kind": "Bar", 4 | "rbacBindings": [ 5 | { 6 | "clusterRoleBindings": [ 7 | { 8 | "clusterRole": "view" 9 | } 10 | ], 11 | "name": "security-audit-viewer-vault", 12 | "subjects": [ 13 | { 14 | "kind": "Group", 15 | "name": "vault:some-team@domain.tld" 16 | } 17 | ] 18 | } 19 | ], 20 | "metadata": { 21 | "generation": 46, 22 | "labels": { 23 | "argocd.argoproj.io/instance": "foobar-plg-gcp-eu-west1-v1" 24 | }, 25 | "managedFields": [ 26 | { 27 | "apiVersion": "commercetools.io/v1alpha1", 28 | "fieldsType": "FieldsV1", 29 | "fieldsV1": { 30 | "f:metadata": { 31 | "f:labels": { 32 | "f:argocd.argoproj.io/instance": {} 33 | } 34 | }, 35 | "f:spec": { 36 | "f:deploymentName": {}, 37 | "f:replicas": {} 38 | } 39 | }, 40 | "manager": "argocd-controller", 41 | "operation": "Apply", 42 | "time": "2024-08-30T15:08:52Z" 43 | } 44 | ], 45 | "name": "example-baz-bar", 46 | "namespace": "gitops-demo-ns2", 47 | "resourceVersion": "2168525640", 48 | "uid": "f845fd72-d6d9-48f2-b0f2-2def6807deb8" 49 | }, 50 | "spec": { 51 | "deploymentName": "example-baz-bar", 52 | "replicas": 63 53 | } 54 | } 55 | -------------------------------------------------------------------------------- /internal/pkg/argocd/testdata/TestRenderDiff.md: -------------------------------------------------------------------------------- 1 | ```diff 2 | diff live target 3 | --- live 4 | +++ target 5 | @@ -21,14 +21,14 @@ 6 | name: example-baz-bar 7 | namespace: gitops-demo-ns2 8 | resourceVersion: "2168525640" 9 | uid: f845fd72-d6d9-48f2-b0f2-2def6807deb8 10 | rbacBindings: 11 | - clusterRoleBindings: 12 | - clusterRole: view 13 | name: security-audit-viewer-vault 14 | subjects: 15 | - kind: Group 16 | - name: vault:some-team@domain.tld 17 | + name: vault:some-team-name@domain.tld 18 | spec: 19 | deploymentName: example-baz-bar 20 | - replicas: 63 21 | + replicas: 42 22 | ``` 23 | -------------------------------------------------------------------------------- /internal/pkg/argocd/testdata/TestRenderDiff.target: -------------------------------------------------------------------------------- 1 | { 2 | "apiVersion": "commercetools.io/v1alpha1", 3 | "kind": "Bar", 4 | "rbacBindings": [ 5 | { 6 | "clusterRoleBindings": [ 7 | { 8 | "clusterRole": "view" 9 | } 10 | ], 11 | "name": "security-audit-viewer-vault", 12 | "subjects": [ 13 | { 14 | "kind": "Group", 15 | "name": "vault:some-team-name@domain.tld" 16 | } 17 | ] 18 | } 19 | ], 20 | "metadata": { 21 | "generation": 46, 22 | "labels": { 23 | "argocd.argoproj.io/instance": "foobar-plg-gcp-eu-west1-v1" 24 | }, 25 | "managedFields": [ 26 | { 27 | "apiVersion": "commercetools.io/v1alpha1", 28 | "fieldsType": "FieldsV1", 29 | "fieldsV1": { 30 | "f:metadata": { 31 | "f:labels": { 32 | "f:argocd.argoproj.io/instance": {} 33 | } 34 | }, 35 | "f:spec": { 36 | "f:deploymentName": {}, 37 | "f:replicas": {} 38 | } 39 | }, 40 | "manager": "argocd-controller", 41 | "operation": "Apply", 42 | "time": "2024-08-30T15:08:52Z" 43 | } 44 | ], 45 | "name": "example-baz-bar", 46 | "namespace": "gitops-demo-ns2", 47 | "resourceVersion": "2168525640", 48 | "uid": "f845fd72-d6d9-48f2-b0f2-2def6807deb8" 49 | }, 50 | "spec": { 51 | "deploymentName": "example-baz-bar", 52 | "replicas": 42 53 | } 54 | } 55 | -------------------------------------------------------------------------------- /internal/pkg/configuration/config.go: -------------------------------------------------------------------------------- 1 | package configuration 2 | 3 | import ( 4 | yaml "gopkg.in/yaml.v2" 5 | ) 6 | 7 | type WebhookEndpointRegex struct { 8 | Expression string `yaml:"expression"` 9 | Replacements []string `yaml:"replacements"` 10 | } 11 | 12 | type ComponentConfig struct { 13 | PromotionTargetAllowList []string `yaml:"promotionTargetAllowList"` 14 | PromotionTargetBlockList []string `yaml:"promotionTargetBlockList"` 15 | DisableArgoCDDiff bool `yaml:"disableArgoCDDiff"` 16 | } 17 | 18 | type Condition struct { 19 | PrHasLabels []string `yaml:"prHasLabels"` 20 | AutoMerge bool `yaml:"autoMerge"` 21 | } 22 | 23 | type PromotionPr struct { 24 | TargetDescription string `yaml:"targetDescription"` 25 | TargetPaths []string `yaml:"targetPaths"` 26 | } 27 | 28 | type PromotionPath struct { 29 | Conditions Condition `yaml:"conditions"` 30 | ComponentPathExtraDepth int `yaml:"componentPathExtraDepth"` 31 | SourcePath string `yaml:"sourcePath"` 32 | PromotionPrs []PromotionPr `yaml:"promotionPrs"` 33 | } 34 | 35 | type Config struct { 36 | // What paths trigger promotion to which paths 37 | PromotionPaths []PromotionPath `yaml:"promotionPaths"` 38 | 39 | // Generic configuration 40 | PromtionPrLables []string `yaml:"promtionPRlables"` 41 | DryRunMode bool `yaml:"dryRunMode"` 42 | AutoApprovePromotionPrs bool `yaml:"autoApprovePromotionPrs"` 43 | ToggleCommitStatus map[string]string `yaml:"toggleCommitStatus"` 44 | WebhookEndpointRegexs []WebhookEndpointRegex `yaml:"webhookEndpointRegexs"` 45 | WhProxtSkipTLSVerifyUpstream bool `yaml:"whProxtSkipTLSVerifyUpstream"` 46 | Argocd ArgocdConfig `yaml:"argocd"` 47 | } 48 | 49 | type ArgocdConfig struct { 50 | CommentDiffonPR bool `yaml:"commentDiffonPR"` 51 | AutoMergeNoDiffPRs bool `yaml:"autoMergeNoDiffPRs"` 52 | AllowSyncfromBranchPathRegex string `yaml:"allowSyncfromBranchPathRegex"` 53 | UseSHALabelForAppDiscovery bool `yaml:"useSHALabelForAppDiscovery"` 54 | CreateTempAppObjectFroNewApps bool `yaml:"createTempAppObjectFromNewApps"` 55 | } 56 | 57 | func ParseConfigFromYaml(y string) (*Config, error) { 58 | config := &Config{} 59 | 60 | err := yaml.Unmarshal([]byte(y), config) 61 | 62 | return config, err 63 | } 64 | -------------------------------------------------------------------------------- /internal/pkg/configuration/config_test.go: -------------------------------------------------------------------------------- 1 | package configuration 2 | 3 | import ( 4 | "os" 5 | "testing" 6 | 7 | "github.com/go-test/deep" 8 | ) 9 | 10 | func TestConfigurationParse(t *testing.T) { 11 | t.Parallel() 12 | 13 | configurationFileContent, _ := os.ReadFile("tests/testConfigurationParsing.yaml") 14 | 15 | config, err := ParseConfigFromYaml(string(configurationFileContent)) 16 | if err != nil { 17 | t.Fatalf("config parsing failed: err=%s", err) 18 | } 19 | 20 | if config.PromotionPaths == nil { 21 | t.Fatalf("config is missing PromotionPaths, %v", config.PromotionPaths) 22 | } 23 | 24 | expectedConfig := &Config{ 25 | PromotionPaths: []PromotionPath{ 26 | { 27 | SourcePath: "workspace/", 28 | Conditions: Condition{ 29 | PrHasLabels: []string{ 30 | "some-label", 31 | }, 32 | AutoMerge: true, 33 | }, 34 | PromotionPrs: []PromotionPr{ 35 | { 36 | TargetPaths: []string{ 37 | "env/staging/us-east4/c1/", 38 | }, 39 | }, 40 | { 41 | TargetPaths: []string{ 42 | "env/staging/europe-west4/c1/", 43 | }, 44 | }, 45 | }, 46 | }, 47 | { 48 | SourcePath: "env/staging/us-east4/c1/", 49 | Conditions: Condition{ 50 | AutoMerge: false, 51 | }, 52 | PromotionPrs: []PromotionPr{ 53 | { 54 | TargetPaths: []string{ 55 | "env/prod/us-central1/c2/", 56 | }, 57 | }, 58 | }, 59 | }, 60 | { 61 | SourcePath: "env/prod/us-central1/c2/", 62 | Conditions: Condition{ 63 | AutoMerge: false, 64 | }, 65 | PromotionPrs: []PromotionPr{ 66 | { 67 | TargetPaths: []string{ 68 | "env/prod/us-west1/c2/", 69 | "env/prod/us-central1/c3/", 70 | }, 71 | }, 72 | }, 73 | }, 74 | }, 75 | } 76 | 77 | if diff := deep.Equal(expectedConfig, config); diff != nil { 78 | t.Error(diff) 79 | } 80 | } 81 | -------------------------------------------------------------------------------- /internal/pkg/configuration/tests/testConfigurationParsing.yaml: -------------------------------------------------------------------------------- 1 | promotionPaths: 2 | - sourcePath: "workspace/" 3 | conditions: 4 | prHasLabels: 5 | - "some-label" 6 | autoMerge: true 7 | promotionPrs: 8 | - targetPaths: 9 | - "env/staging/us-east4/c1/" 10 | - targetPaths: 11 | - "env/staging/europe-west4/c1/" 12 | - sourcePath: "env/staging/us-east4/c1/" 13 | conditions: 14 | autoMerge: false 15 | promotionPrs: 16 | - targetPaths: 17 | - "env/prod/us-central1/c2/" 18 | - sourcePath: "env/prod/us-central1/c2/" 19 | conditions: 20 | promotionPrs: 21 | - targetPaths: 22 | - "env/prod/us-west1/c2/" 23 | - "env/prod/us-central1/c3/" 24 | 25 | promtionPrLables: 26 | - "promotion" 27 | promotionBranchNameTemplte: "promotions/{{.safeBranchName}}" 28 | promtionPrBodyTemplate: | 29 | This is a promotion of {{ .originalPrNumber }} 30 | Bla Bla 31 | -------------------------------------------------------------------------------- /internal/pkg/githubapi/.tmpMJcSWN: -------------------------------------------------------------------------------- 1 | package githubapi 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "path" 7 | "strings" 8 | 9 | "github.com/google/go-github/v48/github" 10 | log "github.com/sirupsen/logrus" 11 | prom "github.com/wayfair-incubator/telefonistka/internal/pkg/prometheus" 12 | ) 13 | 14 | type GhPrClientDetails struct { 15 | Ghclient *github.Client 16 | // This whole struct describe the metadata of the PR, so it makes sense to share the context with everything to generate HTTP calls related to that PR, right? 17 | Ctx context.Context //nolint:containedctx 18 | DefaultBranch string 19 | Owner string 20 | Repo string 21 | PrAuthor string 22 | PrNumber int 23 | PrSHA string 24 | Ref string 25 | PrLogger *log.Entry 26 | Labels []*github.Label 27 | } 28 | 29 | func (p GhPrClientDetails) CommentOnPr(commentBody string) error { 30 | commentBody = "\n" + commentBody 31 | 32 | comment := &github.IssueComment{Body: &commentBody} 33 | _, resp, err := p.Ghclient.Issues.CreateComment(p.Ctx, p.Owner, p.Repo, p.PrNumber, comment) 34 | prom.InstrumentGhCall(resp) 35 | if err != nil { 36 | p.PrLogger.Errorf("Could not comment in PR: err=%s\n%v\n", err, resp) 37 | } 38 | return err 39 | } 40 | 41 | func DoesPrHasLabel(eventPayload github.PullRequestEvent, name string) bool { 42 | result := false 43 | for _, prLabel := range eventPayload.PullRequest.Labels { 44 | if *prLabel.Name == name { 45 | result = true 46 | break 47 | } 48 | } 49 | return result 50 | } 51 | 52 | func (p *GhPrClientDetails) ToggleCommitStatus(context string, user string) error { 53 | var r error 54 | listOpts := &github.ListOptions{} 55 | 56 | initialStatuses, resp, err := p.Ghclient.Repositories.ListStatuses(p.Ctx, p.Owner, p.Repo, p.Ref, listOpts) 57 | prom.InstrumentGhCall(resp) 58 | if err != nil { 59 | p.PrLogger.Errorf("Failed to fetch existing statuses for commit %s, err=%s", p.Ref, err) 60 | r = err 61 | } 62 | 63 | for _, commitStatus := range initialStatuses { 64 | if *commitStatus.Context == context { 65 | if *commitStatus.State != "success" { 66 | p.PrLogger.Infof("%s Toggled %s(%s) to success", user, context, *commitStatus.State) 67 | *commitStatus.State = "success" 68 | _, resp, err := p.Ghclient.Repositories.CreateStatus(p.Ctx, p.Owner, p.Repo, p.PrSHA, commitStatus) 69 | prom.InstrumentGhCall(resp) 70 | if err != nil { 71 | p.PrLogger.Errorf("Failed to create context %s, err=%s", context, err) 72 | r = err 73 | } 74 | } else { 75 | p.PrLogger.Infof("%s Toggled %s(%s) to failure", user, context, *commitStatus.State) 76 | *commitStatus.State = "failure" 77 | _, resp, err := p.Ghclient.Repositories.CreateStatus(p.Ctx, p.Owner, p.Repo, p.PrSHA, commitStatus) 78 | prom.InstrumentGhCall(resp) 79 | if err != nil { 80 | p.PrLogger.Errorf("Failed to create context %s, err=%s", context, err) 81 | r = err 82 | } 83 | } 84 | break 85 | } 86 | } 87 | 88 | return r 89 | } 90 | 91 | func SetCommitStatus(ghPrClientDetails GhPrClientDetails, state string) { 92 | // TODO change all these values 93 | context := " -------------------------------------------------------------------------------- /internal/pkg/githubapi/clients.go: -------------------------------------------------------------------------------- 1 | package githubapi 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "net/http" 7 | "os" 8 | "strconv" 9 | "time" 10 | 11 | "github.com/bradleyfalzon/ghinstallation/v2" 12 | "github.com/google/go-github/v62/github" 13 | lru "github.com/hashicorp/golang-lru/v2" 14 | "github.com/shurcooL/githubv4" 15 | log "github.com/sirupsen/logrus" 16 | "golang.org/x/oauth2" 17 | ) 18 | 19 | func getEnv(key, fallback string) string { 20 | if value, ok := os.LookupEnv(key); ok { 21 | return value 22 | } 23 | return fallback 24 | } 25 | 26 | func getCrucialEnv(key string) string { 27 | if value, ok := os.LookupEnv(key); ok { 28 | return value 29 | } 30 | log.Fatalf("%s environment variable is required", key) 31 | os.Exit(3) 32 | return "" 33 | } 34 | 35 | type GhClientPair struct { 36 | v3Client *github.Client 37 | v4Client *githubv4.Client 38 | } 39 | 40 | func getAppInstallationId(githubAppPrivateKeyPath string, githubAppId int64, githubRestAltURL string, ctx context.Context, owner string) (int64, error) { 41 | atr, err := ghinstallation.NewAppsTransportKeyFromFile(http.DefaultTransport, githubAppId, githubAppPrivateKeyPath) 42 | if err != nil { 43 | panic(err) 44 | } 45 | tempClient := github.NewClient( 46 | &http.Client{ 47 | Transport: atr, 48 | Timeout: time.Second * 30, 49 | }) 50 | 51 | if githubRestAltURL != "" { 52 | tempClient, err = tempClient.WithEnterpriseURLs(githubRestAltURL, githubRestAltURL) 53 | if err != nil { 54 | log.Fatalf("failed to create git client for app: %v\n", err) 55 | } 56 | } 57 | 58 | installations, _, err := tempClient.Apps.ListInstallations(ctx, &github.ListOptions{}) 59 | if err != nil { 60 | log.Fatalf("failed to list installations: %v\n", err) 61 | } 62 | 63 | var installID int64 64 | for _, i := range installations { 65 | if *i.Account.Login == owner { 66 | installID = i.GetID() 67 | log.Infof("Installation ID for GitHub Application # %v is: %v", githubAppId, installID) 68 | return installID, nil 69 | } 70 | } 71 | 72 | return 0, err 73 | } 74 | 75 | func createGithubAppRestClient(githubAppPrivateKeyPath string, githubAppId int64, githubAppInstallationId int64, githubRestAltURL string, ctx context.Context) *github.Client { 76 | itr, err := ghinstallation.NewKeyFromFile(http.DefaultTransport, githubAppId, githubAppInstallationId, githubAppPrivateKeyPath) 77 | if err != nil { 78 | log.Fatal(err) 79 | } 80 | var client *github.Client 81 | 82 | if githubRestAltURL != "" { 83 | itr.BaseURL = githubRestAltURL 84 | client, _ = github.NewClient(&http.Client{Transport: itr}).WithEnterpriseURLs(githubRestAltURL, githubRestAltURL) 85 | } else { 86 | client = github.NewClient(&http.Client{Transport: itr}) 87 | } 88 | return client 89 | } 90 | 91 | func createGithubRestClient(githubOauthToken string, githubRestAltURL string, ctx context.Context) *github.Client { 92 | ts := oauth2.StaticTokenSource( 93 | &oauth2.Token{AccessToken: githubOauthToken}, 94 | ) 95 | tc := oauth2.NewClient(ctx, ts) 96 | client := github.NewClient(tc) 97 | if githubRestAltURL != "" { 98 | client, _ = client.WithEnterpriseURLs(githubRestAltURL, githubRestAltURL) 99 | } 100 | 101 | return client 102 | } 103 | 104 | func createGithubAppGraphQlClient(githubAppPrivateKeyPath string, githubAppId int64, githubAppInstallationId int64, githubGraphqlAltURL string, githubRestAltURL string, ctx context.Context) *githubv4.Client { 105 | itr, err := ghinstallation.NewKeyFromFile(http.DefaultTransport, githubAppId, githubAppInstallationId, githubAppPrivateKeyPath) 106 | if err != nil { 107 | log.Fatal(err) 108 | } 109 | var client *githubv4.Client 110 | 111 | if githubGraphqlAltURL != "" { 112 | itr.BaseURL = githubRestAltURL 113 | client = githubv4.NewEnterpriseClient(githubGraphqlAltURL, &http.Client{Transport: itr}) 114 | } else { 115 | client = githubv4.NewClient(&http.Client{Transport: itr}) 116 | } 117 | return client 118 | } 119 | 120 | func createGithubGraphQlClient(githubOauthToken string, githubGraphqlAltURL string) *githubv4.Client { 121 | ts := oauth2.StaticTokenSource( 122 | &oauth2.Token{AccessToken: githubOauthToken}, 123 | ) 124 | httpClient := oauth2.NewClient(context.Background(), ts) 125 | var client *githubv4.Client 126 | if githubGraphqlAltURL != "" { 127 | client = githubv4.NewEnterpriseClient(githubGraphqlAltURL, httpClient) 128 | } else { 129 | client = githubv4.NewClient(httpClient) 130 | } 131 | return client 132 | } 133 | 134 | func createGhAppClientPair(ctx context.Context, githubAppId int64, owner string, ghAppPKeyPathEnvVarName string) GhClientPair { 135 | var githubRestAltURL string 136 | var githubGraphqlAltURL string 137 | githubAppPrivateKeyPath := getCrucialEnv(ghAppPKeyPathEnvVarName) 138 | githubHost := getEnv("GITHUB_HOST", "") 139 | if githubHost != "" { 140 | githubRestAltURL = fmt.Sprintf("https://%s/api/v3", githubHost) 141 | githubGraphqlAltURL = fmt.Sprintf("https://%s/api/graphql", githubHost) 142 | log.Infof("Github REST API endpoint is configured to %s", githubRestAltURL) 143 | log.Infof("Github graphql API endpoint is configured to %s", githubGraphqlAltURL) 144 | } else { 145 | log.Debugf("Using public Github API endpoint") 146 | } 147 | 148 | githubAppInstallationId, err := getAppInstallationId(githubAppPrivateKeyPath, githubAppId, githubRestAltURL, ctx, owner) 149 | if err != nil { 150 | log.Errorf("Couldn't find installation for app ID %v and repo owner %s", githubAppId, owner) 151 | } 152 | 153 | return GhClientPair{ 154 | v3Client: createGithubAppRestClient(githubAppPrivateKeyPath, githubAppId, githubAppInstallationId, githubRestAltURL, ctx), 155 | v4Client: createGithubAppGraphQlClient(githubAppPrivateKeyPath, githubAppId, githubAppInstallationId, githubGraphqlAltURL, githubRestAltURL, ctx), 156 | } 157 | } 158 | 159 | func createGhTokenClientPair(ctx context.Context, ghOauthToken string) GhClientPair { 160 | var githubRestAltURL string 161 | var githubGraphqlAltURL string 162 | githubHost := getEnv("GITHUB_HOST", "") 163 | if githubHost != "" { 164 | githubRestAltURL = fmt.Sprintf("https://%s/api/v3", githubHost) 165 | githubGraphqlAltURL = fmt.Sprintf("https://%s/api/graphql", githubHost) 166 | log.Infof("Github REST API endpoint is configured to %s", githubRestAltURL) 167 | log.Infof("Github graphql API endpoint is configured to %s", githubGraphqlAltURL) 168 | } else { 169 | log.Debugf("Using public Github API endpoint") 170 | } 171 | 172 | return GhClientPair{ 173 | v3Client: createGithubRestClient(ghOauthToken, githubRestAltURL, ctx), 174 | v4Client: createGithubGraphQlClient(ghOauthToken, githubGraphqlAltURL), 175 | } 176 | } 177 | 178 | func (gcp *GhClientPair) GetAndCache(ghClientCache *lru.Cache[string, GhClientPair], ghAppIdEnvVarName string, ghAppPKeyPathEnvVarName string, ghOauthTokenEnvVarName string, repoOwner string, ctx context.Context) { 179 | githubAppId := getEnv(ghAppIdEnvVarName, "") 180 | var keyExist bool 181 | if githubAppId != "" { 182 | *gcp, keyExist = ghClientCache.Get(repoOwner) 183 | if keyExist { 184 | log.Debugf("Found cached client for %s", repoOwner) 185 | } else { 186 | log.Infof("Did not found cached client for %s, creating one with %s/%s env vars", repoOwner, ghAppIdEnvVarName, ghAppPKeyPathEnvVarName) 187 | githubAppIdint, err := strconv.ParseInt(githubAppId, 10, 64) 188 | if err != nil { 189 | log.Fatalf("GITHUB_APP_ID value could not converted to int64, %v", err) 190 | } 191 | *gcp = createGhAppClientPair(ctx, githubAppIdint, repoOwner, ghAppPKeyPathEnvVarName) 192 | ghClientCache.Add(repoOwner, *gcp) 193 | } 194 | } else { 195 | *gcp, keyExist = ghClientCache.Get("global") 196 | if keyExist { 197 | log.Debug("Found global cached client") 198 | } else { 199 | log.Infof("Did not found global cached client, creating one with %s env var", ghOauthTokenEnvVarName) 200 | ghOauthToken := getCrucialEnv(ghOauthTokenEnvVarName) 201 | 202 | *gcp = createGhTokenClientPair(ctx, ghOauthToken) 203 | ghClientCache.Add("global", *gcp) 204 | } 205 | } 206 | } 207 | -------------------------------------------------------------------------------- /internal/pkg/githubapi/drift_detection.go: -------------------------------------------------------------------------------- 1 | package githubapi 2 | 3 | import ( 4 | "bytes" 5 | "fmt" 6 | "strings" 7 | 8 | "github.com/google/go-github/v62/github" 9 | "github.com/hexops/gotextdiff" 10 | "github.com/hexops/gotextdiff/myers" 11 | "github.com/hexops/gotextdiff/span" 12 | prom "github.com/wayfair-incubator/telefonistka/internal/pkg/prometheus" 13 | ) 14 | 15 | func generateDiffOutput(ghPrClientDetails GhPrClientDetails, defaultBranch string, sourceFilesSHAs map[string]string, targetFilesSHAs map[string]string, sourcePath string, targetPath string) (bool, string, error) { 16 | var hasDiff bool 17 | var diffOutput bytes.Buffer 18 | var filesWithDiff []string 19 | diffOutput.WriteString("\n```diff\n") 20 | 21 | // staring with collecting files with different content and file only present in the source dir 22 | for filename, sha := range sourceFilesSHAs { 23 | ghPrClientDetails.PrLogger.Debugf("Looking at file %s", filename) 24 | if targetPathfileSha, found := targetFilesSHAs[filename]; found { 25 | if sha != targetPathfileSha { 26 | ghPrClientDetails.PrLogger.Debugf("%s is different from %s", sourcePath+"/"+filename, targetPath+"/"+filename) 27 | hasDiff = true 28 | sourceFileContent, _, _ := GetFileContent(ghPrClientDetails, defaultBranch, sourcePath+"/"+filename) 29 | targetFileContent, _, _ := GetFileContent(ghPrClientDetails, defaultBranch, targetPath+"/"+filename) 30 | 31 | edits := myers.ComputeEdits(span.URIFromPath(filename), sourceFileContent, targetFileContent) 32 | diffOutput.WriteString(fmt.Sprint(gotextdiff.ToUnified(sourcePath+"/"+filename, targetPath+"/"+filename, sourceFileContent, edits))) 33 | filesWithDiff = append(filesWithDiff, sourcePath+"/"+filename) 34 | } else { 35 | ghPrClientDetails.PrLogger.Debugf("%s is identical to %s", sourcePath+"/"+filename, targetPath+"/"+filename) 36 | } 37 | } else { 38 | hasDiff = true 39 | diffOutput.WriteString(fmt.Sprintf("--- %s/%s (missing from target dir %s)\n", sourcePath, filename, targetPath)) 40 | } 41 | } 42 | 43 | // then going over the target to check files that only exists there 44 | for filename := range targetFilesSHAs { 45 | if _, found := sourceFilesSHAs[filename]; !found { 46 | diffOutput.WriteString(fmt.Sprintf("+++ %s/%s (missing from source dir %s)\n", targetPath, filename, sourcePath)) 47 | hasDiff = true 48 | } 49 | } 50 | 51 | diffOutput.WriteString("\n```\n") 52 | 53 | if len(filesWithDiff) != 0 { 54 | diffOutput.WriteString("\n### Blame Links:\n") 55 | blameUrlPrefix := ghPrClientDetails.getBlameURLPrefix() 56 | 57 | for _, f := range filesWithDiff { 58 | diffOutput.WriteString("[" + f + "](" + blameUrlPrefix + "/HEAD/" + f + ")\n") // TODO consider switching HEAD to specific SHA 59 | } 60 | } 61 | 62 | return hasDiff, diffOutput.String(), nil 63 | } 64 | 65 | func CompareRepoDirectories(ghPrClientDetails GhPrClientDetails, sourcePath string, targetPath string, defaultBranch string) (bool, string, error) { 66 | // Compares two directories content 67 | 68 | // comparing sourcePath targetPath Git object SHA to avoid costly tree compare: 69 | sourcePathGitObjectSha, err := getDirecotyGitObjectSha(ghPrClientDetails, sourcePath, defaultBranch) 70 | if err != nil { 71 | ghPrClientDetails.PrLogger.Errorf("Couldn't get %v, Git object sha: %v", sourcePath, err) 72 | return false, "", err 73 | } 74 | targetPathGitObjectSha, err := getDirecotyGitObjectSha(ghPrClientDetails, targetPath, defaultBranch) 75 | if err != nil { 76 | ghPrClientDetails.PrLogger.Errorf("Couldn't get %v, Git object sha: %v", targetPath, err) 77 | return false, "", err 78 | } 79 | 80 | if sourcePathGitObjectSha == targetPathGitObjectSha { 81 | ghPrClientDetails.PrLogger.Debugf("%s(%s) vs %s(%s) git object SHA matched.", sourcePath, sourcePathGitObjectSha, targetPath, targetPathGitObjectSha) 82 | return false, "", nil 83 | } else { 84 | ghPrClientDetails.PrLogger.Debugf("%s(%s) vs %s(%s) git object SHA didn't match! Will do a full tree compare", sourcePath, sourcePathGitObjectSha, targetPath, targetPathGitObjectSha) 85 | sourceFilesSHAs := make(map[string]string) 86 | targetFilesSHAs := make(map[string]string) 87 | hasDiff := false 88 | 89 | generateFlatMapfromFileTree(&ghPrClientDetails, &sourcePath, &sourcePath, &defaultBranch, sourceFilesSHAs) 90 | generateFlatMapfromFileTree(&ghPrClientDetails, &targetPath, &targetPath, &defaultBranch, targetFilesSHAs) 91 | // ghPrClientDetails.PrLogger.Infoln(sourceFilesSHAs) 92 | hasDiff, diffOutput, err := generateDiffOutput(ghPrClientDetails, defaultBranch, sourceFilesSHAs, targetFilesSHAs, sourcePath, targetPath) 93 | 94 | return hasDiff, diffOutput, err 95 | } 96 | } 97 | 98 | func generateFlatMapfromFileTree(ghPrClientDetails *GhPrClientDetails, workingPath *string, rootPath *string, branch *string, listOfFiles map[string]string) { 99 | getContentOpts := &github.RepositoryContentGetOptions{ 100 | Ref: *branch, 101 | } 102 | _, directoryContent, resp, _ := ghPrClientDetails.GhClientPair.v3Client.Repositories.GetContents(ghPrClientDetails.Ctx, ghPrClientDetails.Owner, ghPrClientDetails.Repo, *workingPath, getContentOpts) 103 | prom.InstrumentGhCall(resp) 104 | for _, elementInDir := range directoryContent { 105 | if *elementInDir.Type == "file" { 106 | relativeName := strings.TrimPrefix(*elementInDir.Path, *rootPath+"/") 107 | listOfFiles[relativeName] = *elementInDir.SHA 108 | } else if *elementInDir.Type == "dir" { 109 | generateFlatMapfromFileTree(ghPrClientDetails, elementInDir.Path, rootPath, branch, listOfFiles) 110 | } else { 111 | ghPrClientDetails.PrLogger.Infof("Ignoring type %s for path %s", *elementInDir.Type, *elementInDir.Path) 112 | } 113 | } 114 | } 115 | -------------------------------------------------------------------------------- /internal/pkg/githubapi/drift_detection_test.go: -------------------------------------------------------------------------------- 1 | package githubapi 2 | 3 | import ( 4 | "context" 5 | "testing" 6 | 7 | "github.com/go-test/deep" 8 | "github.com/google/go-github/v62/github" 9 | "github.com/hexops/gotextdiff" 10 | "github.com/hexops/gotextdiff/myers" 11 | "github.com/hexops/gotextdiff/span" 12 | "github.com/migueleliasweb/go-github-mock/src/mock" 13 | log "github.com/sirupsen/logrus" 14 | ) 15 | 16 | func TestGenerateFlatMapfromFileTree(t *testing.T) { 17 | t.Parallel() 18 | ctx := context.Background() 19 | filesSHAs := make(map[string]string) 20 | 21 | mockedHTTPClient := mock.NewMockedHTTPClient( 22 | mock.WithRequestMatch( 23 | mock.GetReposContentsByOwnerByRepoByPath, 24 | []github.RepositoryContent{ 25 | { 26 | Type: github.String("file"), 27 | Path: github.String("some/path/file1"), 28 | SHA: github.String("fffff1"), 29 | }, 30 | { 31 | Type: github.String("file"), 32 | Path: github.String("some/path/file2"), 33 | SHA: github.String("fffff2"), 34 | }, 35 | { 36 | Type: github.String("dir"), 37 | Path: github.String("some/path/dir1"), 38 | SHA: github.String("fffff3"), 39 | }, 40 | }, 41 | []github.RepositoryContent{ 42 | { 43 | Type: github.String("file"), 44 | Path: github.String("some/path/dir1/file4"), 45 | SHA: github.String("fffff4"), 46 | }, 47 | { 48 | Type: github.String("dir"), 49 | Path: github.String("some/path/dir1/nested_dir1/"), 50 | SHA: github.String("fffff3"), 51 | }, 52 | }, 53 | []github.RepositoryContent{ 54 | { 55 | Type: github.String("file"), 56 | Path: github.String("some/path/dir1/nested_dir1/file5"), 57 | SHA: github.String("fffff5"), 58 | }, 59 | }, 60 | ), 61 | ) 62 | ghClientPair := GhClientPair{v3Client: github.NewClient(mockedHTTPClient)} 63 | 64 | ghPrClientDetails := GhPrClientDetails{ 65 | Ctx: ctx, 66 | GhClientPair: &ghClientPair, 67 | Owner: "AnOwner", 68 | Repo: "Arepo", 69 | PrNumber: 120, 70 | Ref: "Abranch", 71 | PrLogger: log.WithFields(log.Fields{ 72 | "repo": "AnOwner/Arepo", 73 | "prNumber": 120, 74 | }), 75 | } 76 | expectedFilesSHAs := map[string]string{ 77 | "file1": "fffff1", 78 | "file2": "fffff2", 79 | "dir1/file4": "fffff4", 80 | "dir1/nested_dir1/file5": "fffff5", 81 | } 82 | 83 | defaultBranch := "main" 84 | targetPath := "some/path" 85 | generateFlatMapfromFileTree(&ghPrClientDetails, &targetPath, &targetPath, &defaultBranch, filesSHAs) 86 | if diff := deep.Equal(expectedFilesSHAs, filesSHAs); diff != nil { 87 | for _, l := range diff { 88 | t.Error(l) 89 | } 90 | } 91 | } 92 | 93 | func TestGenerateDiffOutputDiffFileContent(t *testing.T) { 94 | t.Parallel() 95 | ctx := context.Background() 96 | 97 | mockedHTTPClient := mock.NewMockedHTTPClient( 98 | mock.WithRequestMatch( 99 | mock.GetReposContentsByOwnerByRepoByPath, 100 | github.RepositoryContent{ 101 | Content: github.String("File A content\n"), 102 | }, 103 | github.RepositoryContent{ 104 | Content: github.String("File B content\n"), 105 | }, 106 | ), 107 | ) 108 | 109 | ghClientPair := GhClientPair{v3Client: github.NewClient(mockedHTTPClient)} 110 | 111 | ghPrClientDetails := GhPrClientDetails{ 112 | Ctx: ctx, 113 | GhClientPair: &ghClientPair, 114 | Owner: "AnOwner", 115 | Repo: "Arepo", 116 | PrNumber: 120, 117 | Ref: "Abranch", 118 | PrLogger: log.WithFields(log.Fields{ 119 | "repo": "AnOwner/Arepo", 120 | "prNumber": 120, 121 | }), 122 | } 123 | 124 | // TODO move this to file 125 | expectedDiffOutput := "\n```" + `diff 126 | --- source-path/file-1.text 127 | +++ target-path/file-1.text 128 | @@ -1 +1 @@ 129 | -File A content 130 | +File B content` + "\n\n```\n\n" + `### Blame Links: 131 | [source-path/file-1.text](https://github.com/AnOwner/Arepo/blame/HEAD/source-path/file-1.text) 132 | ` 133 | 134 | var sourceFilesSHAs map[string]string 135 | var targetFilesSHAs map[string]string 136 | 137 | sourceFilesSHAs = make(map[string]string) 138 | targetFilesSHAs = make(map[string]string) 139 | 140 | sourceFilesSHAs["file-1.text"] = "000001" 141 | targetFilesSHAs["file-1.text"] = "000002" 142 | 143 | isDiff, diffOutput, err := generateDiffOutput(ghPrClientDetails, "main", sourceFilesSHAs, targetFilesSHAs, "source-path", "target-path") 144 | if err != nil { 145 | t.Fatalf("generating diff output failed: err=%s", err) 146 | } 147 | 148 | if diffOutput != expectedDiffOutput { 149 | edits := myers.ComputeEdits(span.URIFromPath("diff.text"), diffOutput, expectedDiffOutput) 150 | t.Fatalf("Diff Output is wrong:\n%s", gotextdiff.ToUnified("computed", "expected", diffOutput, edits)) 151 | } 152 | if !isDiff { 153 | t.Fatal("Did not detect diff in in files with different SHAs/content") 154 | } 155 | } 156 | 157 | func TestGenerateDiffOutputIdenticalFiles(t *testing.T) { 158 | t.Parallel() 159 | ctx := context.Background() 160 | 161 | mockedHTTPClient := mock.NewMockedHTTPClient() 162 | 163 | ghClientPair := GhClientPair{v3Client: github.NewClient(mockedHTTPClient)} 164 | 165 | ghPrClientDetails := GhPrClientDetails{ 166 | Ctx: ctx, 167 | GhClientPair: &ghClientPair, 168 | Owner: "AnOwner", 169 | Repo: "Arepo", 170 | PrNumber: 120, 171 | Ref: "Abranch", 172 | PrLogger: log.WithFields(log.Fields{ 173 | "repo": "AnOwner/Arepo", 174 | "prNumber": 120, 175 | }), 176 | } 177 | 178 | var sourceFilesSHAs map[string]string 179 | var targetFilesSHAs map[string]string 180 | 181 | sourceFilesSHAs = make(map[string]string) 182 | targetFilesSHAs = make(map[string]string) 183 | 184 | sourceFilesSHAs["file-1.text"] = "000001" 185 | targetFilesSHAs["file-1.text"] = "000001" 186 | 187 | isDiff, _, err := generateDiffOutput(ghPrClientDetails, "main", sourceFilesSHAs, targetFilesSHAs, "source-path", "target-path") 188 | if err != nil { 189 | t.Fatalf("generating diff output failed: err=%s", err) 190 | } 191 | 192 | if isDiff { 193 | t.Error("Found a Diff in files with identical SHA/content") 194 | } 195 | } 196 | 197 | func TestGenerateDiffOutputMissingSourceFile(t *testing.T) { 198 | t.Parallel() 199 | ctx := context.Background() 200 | 201 | mockedHTTPClient := mock.NewMockedHTTPClient( 202 | mock.WithRequestMatch( 203 | mock.GetReposContentsByOwnerByRepoByPath, 204 | github.RepositoryContent{ 205 | Content: github.String("File A content\n"), 206 | }, 207 | ), 208 | ) 209 | 210 | ghClientPair := GhClientPair{v3Client: github.NewClient(mockedHTTPClient)} 211 | 212 | ghPrClientDetails := GhPrClientDetails{ 213 | Ctx: ctx, 214 | GhClientPair: &ghClientPair, 215 | Owner: "AnOwner", 216 | Repo: "Arepo", 217 | PrNumber: 120, 218 | Ref: "Abranch", 219 | PrLogger: log.WithFields(log.Fields{ 220 | "repo": "AnOwner/Arepo", 221 | "prNumber": 120, 222 | }), 223 | } 224 | 225 | expectedDiffOutput := "\n```" + `diff 226 | +++ target-path/file-1.text (missing from source dir source-path)` + "\n\n```\n" 227 | 228 | var sourceFilesSHAs map[string]string 229 | var targetFilesSHAs map[string]string 230 | 231 | sourceFilesSHAs = make(map[string]string) 232 | targetFilesSHAs = make(map[string]string) 233 | 234 | targetFilesSHAs["file-1.text"] = "000001" 235 | 236 | isDiff, diffOutput, err := generateDiffOutput(ghPrClientDetails, "main", sourceFilesSHAs, targetFilesSHAs, "source-path", "target-path") 237 | if err != nil { 238 | t.Fatalf("generating diff output failed: err=%s", err) 239 | } 240 | 241 | if diffOutput != expectedDiffOutput { 242 | t.Errorf("Diff Output is wrong:\n%s\n vs:\n%s\n", diffOutput, expectedDiffOutput) 243 | } 244 | if !isDiff { 245 | t.Errorf("Did not detect diff in in files with different SHAs/content, isDiff=%t", isDiff) 246 | } 247 | } 248 | 249 | func TestGenerateDiffOutputMissingTargetFile(t *testing.T) { 250 | t.Parallel() 251 | ctx := context.Background() 252 | 253 | mockedHTTPClient := mock.NewMockedHTTPClient( 254 | mock.WithRequestMatch( 255 | mock.GetReposContentsByOwnerByRepoByPath, 256 | github.RepositoryContent{ 257 | Content: github.String("File A content\n"), 258 | }, 259 | ), 260 | ) 261 | 262 | ghClientPair := GhClientPair{v3Client: github.NewClient(mockedHTTPClient)} 263 | 264 | ghPrClientDetails := GhPrClientDetails{ 265 | Ctx: ctx, 266 | GhClientPair: &ghClientPair, 267 | Owner: "AnOwner", 268 | Repo: "Arepo", 269 | PrNumber: 120, 270 | Ref: "Abranch", 271 | PrLogger: log.WithFields(log.Fields{ 272 | "repo": "AnOwner/Arepo", 273 | "prNumber": 120, 274 | }), 275 | } 276 | 277 | expectedDiffOutput := "\n```" + `diff 278 | --- source-path/file-1.text (missing from target dir target-path)` + "\n\n```\n" 279 | 280 | var sourceFilesSHAs map[string]string 281 | var targetFilesSHAs map[string]string 282 | 283 | sourceFilesSHAs = make(map[string]string) 284 | targetFilesSHAs = make(map[string]string) 285 | 286 | sourceFilesSHAs["file-1.text"] = "000001" 287 | 288 | isDiff, diffOutput, err := generateDiffOutput(ghPrClientDetails, "main", sourceFilesSHAs, targetFilesSHAs, "source-path", "target-path") 289 | if err != nil { 290 | t.Fatalf("generating diff output failed: err=%s", err) 291 | } 292 | 293 | if diffOutput != expectedDiffOutput { 294 | t.Errorf("Diff Output is wrong(computed):\n%s\n vs(expected):\n%s\n", diffOutput, expectedDiffOutput) 295 | } 296 | if !isDiff { 297 | t.Errorf("Did not detect diff in in files with different SHAs/content, isDiff=%t", isDiff) 298 | } 299 | } 300 | -------------------------------------------------------------------------------- /internal/pkg/githubapi/github_graphql.go: -------------------------------------------------------------------------------- 1 | package githubapi 2 | 3 | import ( 4 | "context" 5 | "strings" 6 | 7 | "github.com/shurcooL/githubv4" 8 | log "github.com/sirupsen/logrus" 9 | ) 10 | 11 | // go-github is my peffered way to interact with GitHub because of the better developer expirience(pre made types, easy API mocking). 12 | // But some functionality is not availalble in GH V3 rest API, like PR comment minimization, so here we are: 13 | func GetBotGhIdentity(githubGraphQlClient *githubv4.Client, ctx context.Context) (string, error) { 14 | var getBotGhIdentityQuery struct { 15 | Viewer struct { 16 | Login githubv4.String 17 | } 18 | } 19 | 20 | err := githubGraphQlClient.Query(ctx, &getBotGhIdentityQuery, nil) 21 | botIdentity := getBotGhIdentityQuery.Viewer.Login 22 | if err != nil { 23 | log.Errorf("Failed to fetch token owner name: err=%s\n", err) 24 | return "", err 25 | } 26 | return string(botIdentity), nil 27 | } 28 | 29 | func MimizeStalePrComments(ghPrClientDetails GhPrClientDetails, githubGraphQlClient *githubv4.Client, botIdentity string) error { 30 | var getCommentNodeIdsQuery struct { 31 | Repository struct { 32 | PullRequest struct { 33 | Title githubv4.String 34 | Comments struct { 35 | Edges []struct { 36 | Node struct { 37 | Id githubv4.ID 38 | IsMinimized githubv4.Boolean 39 | Body githubv4.String 40 | Author struct { 41 | Login githubv4.String 42 | } 43 | } 44 | } 45 | } `graphql:"comments(last: 100)"` 46 | } `graphql:"pullRequest(number: $prNumber )"` 47 | } `graphql:"repository(owner: $owner, name: $repo)"` 48 | } // Mimizing stale comment is not crutial so only taking the last 100 comments, should cover most cases. 49 | // Would be nice if I could filter based on Author and isMinized here, in the query, to get just the relevant ones, 50 | // but I don't think GH graphQL supports it, so for now I just filter in code, see conditioanl near the end of this function. 51 | 52 | getCommentNodeIdsParams := map[string]interface{}{ 53 | "owner": githubv4.String(ghPrClientDetails.Owner), 54 | "repo": githubv4.String(ghPrClientDetails.Repo), 55 | "prNumber": githubv4.Int(ghPrClientDetails.PrNumber), //nolint:gosec // G115: type mismatch between shurcooL/githubv4 and google/go-github. Number taken from latter for use in query using former. 56 | } 57 | 58 | var minimizeCommentMutation struct { 59 | MinimizeComment struct { 60 | ClientMutationId githubv4.ID 61 | MinimizedComment struct { 62 | IsMinimized githubv4.Boolean 63 | } 64 | } `graphql:"minimizeComment(input: $input)"` 65 | } 66 | 67 | err := githubGraphQlClient.Query(ghPrClientDetails.Ctx, &getCommentNodeIdsQuery, getCommentNodeIdsParams) 68 | if err != nil { 69 | ghPrClientDetails.PrLogger.Errorf("Failed to minimize stale comments: err=%s\n", err) 70 | } 71 | bi := githubv4.String(strings.TrimSuffix(botIdentity, "[bot]")) 72 | for _, prComment := range getCommentNodeIdsQuery.Repository.PullRequest.Comments.Edges { 73 | if !prComment.Node.IsMinimized && prComment.Node.Author.Login == bi { 74 | if strings.Contains(string(prComment.Node.Body), "") { 75 | ghPrClientDetails.PrLogger.Infof("Minimizing Comment %s", prComment.Node.Id) 76 | minimizeCommentInput := githubv4.MinimizeCommentInput{ 77 | SubjectID: prComment.Node.Id, 78 | Classifier: githubv4.ReportedContentClassifiers("OUTDATED"), 79 | ClientMutationID: &bi, 80 | } 81 | err := githubGraphQlClient.Mutate(ghPrClientDetails.Ctx, &minimizeCommentMutation, minimizeCommentInput, nil) 82 | // As far as I can tell minimizeComment Github's grpahQL method doesn't accept list do doing one call per comment 83 | if err != nil { 84 | ghPrClientDetails.PrLogger.Errorf("Failed to minimize comment ID %s\n err=%s", prComment.Node.Id, err) 85 | // Handle error. 86 | } 87 | } else { 88 | ghPrClientDetails.PrLogger.Debugln("Ignoring comment without identification tag") 89 | } 90 | } 91 | } 92 | 93 | return err 94 | } 95 | -------------------------------------------------------------------------------- /internal/pkg/githubapi/pr_metrics.go: -------------------------------------------------------------------------------- 1 | package githubapi 2 | 3 | import ( 4 | "context" 5 | "time" 6 | 7 | "github.com/google/go-github/v62/github" 8 | lru "github.com/hashicorp/golang-lru/v2" 9 | log "github.com/sirupsen/logrus" 10 | prom "github.com/wayfair-incubator/telefonistka/internal/pkg/prometheus" 11 | ) 12 | 13 | const ( 14 | timeToDefineStale = 20 * time.Minute 15 | metricRefreshTime = 60 * time.Second 16 | ) 17 | 18 | func MainGhMetricsLoop(mainGhClientCache *lru.Cache[string, GhClientPair]) { 19 | for t := range time.Tick(metricRefreshTime) { 20 | log.Debugf("Updating pr metrics at %v", t) 21 | getPrMetrics(mainGhClientCache) 22 | } 23 | } 24 | 25 | func getRepoPrMetrics(ctx context.Context, ghClient GhClientPair, repo *github.Repository) (pc prom.PrCounters, err error) { 26 | log.Debugf("Checking repo %s", repo.GetName()) 27 | ghOwner := repo.GetOwner().GetLogin() 28 | prListOpts := &github.PullRequestListOptions{ 29 | State: "open", 30 | } 31 | prs := []*github.PullRequest{} 32 | 33 | // paginate through PRs, there might be lots of them. 34 | for { 35 | perPagePrs, resp, err := ghClient.v3Client.PullRequests.List(ctx, ghOwner, repo.GetName(), prListOpts) 36 | _ = prom.InstrumentGhCall(resp) 37 | if err != nil { 38 | log.Errorf("error getting PRs for %s/%s: %v", ghOwner, repo.GetName(), err) 39 | } 40 | prs = append(prs, perPagePrs...) 41 | if resp.NextPage == 0 { 42 | break 43 | } 44 | prListOpts.Page = resp.NextPage 45 | } 46 | 47 | for _, pr := range prs { 48 | if DoesPrHasLabel(pr.Labels, "promotion") { 49 | pc.OpenPromotionPrs++ 50 | } 51 | 52 | log.Debugf("Checking PR %d", pr.GetNumber()) 53 | commitStatuses, resp, err := ghClient.v3Client.Repositories.GetCombinedStatus(ctx, ghOwner, repo.GetName(), pr.GetHead().GetSHA(), nil) 54 | _ = prom.InstrumentGhCall(resp) 55 | if err != nil { 56 | log.Errorf("error getting statuses for %s/%s/%d: %v", ghOwner, repo.GetName(), pr.GetNumber(), err) 57 | continue 58 | } 59 | if isPrStalePending(commitStatuses, timeToDefineStale) { 60 | pc.PrWithStaleChecks++ 61 | } 62 | } 63 | pc.OpenPrs = len(prs) 64 | 65 | return 66 | } 67 | 68 | // isPrStalePending checks if the a combinedStatus has a "telefonistka" context pending status that is older than timeToDefineStale and is in pending state 69 | func isPrStalePending(commitStatuses *github.CombinedStatus, timeToDefineStale time.Duration) bool { 70 | for _, status := range commitStatuses.Statuses { 71 | if *status.Context == "telefonistka" && 72 | *status.State == "pending" && 73 | status.UpdatedAt.GetTime().Before(time.Now().Add(timeToDefineStale*-1)) { 74 | log.Debugf("Adding status %s-%v-%s !!!", *status.Context, status.UpdatedAt.GetTime(), *status.State) 75 | return true 76 | } else { 77 | log.Debugf("Ignoring status %s-%v-%s", *status.Context, status.UpdatedAt.GetTime(), *status.State) 78 | } 79 | } 80 | 81 | return false 82 | } 83 | 84 | // getPrMetrics iterates through all clients , gets all repos and then all PRs and calculates metrics 85 | // getPrMetrics assumes Telefonsitka uses a GitHub App style of authentication as it uses the Apps.ListRepos call 86 | // When using personal access token authentication, Telefonistka is unaware of the "relevant" repos (at least it get a webhook from them). 87 | func getPrMetrics(mainGhClientCache *lru.Cache[string, GhClientPair]) { 88 | ctx, cancel := context.WithTimeout(context.Background(), 1*time.Minute) 89 | defer cancel() 90 | for _, ghOwner := range mainGhClientCache.Keys() { 91 | log.Debugf("Checking gh Owner %s", ghOwner) 92 | ghClient, _ := mainGhClientCache.Get(ghOwner) 93 | repos, resp, err := ghClient.v3Client.Apps.ListRepos(ctx, nil) 94 | _ = prom.InstrumentGhCall(resp) 95 | if err != nil { 96 | log.Errorf("error getting repos for %s: %v", ghOwner, err) 97 | continue 98 | } 99 | for _, repo := range repos.Repositories { 100 | pc, err := getRepoPrMetrics(ctx, ghClient, repo) 101 | if err != nil { 102 | log.Errorf("error getting repos for %s: %v", ghOwner, err) 103 | continue 104 | } 105 | prom.PublishPrMetrics(pc, repo.GetFullName()) 106 | } 107 | } 108 | } 109 | -------------------------------------------------------------------------------- /internal/pkg/githubapi/pr_metrics_test.go: -------------------------------------------------------------------------------- 1 | package githubapi 2 | 3 | import ( 4 | "testing" 5 | "time" 6 | 7 | "github.com/google/go-github/v62/github" 8 | ) 9 | 10 | func TestIsPrStalePending(t *testing.T) { 11 | t.Parallel() 12 | timeToDefineStale := 15 * time.Minute 13 | 14 | currentTime := time.Now() 15 | tests := map[string]struct { 16 | input github.CombinedStatus 17 | result bool 18 | }{ 19 | "All Success": { 20 | input: github.CombinedStatus{ 21 | Statuses: []*github.RepoStatus{ 22 | { 23 | State: github.String("success"), 24 | Context: github.String("telefonistka"), 25 | UpdatedAt: &github.Timestamp{ 26 | Time: currentTime.Add(-10 * time.Minute), 27 | }, 28 | }, 29 | { 30 | State: github.String("success"), 31 | Context: github.String("circleci"), 32 | UpdatedAt: &github.Timestamp{ 33 | Time: currentTime.Add(-10 * time.Minute), 34 | }, 35 | }, 36 | { 37 | State: github.String("success"), 38 | Context: github.String("foobar"), 39 | UpdatedAt: &github.Timestamp{ 40 | Time: currentTime.Add(-10 * time.Minute), 41 | }, 42 | }, 43 | }, 44 | }, 45 | result: false, 46 | }, 47 | "Pending but not stale": { 48 | input: github.CombinedStatus{ 49 | Statuses: []*github.RepoStatus{ 50 | { 51 | State: github.String("pending"), 52 | Context: github.String("telefonistka"), 53 | UpdatedAt: &github.Timestamp{ 54 | Time: currentTime.Add(-1 * time.Minute), 55 | }, 56 | }, 57 | }, 58 | }, 59 | result: false, 60 | }, 61 | 62 | "Pending and stale": { 63 | input: github.CombinedStatus{ 64 | Statuses: []*github.RepoStatus{ 65 | { 66 | State: github.String("pending"), 67 | Context: github.String("telefonistka"), 68 | UpdatedAt: &github.Timestamp{ 69 | Time: currentTime.Add(-20 * time.Minute), 70 | }, 71 | }, 72 | }, 73 | }, 74 | result: true, 75 | }, 76 | } 77 | 78 | for name, tc := range tests { 79 | name := name 80 | tc := tc 81 | t.Run(name, func(t *testing.T) { 82 | t.Parallel() 83 | result := isPrStalePending(&tc.input, timeToDefineStale) 84 | if result != tc.result { 85 | t.Errorf("(%s)Expected %v, got %v", name, tc.result, result) 86 | } 87 | }) 88 | } 89 | } 90 | -------------------------------------------------------------------------------- /internal/pkg/githubapi/promotion.go: -------------------------------------------------------------------------------- 1 | package githubapi 2 | 3 | import ( 4 | "fmt" 5 | "regexp" 6 | "sort" 7 | "strings" 8 | 9 | "github.com/google/go-github/v62/github" 10 | log "github.com/sirupsen/logrus" 11 | cfg "github.com/wayfair-incubator/telefonistka/internal/pkg/configuration" 12 | prom "github.com/wayfair-incubator/telefonistka/internal/pkg/prometheus" 13 | yaml "gopkg.in/yaml.v2" 14 | ) 15 | 16 | type PromotionInstance struct { 17 | Metadata PromotionInstanceMetaData `deep:"-"` // Unit tests ignore Metadata currently 18 | ComputedSyncPaths map[string]string // key is target, value is source 19 | } 20 | 21 | type PromotionInstanceMetaData struct { 22 | SourcePath string 23 | TargetPaths []string 24 | TargetDescription string 25 | PerComponentSkippedTargetPaths map[string][]string // ComponentName is the key, 26 | ComponentNames []string 27 | AutoMerge bool 28 | } 29 | 30 | func containMatchingRegex(patterns []string, str string) bool { 31 | for _, pattern := range patterns { 32 | doesElementMatchPattern, err := regexp.MatchString(pattern, str) 33 | if err != nil { 34 | log.Errorf("failed to match regex %s vs %s\n%s", pattern, str, err) 35 | return false 36 | } 37 | if doesElementMatchPattern { 38 | return true 39 | } 40 | } 41 | return false 42 | } 43 | 44 | func contains(s []string, str string) bool { 45 | for _, v := range s { 46 | if v == str { 47 | return true 48 | } 49 | } 50 | return false 51 | } 52 | 53 | func DetectDrift(ghPrClientDetails GhPrClientDetails) error { 54 | ghPrClientDetails.PrLogger.Debugln("Checking for Drift") 55 | if ghPrClientDetails.Ctx.Err() != nil { 56 | return ghPrClientDetails.Ctx.Err() 57 | } 58 | diffOutputMap := make(map[string]string) 59 | defaultBranch, _ := ghPrClientDetails.GetDefaultBranch() 60 | config, err := GetInRepoConfig(ghPrClientDetails, defaultBranch) 61 | if err != nil { 62 | _ = ghPrClientDetails.CommentOnPr(fmt.Sprintf("Failed to get configuration\n```\n%s\n```\n", err)) 63 | return err 64 | } 65 | 66 | promotions, _ := GeneratePromotionPlan(ghPrClientDetails, config, ghPrClientDetails.Ref) 67 | 68 | for _, promotion := range promotions { 69 | ghPrClientDetails.PrLogger.Debugf("Checking drift for %s", promotion.Metadata.SourcePath) 70 | for trgt, src := range promotion.ComputedSyncPaths { 71 | hasDiff, diffOutput, _ := CompareRepoDirectories(ghPrClientDetails, src, trgt, defaultBranch) 72 | if hasDiff { 73 | mapKey := fmt.Sprintf("`%s` ↔️ `%s`", src, trgt) 74 | diffOutputMap[mapKey] = diffOutput 75 | ghPrClientDetails.PrLogger.Debugf("Found diff @ %s", mapKey) 76 | } 77 | } 78 | } 79 | if len(diffOutputMap) != 0 { 80 | templateOutput, err := executeTemplate("driftMsg", defaultTemplatesFullPath("drift-pr-comment.gotmpl"), diffOutputMap) 81 | if err != nil { 82 | return err 83 | } 84 | 85 | err = commentPR(ghPrClientDetails, templateOutput) 86 | if err != nil { 87 | return err 88 | } 89 | } else { 90 | ghPrClientDetails.PrLogger.Infof("No drift found") 91 | } 92 | 93 | return nil 94 | } 95 | 96 | func getComponentConfig(ghPrClientDetails GhPrClientDetails, componentPath string, branch string) (*cfg.ComponentConfig, error) { 97 | componentConfig := &cfg.ComponentConfig{} 98 | rGetContentOps := &github.RepositoryContentGetOptions{Ref: branch} 99 | componentConfigFileContent, _, resp, err := ghPrClientDetails.GhClientPair.v3Client.Repositories.GetContents(ghPrClientDetails.Ctx, ghPrClientDetails.Owner, ghPrClientDetails.Repo, componentPath+"/telefonistka.yaml", rGetContentOps) 100 | prom.InstrumentGhCall(resp) 101 | if (err != nil) && (resp.StatusCode != 404) { // The file is optional 102 | ghPrClientDetails.PrLogger.Errorf("could not get file list from GH API: err=%s\nresponse=%v", err, resp) 103 | return nil, err 104 | } else if resp.StatusCode == 404 { 105 | ghPrClientDetails.PrLogger.Debugf("No in-component config in %s", componentPath) 106 | return &cfg.ComponentConfig{}, nil 107 | } 108 | componentConfigFileContentString, _ := componentConfigFileContent.GetContent() 109 | err = yaml.Unmarshal([]byte(componentConfigFileContentString), componentConfig) 110 | if err != nil { 111 | ghPrClientDetails.PrLogger.Errorf("Failed to parse configuration: err=%s\n", err) // TODO comment this error to PR 112 | return nil, err 113 | } 114 | return componentConfig, nil 115 | } 116 | 117 | // This function generates a list of "components" that where changed in the PR and are relevant for promotion) 118 | func generateListOfRelevantComponents(ghPrClientDetails GhPrClientDetails, config *cfg.Config) (relevantComponents map[relevantComponent]struct{}, err error) { 119 | relevantComponents = make(map[relevantComponent]struct{}) 120 | 121 | // Get the list of files in the PR, with pagination 122 | opts := &github.ListOptions{} 123 | prFiles := []*github.CommitFile{} 124 | 125 | for { 126 | perPagePrFiles, resp, err := ghPrClientDetails.GhClientPair.v3Client.PullRequests.ListFiles(ghPrClientDetails.Ctx, ghPrClientDetails.Owner, ghPrClientDetails.Repo, ghPrClientDetails.PrNumber, opts) 127 | prom.InstrumentGhCall(resp) 128 | if err != nil { 129 | ghPrClientDetails.PrLogger.Errorf("could not get file list from GH API: err=%s\nstatus code=%v", err, resp.Response.Status) 130 | return nil, err 131 | } 132 | prFiles = append(prFiles, perPagePrFiles...) 133 | if resp.NextPage == 0 { 134 | break 135 | } 136 | opts.Page = resp.NextPage 137 | } 138 | 139 | for _, changedFile := range prFiles { 140 | for _, promotionPathConfig := range config.PromotionPaths { 141 | if match, _ := regexp.MatchString("^"+promotionPathConfig.SourcePath+".*", *changedFile.Filename); match { 142 | // "components" here are the sub directories of the SourcePath 143 | // but with promotionPathConfig.ComponentPathExtraDepth we can grab multiple levels of subdirectories, 144 | // to support cases where components are nested deeper(e.g. [SourcePath]/owningTeam/namespace/component1) 145 | componentPathRegexSubSstrings := []string{} 146 | for i := 0; i <= promotionPathConfig.ComponentPathExtraDepth; i++ { 147 | componentPathRegexSubSstrings = append(componentPathRegexSubSstrings, "[^/]*") 148 | } 149 | componentPathRegexSubString := strings.Join(componentPathRegexSubSstrings, "/") 150 | getComponentRegexString := regexp.MustCompile("^" + promotionPathConfig.SourcePath + "(" + componentPathRegexSubString + ")/.*") 151 | componentName := getComponentRegexString.ReplaceAllString(*changedFile.Filename, "${1}") 152 | 153 | getSourcePathRegexString := regexp.MustCompile("^(" + promotionPathConfig.SourcePath + ")" + componentName + "/.*") 154 | compiledSourcePath := getSourcePathRegexString.ReplaceAllString(*changedFile.Filename, "${1}") 155 | relevantComponentsElement := relevantComponent{ 156 | SourcePath: compiledSourcePath, 157 | ComponentName: componentName, 158 | AutoMerge: promotionPathConfig.Conditions.AutoMerge, 159 | } 160 | relevantComponents[relevantComponentsElement] = struct{}{} 161 | break // a file can only be a single "source dir" 162 | } 163 | } 164 | } 165 | return relevantComponents, nil 166 | } 167 | 168 | type relevantComponent struct { 169 | SourcePath string 170 | ComponentName string 171 | AutoMerge bool 172 | } 173 | 174 | func generateListOfChangedComponentPaths(ghPrClientDetails GhPrClientDetails, config *cfg.Config) (changedComponentPaths []string, err error) { 175 | // If the PR has a list of promoted paths in the PR Telefonistika metadata(=is a promotion PR), we use that 176 | if len(ghPrClientDetails.PrMetadata.PromotedPaths) > 0 { 177 | changedComponentPaths = ghPrClientDetails.PrMetadata.PromotedPaths 178 | return changedComponentPaths, nil 179 | } 180 | 181 | // If not we will use in-repo config to generate it, and turns the map with struct keys into a list of strings 182 | relevantComponents, err := generateListOfRelevantComponents(ghPrClientDetails, config) 183 | if err != nil { 184 | return nil, err 185 | } 186 | for component := range relevantComponents { 187 | changedComponentPaths = append(changedComponentPaths, component.SourcePath+component.ComponentName) 188 | } 189 | return changedComponentPaths, nil 190 | } 191 | 192 | // This function generates a promotion plan based on the list of relevant components that where "touched" and the in-repo telefonitka configuration 193 | func generatePlanBasedOnChangeddComponent(ghPrClientDetails GhPrClientDetails, config *cfg.Config, relevantComponents map[relevantComponent]struct{}, configBranch string) (promotions map[string]PromotionInstance, err error) { 194 | promotions = make(map[string]PromotionInstance) 195 | for componentToPromote := range relevantComponents { 196 | componentConfig, err := getComponentConfig(ghPrClientDetails, componentToPromote.SourcePath+componentToPromote.ComponentName, configBranch) 197 | if err != nil { 198 | ghPrClientDetails.PrLogger.Errorf("Failed to get in component configuration, err=%s\nskipping %s", err, componentToPromote.SourcePath+componentToPromote.ComponentName) 199 | } 200 | 201 | for _, configPromotionPath := range config.PromotionPaths { 202 | if match, _ := regexp.MatchString(configPromotionPath.SourcePath, componentToPromote.SourcePath); match { 203 | // This section checks if a PromotionPath has a condition and skips it if needed 204 | if configPromotionPath.Conditions.PrHasLabels != nil { 205 | thisPrHasTheRightLabel := false 206 | for _, l := range ghPrClientDetails.Labels { 207 | if contains(configPromotionPath.Conditions.PrHasLabels, *l.Name) { 208 | thisPrHasTheRightLabel = true 209 | break 210 | } 211 | } 212 | if !thisPrHasTheRightLabel { 213 | continue 214 | } 215 | } 216 | 217 | for _, ppr := range configPromotionPath.PromotionPrs { 218 | sort.Strings(ppr.TargetPaths) 219 | 220 | mapKey := configPromotionPath.SourcePath + ">" + strings.Join(ppr.TargetPaths, "|") // This key is used to aggregate the PR based on source and target combination 221 | if entry, ok := promotions[mapKey]; !ok { 222 | ghPrClientDetails.PrLogger.Debugf("Adding key %s", mapKey) 223 | if ppr.TargetDescription == "" { 224 | ppr.TargetDescription = strings.Join(ppr.TargetPaths, " ") 225 | } 226 | promotions[mapKey] = PromotionInstance{ 227 | Metadata: PromotionInstanceMetaData{ 228 | TargetPaths: ppr.TargetPaths, 229 | TargetDescription: ppr.TargetDescription, 230 | SourcePath: componentToPromote.SourcePath, 231 | ComponentNames: []string{componentToPromote.ComponentName}, 232 | PerComponentSkippedTargetPaths: map[string][]string{}, 233 | AutoMerge: componentToPromote.AutoMerge, 234 | }, 235 | ComputedSyncPaths: map[string]string{}, 236 | } 237 | } else if !contains(entry.Metadata.ComponentNames, componentToPromote.ComponentName) { 238 | entry.Metadata.ComponentNames = append(entry.Metadata.ComponentNames, componentToPromote.ComponentName) 239 | promotions[mapKey] = entry 240 | } 241 | 242 | for _, indevidualPath := range ppr.TargetPaths { 243 | if componentConfig != nil { 244 | // BlockList supersedes Allowlist, if something matched there the entry is ignored regardless of allowlist 245 | if componentConfig.PromotionTargetBlockList != nil { 246 | if containMatchingRegex(componentConfig.PromotionTargetBlockList, indevidualPath) { 247 | promotions[mapKey].Metadata.PerComponentSkippedTargetPaths[componentToPromote.ComponentName] = append(promotions[mapKey].Metadata.PerComponentSkippedTargetPaths[componentToPromote.ComponentName], indevidualPath) 248 | continue 249 | } 250 | } 251 | if componentConfig.PromotionTargetAllowList != nil { 252 | if !containMatchingRegex(componentConfig.PromotionTargetAllowList, indevidualPath) { 253 | promotions[mapKey].Metadata.PerComponentSkippedTargetPaths[componentToPromote.ComponentName] = append(promotions[mapKey].Metadata.PerComponentSkippedTargetPaths[componentToPromote.ComponentName], indevidualPath) 254 | continue 255 | } 256 | } 257 | } 258 | promotions[mapKey].ComputedSyncPaths[indevidualPath+componentToPromote.ComponentName] = componentToPromote.SourcePath + componentToPromote.ComponentName 259 | } 260 | } 261 | break 262 | } 263 | } 264 | } 265 | return promotions, nil 266 | } 267 | 268 | func GeneratePromotionPlan(ghPrClientDetails GhPrClientDetails, config *cfg.Config, configBranch string) (map[string]PromotionInstance, error) { 269 | // TODO refactor tests to use the two functions below instead of this one 270 | relevantComponents, err := generateListOfRelevantComponents(ghPrClientDetails, config) 271 | if err != nil { 272 | return nil, err 273 | } 274 | promotions, err := generatePlanBasedOnChangeddComponent(ghPrClientDetails, config, relevantComponents, configBranch) 275 | return promotions, err 276 | } 277 | -------------------------------------------------------------------------------- /internal/pkg/githubapi/testdata/custom_commit_status_invalid_template.gotmpl: -------------------------------------------------------------------------------- 1 | https://custom-url.com?time={{.InvalidField}} 2 | -------------------------------------------------------------------------------- /internal/pkg/githubapi/testdata/custom_commit_status_valid_template.gotmpl: -------------------------------------------------------------------------------- 1 | {{ $calculated_time := .CommitTime.Add -600000000000 }}https://custom-url.com?time={{.CommitTime.UnixMilli}}&calculated_time={{$calculated_time.UnixMilli}} 2 | -------------------------------------------------------------------------------- /internal/pkg/githubapi/testdata/diff_comment_data_test.json: -------------------------------------------------------------------------------- 1 | 2 | { 3 | "DiffOfChangedComponents": [ 4 | { 5 | "ComponentPath": "clusters/playground/aws/eu-central-1/v1/special-delivery/ssllab-test/ssllab-test", 6 | "ArgoCdAppName": "temp-ssllab-test-plg-aws-eu-central1-v1", 7 | "ArgoCdAppURL": "https://argocd-lab.example.com/applications/temp-ssllab-test-plg-aws-eu-central1-v1", 8 | "DiffElements": [ 9 | { 10 | "ObjectGroup": "", 11 | "ObjectName": "ssllabs-exporter", 12 | "ObjectKind": "Service", 13 | "ObjectNamespace": "", 14 | "Diff": " (*unstructured.Unstructured)(\n- \tnil,\n+ \t\u0026{\n+ \t\tObject: map[string]any{\n+ \t\t\t\"apiVersion\": string(\"v1\"),\n+ \t\t\t\"kind\": string(\"Service\"),\n+ \t\t\t\"metadata\": map[string]any{\"labels\": map[string]any{...}, \"name\": string(\"ssllabs-exporter\")},\n+ \t\t\t\"spec\": map[string]any{\n+ \t\t\t\t\"ports\": []any{...},\n+ \t\t\t\t\"selector\": map[string]any{...},\n+ \t\t\t\t\"type\": string(\"ClusterIP\"),\n+ \t\t\t},\n+ \t\t},\n+ \t},\n )\n" 15 | }, 16 | { 17 | "ObjectGroup": "apps", 18 | "ObjectName": "ssllabs-exporter", 19 | "ObjectKind": "Deployment", 20 | "ObjectNamespace": "", 21 | "Diff": " (*unstructured.Unstructured)(\n- \tnil,\n+ \t\u0026{\n+ \t\tObject: map[string]any{\n+ \t\t\t\"apiVersion\": string(\"apps/v1\"),\n+ \t\t\t\"kind\": string(\"Deployment\"),\n+ \t\t\t\"metadata\": map[string]any{\"labels\": map[string]any{...}, \"name\": string(\"ssllabs-exporter\")},\n+ \t\t\t\"spec\": map[string]any{\n+ \t\t\t\t\"replicas\": int64(2),\n+ \t\t\t\t\"selector\": map[string]any{...},\n+ \t\t\t\t\"template\": map[string]any{...},\n+ \t\t\t},\n+ \t\t},\n+ \t},\n )\n" 22 | } 23 | ], 24 | "HasDiff": true, 25 | "DiffError": null, 26 | "AppWasTemporarilyCreated": false 27 | }, 28 | { 29 | "ComponentPath": "clusters/playground/aws/eu-central-1/v2/special-delivery/ssllab-test/ssllab-test", 30 | "ArgoCdAppName": "temp-ssllab-test-plg-aws-eu-central1-v2", 31 | "ArgoCdAppURL": "https://argocd-lab.example.com/applications/temp-ssllab-test-plg-aws-eu-central1-v1", 32 | "DiffElements": [ 33 | { 34 | "ObjectGroup": "", 35 | "ObjectName": "ssllabs-exporter", 36 | "ObjectKind": "Service", 37 | "ObjectNamespace": "", 38 | "Diff": " (*unstructured.Unstructured)(\n- \tnil,\n+ \t\u0026{\n+ \t\tObject: map[string]any{\n+ \t\t\t\"apiVersion\": string(\"v1\"),\n+ \t\t\t\"kind\": string(\"Service\"),\n+ \t\t\t\"metadata\": map[string]any{\"labels\": map[string]any{...}, \"name\": string(\"ssllabs-exporter\")},\n+ \t\t\t\"spec\": map[string]any{\n+ \t\t\t\t\"ports\": []any{...},\n+ \t\t\t\t\"selector\": map[string]any{...},\n+ \t\t\t\t\"type\": string(\"ClusterIP\"),\n+ \t\t\t},\n+ \t\t},\n+ \t},\n )\n" 39 | }, 40 | { 41 | "ObjectGroup": "apps", 42 | "ObjectName": "ssllabs-exporter", 43 | "ObjectKind": "Deployment", 44 | "ObjectNamespace": "", 45 | "Diff": " (*unstructured.Unstructured)(\n- \tnil,\n+ \t\u0026{\n+ \t\tObject: map[string]any{\n+ \t\t\t\"apiVersion\": string(\"apps/v1\"),\n+ \t\t\t\"kind\": string(\"Deployment\"),\n+ \t\t\t\"metadata\": map[string]any{\"labels\": map[string]any{...}, \"name\": string(\"ssllabs-exporter\")},\n+ \t\t\t\"spec\": map[string]any{\n+ \t\t\t\t\"replicas\": int64(2),\n+ \t\t\t\t\"selector\": map[string]any{...},\n+ \t\t\t\t\"template\": map[string]any{...},\n+ \t\t\t},\n+ \t\t},\n+ \t},\n )\n" 46 | } 47 | ], 48 | "HasDiff": true, 49 | "DiffError": null, 50 | "AppWasTemporarilyCreated": false 51 | }, 52 | { 53 | "ComponentPath": "clusters/playground/aws/eu-central-1/v3/special-delivery/ssllab-test/ssllab-test", 54 | "ArgoCdAppName": "temp-ssllab-test-plg-aws-eu-central1-v3", 55 | "ArgoCdAppURL": "https://argocd-lab.example.com/applications/temp-ssllab-test-plg-aws-eu-central1-v1", 56 | "DiffElements": [ 57 | { 58 | "ObjectGroup": "", 59 | "ObjectName": "ssllabs-exporter", 60 | "ObjectKind": "Service", 61 | "ObjectNamespace": "", 62 | "Diff": " (*unstructured.Unstructured)(\n- \tnil,\n+ \t\u0026{\n+ \t\tObject: map[string]any{\n+ \t\t\t\"apiVersion\": string(\"v1\"),\n+ \t\t\t\"kind\": string(\"Service\"),\n+ \t\t\t\"metadata\": map[string]any{\"labels\": map[string]any{...}, \"name\": string(\"ssllabs-exporter\")},\n+ \t\t\t\"spec\": map[string]any{\n+ \t\t\t\t\"ports\": []any{...},\n+ \t\t\t\t\"selector\": map[string]any{...},\n+ \t\t\t\t\"type\": string(\"ClusterIP\"),\n+ \t\t\t},\n+ \t\t},\n+ \t},\n )\n" 63 | }, 64 | { 65 | "ObjectGroup": "apps", 66 | "ObjectName": "ssllabs-exporter", 67 | "ObjectKind": "Deployment", 68 | "ObjectNamespace": "", 69 | "Diff": " (*unstructured.Unstructured)(\n- \tnil,\n+ \t\u0026{\n+ \t\tObject: map[string]any{\n+ \t\t\t\"apiVersion\": string(\"apps/v1\"),\n+ \t\t\t\"kind\": string(\"Deployment\"),\n+ \t\t\t\"metadata\": map[string]any{\"labels\": map[string]any{...}, \"name\": string(\"ssllabs-exporter\")},\n+ \t\t\t\"spec\": map[string]any{\n+ \t\t\t\t\"replicas\": int64(2),\n+ \t\t\t\t\"selector\": map[string]any{...},\n+ \t\t\t\t\"template\": map[string]any{...},\n+ \t\t\t},\n+ \t\t},\n+ \t},\n )\n" 70 | } 71 | ], 72 | "HasDiff": true, 73 | "DiffError": null, 74 | "AppWasTemporarilyCreated": false 75 | } 76 | ], 77 | "DisplaySyncBranchCheckBox": false, 78 | "BranchName": "promotions/284-simulate-error-5c159151017f", 79 | "Header": "" 80 | } 81 | -------------------------------------------------------------------------------- /internal/pkg/githubapi/testdata/pr_body.golden.md: -------------------------------------------------------------------------------- 1 | ↘️ #1 `sourcePath1` ➡️ 2 |     `targetPath1` 3 |     `targetPath2` 4 |     ↘️ #2 `sourcePath2` ➡️ 5 |         `targetPath4` 6 |         ↘️ #3 `sourcePath3` ➡️ 7 |             `targetPath5` 8 |             `targetPath6` 9 | -------------------------------------------------------------------------------- /internal/pkg/githubapi/testdata/pr_body_multi_component.golden.md: -------------------------------------------------------------------------------- 1 | ↘️ #1 `sourcePath1` ➡️ 2 |     `targetPath1` 3 |     ↘️ #2 `sourcePath2` ➡️ 4 |         `targetPath2` 5 | -------------------------------------------------------------------------------- /internal/pkg/githubapi/webhook_proxy.go: -------------------------------------------------------------------------------- 1 | package githubapi 2 | 3 | import ( 4 | "bytes" 5 | "context" 6 | "crypto/tls" 7 | "fmt" 8 | "io" 9 | "net/http" 10 | "regexp" 11 | "strings" 12 | 13 | "github.com/google/go-github/v62/github" 14 | log "github.com/sirupsen/logrus" 15 | "github.com/wayfair-incubator/telefonistka/internal/pkg/configuration" 16 | prom "github.com/wayfair-incubator/telefonistka/internal/pkg/prometheus" 17 | "golang.org/x/exp/maps" 18 | ) 19 | 20 | // @Title 21 | // @Description 22 | // @Author 23 | // @Update 24 | 25 | func generateListOfChangedFiles(eventPayload *github.PushEvent) []string { 26 | fileList := map[string]bool{} // using map for uniqueness 27 | 28 | for _, commit := range eventPayload.Commits { 29 | for _, file := range commit.Added { 30 | fileList[file] = true 31 | } 32 | for _, file := range commit.Modified { 33 | fileList[file] = true 34 | } 35 | for _, file := range commit.Removed { 36 | fileList[file] = true 37 | } 38 | } 39 | 40 | return maps.Keys(fileList) 41 | } 42 | 43 | func generateListOfEndpoints(listOfChangedFiles []string, config *configuration.Config) []string { 44 | endpoints := map[string]bool{} // using map for uniqueness 45 | for _, file := range listOfChangedFiles { 46 | for _, regex := range config.WebhookEndpointRegexs { 47 | m := regexp.MustCompile(regex.Expression) 48 | 49 | if m.MatchString(file) { 50 | for _, replacement := range regex.Replacements { 51 | endpoints[m.ReplaceAllString(file, replacement)] = true 52 | } 53 | break 54 | } 55 | } 56 | } 57 | 58 | return maps.Keys(endpoints) 59 | } 60 | 61 | func proxyRequest(ctx context.Context, skipTLSVerify bool, originalHttpRequest *http.Request, body []byte, endpoint string, responses chan<- string) { 62 | tr := &http.Transport{} 63 | if skipTLSVerify { 64 | tr = &http.Transport{ 65 | TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, // #nosec G402 - letting the user decide if they want to skip TLS verification, for some in-cluster scenarios its a reasonable compromise 66 | } 67 | } 68 | client := &http.Client{Transport: tr} 69 | req, err := http.NewRequestWithContext(ctx, originalHttpRequest.Method, endpoint, bytes.NewBuffer(body)) 70 | if err != nil { 71 | log.Errorf("Error creating request to %s: %v", endpoint, err) 72 | responses <- fmt.Sprintf("Failed to create request to %s", endpoint) 73 | return 74 | } 75 | req.Header = originalHttpRequest.Header.Clone() 76 | // Because payload and headers are passed as-is, I'm hoping webhook signature validation will "just work" 77 | 78 | resp, err := client.Do(req) 79 | if err != nil { 80 | log.Errorf("Error proxying request to %s: %v", endpoint, err) 81 | responses <- fmt.Sprintf("Failed to proxy request to %s", endpoint) 82 | return 83 | } else { 84 | log.Debugf("Webhook successfully forwarded to %s", endpoint) 85 | } 86 | defer resp.Body.Close() 87 | 88 | _ = prom.InstrumentProxyUpstreamRequest(resp) 89 | 90 | respBody, err := io.ReadAll(resp.Body) 91 | 92 | if !strings.HasPrefix(resp.Status, "2") { 93 | log.Errorf("Got non 2XX HTTP status from %s: status=%s body=%v", endpoint, resp.Status, body) 94 | } 95 | 96 | if err != nil { 97 | log.Errorf("Error reading response body from %s: %v", endpoint, err) 98 | responses <- fmt.Sprintf("Failed to read response from %s", endpoint) 99 | return 100 | } 101 | 102 | responses <- string(respBody) 103 | } 104 | 105 | func handlePushEvent(ctx context.Context, eventPayload *github.PushEvent, httpRequest *http.Request, payload []byte, ghPrClientDetails GhPrClientDetails) { 106 | listOfChangedFiles := generateListOfChangedFiles(eventPayload) 107 | log.Debugf("Changed files in push event: %v", listOfChangedFiles) 108 | 109 | defaultBranch := eventPayload.Repo.DefaultBranch 110 | 111 | if *eventPayload.Ref == "refs/heads/"+*defaultBranch { 112 | // TODO this need to be cached with TTL + invalidate if configfile in listOfChangedFiles? 113 | // This is possible because these webhooks are defined as "best effort" for the designed use case: 114 | // Speeding up ArgoCD reconcile loops 115 | config, _ := GetInRepoConfig(ghPrClientDetails, *defaultBranch) 116 | endpoints := generateListOfEndpoints(listOfChangedFiles, config) 117 | 118 | // Create a channel to receive responses from the goroutines 119 | responses := make(chan string) 120 | 121 | // Use a buffered channel with the same size as the number of endpoints 122 | // to prevent goroutines from blocking in case of slow endpoints 123 | results := make(chan string, len(endpoints)) 124 | 125 | // Start a goroutine for each endpoint 126 | for _, endpoint := range endpoints { 127 | go proxyRequest(ctx, config.WhProxtSkipTLSVerifyUpstream, httpRequest, payload, endpoint, responses) 128 | } 129 | 130 | // Wait for all goroutines to finish and collect the responses 131 | for i := 0; i < len(endpoints); i++ { 132 | result := <-responses 133 | results <- result 134 | } 135 | 136 | close(responses) 137 | close(results) 138 | } 139 | } 140 | -------------------------------------------------------------------------------- /internal/pkg/githubapi/webhook_proxy_test.go: -------------------------------------------------------------------------------- 1 | package githubapi 2 | 3 | // @Title 4 | // @Description 5 | // @Author 6 | // @Update 7 | import ( 8 | "sort" 9 | "testing" 10 | 11 | "github.com/go-test/deep" 12 | "github.com/google/go-github/v62/github" 13 | cfg "github.com/wayfair-incubator/telefonistka/internal/pkg/configuration" 14 | ) 15 | 16 | func TestGenerateListOfEndpoints(t *testing.T) { 17 | t.Parallel() 18 | config := &cfg.Config{ 19 | WebhookEndpointRegexs: []cfg.WebhookEndpointRegex{ 20 | { 21 | Expression: `^workspace\/[^/]*\/.*`, 22 | Replacements: []string{ 23 | `https://blabla.com/webhook`, 24 | }, 25 | }, 26 | { 27 | Expression: `^clusters\/([^/]*)\/([^/]*)\/([^/]*)\/.*`, 28 | Replacements: []string{ 29 | `https://ingress-a-${1}-${2}-${3}.example.com/webhook`, 30 | `https://ingress-b-${1}-${2}-${3}.example.com/webhook`, 31 | }, 32 | }, 33 | }, 34 | } 35 | listOfFiles := []string{ 36 | "workspace/csi-verify/values/global.yaml", 37 | "clusters/sdeprod/dsm1/c1/csi-verify/values/global.yaml", 38 | } 39 | 40 | endpoints := generateListOfEndpoints(listOfFiles, config) 41 | expectedEndpoints := []string{ 42 | "https://blabla.com/webhook", 43 | "https://ingress-a-sdeprod-dsm1-c1.example.com/webhook", 44 | "https://ingress-b-sdeprod-dsm1-c1.example.com/webhook", 45 | } 46 | 47 | sort.Strings(endpoints) 48 | sort.Strings(expectedEndpoints) 49 | if diff := deep.Equal(endpoints, expectedEndpoints); diff != nil { 50 | t.Error(diff) 51 | } 52 | } 53 | 54 | func TestGenerateListOfChangedFiles(t *testing.T) { 55 | t.Parallel() 56 | eventPayload := &github.PushEvent{ 57 | Commits: []*github.HeadCommit{ 58 | { 59 | Added: []string{ 60 | "workspace/csi-verify/values/global-new.yaml", 61 | }, 62 | Removed: []string{ 63 | "workspace/csi-verify/values/global-old.yaml", 64 | }, 65 | SHA: github.String("000001"), 66 | }, 67 | { 68 | Modified: []string{ 69 | "clusters/sdeprod/dsm1/c1/csi-verify/values/global.yaml", 70 | }, 71 | SHA: github.String("000002"), 72 | }, 73 | }, 74 | } 75 | 76 | listOfFiles := generateListOfChangedFiles(eventPayload) 77 | expectedListOfFiles := []string{ 78 | "workspace/csi-verify/values/global-new.yaml", 79 | "workspace/csi-verify/values/global-old.yaml", 80 | "clusters/sdeprod/dsm1/c1/csi-verify/values/global.yaml", 81 | } 82 | 83 | sort.Strings(listOfFiles) 84 | sort.Strings(expectedListOfFiles) 85 | 86 | if diff := deep.Equal(listOfFiles, expectedListOfFiles); diff != nil { 87 | t.Error(diff) 88 | } 89 | } 90 | -------------------------------------------------------------------------------- /internal/pkg/mocks/.gitignore: -------------------------------------------------------------------------------- 1 | /argocd_application.go 2 | -------------------------------------------------------------------------------- /internal/pkg/mocks/mocks.go: -------------------------------------------------------------------------------- 1 | package mocks 2 | 3 | // This package contains generated mocks 4 | 5 | //go:generate go run github.com/golang/mock/mockgen@v1.6.0 -destination=argocd_application.go -package=mocks github.com/argoproj/argo-cd/v2/pkg/apiclient/application ApplicationServiceClient 6 | 7 | //go:generate go run github.com/golang/mock/mockgen@v1.6.0 -destination=argocd_settings.go -package=mocks github.com/argoproj/argo-cd/v2/pkg/apiclient/settings SettingsServiceClient 8 | 9 | //go:generate go run github.com/golang/mock/mockgen@v1.6.0 -destination=argocd_project.go -package=mocks github.com/argoproj/argo-cd/v2/pkg/apiclient/project ProjectServiceClient 10 | -------------------------------------------------------------------------------- /internal/pkg/prometheus/prometheus.go: -------------------------------------------------------------------------------- 1 | package prometheus 2 | 3 | import ( 4 | "net/http" 5 | "strconv" 6 | "strings" 7 | 8 | "github.com/google/go-github/v62/github" 9 | "github.com/prometheus/client_golang/prometheus" 10 | "github.com/prometheus/client_golang/prometheus/promauto" 11 | ) 12 | 13 | type PrCounters struct { 14 | OpenPrs int 15 | OpenPromotionPrs int 16 | PrWithStaleChecks int 17 | } 18 | 19 | var ( 20 | webhookHitsVec = promauto.NewCounterVec(prometheus.CounterOpts{ 21 | Name: "webhook_hits_total", 22 | Help: "The total number of validated webhook hits", 23 | Namespace: "telefonistka", 24 | Subsystem: "webhook_server", 25 | }, []string{"parsing"}) 26 | 27 | ghRateLimitCounter = promauto.NewGaugeVec(prometheus.GaugeOpts{ 28 | Name: "github_rest_api_client_rate_limit", 29 | Help: "The number of requests per hour the client is currently limited to", 30 | Namespace: "telefonistka", 31 | Subsystem: "github", 32 | }, []string{"repo_owner"}) 33 | 34 | ghRateRemainingCounter = promauto.NewGaugeVec(prometheus.GaugeOpts{ 35 | Name: "github_rest_api_client_rate_remaining", 36 | Help: "The number of remaining requests the client can make this hour", 37 | Namespace: "telefonistka", 38 | Subsystem: "github", 39 | }, []string{"repo_owner"}) 40 | 41 | githubOpsCountVec = promauto.NewCounterVec(prometheus.CounterOpts{ 42 | Name: "github_operations_total", 43 | Help: "The total number of Github API operations", 44 | Namespace: "telefonistka", 45 | Subsystem: "github", 46 | }, []string{"api_group", "api_path", "repo_slug", "status", "method"}) 47 | 48 | ghOpenPrsGauge = promauto.NewGaugeVec(prometheus.GaugeOpts{ 49 | Name: "open_prs", 50 | Help: "The total number of open PRs", 51 | Namespace: "telefonistka", 52 | Subsystem: "github", 53 | }, []string{"repo_slug"}) 54 | 55 | ghOpenPromotionPrsGauge = promauto.NewGaugeVec(prometheus.GaugeOpts{ 56 | Name: "open_promotion_prs", 57 | Help: "The total number of open PRs with promotion label", 58 | Namespace: "telefonistka", 59 | Subsystem: "github", 60 | }, []string{"repo_slug"}) 61 | 62 | ghOpenPrsWithPendingCheckGauge = promauto.NewGaugeVec(prometheus.GaugeOpts{ 63 | Name: "open_prs_with_pending_telefonistka_checks", 64 | Help: "The total number of open PRs with pending Telefonistka checks(excluding PRs with very recent commits)", 65 | Namespace: "telefonistka", 66 | Subsystem: "github", 67 | }, []string{"repo_slug"}) 68 | 69 | commitStatusUpdates = promauto.NewCounterVec(prometheus.CounterOpts{ 70 | Name: "commit_status_updates_total", 71 | Help: "The total number of commit status updates, and their status (success/pending/failure)", 72 | Namespace: "telefonistka", 73 | Subsystem: "github", 74 | }, []string{"repo_slug", "status"}) 75 | 76 | whUpstreamRequestsCountVec = promauto.NewCounterVec(prometheus.CounterOpts{ 77 | Name: "upstream_requests_total", 78 | Help: "The total number of requests forwarded upstream servers", 79 | Namespace: "telefonistka", 80 | Subsystem: "webhook_proxy", 81 | }, []string{"status", "method", "url"}) 82 | ) 83 | 84 | func IncCommitStatusUpdateCounter(repoSlug string, status string) { 85 | commitStatusUpdates.With(prometheus.Labels{ 86 | "repo_slug": repoSlug, 87 | "status": status, 88 | }).Inc() 89 | } 90 | 91 | func PublishPrMetrics(pc PrCounters, repoSlug string) { 92 | metricLables := prometheus.Labels{ 93 | "repo_slug": repoSlug, 94 | } 95 | ghOpenPrsGauge.With(metricLables).Set(float64(pc.OpenPrs)) 96 | ghOpenPromotionPrsGauge.With(metricLables).Set(float64(pc.OpenPromotionPrs)) 97 | ghOpenPrsWithPendingCheckGauge.With(metricLables).Set(float64(pc.PrWithStaleChecks)) 98 | } 99 | 100 | // This function instrument Webhook hits and parsing of their content 101 | func InstrumentWebhookHit(parsing_status string) { 102 | webhookHitsVec.With(prometheus.Labels{"parsing": parsing_status}).Inc() 103 | } 104 | 105 | // This function instrument API calls to GitHub API 106 | func InstrumentGhCall(resp *github.Response) prometheus.Labels { 107 | if resp == nil { 108 | return prometheus.Labels{} 109 | } 110 | requestPathSlice := strings.Split(resp.Request.URL.Path, "/") 111 | var relevantRequestPathSlice []string 112 | // GitHub enterprise API as an additional "api/v3" perfix 113 | if requestPathSlice[1] == "api" && requestPathSlice[2] == "v3" { 114 | relevantRequestPathSlice = requestPathSlice[3:] 115 | } else { 116 | relevantRequestPathSlice = requestPathSlice[1:] 117 | } 118 | var apiPath string 119 | var repoSlug string 120 | var repoOwner string 121 | 122 | if len(relevantRequestPathSlice) < 4 { 123 | apiPath = "" 124 | if len(relevantRequestPathSlice) < 3 { 125 | repoSlug = "" 126 | repoOwner = "" 127 | } else { 128 | repoSlug = strings.Join(relevantRequestPathSlice[1:3], "/") 129 | repoOwner = relevantRequestPathSlice[1] 130 | } 131 | } else { 132 | apiPath = relevantRequestPathSlice[3] 133 | repoSlug = strings.Join(relevantRequestPathSlice[1:3], "/") 134 | repoOwner = relevantRequestPathSlice[1] 135 | } 136 | 137 | labels := prometheus.Labels{ 138 | "api_group": relevantRequestPathSlice[0], 139 | "api_path": apiPath, 140 | "repo_slug": repoSlug, 141 | "method": resp.Request.Method, 142 | "status": strconv.Itoa(resp.Response.StatusCode), 143 | } 144 | 145 | rateLimitLables := prometheus.Labels{ 146 | "repo_owner": repoOwner, 147 | } 148 | ghRateLimitCounter.With(rateLimitLables).Set(float64(resp.Rate.Limit)) 149 | ghRateRemainingCounter.With(rateLimitLables).Set(float64(resp.Rate.Remaining)) 150 | 151 | githubOpsCountVec.With(labels).Inc() 152 | // resp.Request. 153 | 154 | return labels 155 | } 156 | 157 | // This function instrument upstream webhooks for the WH forwarding/multiplexing feature 158 | func InstrumentProxyUpstreamRequest(resp *http.Response) prometheus.Labels { 159 | if resp == nil { 160 | return prometheus.Labels{} 161 | } 162 | 163 | labels := prometheus.Labels{ 164 | "method": resp.Request.Method, 165 | "status": strconv.Itoa(resp.StatusCode), 166 | "url": resp.Request.URL.String(), 167 | } 168 | whUpstreamRequestsCountVec.With(labels).Inc() 169 | return labels 170 | } 171 | -------------------------------------------------------------------------------- /internal/pkg/prometheus/prometheus_test.go: -------------------------------------------------------------------------------- 1 | package prometheus 2 | 3 | import ( 4 | "net/http" 5 | "net/url" 6 | "testing" 7 | 8 | "github.com/go-test/deep" 9 | "github.com/google/go-github/v62/github" 10 | "github.com/prometheus/client_golang/prometheus" 11 | ) 12 | 13 | func TestUserGetUrl(t *testing.T) { 14 | t.Parallel() 15 | expectedLabels := prometheus.Labels{ 16 | "api_group": "user", 17 | "api_path": "", 18 | "repo_slug": "", 19 | "status": "404", 20 | "method": "GET", 21 | } 22 | instrumentGhCallTestHelper(t, "/api/v3/user", expectedLabels) 23 | } 24 | 25 | func TestRepoGetUrl(t *testing.T) { 26 | t.Parallel() 27 | expectedLabels := prometheus.Labels{ 28 | "api_group": "repos", 29 | "api_path": "", 30 | "repo_slug": "shared/k8s-helmfile", 31 | "status": "404", 32 | "method": "GET", 33 | } 34 | instrumentGhCallTestHelper(t, "/api/v3/repos/shared/k8s-helmfile", expectedLabels) 35 | } 36 | 37 | func TestContentUrl(t *testing.T) { 38 | t.Parallel() 39 | expectedLabels := prometheus.Labels{ 40 | "api_group": "repos", 41 | "api_path": "contents", 42 | "repo_slug": "shared/k8s-helmfile", 43 | "status": "404", 44 | "method": "GET", 45 | } 46 | instrumentGhCallTestHelper(t, "/api/v3/repos/shared/k8s-helmfile/contents/workspace/telefonistka/telefonistka.yaml", expectedLabels) 47 | } 48 | 49 | func TestPullUrl(t *testing.T) { 50 | t.Parallel() 51 | expectedLabels := prometheus.Labels{ 52 | "api_group": "repos", 53 | "api_path": "pulls", 54 | "repo_slug": "AnOwner/Arepo", 55 | "status": "404", 56 | "method": "GET", 57 | } 58 | instrumentGhCallTestHelper(t, "/repos/AnOwner/Arepo/pulls/33", expectedLabels) 59 | } 60 | 61 | func TestShortUrl(t *testing.T) { 62 | t.Parallel() 63 | expectedLabels := prometheus.Labels{ 64 | "api_group": "repos", 65 | "api_path": "contents", 66 | "repo_slug": "AnOwner/Arepo", 67 | "status": "404", 68 | "method": "GET", 69 | } 70 | instrumentGhCallTestHelper(t, "/repos/AnOwner/Arepo/contents/telefonistka.yaml", expectedLabels) 71 | } 72 | 73 | func TestApiUrl(t *testing.T) { 74 | t.Parallel() 75 | expectedLabels := prometheus.Labels{ 76 | "api_group": "repos", 77 | "api_path": "contents", 78 | "repo_slug": "AnOwner/Arepo", 79 | "status": "404", 80 | "method": "GET", 81 | } 82 | instrumentGhCallTestHelper(t, "/api/v3/repos/AnOwner/Arepo/contents/telefonistka.yaml", expectedLabels) 83 | } 84 | 85 | func TestInstrumentProxyUpstreamRequestLables(t *testing.T) { 86 | t.Parallel() 87 | 88 | mockURL, _ := url.Parse("https://argocd.example.com/webhook") 89 | 90 | httpReq := &http.Request{ 91 | URL: mockURL, 92 | Method: "POST", 93 | } 94 | 95 | httpResp := &http.Response{ 96 | Request: httpReq, 97 | StatusCode: 200, 98 | } 99 | 100 | expectedLabels := prometheus.Labels{ 101 | "status": "200", 102 | "method": "POST", 103 | "url": "https://argocd.example.com/webhook", 104 | } 105 | labels := InstrumentProxyUpstreamRequest(httpResp) 106 | if diff := deep.Equal(expectedLabels, labels); diff != nil { 107 | t.Error(diff) 108 | } 109 | } 110 | 111 | func instrumentGhCallTestHelper(t *testing.T, httpURL string, expectedLabels prometheus.Labels) { 112 | t.Helper() 113 | mockURL, _ := url.Parse("https://github.com/api/v3/content/foo/bar/file.txt") 114 | 115 | httpReq := &http.Request{ 116 | URL: mockURL, 117 | Method: "GET", 118 | } 119 | 120 | httpResp := &http.Response{ 121 | Request: httpReq, 122 | StatusCode: 404, 123 | } 124 | 125 | resp := &github.Response{ 126 | Response: httpResp, 127 | } 128 | resp.Request.URL.Path = httpURL 129 | labels := InstrumentGhCall(resp) 130 | 131 | if diff := deep.Equal(expectedLabels, labels); diff != nil { 132 | t.Error(diff) 133 | } 134 | } 135 | -------------------------------------------------------------------------------- /internal/pkg/testutils/testutils.go: -------------------------------------------------------------------------------- 1 | package testutils 2 | 3 | import ( 4 | "io" 5 | "os" 6 | 7 | log "github.com/sirupsen/logrus" 8 | ) 9 | 10 | // Quiet suppresses logs when running go test. 11 | func Quiet() func() { 12 | log.SetOutput(io.Discard) 13 | return func() { 14 | log.SetOutput(os.Stdout) 15 | } 16 | } 17 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "github.com/wayfair-incubator/telefonistka/cmd/telefonistka" 5 | ) 6 | 7 | func main() { 8 | telefonistka.Execute() 9 | } 10 | -------------------------------------------------------------------------------- /mirrord.json: -------------------------------------------------------------------------------- 1 | { 2 | "target": { 3 | "path": "deployment/telefonistka", 4 | "namespace": "telefonistka" 5 | }, 6 | "agent": { 7 | "namespace": "mirrord" 8 | }, 9 | "feature": { 10 | "fs": { 11 | "mode": "read", 12 | "read_write": ".+\\.json" , 13 | "read_only": [ "^/etc/telefonistka-gh-app-creds/.*", "^/etc/telefonistka-gh-app-config/.*" ] 14 | }, 15 | "network": { 16 | "incoming": "steal", 17 | "outgoing": true 18 | } 19 | }, 20 | "operator": false, 21 | "kubeconfig": "~/.kube/config", 22 | "sip_binaries": "bash", 23 | "telemetry": true 24 | } 25 | 26 | -------------------------------------------------------------------------------- /renovate.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://docs.renovatebot.com/renovate-schema.json", 3 | "description": "Wayfair OSPO recommended presets (https://github.com/wayfair/ospo-automation/blob/main/default.json)", 4 | "extends": [ 5 | "github>wayfair/ospo-automation" 6 | ] 7 | } 8 | -------------------------------------------------------------------------------- /templates/argoCD-diff-pr-comment-concise.gotmpl: -------------------------------------------------------------------------------- 1 | {{define "argoCdDiffConcise"}} 2 | Diff of ArgoCD applications(⚠️ concise view, full diff didn't fit GH comment): 3 | {{ range $appDiffResult := .DiffOfChangedComponents }} 4 | 5 | 6 | {{if $appDiffResult.DiffError }} 7 | > [!CAUTION] 8 | > **Error getting diff from ArgoCD** (`{{ $appDiffResult.ComponentPath }}`) 9 | 10 | ``` 11 | {{ $appDiffResult.DiffError }} 12 | 13 | ``` 14 | 15 | {{- else }} 16 | **[{{ $appDiffResult.ArgoCdAppName }}]({{ $appDiffResult.ArgoCdAppURL }})** @ `{{ $appDiffResult.ComponentPath }}` 17 | {{if $appDiffResult.HasDiff }} 18 | 19 |
ArgoCD list of changed objects(Click to expand): 20 | 21 | {{ range $objectDiff := $appDiffResult.DiffElements }} 22 | {{- if $objectDiff.Diff}} 23 | `{{ $objectDiff.ObjectNamespace }}/{{ $objectDiff.ObjectKind}}/{{ $objectDiff.ObjectName }}` 24 | {{- end}} 25 | {{- end }} 26 | 27 |
28 | {{- else }} 29 | {{ if $appDiffResult.AppSyncedFromPRBranch }} 30 | > [!NOTE] 31 | > The app already has this branch set as the source target revision, and autosync is enabled. Diff calculation was skipped. 32 | {{- else }} 33 | 34 | No diff 🤷 35 | {{- end}} 36 | {{if $appDiffResult.AppWasTemporarilyCreated }} 37 | > [!NOTE] 38 | > Telefonistka has temporarily created an ArgoCD app object to render manifest previews. 39 | Please be aware: 40 | > * The app will only appear in the ArgoCD UI for a few seconds. 41 | {{- end}} 42 | 43 | {{- end }} 44 | {{- end }} 45 | 46 | {{- end }} 47 | 48 | {{- if .DisplaySyncBranchCheckBox }} 49 | 50 | - [ ] Set ArgoCD apps Target Revision to `{{ .BranchName }}` 51 | 52 | {{ end}} 53 | 54 | 55 | {{- end }} 56 | -------------------------------------------------------------------------------- /templates/argoCD-diff-pr-comment.gotmpl: -------------------------------------------------------------------------------- 1 | {{define "argoCdDiff"}} 2 | {{ if .Header }} 3 | {{ .Header }} 4 | {{- end}} 5 | Diff of ArgoCD applications: 6 | {{ range $appDiffResult := .DiffOfChangedComponents }} 7 | 8 | 9 | {{if $appDiffResult.DiffError }} 10 | > [!CAUTION] 11 | > **Error getting diff from ArgoCD** (`{{ $appDiffResult.ComponentPath }}`) 12 | 13 | Please check the App Conditions of **[{{ $appDiffResult.ArgoCdAppName }}]({{ $appDiffResult.ArgoCdAppURL }})** for more details. 14 | {{- if $appDiffResult.AppWasTemporarilyCreated }} 15 | > [!WARNING] 16 | > For investigation we kept the temporary application, please make sure to clean it up later! 17 | 18 | {{- end}} 19 | ``` 20 | {{ $appDiffResult.DiffError }} 21 | 22 | ``` 23 | 24 | {{- else }} 25 | **[{{ $appDiffResult.ArgoCdAppName }}]({{ $appDiffResult.ArgoCdAppURL }})** @ `{{ $appDiffResult.ComponentPath }}` 26 | {{if $appDiffResult.HasDiff }} 27 | 28 |
ArgoCD Diff(Click to expand): 29 | 30 | ```diff 31 | {{ range $objectDiff := $appDiffResult.DiffElements }} 32 | {{- if $objectDiff.Diff}} 33 | {{ $objectDiff.ObjectNamespace }}/{{ $objectDiff.ObjectKind}}/{{ $objectDiff.ObjectName }}: 34 | {{$objectDiff.Diff}} 35 | {{- end}} 36 | {{- end }} 37 | ``` 38 | 39 |
40 | {{- else }} 41 | {{ if $appDiffResult.AppSyncedFromPRBranch }} 42 | > [!NOTE] 43 | > The app already has this branch set as the source target revision, and autosync is enabled. Diff calculation was skipped. 44 | {{- else }} 45 | 46 | No diff 🤷 47 | {{- end}} 48 | {{if $appDiffResult.AppWasTemporarilyCreated }} 49 | > [!NOTE] 50 | > Telefonistka has temporarily created an ArgoCD app object to render manifest previews. 51 | Please be aware: 52 | > * The app will only appear in the ArgoCD UI for a few seconds. 53 | {{- end}} 54 | 55 | {{- end }} 56 | {{- end }} 57 | 58 | {{- end }} 59 | 60 | {{- if .DisplaySyncBranchCheckBox }} 61 | 62 | - [ ] Set ArgoCD apps Target Revision to `{{ .BranchName }}` 63 | 64 | {{ end}} 65 | 66 | 67 | {{- end }} 68 | -------------------------------------------------------------------------------- /templates/auto-merge-comment.gotmpl: -------------------------------------------------------------------------------- 1 | {{define "autoMerge"}} 2 | ✅ Auto merge is enabled 3 | 🚀 Merging promotion PR: #{{.prNumber}} 4 | {{ end }} 5 | 6 | -------------------------------------------------------------------------------- /templates/drift-pr-comment.gotmpl: -------------------------------------------------------------------------------- 1 | {{define "driftMsg"}} 2 | # ⚠️ Found drift between environments ⚠️ 3 | 4 | ## Intro 5 | Drift detection runs on the files in the main branch irrespective of the changes of the PR. 6 | 7 | This could happen in two scenarios: 8 | 1. A promotion that affects these components is still in progress or was cancelled before completion. 9 | This means that your automated promotion PR will **include these changes** in addition to your changes! 10 | The "Blame Links" at the bottom of this comment can be a good place to start looking for the culprit. 11 | 12 | 13 | 2. Someone made a change directly to one of the directories representing promotion targets. 14 | These change will be **overridden** by the automated promotion PRs unless changes are made to their respective branches. 15 | 16 | 17 | ## Diffs 18 | 19 | 20 | {{- range $title, $diffOutput := . }} 21 | 22 | {{ $title }} 23 | 24 |
Diff (Click to expand) 25 | 26 | {{ $diffOutput }} 27 | 28 |
29 | 30 | {{- end }} 31 | 32 | {{- end }} 33 | -------------------------------------------------------------------------------- /templates/dry-run-pr-comment.gotmpl: -------------------------------------------------------------------------------- 1 | {{define "dryRunMsg"}} 2 | ## Telefonistka Promotion Dry Run Message: 3 | 4 | This is the plan for opening promotion PRs: 5 | 6 | 7 | {{ range $key, $value := . }} 8 | 9 | 10 | ``` 11 | PR :{{ $value.Metadata.SourcePath }} 12 | {{- range $trgt, $src := $value.ComputedSyncPaths }} 13 | ✅ {{ $src }} ➡️ {{ $trgt }} 14 | {{- end }} 15 | {{- if $value.Metadata.PerComponentSkippedTargetPaths}} 16 | Skipped target paths: 17 | {{- range $k, $v := $value.Metadata.PerComponentSkippedTargetPaths}} 18 | 🚫 {{ $value.Metadata.SourcePath }}/{{$k}} ➡️ {{$v}} 19 | {{- end}} 20 | {{- end}} 21 | ``` 22 | 23 | {{- end }} 24 | {{ end }} 25 | 26 | --------------------------------------------------------------------------------