├── .dockerignore ├── changelogs ├── CHANGELOG-1.11.0.md ├── CHANGELOG-1.3.0.md ├── CHANGELOG-1.13.0.md ├── CHANGELOG-1.2.1.md ├── CHANGELOG-1.6.0.md ├── CHANGELOG-1.1.2.md ├── CHANGELOG-1.1.1.md ├── CHANGELOG-1.10.0.md ├── CHANGELOG-1.4.0.md ├── CHANGELOG-1.7.0.md ├── CHANGELOG-1.9.0.md ├── CHANGELOG-1.2.0.md ├── CHANGELOG-1.5.0.md └── CHANGELOG-1.8.0.md ├── hack ├── ci │ └── build_util.sh ├── cp-plugin │ └── main.go ├── release-tools │ └── changelog.sh ├── build.sh └── docker-push.sh ├── .gitignore ├── tilt-provider.json ├── .github ├── auto-assignees.yml └── workflows │ ├── auto_request_review.yml │ ├── pr.yaml │ ├── auto_assign_prs.yml │ └── push.yml ├── velero-plugin-for-microsoft-azure ├── main.go ├── object_store_test.go ├── object_store_integration_test.go ├── volume_snapshotter_test.go ├── object_store.go └── volume_snapshotter.go ├── Dockerfile ├── volumesnapshotlocation.md ├── CONTRIBUTING.md ├── backupstoragelocation.md ├── Makefile ├── go.mod ├── CODE_OF_CONDUCT.md ├── LICENSE ├── go.sum └── README.md /.dockerignore: -------------------------------------------------------------------------------- 1 | _output 2 | Dockerfile -------------------------------------------------------------------------------- /changelogs/CHANGELOG-1.11.0.md: -------------------------------------------------------------------------------- 1 | ## All changes 2 | 3 | - Bump up version of Velero (#254, @ywk253100) 4 | -------------------------------------------------------------------------------- /changelogs/CHANGELOG-1.3.0.md: -------------------------------------------------------------------------------- 1 | ## All changes 2 | 3 | - Change the base image to `distroless`. (#106, @ywk253100) 4 | -------------------------------------------------------------------------------- /changelogs/CHANGELOG-1.13.0.md: -------------------------------------------------------------------------------- 1 | ## All changes 2 | 3 | - Remove GCR and update some action versions. (#290, @blackpiglet) 4 | -------------------------------------------------------------------------------- /changelogs/CHANGELOG-1.2.1.md: -------------------------------------------------------------------------------- 1 | ## All changes 2 | 3 | This release contains no user facing changes but includes fixes for CVE-2021-3121 and CVE-2021-3580. 4 | -------------------------------------------------------------------------------- /changelogs/CHANGELOG-1.6.0.md: -------------------------------------------------------------------------------- 1 | ## All changes 2 | 3 | - Update the snapshotter to support VSL credential (#147, @ywk253100) 4 | - Bump up golang to 1.18 to fix CVEs(#150, @ywk253100) 5 | -------------------------------------------------------------------------------- /changelogs/CHANGELOG-1.1.2.md: -------------------------------------------------------------------------------- 1 | ## All changes 2 | 3 | - Fixed an issue where only the first page of results from ListBlobs was processed. Now, all pages of results are processed. (#87, @justenwalker) 4 | -------------------------------------------------------------------------------- /hack/ci/build_util.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -x 3 | 4 | set -e 5 | 6 | function uploader { 7 | gsutil cp $1 gs://$2/$1 8 | gsutil -D setacl public-read gs://$2/$1 &> /dev/null 9 | } 10 | -------------------------------------------------------------------------------- /changelogs/CHANGELOG-1.1.1.md: -------------------------------------------------------------------------------- 1 | ## All changes 2 | 3 | - add support for `aad-pod-identity` authentication (#51, @gitirabassi) 4 | - add support for incremental snapshots of Azure disks (#52, @stephanwehr) 5 | -------------------------------------------------------------------------------- /changelogs/CHANGELOG-1.10.0.md: -------------------------------------------------------------------------------- 1 | ## All changes 2 | 3 | - Add tips for incremental snapshot (#228, @anshulahuja98) 4 | - Add instructions to enable the certificate-based authentication (#235, @ywk253100) 5 | -------------------------------------------------------------------------------- /changelogs/CHANGELOG-1.4.0.md: -------------------------------------------------------------------------------- 1 | ## All changes 2 | 3 | - Support snapshotting pv provisioned by csi driver on Azure (#109, @ywk253100) 4 | - Upgrade golang to 1.17 and upgrade the packages to the latest versions (#114, @ywk253100) -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Folders 2 | _output 3 | 4 | *.exe 5 | *.test 6 | *.prof 7 | debug 8 | 9 | .container-* 10 | .push-* 11 | 12 | # Editor related 13 | .vimrc 14 | .go 15 | .DS_Store 16 | .vscode 17 | *.diff 18 | .vs 19 | 20 | .idea/ 21 | 22 | _tiltbuild -------------------------------------------------------------------------------- /tilt-provider.json: -------------------------------------------------------------------------------- 1 | { 2 | "plugin_name": "velero-plugin-for-microsoft-azure", 3 | "context": ".", 4 | "image": "velero/velero-plugin-for-microsoft-azure", 5 | "live_reload_deps": [ 6 | "velero-plugin-for-microsoft-azure" 7 | ], 8 | "go_main": "./velero-plugin-for-microsoft-azure" 9 | } -------------------------------------------------------------------------------- /changelogs/CHANGELOG-1.7.0.md: -------------------------------------------------------------------------------- 1 | ## All changes 2 | 3 | - Rewrite the implementation with new Azure SDK (#111, @yvespp) 4 | - Fix CVEs reported by trivy scanner (#162, @blackpiglet) 5 | - Fix dependabot alerts(#170, @blackpiglet) 6 | - Update readme to clarify the content handled by object store plugin (#173, @ywk253100) 7 | - Bump up Golang to 1.19 (#175, @ywk253100) -------------------------------------------------------------------------------- /changelogs/CHANGELOG-1.9.0.md: -------------------------------------------------------------------------------- 1 | ## All changes 2 | 3 | - Refactor the Azure plugin (#206, @ywk253100) 4 | - Adjust the buffer size to avoid OOM (#210, @ywk253100) 5 | - Add MSI Support for Azure plugin (#212, @yanggangtony) 6 | - Bump version of Azure Compute SDK to support Prem SSD V2 (#215, @ywk253100) 7 | - Fix docs around storageaccounturi (#218, @anshulahuja98) -------------------------------------------------------------------------------- /changelogs/CHANGELOG-1.2.0.md: -------------------------------------------------------------------------------- 1 | ## All changes 2 | 3 | - Add section to README describing least required privileges in Azure (#75, @justbert) 4 | - Add support for the new `credentialsFile` config key which enables per-BSL credentials. If set, the plugin will use this path as the credentials file for authentication rather than the credentials file path in the environment (#90, @zubron) -------------------------------------------------------------------------------- /changelogs/CHANGELOG-1.5.0.md: -------------------------------------------------------------------------------- 1 | ## All changes 2 | 3 | - Support for zone-redundant storage (ZRS) managed disks (#131, @yyvess) 4 | - Refine the minimum permissions needed by Velero in README(#133, @ywk253100) 5 | - Use scratch as baseimage for plugin(#137, @yvespp) 6 | - Use the environment variable when creating the custom role in README(#139, @tbuchi888) 7 | - Add new config item "tags" to volume snapshot location for Azure(#140, @ywk253100) -------------------------------------------------------------------------------- /.github/auto-assignees.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # This assigns a PR to its author 3 | addAssignees: author 4 | 5 | reviewers: 6 | # The default reviewers 7 | defaults: 8 | - maintainers 9 | 10 | groups: 11 | maintainers: 12 | - sseago 13 | - reasonerjt 14 | - ywk253100 15 | - blackpiglet 16 | - qiuming-best 17 | - shubham-pampattiwar 18 | 19 | options: 20 | ignore_draft: true 21 | ignored_keywords: 22 | - WIP 23 | - wip 24 | - DO NOT MERGE 25 | enable_group_assignment: true 26 | number_of_reviewers: 2 27 | -------------------------------------------------------------------------------- /.github/workflows/auto_request_review.yml: -------------------------------------------------------------------------------- 1 | --- 2 | name: "Auto Request Review" 3 | 4 | on: 5 | pull_request_target: 6 | types: [opened, ready_for_review, reopened] 7 | 8 | jobs: 9 | auto-request-review: 10 | name: Auto Request Review 11 | runs-on: ubuntu-latest 12 | steps: 13 | - name: Request a PR review based on files types/paths, and/or groups the author belongs to 14 | uses: necojackarc/auto-request-review@v0.7.0 15 | with: 16 | token: ${{ secrets.GITHUB_TOKEN }} 17 | config: .github/auto-assignees.yml 18 | -------------------------------------------------------------------------------- /.github/workflows/pr.yaml: -------------------------------------------------------------------------------- 1 | name: Run CI 2 | on: [pull_request] 3 | jobs: 4 | 5 | build: 6 | name: Run CI 7 | runs-on: ubuntu-latest 8 | steps: 9 | 10 | - name: Check out the code 11 | uses: actions/checkout@v4 12 | 13 | - name: Set up Go 14 | uses: actions/setup-go@v5 15 | with: 16 | go-version-file: 'go.mod' 17 | id: go 18 | 19 | 20 | - name: Make CI 21 | run: make ci 22 | 23 | - name: Upload test coverage 24 | uses: codecov/codecov-action@v2 25 | with: 26 | token: ${{ secrets.CODECOV_TOKEN }} 27 | files: coverage.out 28 | verbose: true 29 | -------------------------------------------------------------------------------- /changelogs/CHANGELOG-1.8.0.md: -------------------------------------------------------------------------------- 1 | ## All changes 2 | 3 | - Add pushing image to gcr.io in push action (#183, @blackpiglet) 4 | - Support AzureDNSZone storage account (#185, @anshulahuja98) 5 | - Load environment variables early so that functions requiring those variables work correctly (#186, @gwynforthewyn) 6 | - Support auth via Workload Identity (#188, @tareksha) 7 | - Fixed json parsing errors in README.md (#193, @rajatumrao) 8 | - Support configuring Azure AD endpoint(#195, @Jeremy-Boyle) 9 | - Update doc to fix the "DataActions" (#200, @ywk253100) 10 | - Update README to include the "AZURE_ENVIRONMENT" in the credential file to workaround non public cloud issue (#201, @ywk253100) -------------------------------------------------------------------------------- /.github/workflows/auto_assign_prs.yml: -------------------------------------------------------------------------------- 1 | --- 2 | name: "Auto Assign Author" 3 | 4 | # pull_request_target means that this will run on pull requests, but in 5 | # the context of the base repo. This should mean PRs from forks are supported. 6 | on: 7 | pull_request_target: 8 | types: [opened, reopened, ready_for_review] 9 | 10 | jobs: 11 | # Automatically assigns reviewers and owner 12 | add-reviews: 13 | runs-on: ubuntu-latest 14 | steps: 15 | - name: Set the author of a PR as the assignee 16 | uses: kentaro-m/auto-assign-action@v1.1.2 17 | with: 18 | configuration-path: ".github/auto-assignees.yml" 19 | repo-token: "${{ secrets.GITHUB_TOKEN }}" 20 | -------------------------------------------------------------------------------- /hack/cp-plugin/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "errors" 5 | "fmt" 6 | "io" 7 | "os" 8 | ) 9 | 10 | func main() { 11 | if len(os.Args) != 3 { 12 | fmt.Println( 13 | `Error: This command requires two arguments. 14 | Usage: cp-plugin src dst`) 15 | os.Exit(1) 16 | } 17 | src, dst := os.Args[1], os.Args[2] 18 | fmt.Printf("Copying %s to %s ... ", src, dst) 19 | srcFile, err := os.Open(src) 20 | if err != nil { 21 | panic(err) 22 | } 23 | defer srcFile.Close() 24 | if _, err := os.Stat(dst); errors.Is(err, os.ErrNotExist) { 25 | _, err = os.Create(dst) 26 | if err != nil { 27 | panic(err) 28 | } 29 | } 30 | dstFile, err := os.OpenFile(dst, os.O_WRONLY, 0755) 31 | if err != nil { 32 | panic(err) 33 | } 34 | defer dstFile.Close() 35 | buf := make([]byte, 1024*128) 36 | _, err = io.CopyBuffer(dstFile, srcFile, buf) 37 | if err != nil { 38 | panic(err) 39 | } 40 | os.Chmod(dst, 0755) 41 | fmt.Println("done.") 42 | } 43 | -------------------------------------------------------------------------------- /hack/release-tools/changelog.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Copyright 2018 the Velero contributors. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | set -o errexit 18 | set -o nounset 19 | set -o pipefail 20 | 21 | function join { local IFS="$1"; shift; echo "$*"; } 22 | 23 | CHANGELOG_PATH='changelogs/unreleased' 24 | UNRELEASED=$(ls -t ${CHANGELOG_PATH}) 25 | echo -e "Generating CHANGELOG markdown from ${CHANGELOG_PATH}\n" 26 | for entry in $UNRELEASED 27 | do 28 | IFS=$'-' read -ra pruser <<<"$entry" 29 | contents=$(cat ${CHANGELOG_PATH}/${entry}) 30 | pr=${pruser[0]} 31 | user=$(join '-' ${pruser[@]:1}) 32 | echo " * ${contents} (#${pr}, @${user})" 33 | done 34 | echo -e "\nCopy and paste the list above in to the appropriate CHANGELOG file." 35 | echo "Be sure to run: git rm ${CHANGELOG_PATH}/*" 36 | -------------------------------------------------------------------------------- /velero-plugin-for-microsoft-azure/main.go: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright 2017, 2019 the Velero contributors. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"); 5 | you may not use this file except in compliance with the License. 6 | You may obtain a copy of the License at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | Unless required by applicable law or agreed to in writing, software 11 | distributed under the License is distributed on an "AS IS" BASIS, 12 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | See the License for the specific language governing permissions and 14 | limitations under the License. 15 | */ 16 | 17 | package main 18 | 19 | import ( 20 | "github.com/sirupsen/logrus" 21 | "github.com/spf13/pflag" 22 | veleroplugin "github.com/vmware-tanzu/velero/pkg/plugin/framework" 23 | ) 24 | 25 | func main() { 26 | veleroplugin.NewServer(). 27 | BindFlags(pflag.CommandLine). 28 | RegisterObjectStore("velero.io/azure", newAzureObjectStore). 29 | RegisterVolumeSnapshotter("velero.io/azure", newAzureVolumeSnapshotter). 30 | Serve() 31 | } 32 | 33 | func newAzureObjectStore(logger logrus.FieldLogger) (interface{}, error) { 34 | return newObjectStore(logger), nil 35 | } 36 | 37 | func newAzureVolumeSnapshotter(logger logrus.FieldLogger) (interface{}, error) { 38 | return newVolumeSnapshotter(logger), nil 39 | } 40 | -------------------------------------------------------------------------------- /hack/build.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Copyright 2016 The Kubernetes Authors. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | set -o errexit 18 | set -o nounset 19 | set -o pipefail 20 | 21 | if [ -z "${PKG}" ]; then 22 | echo "PKG must be set" 23 | exit 1 24 | fi 25 | if [ -z "${BIN}" ]; then 26 | echo "BIN must be set" 27 | exit 1 28 | fi 29 | if [ -z "${GOOS}" ]; then 30 | echo "GOOS must be set" 31 | exit 1 32 | fi 33 | if [ -z "${GOARCH}" ]; then 34 | echo "GOARCH must be set" 35 | exit 1 36 | fi 37 | 38 | export CGO_ENABLED=0 39 | 40 | if [[ -z "${OUTPUT_DIR:-}" ]]; then 41 | OUTPUT_DIR=. 42 | fi 43 | OUTPUT=${OUTPUT_DIR}/${BIN} 44 | if [[ "${GOOS}" = "windows" ]]; then 45 | OUTPUT="${OUTPUT}.exe" 46 | fi 47 | 48 | go build \ 49 | -o ${OUTPUT} \ 50 | -installsuffix "static" \ 51 | ${PKG}/${BIN} 52 | -------------------------------------------------------------------------------- /.github/workflows/push.yml: -------------------------------------------------------------------------------- 1 | name: Main CI 2 | 3 | on: 4 | push: 5 | branches: 6 | - 'main' 7 | - 'release-**' 8 | tags: 9 | - '*' 10 | 11 | jobs: 12 | 13 | build: 14 | name: Build 15 | runs-on: ubuntu-latest 16 | steps: 17 | 18 | - name: Check out code into the Go module directory 19 | uses: actions/checkout@v4 20 | 21 | - name: Set up Go 22 | uses: actions/setup-go@v5 23 | with: 24 | go-version-file: 'go.mod' 25 | id: go 26 | 27 | - name: Set up QEMU 28 | id: qemu 29 | uses: docker/setup-qemu-action@v1 30 | with: 31 | platforms: all 32 | 33 | - name: Set up Docker Buildx 34 | id: buildx 35 | uses: docker/setup-buildx-action@v1 36 | with: 37 | version: latest 38 | 39 | - name: Build 40 | run: make local 41 | 42 | - name: Test 43 | run: make test 44 | 45 | - name: Upload test coverage 46 | uses: codecov/codecov-action@v2 47 | with: 48 | token: ${{ secrets.CODECOV_TOKEN }} 49 | files: coverage.out 50 | verbose: true 51 | 52 | # Only try to publish the container image from the root repo; forks don't have permission to do so and will always get failures. 53 | - name: Publish container image 54 | if: github.repository == 'vmware-tanzu/velero-plugin-for-microsoft-azure' 55 | run: | 56 | docker login -u ${{ secrets.DOCKER_USER }} -p ${{ secrets.DOCKER_PASSWORD }} 57 | ./hack/docker-push.sh 58 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | # Copyright the Velero contributors. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | FROM --platform=$BUILDPLATFORM golang:1.24-bookworm AS build 16 | 17 | ARG TARGETOS 18 | ARG TARGETARCH 19 | ARG TARGETVARIANT 20 | ARG GOPROXY 21 | 22 | ENV GOOS=${TARGETOS} \ 23 | GOARCH=${TARGETARCH} \ 24 | GOARM=${TARGETVARIANT} \ 25 | GOPROXY=${GOPROXY} 26 | 27 | COPY . /go/src/velero-plugin-for-microsoft-azure 28 | WORKDIR /go/src/velero-plugin-for-microsoft-azure 29 | RUN export GOARM=$( echo "${GOARM}" | cut -c2-) && \ 30 | CGO_ENABLED=0 go build -v -o /go/bin/velero-plugin-for-microsoft-azure ./velero-plugin-for-microsoft-azure && \ 31 | CGO_ENABLED=0 go build -v -o /go/bin/cp-plugin ./hack/cp-plugin 32 | 33 | FROM scratch 34 | COPY --from=build /go/bin/velero-plugin-for-microsoft-azure /plugins/ 35 | COPY --from=build /go/bin/cp-plugin /bin/cp-plugin 36 | USER 65532:65532 37 | ENTRYPOINT ["cp-plugin", "/plugins/velero-plugin-for-microsoft-azure", "/target/velero-plugin-for-microsoft-azure"] 38 | -------------------------------------------------------------------------------- /volumesnapshotlocation.md: -------------------------------------------------------------------------------- 1 | # Volume Snapshot Location 2 | 3 | The following sample Azure `VolumeSnapshotLocation` YAML shows all of the configurable parameters. The items under `spec.config` can be provided as key-value pairs to the `velero install` command's `--snapshot-location-config` flag -- for example, `apiTimeout=5m,resourceGroup=my-rg,...`. 4 | 5 | ```yaml 6 | apiVersion: velero.io/v1 7 | kind: VolumeSnapshotLocation 8 | metadata: 9 | name: azure-default 10 | namespace: velero 11 | spec: 12 | # Name of the volume snapshotter plugin to use to connect to this location. 13 | # 14 | # Required. 15 | provider: velero.io/azure 16 | 17 | config: 18 | # How long to wait for an Azure API request to complete before timeout. 19 | # 20 | # Optional (defaults to 2m0s). 21 | apiTimeout: 5m 22 | 23 | # The name of the resource group where volume snapshots should be stored, if different 24 | # from the cluster's resource group. 25 | # 26 | # Optional. 27 | resourceGroup: my-rg 28 | 29 | # The ID of the subscription where volume snapshots should be stored, if different 30 | # from the cluster's subscription. Requires "resourceGroup" to also be set. 31 | # 32 | # Optional. 33 | subscriptionId: alt-subscription 34 | 35 | # URI of the AAD endpoint of the volume snapshots account. 36 | # 37 | # Note that the fully qualified AAD URI with http(s):// scheme is required to authenticate 38 | # 39 | # Optional. This will ensure that velero uses the provided AAD URI to authenticate to the volume snapshots account. 40 | activeDirectoryAuthorityURI: https://login.microsoftonline.us/ 41 | 42 | # Azure offers the option to take full or incremental snapshots of managed disks. 43 | # - Set this parameter to true, to take incremental snapshots. 44 | # - If the parameter is omitted or set to false, full snapshots are taken (default). 45 | # 46 | # Optional. 47 | incremental: "" 48 | 49 | # The tags added to the volume snapshots during the backup 50 | # 51 | # Optional. 52 | tags: key1=value1,key2=value2 53 | ``` 54 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing 2 | 3 | ## CHANGELOG 4 | 5 | Authors are expected to include a changelog file with their pull requests. The changelog file 6 | should be a new file created in the `changelogs/unreleased` folder. The file should follow the 7 | naming convention of `pr-username` and the contents of the file should be your text for the 8 | changelog. 9 | 10 | velero-plugin-for-microsoft-azure/changelogs/unreleased <- folder 11 | 000-username <- file 12 | 13 | 14 | ## DCO Sign off 15 | 16 | All authors to the project retain copyright to their work. However, to ensure 17 | that they are only submitting work that they have rights to, we are requiring 18 | everyone to acknowledge this by signing their work. 19 | 20 | Any copyright notices in this repo should specify the authors as "the Velero contributors". 21 | 22 | To sign your work, just add a line like this at the end of your commit message: 23 | 24 | ``` 25 | Signed-off-by: Joe Beda 26 | ``` 27 | 28 | This can easily be done with the `--signoff` option to `git commit`. 29 | 30 | By doing this you state that you can certify the following (from https://developercertificate.org/): 31 | 32 | ``` 33 | Developer Certificate of Origin 34 | Version 1.1 35 | 36 | Copyright (C) 2004, 2006 The Linux Foundation and its contributors. 37 | 1 Letterman Drive 38 | Suite D4700 39 | San Francisco, CA, 94129 40 | 41 | Everyone is permitted to copy and distribute verbatim copies of this 42 | license document, but changing it is not allowed. 43 | 44 | 45 | Developer's Certificate of Origin 1.1 46 | 47 | By making a contribution to this project, I certify that: 48 | 49 | (a) The contribution was created in whole or in part by me and I 50 | have the right to submit it under the open source license 51 | indicated in the file; or 52 | 53 | (b) The contribution is based upon previous work that, to the best 54 | of my knowledge, is covered under an appropriate open source 55 | license and I have the right under that license to submit that 56 | work with modifications, whether created in whole or in part 57 | by me, under the same open source license (unless I am 58 | permitted to submit under a different license), as indicated 59 | in the file; or 60 | 61 | (c) The contribution was provided directly to me by some other 62 | person who certified (a), (b) or (c) and I have not modified 63 | it. 64 | 65 | (d) I understand and agree that this project and the contribution 66 | are public and that a record of the contribution (including all 67 | personal information I submit with it, including my sign-off) is 68 | maintained indefinitely and may be redistributed consistent with 69 | this project or the open source license(s) involved. 70 | ``` 71 | -------------------------------------------------------------------------------- /backupstoragelocation.md: -------------------------------------------------------------------------------- 1 | # Backup Storage Location 2 | 3 | The following sample Azure `BackupStorageLocation` YAML shows all of the configurable parameters. The items under `spec.config` can be provided as key-value pairs to the `velero install` command's `--backup-location-config` flag -- for example, `resourceGroup=my-rg,storageAccount=my-sa,...`. 4 | 5 | ```yaml 6 | apiVersion: velero.io/v1 7 | kind: BackupStorageLocation 8 | metadata: 9 | name: default 10 | namespace: velero 11 | spec: 12 | # Name of the object store plugin to use to connect to this location. 13 | # 14 | # Required. 15 | provider: velero.io/azure 16 | 17 | objectStorage: 18 | # The bucket/blob container in which to store backups. 19 | # 20 | # Required. 21 | bucket: my-bucket 22 | 23 | # The prefix within the bucket under which to store backups. 24 | # 25 | # Optional. 26 | prefix: my-prefix 27 | 28 | config: 29 | # Name of the resource group containing the storage account for this backup storage location. 30 | # 31 | # Required. 32 | resourceGroup: my-backup-resource-group 33 | 34 | # Name of the storage account for this backup storage location. 35 | # 36 | # Required. 37 | storageAccount: my-backup-storage-account 38 | 39 | # Name of the environment variable in $AZURE_CREDENTIALS_FILE that contains storage account key for this backup storage location. 40 | # 41 | # Required if using a storage account access key to authenticate rather than a service principal. 42 | storageAccountKeyEnvVar: MY_BACKUP_STORAGE_ACCOUNT_KEY_ENV_VAR 43 | 44 | # ID of the subscription for this backup storage location. 45 | # 46 | # Optional. 47 | subscriptionId: my-subscription 48 | 49 | # URI of the blob endpoint of the storage account. 50 | # 51 | # Optional. This will ensure that velero uses the provided URI to communicate to the Storage Account, 52 | # and it will not try to fetch the Endpoint by making an ARM call. 53 | # If this field is provided then resourceGroup, subscriptionId can be left empty 54 | storageAccountURI: https://my-sa.blob.core.windows.net 55 | 56 | # Boolean parameter to decide whether to use AAD for authenticating with the storage account. 57 | # If false/ not provided, plugin will fallback to using ListKeys 58 | # 59 | # Optional. Recommended. 60 | useAAD: "true" 61 | 62 | # URI of the AAD endpoint of the storage account. 63 | # 64 | # Note that useAAD: should be set to "true" in order to use the provided AAD URI and http(s):// scheme is required to authenticate 65 | # 66 | # Optional. This will ensure that velero uses the provided AAD URI to authenticate to the Storage Account. 67 | activeDirectoryAuthorityURI: https://login.microsoftonline.us/ 68 | 69 | # The block size, in bytes, to use when uploading objects to Azure blob storage. 70 | # See https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-block-blobs 71 | # for more information on block blobs. 72 | # 73 | # Optional (defaults to 1048576, i.e. 1MB, maximum 104857600, i.e. 100MB). 74 | blockSizeInBytes: "1048576" 75 | ``` 76 | -------------------------------------------------------------------------------- /hack/docker-push.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Copyright the Velero contributors. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | # docker-push is invoked by the CI/CD system to deploy docker images to Docker Hub. 18 | # It will build images for all commits to main and all git tags. 19 | # The highest, non-prerelease semantic version will also be given the `latest` tag. 20 | 21 | set +x 22 | 23 | if [[ -z "$CI" ]]; then 24 | echo "This script is intended to be run only on Github Actions." >&2 25 | exit 1 26 | fi 27 | 28 | # Return value is written into HIGHEST 29 | HIGHEST="" 30 | function highest_release() { 31 | # Loop through the tags since pre-release versions come before the actual versions. 32 | # Iterate til we find the first non-pre-release 33 | 34 | # This is not necessarily the most recently made tag; instead, we want it to be the highest semantic version. 35 | # The most recent tag could potentially be a lower semantic version, made as a point release for a previous series. 36 | # As an example, if v1.3.0 exists and we create v1.2.2, v1.3.0 should still be `latest`. 37 | # `git describe --tags $(git rev-list --tags --max-count=1)` would return the most recently made tag. 38 | 39 | for t in $(git tag -l --sort=-v:refname); 40 | do 41 | # If the tag has alpha, beta or rc in it, it's not "latest" 42 | if [[ "$t" == *"beta"* || "$t" == *"alpha"* || "$t" == *"rc"* ]]; then 43 | continue 44 | fi 45 | HIGHEST="$t" 46 | break 47 | done 48 | } 49 | 50 | triggeredBy=$(echo $GITHUB_REF | cut -d / -f 2) 51 | if [[ "$triggeredBy" == "heads" ]]; then 52 | BRANCH=$(echo $GITHUB_REF | cut -d / -f 3) 53 | TAG= 54 | elif [[ "$triggeredBy" == "tags" ]]; then 55 | BRANCH= 56 | TAG=$(echo $GITHUB_REF | cut -d / -f 3) 57 | fi 58 | 59 | TAG_LATEST=false 60 | if [[ ! -z "$TAG" ]]; then 61 | echo "We're building tag $TAG" 62 | VERSION="$TAG" 63 | # Explicitly checkout tags when building from a git tag. 64 | # This is not needed when building from main 65 | git fetch --tags 66 | # Calculate the latest release if there's a tag. 67 | highest_release 68 | if [[ "$TAG" == "$HIGHEST" ]]; then 69 | TAG_LATEST=true 70 | fi 71 | else 72 | echo "We're on branch $BRANCH" 73 | VERSION="$BRANCH" 74 | if [[ "$VERSION" == release-* ]]; then 75 | VERSION=${VERSION}-dev 76 | fi 77 | fi 78 | 79 | if [[ -z "$BUILDX_PLATFORMS" ]]; then 80 | BUILDX_PLATFORMS="linux/amd64,linux/arm64,linux/arm/v7" 81 | fi 82 | 83 | # Debugging info 84 | echo "Highest tag found: $HIGHEST" 85 | echo "BRANCH: $BRANCH" 86 | echo "TAG: $TAG" 87 | echo "TAG_LATEST: $TAG_LATEST" 88 | echo "BUILDX_PLATFORMS: $BUILDX_PLATFORMS" 89 | echo "VERSION: $VERSION" 90 | 91 | echo "Building and pushing container images." 92 | 93 | # The use of "registry" as the buildx output type below instructs 94 | # Docker to push the image 95 | 96 | VERSION="$VERSION" \ 97 | TAG_LATEST="$TAG_LATEST" \ 98 | BUILDX_PLATFORMS="$BUILDX_PLATFORMS" \ 99 | BUILDX_OUTPUT_TYPE="registry" \ 100 | make container -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Copyright the Velero contributors. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | # The binary to build (just the basename). 16 | BIN ?= velero-plugin-for-microsoft-azure 17 | 18 | # This repo's root import path (under GOPATH). 19 | PKG := github.com/vmware-tanzu/velero-plugin-for-microsoft-azure 20 | 21 | # Where to push the docker image. 22 | REGISTRY ?= velero 23 | 24 | # Image name 25 | IMAGE ?= $(REGISTRY)/$(BIN) 26 | 27 | # We allow the Dockerfile to be configurable to enable the use of custom Dockerfiles 28 | # that pull base images from different registries. 29 | VELERO_DOCKERFILE ?= Dockerfile 30 | 31 | # Which architecture to build - see $(ALL_ARCH) for options. 32 | # if the 'local' rule is being run, detect the ARCH from 'go env' 33 | # if it wasn't specified by the caller. 34 | local : ARCH ?= $(shell go env GOOS)-$(shell go env GOARCH) 35 | ARCH ?= linux-amd64 36 | 37 | VERSION ?= main 38 | 39 | TAG_LATEST ?= false 40 | 41 | ifeq ($(TAG_LATEST), true) 42 | IMAGE_TAGS ?= $(IMAGE):$(VERSION) $(IMAGE):latest 43 | else 44 | IMAGE_TAGS ?= $(IMAGE):$(VERSION) 45 | endif 46 | 47 | ifeq ($(shell docker buildx inspect 2>/dev/null | awk '/Status/ { print $$2 }'), running) 48 | BUILDX_ENABLED ?= true 49 | else 50 | BUILDX_ENABLED ?= false 51 | endif 52 | 53 | define BUILDX_ERROR 54 | buildx not enabled, refusing to run this recipe 55 | see: https://velero.io/docs/main/build-from-source/#making-images-and-updating-velero for more info 56 | endef 57 | 58 | CLI_PLATFORMS ?= linux-amd64 linux-arm linux-arm64 darwin-amd64 darwin-arm64 windows-amd64 linux-ppc64le 59 | BUILDX_PLATFORMS ?= $(subst -,/,$(ARCH)) 60 | BUILDX_OUTPUT_TYPE ?= docker 61 | 62 | # set git sha and tree state 63 | GIT_SHA = $(shell git rev-parse HEAD) 64 | ifneq ($(shell git status --porcelain 2> /dev/null),) 65 | GIT_TREE_STATE ?= dirty 66 | else 67 | GIT_TREE_STATE ?= clean 68 | endif 69 | 70 | ### 71 | ### These variables should not need tweaking. 72 | ### 73 | 74 | platform_temp = $(subst -, ,$(ARCH)) 75 | GOOS = $(word 1, $(platform_temp)) 76 | GOARCH = $(word 2, $(platform_temp)) 77 | GOPROXY ?= https://proxy.golang.org 78 | 79 | local: build-dirs 80 | GOOS=$(GOOS) \ 81 | GOARCH=$(GOARCH) \ 82 | VERSION=$(VERSION) \ 83 | REGISTRY=$(REGISTRY) \ 84 | PKG=$(PKG) \ 85 | BIN=$(BIN) \ 86 | GIT_SHA=$(GIT_SHA) \ 87 | GIT_TREE_STATE=$(GIT_TREE_STATE) \ 88 | OUTPUT_DIR=$$(pwd)/_output/bin/$(GOOS)/$(GOARCH) \ 89 | ./hack/build.sh 90 | 91 | # test runs unit tests using 'go test' in the local environment. 92 | test: 93 | CGO_ENABLED=0 go test -v -coverprofile=coverage.out -timeout 60s ./... 94 | 95 | # ci is a convenience target for CI builds. 96 | ci: verify-modules test 97 | 98 | container: 99 | ifneq ($(BUILDX_ENABLED), true) 100 | $(error $(BUILDX_ERROR)) 101 | endif 102 | @docker buildx build --pull \ 103 | --output=type=$(BUILDX_OUTPUT_TYPE) \ 104 | --platform $(BUILDX_PLATFORMS) \ 105 | $(addprefix -t , $(IMAGE_TAGS)) \ 106 | --build-arg=GOPROXY=$(GOPROXY) \ 107 | --build-arg=PKG=$(PKG) \ 108 | --build-arg=BIN=$(BIN) \ 109 | --build-arg=VERSION=$(VERSION) \ 110 | --build-arg=GIT_SHA=$(GIT_SHA) \ 111 | --build-arg=GIT_TREE_STATE=$(GIT_TREE_STATE) \ 112 | --build-arg=REGISTRY=$(REGISTRY) \ 113 | -f $(VELERO_DOCKERFILE) . 114 | @echo "container: $(IMAGE):$(VERSION)" 115 | ifeq ($(BUILDX_OUTPUT_TYPE)_$(REGISTRY), registry_velero) 116 | docker pull $(IMAGE):$(VERSION) 117 | rm -f $(BIN)-$(VERSION).tar 118 | docker save $(IMAGE):$(VERSION) -o $(BIN)-$(VERSION).tar 119 | gzip -f $(BIN)-$(VERSION).tar 120 | endif 121 | 122 | build-dirs: 123 | @mkdir -p _output/bin/$(GOOS)/$(GOARCH) 124 | 125 | .PHONY: modules 126 | modules: 127 | go mod tidy -compat=1.17 128 | 129 | .PHONY: verify-modules 130 | verify-modules: modules 131 | @if !(git diff --quiet HEAD -- go.sum go.mod); then \ 132 | echo "go module files are out of date, please commit the changes to go.mod and go.sum"; exit 1; \ 133 | fi 134 | 135 | changelog: 136 | hack/release-tools/changelog.sh 137 | 138 | # clean removes build artifacts from the local environment. 139 | clean: 140 | @echo "cleaning" 141 | rm -rf _output 142 | -------------------------------------------------------------------------------- /velero-plugin-for-microsoft-azure/object_store_test.go: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright 2018, 2019 the Velero contributors. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"); 5 | you may not use this file except in compliance with the License. 6 | You may obtain a copy of the License at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | Unless required by applicable law or agreed to in writing, software 11 | distributed under the License is distributed on an "AS IS" BASIS, 12 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | See the License for the specific language governing permissions and 14 | limitations under the License. 15 | */ 16 | 17 | package main 18 | 19 | import ( 20 | "io" 21 | "testing" 22 | "time" 23 | 24 | "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob" 25 | "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob" 26 | "github.com/pkg/errors" 27 | "github.com/sirupsen/logrus" 28 | "github.com/stretchr/testify/assert" 29 | "github.com/stretchr/testify/mock" 30 | "github.com/stretchr/testify/require" 31 | ) 32 | 33 | func TestObjectExists(t *testing.T) { 34 | tests := []struct { 35 | name string 36 | getBlobError error 37 | exists bool 38 | errorResponse error 39 | expectedExists bool 40 | expectedError string 41 | }{ 42 | { 43 | name: "getBlob error", 44 | exists: false, 45 | errorResponse: errors.New("getBlob"), 46 | expectedExists: false, 47 | expectedError: "getBlob", 48 | }, 49 | { 50 | name: "exists", 51 | exists: true, 52 | errorResponse: nil, 53 | expectedExists: true, 54 | }, 55 | { 56 | name: "doesn't exist", 57 | exists: false, 58 | errorResponse: nil, 59 | expectedExists: false, 60 | }, 61 | { 62 | name: "error checking for existence", 63 | exists: false, 64 | errorResponse: errors.New("bad"), 65 | expectedExists: false, 66 | expectedError: "bad", 67 | }, 68 | } 69 | 70 | for _, tc := range tests { 71 | t.Run(tc.name, func(t *testing.T) { 72 | blobGetter := new(mockBlobGetter) 73 | defer blobGetter.AssertExpectations(t) 74 | 75 | o := &ObjectStore{ 76 | blobGetter: blobGetter, 77 | } 78 | 79 | bucket := "b" 80 | key := "k" 81 | 82 | blob := new(mockBlob) 83 | defer blob.AssertExpectations(t) 84 | blobGetter.On("getBlob", bucket, key).Return(blob) 85 | 86 | blob.On("Exists").Return(tc.exists, tc.errorResponse) 87 | 88 | exists, err := o.ObjectExists(bucket, key) 89 | 90 | if tc.expectedError != "" { 91 | assert.EqualError(t, err, tc.expectedError) 92 | return 93 | } 94 | require.NoError(t, err) 95 | 96 | assert.Equal(t, tc.expectedExists, exists) 97 | }) 98 | } 99 | } 100 | 101 | type mockBlobGetter struct { 102 | mock.Mock 103 | } 104 | 105 | func (m *mockBlobGetter) getBlob(bucket string, key string) blob { 106 | args := m.Called(bucket, key) 107 | return args.Get(0).(blob) 108 | } 109 | 110 | type mockBlob struct { 111 | mock.Mock 112 | } 113 | 114 | func (m *mockBlob) PutBlock(blockID string, chunk []byte, options *blockblob.StageBlockOptions) error { 115 | args := m.Called(blockID, chunk, options) 116 | return args.Error(0) 117 | } 118 | func (m *mockBlob) PutBlockList(blocks []string, options *blockblob.CommitBlockListOptions) error { 119 | args := m.Called(blocks, options) 120 | return args.Error(0) 121 | } 122 | 123 | func (m *mockBlob) Exists() (bool, error) { 124 | args := m.Called() 125 | return args.Bool(0), args.Error(1) 126 | } 127 | 128 | func (m *mockBlob) Get(options *azblob.DownloadStreamOptions) (io.ReadCloser, error) { 129 | args := m.Called(options) 130 | return args.Get(0).(io.ReadCloser), args.Error(1) 131 | } 132 | 133 | func (m *mockBlob) Delete(options *azblob.DeleteBlobOptions) error { 134 | args := m.Called(options) 135 | return args.Error(0) 136 | } 137 | 138 | func (m *mockBlob) GetSASURI(ttl time.Duration, sharedKeyCredential *azblob.SharedKeyCredential) (string, error) { 139 | args := m.Called(ttl, sharedKeyCredential) 140 | return args.String(0), args.Error(1) 141 | } 142 | 143 | func TestGetBlockSize(t *testing.T) { 144 | logger := logrus.New() 145 | config := map[string]string{} 146 | // not specified 147 | size := getBlockSize(logger, config) 148 | assert.Equal(t, defaultBlockSize, size) 149 | 150 | // invalid value specified 151 | config[blockSizeConfigKey] = "invalid" 152 | size = getBlockSize(logger, config) 153 | assert.Equal(t, defaultBlockSize, size) 154 | 155 | // value < 0 specified 156 | config[blockSizeConfigKey] = "0" 157 | size = getBlockSize(logger, config) 158 | assert.Equal(t, defaultBlockSize, size) 159 | 160 | // value > max size specified 161 | config[blockSizeConfigKey] = "1048576000" 162 | size = getBlockSize(logger, config) 163 | assert.Equal(t, maxBlockSize, size) 164 | 165 | // valid value specified 166 | config[blockSizeConfigKey] = "1048570" 167 | size = getBlockSize(logger, config) 168 | assert.Equal(t, 1048570, size) 169 | } 170 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/vmware-tanzu/velero-plugin-for-microsoft-azure 2 | 3 | go 1.24.0 4 | 5 | require ( 6 | github.com/Azure/azure-sdk-for-go/sdk/azcore v1.18.1 7 | github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v4 v4.2.1 8 | github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.1 9 | github.com/gofrs/uuid v4.3.1+incompatible 10 | github.com/pkg/errors v0.9.1 11 | github.com/sirupsen/logrus v1.9.3 12 | github.com/spf13/pflag v1.0.5 13 | github.com/stretchr/testify v1.10.0 14 | github.com/vmware-tanzu/velero v0.0.0-20250826085519-79b027577e6a 15 | k8s.io/api v0.31.3 16 | k8s.io/apimachinery v0.31.3 17 | sigs.k8s.io/azuredisk-csi-driver v1.26.0 18 | ) 19 | 20 | require ( 21 | github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.10.1 // indirect 22 | github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.0 // indirect 23 | github.com/joho/godotenv v1.4.0 // indirect 24 | google.golang.org/genproto/googleapis/rpc v0.0.0-20250603155806-513f23925822 // indirect 25 | ) 26 | 27 | require ( 28 | github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.1 // indirect 29 | github.com/AzureAD/microsoft-authentication-library-for-go v1.4.2 // indirect 30 | github.com/beorn7/perks v1.0.1 // indirect 31 | github.com/cespare/xxhash/v2 v2.3.0 // indirect 32 | github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect 33 | github.com/emicklei/go-restful/v3 v3.11.0 // indirect 34 | github.com/evanphx/json-patch/v5 v5.9.0 // indirect 35 | github.com/fatih/color v1.18.0 // indirect 36 | github.com/fxamacker/cbor/v2 v2.7.0 // indirect 37 | github.com/go-logr/logr v1.4.3 // indirect 38 | github.com/go-openapi/jsonpointer v0.19.6 // indirect 39 | github.com/go-openapi/jsonreference v0.20.2 // indirect 40 | github.com/go-openapi/swag v0.22.4 // indirect 41 | github.com/gobwas/glob v0.2.3 // indirect 42 | github.com/gogo/protobuf v1.3.2 // indirect 43 | github.com/golang-jwt/jwt/v5 v5.2.2 // indirect 44 | github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect 45 | github.com/golang/protobuf v1.5.4 // indirect 46 | github.com/google/gnostic-models v0.6.8 // indirect 47 | github.com/google/go-cmp v0.7.0 // indirect 48 | github.com/google/gofuzz v1.2.0 // indirect 49 | github.com/google/uuid v1.6.0 // indirect 50 | github.com/hashicorp/go-hclog v1.4.0 // indirect 51 | github.com/hashicorp/go-plugin v1.6.0 // indirect 52 | github.com/hashicorp/yamux v0.1.1 // indirect 53 | github.com/imdario/mergo v0.3.13 // indirect 54 | github.com/inconshreveable/mousetrap v1.1.0 // indirect 55 | github.com/josharian/intern v1.0.0 // indirect 56 | github.com/json-iterator/go v1.1.12 // indirect 57 | github.com/kubernetes-csi/external-snapshotter/client/v8 v8.2.0 // indirect 58 | github.com/kylelemons/godebug v1.1.0 // indirect 59 | github.com/mailru/easyjson v0.7.7 // indirect 60 | github.com/mattn/go-colorable v0.1.14 // indirect 61 | github.com/mattn/go-isatty v0.0.20 // indirect 62 | github.com/mitchellh/go-testing-interface v1.14.1 // indirect 63 | github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect 64 | github.com/modern-go/reflect2 v1.0.2 // indirect 65 | github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect 66 | github.com/oklog/run v1.1.0 // indirect 67 | github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect 68 | github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect 69 | github.com/prometheus/client_golang v1.22.0 // indirect 70 | github.com/prometheus/client_model v0.6.2 // indirect 71 | github.com/prometheus/common v0.65.0 // indirect 72 | github.com/prometheus/procfs v0.15.1 // indirect 73 | github.com/spf13/cobra v1.8.1 // indirect 74 | github.com/stretchr/objx v0.5.2 // indirect 75 | github.com/x448/float16 v0.8.4 // indirect 76 | golang.org/x/crypto v0.40.0 // indirect 77 | golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1 // indirect 78 | golang.org/x/net v0.42.0 // indirect 79 | golang.org/x/oauth2 v0.30.0 // indirect 80 | golang.org/x/sys v0.34.0 // indirect 81 | golang.org/x/term v0.33.0 // indirect 82 | golang.org/x/text v0.27.0 // indirect 83 | golang.org/x/time v0.12.0 // indirect 84 | gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect 85 | google.golang.org/grpc v1.73.0 // indirect 86 | google.golang.org/protobuf v1.36.6 // indirect 87 | gopkg.in/inf.v0 v0.9.1 // indirect 88 | gopkg.in/yaml.v2 v2.4.0 // indirect 89 | gopkg.in/yaml.v3 v3.0.1 // indirect 90 | k8s.io/apiextensions-apiserver v0.31.3 // indirect 91 | k8s.io/client-go v0.31.3 // indirect 92 | k8s.io/klog/v2 v2.130.1 // indirect 93 | k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 // indirect 94 | k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 // indirect 95 | sigs.k8s.io/controller-runtime v0.19.3 // indirect 96 | sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect 97 | sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect 98 | sigs.k8s.io/yaml v1.4.0 // indirect 99 | ) 100 | 101 | // fixes: 102 | // * go mod tidy: cloud.google.com/go/compute/metadata: ambiguous import: found package cloud.google.com/go/compute/metadata in multiple modules: 103 | // * go list -modfile=go.mod -m -json -mod=mod all: k8s.io/kubectl@v0.0.0: invalid version: unknown revision v0.0.0 104 | replace ( 105 | cloud.google.com/go => cloud.google.com/go v0.104.0 106 | k8s.io/kubectl => k8s.io/kubectl v0.25.2 107 | ) 108 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | We as members, contributors, and leaders pledge to make participation in the Velero project and our 6 | community a harassment-free experience for everyone, regardless of age, body 7 | size, visible or invisible disability, ethnicity, sex characteristics, gender 8 | identity and expression, level of experience, education, socio-economic status, 9 | nationality, personal appearance, race, religion, or sexual identity 10 | and orientation. 11 | 12 | We pledge to act and interact in ways that contribute to an open, welcoming, 13 | diverse, inclusive, and healthy community. 14 | 15 | ## Our Standards 16 | 17 | Examples of behavior that contributes to a positive environment for our 18 | community include: 19 | 20 | * Demonstrating empathy and kindness toward other people 21 | * Being respectful of differing opinions, viewpoints, and experiences 22 | * Giving and gracefully accepting constructive feedback 23 | * Accepting responsibility and apologizing to those affected by our mistakes, 24 | and learning from the experience 25 | * Focusing on what is best not just for us as individuals, but for the 26 | overall community 27 | 28 | Examples of unacceptable behavior include: 29 | 30 | * The use of sexualized language or imagery, and sexual attention or 31 | advances of any kind 32 | * Trolling, insulting or derogatory comments, and personal or political attacks 33 | * Public or private harassment 34 | * Publishing others' private information, such as a physical or email 35 | address, without their explicit permission 36 | * Other conduct which could reasonably be considered inappropriate in a 37 | professional setting 38 | 39 | ## Enforcement Responsibilities 40 | 41 | Community leaders are responsible for clarifying and enforcing our standards of 42 | acceptable behavior and will take appropriate and fair corrective action in 43 | response to any behavior that they deem inappropriate, threatening, offensive, 44 | or harmful. 45 | 46 | Community leaders have the right and responsibility to remove, edit, or reject 47 | comments, commits, code, wiki edits, issues, and other contributions that are 48 | not aligned to this Code of Conduct, and will communicate reasons for moderation 49 | decisions when appropriate. 50 | 51 | ## Scope 52 | 53 | This Code of Conduct applies within all community spaces, and also applies when 54 | an individual is officially representing the community in public spaces. 55 | Examples of representing our community include using an official e-mail address, 56 | posting via an official social media account, or acting as an appointed 57 | representative at an online or offline event. 58 | 59 | ## Enforcement 60 | 61 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 62 | reported to the community leaders responsible for enforcement at oss-coc@vmware.com. 63 | All complaints will be reviewed and investigated promptly and fairly. 64 | 65 | All community leaders are obligated to respect the privacy and security of the 66 | reporter of any incident. 67 | 68 | ## Enforcement Guidelines 69 | 70 | Community leaders will follow these Community Impact Guidelines in determining 71 | the consequences for any action they deem in violation of this Code of Conduct: 72 | 73 | ### 1. Correction 74 | 75 | **Community Impact**: Use of inappropriate language or other behavior deemed 76 | unprofessional or unwelcome in the community. 77 | 78 | **Consequence**: A private, written warning from community leaders, providing 79 | clarity around the nature of the violation and an explanation of why the 80 | behavior was inappropriate. A public apology may be requested. 81 | 82 | ### 2. Warning 83 | 84 | **Community Impact**: A violation through a single incident or series 85 | of actions. 86 | 87 | **Consequence**: A warning with consequences for continued behavior. No 88 | interaction with the people involved, including unsolicited interaction with 89 | those enforcing the Code of Conduct, for a specified period of time. This 90 | includes avoiding interactions in community spaces as well as external channels 91 | like social media. Violating these terms may lead to a temporary or 92 | permanent ban. 93 | 94 | ### 3. Temporary Ban 95 | 96 | **Community Impact**: A serious violation of community standards, including 97 | sustained inappropriate behavior. 98 | 99 | **Consequence**: A temporary ban from any sort of interaction or public 100 | communication with the community for a specified period of time. No public or 101 | private interaction with the people involved, including unsolicited interaction 102 | with those enforcing the Code of Conduct, is allowed during this period. 103 | Violating these terms may lead to a permanent ban. 104 | 105 | ### 4. Permanent Ban 106 | 107 | **Community Impact**: Demonstrating a pattern of violation of community 108 | standards, including sustained inappropriate behavior, harassment of an 109 | individual, or aggression toward or disparagement of classes of individuals. 110 | 111 | **Consequence**: A permanent ban from any sort of public interaction within 112 | the community. 113 | 114 | ## Attribution 115 | 116 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], 117 | version 2.0, available at 118 | https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. 119 | 120 | Community Impact Guidelines were inspired by [Mozilla's code of conduct 121 | enforcement ladder](https://github.com/mozilla/diversity). 122 | 123 | [homepage]: https://www.contributor-covenant.org 124 | 125 | For answers to common questions about this code of conduct, see the FAQ at 126 | https://www.contributor-covenant.org/faq. Translations are available at 127 | https://www.contributor-covenant.org/translations. -------------------------------------------------------------------------------- /velero-plugin-for-microsoft-azure/object_store_integration_test.go: -------------------------------------------------------------------------------- 1 | //go:build integration 2 | 3 | /* 4 | Copyright 2018, 2019 the Velero contributors. 5 | 6 | Licensed under the Apache License, Version 2.0 (the "License"); 7 | you may not use this file except in compliance with the License. 8 | You may obtain a copy of the License at 9 | 10 | http://www.apache.org/licenses/LICENSE-2.0 11 | 12 | Unless required by applicable law or agreed to in writing, software 13 | distributed under the License is distributed on an "AS IS" BASIS, 14 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 15 | See the License for the specific language governing permissions and 16 | limitations under the License. 17 | */ 18 | 19 | package main 20 | 21 | import ( 22 | "fmt" 23 | "io/ioutil" 24 | "net/http" 25 | "os" 26 | "strings" 27 | "testing" 28 | 29 | "time" 30 | 31 | "github.com/sirupsen/logrus" 32 | "github.com/stretchr/testify/require" 33 | ) 34 | 35 | // Storage account and container must be created manually beforehand 36 | // To test with a shared access key the key must be set via env var AZ_STORAGE_KEY 37 | // Run with: go test -tags integration ./... 38 | func TestE2E(t *testing.T) { 39 | fmt.Println("Starting e2e test") 40 | container := "velero" 41 | blob := "folder/test" 42 | testBody := "test text" 43 | 44 | tests := []struct { 45 | scenario string 46 | config map[string]string 47 | }{ 48 | { 49 | scenario: "GetProperties + ListKeys", 50 | config: map[string]string{ 51 | storageAccountConfigKey: "velerotest", 52 | storageAccountKeyEnvVarConfigKey: "AZ_STORAGE_KEY", 53 | resourceGroupConfigKey: "saRgName", 54 | subscriptionIDConfigKey: "81d18ba6-71e1-4858-a4a4-4c527ccdd4d6", 55 | }, 56 | }, 57 | { 58 | scenario: "GetProperties + ListKeys - AAD disabled", 59 | config: map[string]string{ 60 | storageAccountConfigKey: "velerotest", 61 | storageAccountKeyEnvVarConfigKey: "AZ_STORAGE_KEY", 62 | resourceGroupConfigKey: "saRgName", 63 | subscriptionIDConfigKey: "81d18ba6-71e1-4858-a4a4-4c527ccdd4d6", 64 | useAADConfigKey: "false", 65 | }, 66 | }, 67 | { 68 | scenario: "SA URI is provided - getProperties is not called, ListKeys is used.", 69 | config: map[string]string{ 70 | storageAccountConfigKey: "velerotest", 71 | storageAccountKeyEnvVarConfigKey: "AZ_STORAGE_KEY", 72 | resourceGroupConfigKey: "saRgName", 73 | subscriptionIDConfigKey: "81d18ba6-71e1-4858-a4a4-4c527ccdd4d6", 74 | storageAccountURIConfigKey: "https://velerotest.blob.core.windows.net/", 75 | }, 76 | }, 77 | { 78 | scenario: "SA URI is provided - getProperties is not called, AAD is used", 79 | config: map[string]string{ 80 | storageAccountConfigKey: "velerotest", 81 | storageAccountKeyEnvVarConfigKey: "AZ_STORAGE_KEY", 82 | resourceGroupConfigKey: "saRgName", 83 | subscriptionIDConfigKey: "81d18ba6-71e1-4858-a4a4-4c527ccdd4d6", 84 | storageAccountURIConfigKey: "https://velerotest.blob.core.windows.net/", 85 | useAADConfigKey: "true", 86 | }, 87 | }, 88 | { 89 | scenario: "AAD and SA URI is provided - getProperties is not called, custom AAD is used", 90 | config: map[string]string{ 91 | storageAccountConfigKey: "velerotest", 92 | storageAccountKeyEnvVarConfigKey: "AZ_STORAGE_KEY", 93 | resourceGroupConfigKey: "saRgName", 94 | subscriptionIDConfigKey: "81d18ba6-71e1-4858-a4a4-4c527ccdd4d6", 95 | storageAccountURIConfigKey: "https://velerotest.blob.core.windows.net/", 96 | useAADConfigKey: "true", 97 | activeDirectoryAuthorityURIConfigKey: "https://core.windows.net" 98 | }, 99 | }, 100 | { 101 | scenario: "GetProperties + ListKeys - AAD enabled", 102 | config: map[string]string{ 103 | storageAccountConfigKey: "velerotest", 104 | storageAccountKeyEnvVarConfigKey: "AZ_STORAGE_KEY", 105 | resourceGroupConfigKey: "saRgName", 106 | subscriptionIDConfigKey: "81d18ba6-71e1-4858-a4a4-4c527ccdd4d6", 107 | useAADConfigKey: "true", 108 | }, 109 | }, 110 | } 111 | 112 | for _, test := range tests { 113 | fmt.Println("=======================================") 114 | fmt.Println("Running test: ", test.scenario) 115 | config := test.config 116 | var log = &logrus.Logger{ 117 | Out: os.Stdout, 118 | Formatter: new(logrus.TextFormatter), 119 | Hooks: make(logrus.LevelHooks), 120 | Level: logrus.DebugLevel, 121 | } 122 | 123 | store := &ObjectStore{log: log} 124 | err := store.Init(config) 125 | if err != nil { 126 | t.Fatal(err) 127 | } 128 | defer store.DeleteObject(container, blob) 129 | 130 | err = store.PutObject(container, blob, strings.NewReader(testBody)) 131 | if err != nil { 132 | t.Fatal(err) 133 | } 134 | 135 | exists, err := store.ObjectExists(container, blob) 136 | if err != nil { 137 | t.Fatal(err) 138 | } 139 | require.True(t, exists) 140 | 141 | closer, err := store.GetObject(container, blob) 142 | if err != nil { 143 | t.Fatal(err) 144 | } 145 | body, err := ioutil.ReadAll(closer) 146 | if err != nil { 147 | t.Fatal(err) 148 | } 149 | require.Equal(t, testBody, string(body)) 150 | 151 | objects, err := store.ListObjects(container, "fol") 152 | if err != nil { 153 | t.Fatal(err) 154 | } 155 | require.Equal(t, len(objects), 1) 156 | require.Equal(t, objects[0], blob) 157 | 158 | objects, err = store.ListObjects(container, "doesntexist") 159 | if err != nil { 160 | t.Fatal(err) 161 | } 162 | require.Equal(t, len(objects), 0) 163 | 164 | objects, err = store.ListCommonPrefixes(container, "fo", "/") 165 | if err != nil { 166 | t.Fatal(err) 167 | } 168 | require.Equal(t, len(objects), 1) 169 | require.Equal(t, objects[0], "folder/") 170 | 171 | url, err := store.CreateSignedURL(container, blob, 5*time.Minute) 172 | if err != nil { 173 | t.Fatal(err) 174 | } 175 | fmt.Printf("SAS URL: %s\n", url) 176 | body, err = downloadURL(url) 177 | if err != nil { 178 | t.Fatal(err) 179 | } 180 | require.Equal(t, testBody, string(body)) 181 | 182 | err = store.DeleteObject(container, blob) 183 | if err != nil { 184 | t.Fatal(err) 185 | } 186 | } 187 | } 188 | 189 | func downloadURL(url string) ([]byte, error) { 190 | resp, err := http.Get(url) 191 | if err != nil { 192 | return nil, err 193 | } 194 | defer resp.Body.Close() 195 | responseBody, err := ioutil.ReadAll(resp.Body) 196 | if err != nil { 197 | return nil, err 198 | } 199 | return responseBody, nil 200 | } 201 | -------------------------------------------------------------------------------- /velero-plugin-for-microsoft-azure/volume_snapshotter_test.go: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright 2017 the Velero contributors. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"); 5 | you may not use this file except in compliance with the License. 6 | You may obtain a copy of the License at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | Unless required by applicable law or agreed to in writing, software 11 | distributed under the License is distributed on an "AS IS" BASIS, 12 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | See the License for the specific language governing permissions and 14 | limitations under the License. 15 | */ 16 | 17 | package main 18 | 19 | import ( 20 | "testing" 21 | 22 | "github.com/sirupsen/logrus" 23 | "github.com/stretchr/testify/assert" 24 | "github.com/stretchr/testify/require" 25 | v1 "k8s.io/api/core/v1" 26 | "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" 27 | "k8s.io/apimachinery/pkg/runtime" 28 | ) 29 | 30 | func TestGetVolumeID(t *testing.T) { 31 | b := &VolumeSnapshotter{ 32 | log: logrus.New(), 33 | } 34 | 35 | pv := &unstructured.Unstructured{ 36 | Object: map[string]interface{}{}, 37 | } 38 | 39 | // missing spec.azureDisk -> no error 40 | volumeID, err := b.GetVolumeID(pv) 41 | require.NoError(t, err) 42 | assert.Equal(t, "", volumeID) 43 | 44 | // missing spec.azureDisk.diskName -> error 45 | azure := map[string]interface{}{} 46 | pv.Object["spec"] = map[string]interface{}{ 47 | "azureDisk": azure, 48 | } 49 | volumeID, err = b.GetVolumeID(pv) 50 | assert.Error(t, err) 51 | assert.Equal(t, "", volumeID) 52 | 53 | // valid 54 | azure["diskName"] = "foo" 55 | volumeID, err = b.GetVolumeID(pv) 56 | assert.NoError(t, err) 57 | assert.Equal(t, "foo", volumeID) 58 | 59 | // CSI driver: unknown driver name 60 | csi := map[string]interface{}{ 61 | "driver": "unknown.csi.azure.com", 62 | "volumeHandle": " /subscriptions/subscription-id/resourceGroups/resource-group-name/providers/Microsoft.Compute/disks/bar", 63 | } 64 | pv.Object["spec"].(map[string]interface{})["csi"] = csi 65 | volumeID, err = b.GetVolumeID(pv) 66 | assert.NoError(t, err) 67 | assert.Equal(t, "foo", volumeID) 68 | 69 | // CSI driver: pass 70 | csi = map[string]interface{}{ 71 | "driver": "disk.csi.azure.com", 72 | "volumeHandle": " /subscriptions/subscription-id/resourceGroups/resource-group-name/providers/Microsoft.Compute/disks/bar", 73 | } 74 | pv.Object["spec"].(map[string]interface{})["csi"] = csi 75 | volumeID, err = b.GetVolumeID(pv) 76 | assert.NoError(t, err) 77 | assert.Equal(t, "bar", volumeID) 78 | } 79 | 80 | func TestSetVolumeID(t *testing.T) { 81 | b := &VolumeSnapshotter{ 82 | disksResourceGroup: "rg", 83 | disksSubscription: "sub", 84 | } 85 | 86 | pv := &unstructured.Unstructured{ 87 | Object: map[string]interface{}{}, 88 | } 89 | 90 | // missing spec.azureDisk -> error 91 | updatedPV, err := b.SetVolumeID(pv, "updated") 92 | require.Error(t, err) 93 | 94 | // happy path, no diskURI 95 | azure := map[string]interface{}{} 96 | pv.Object["spec"] = map[string]interface{}{ 97 | "azureDisk": azure, 98 | } 99 | updatedPV, err = b.SetVolumeID(pv, "updated") 100 | require.NoError(t, err) 101 | 102 | res := new(v1.PersistentVolume) 103 | require.NoError(t, runtime.DefaultUnstructuredConverter.FromUnstructured(updatedPV.UnstructuredContent(), res)) 104 | require.NotNil(t, res.Spec.AzureDisk) 105 | assert.Equal(t, "updated", res.Spec.AzureDisk.DiskName) 106 | assert.Equal(t, "/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Compute/disks/updated", res.Spec.AzureDisk.DataDiskURI) 107 | 108 | // with diskURI 109 | azure["diskURI"] = "/foo/bar/updated/blarg" 110 | updatedPV, err = b.SetVolumeID(pv, "revised") 111 | require.NoError(t, err) 112 | 113 | res = new(v1.PersistentVolume) 114 | require.NoError(t, runtime.DefaultUnstructuredConverter.FromUnstructured(updatedPV.UnstructuredContent(), res)) 115 | require.NotNil(t, res.Spec.AzureDisk) 116 | assert.Equal(t, "revised", res.Spec.AzureDisk.DiskName) 117 | assert.Equal(t, "/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Compute/disks/revised", res.Spec.AzureDisk.DataDiskURI) 118 | 119 | // CSI driver: unknown driver name 120 | csi := map[string]interface{}{ 121 | "driver": "unknown.csi.azure.com", 122 | "volumeHandle": " /subscriptions/subscription-id/resourceGroups/resource-group-name/providers/Microsoft.Compute/disks/foo", 123 | } 124 | pv.Object["spec"].(map[string]interface{})["csi"] = csi 125 | _, err = b.SetVolumeID(pv, "updated") 126 | require.Error(t, err) 127 | 128 | // CSI driver: pass 129 | csi = map[string]interface{}{ 130 | "driver": "disk.csi.azure.com", 131 | "volumeHandle": " /subscriptions/subscription-id/resourceGroups/resource-group-name/providers/Microsoft.Compute/disks/foo", 132 | } 133 | pv.Object["spec"].(map[string]interface{})["csi"] = csi 134 | updatedPV, err = b.SetVolumeID(pv, "updated") 135 | require.NoError(t, err) 136 | 137 | res = new(v1.PersistentVolume) 138 | require.NoError(t, runtime.DefaultUnstructuredConverter.FromUnstructured(updatedPV.UnstructuredContent(), res)) 139 | require.NotNil(t, res.Spec.CSI) 140 | assert.Equal(t, "/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Compute/disks/updated", res.Spec.CSI.VolumeHandle) 141 | } 142 | 143 | func TestParseFullSnapshotName(t *testing.T) { 144 | // invalid name 145 | fullName := "foo/bar" 146 | _, err := parseFullSnapshotName(fullName) 147 | assert.Error(t, err) 148 | 149 | // valid name (current format) 150 | fullName = "/subscriptions/sub-1/resourceGroups/rg-1/providers/Microsoft.Compute/snapshots/snap-1" 151 | snap, err := parseFullSnapshotName(fullName) 152 | require.NoError(t, err) 153 | 154 | assert.Equal(t, "sub-1", snap.subscription) 155 | assert.Equal(t, "rg-1", snap.resourceGroup) 156 | assert.Equal(t, "snap-1", snap.name) 157 | } 158 | 159 | func TestGetComputeResourceName(t *testing.T) { 160 | assert.Equal(t, "/subscriptions/sub-1/resourceGroups/rg-1/providers/Microsoft.Compute/disks/disk-1", getComputeResourceName("sub-1", "rg-1", disksResource, "disk-1")) 161 | 162 | assert.Equal(t, "/subscriptions/sub-1/resourceGroups/rg-1/providers/Microsoft.Compute/snapshots/snap-1", getComputeResourceName("sub-1", "rg-1", snapshotsResource, "snap-1")) 163 | } 164 | 165 | func TestGetSnapshotTags(t *testing.T) { 166 | tests := []struct { 167 | name string 168 | veleroTags map[string]string 169 | snapsTags map[string]string 170 | diskTags map[string]*string 171 | expected map[string]*string 172 | }{ 173 | { 174 | name: "degenerate case (no tags)", 175 | veleroTags: nil, 176 | diskTags: nil, 177 | expected: nil, 178 | }, 179 | { 180 | name: "velero tags only get applied", 181 | veleroTags: map[string]string{ 182 | "velero-key1": "velero-val1", 183 | "velero-key2": "velero-val2", 184 | }, 185 | diskTags: nil, 186 | expected: map[string]*string{ 187 | "velero-key1": stringPtr("velero-val1"), 188 | "velero-key2": stringPtr("velero-val2"), 189 | }, 190 | }, 191 | { 192 | name: "slashes in velero tag keys get replaces with dashes", 193 | veleroTags: map[string]string{ 194 | "velero/key1": "velero-val1", 195 | "velero/key/2": "velero-val2", 196 | }, 197 | diskTags: nil, 198 | expected: map[string]*string{ 199 | "velero-key1": stringPtr("velero-val1"), 200 | "velero-key-2": stringPtr("velero-val2"), 201 | }, 202 | }, 203 | { 204 | name: "volume tags only get applied", 205 | veleroTags: nil, 206 | diskTags: map[string]*string{ 207 | "azure-key1": stringPtr("azure-val1"), 208 | "azure-key2": stringPtr("azure-val2"), 209 | }, 210 | expected: map[string]*string{ 211 | "azure-key1": stringPtr("azure-val1"), 212 | "azure-key2": stringPtr("azure-val2"), 213 | }, 214 | }, 215 | { 216 | name: "non-overlapping velero and volume tags both get applied", 217 | veleroTags: map[string]string{"velero-key": "velero-val"}, 218 | diskTags: map[string]*string{"azure-key": stringPtr("azure-val")}, 219 | expected: map[string]*string{ 220 | "velero-key": stringPtr("velero-val"), 221 | "azure-key": stringPtr("azure-val"), 222 | }, 223 | }, 224 | { 225 | name: "when tags overlap, velero tags take precedence", 226 | veleroTags: map[string]string{ 227 | "velero-key": "velero-val", 228 | "overlapping-key": "velero-val", 229 | }, 230 | diskTags: map[string]*string{ 231 | "azure-key": stringPtr("azure-val"), 232 | "overlapping-key": stringPtr("azure-val"), 233 | }, 234 | expected: map[string]*string{ 235 | "velero-key": stringPtr("velero-val"), 236 | "azure-key": stringPtr("azure-val"), 237 | "overlapping-key": stringPtr("velero-val"), 238 | }, 239 | }, 240 | { 241 | name: "velero, snapshot and volume tags all get applied", 242 | veleroTags: map[string]string{"velero-key": "velero-val"}, 243 | snapsTags: map[string]string{"snap-key": "snap-val"}, 244 | diskTags: map[string]*string{"azure-key": stringPtr("azure-val")}, 245 | expected: map[string]*string{ 246 | "velero-key": stringPtr("velero-val"), 247 | "snap-key": stringPtr("snap-val"), 248 | "azure-key": stringPtr("azure-val"), 249 | }, 250 | }, 251 | } 252 | 253 | for _, test := range tests { 254 | t.Run(test.name, func(t *testing.T) { 255 | res := getSnapshotTags(test.veleroTags, test.snapsTags, test.diskTags) 256 | 257 | if test.expected == nil { 258 | assert.Nil(t, res) 259 | return 260 | } 261 | 262 | assert.Equal(t, len(test.expected), len(res)) 263 | for k, v := range test.expected { 264 | assert.Equal(t, v, res[k]) 265 | } 266 | }) 267 | } 268 | } 269 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {yyyy} {name of copyright owner} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /velero-plugin-for-microsoft-azure/object_store.go: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright the Velero contributors. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"); 5 | you may not use this file except in compliance with the License. 6 | You may obtain a copy of the License at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | Unless required by applicable law or agreed to in writing, software 11 | distributed under the License is distributed on an "AS IS" BASIS, 12 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | See the License for the specific language governing permissions and 14 | limitations under the License. 15 | */ 16 | 17 | package main 18 | 19 | import ( 20 | "bytes" 21 | "context" 22 | "fmt" 23 | "io" 24 | "strconv" 25 | "time" 26 | 27 | "github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime" 28 | "github.com/Azure/azure-sdk-for-go/sdk/azcore/to" 29 | "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob" 30 | "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/bloberror" 31 | "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob" 32 | azcontainer "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container" 33 | "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/sas" 34 | "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/service" 35 | "github.com/pkg/errors" 36 | "github.com/sirupsen/logrus" 37 | 38 | veleroplugin "github.com/vmware-tanzu/velero/pkg/plugin/framework" 39 | "github.com/vmware-tanzu/velero/pkg/util/azure" 40 | ) 41 | 42 | const ( 43 | blockSizeConfigKey = "blockSizeInBytes" 44 | // blocks must be less than/equal to 100MB in size 45 | // ref. https://docs.microsoft.com/en-us/rest/api/storageservices/put-block#uri-parameters 46 | maxBlockSize = 100 * 1024 * 1024 47 | defaultBlockSize = 1 * 1024 * 1024 48 | ) 49 | 50 | type containerGetter interface { 51 | getContainer(bucket string) container 52 | } 53 | 54 | type azureContainerGetter struct { 55 | serviceClient *service.Client 56 | } 57 | 58 | func (cg *azureContainerGetter) getContainer(bucket string) container { 59 | containerClient := cg.serviceClient.NewContainerClient(bucket) 60 | 61 | return &azureContainer{ 62 | containerClient: containerClient, 63 | } 64 | } 65 | 66 | type container interface { 67 | ListBlobs(params *azcontainer.ListBlobsFlatOptions) *runtime.Pager[azcontainer.ListBlobsFlatResponse] 68 | ListBlobsHierarchy(delimiter string, listOptions *azcontainer.ListBlobsHierarchyOptions) *runtime.Pager[azcontainer.ListBlobsHierarchyResponse] 69 | } 70 | 71 | type azureContainer struct { 72 | containerClient *azcontainer.Client 73 | } 74 | 75 | func (c *azureContainer) ListBlobs(params *azcontainer.ListBlobsFlatOptions) *runtime.Pager[azcontainer.ListBlobsFlatResponse] { 76 | return c.containerClient.NewListBlobsFlatPager(params) 77 | } 78 | 79 | func (c *azureContainer) ListBlobsHierarchy(delimiter string, listOptions *azcontainer.ListBlobsHierarchyOptions) *runtime.Pager[azcontainer.ListBlobsHierarchyResponse] { 80 | return c.containerClient.NewListBlobsHierarchyPager(delimiter, listOptions) 81 | } 82 | 83 | type blobGetter interface { 84 | getBlob(bucket, key string) blob 85 | } 86 | 87 | type azureBlobGetter struct { 88 | serviceClient *service.Client 89 | } 90 | 91 | func (bg *azureBlobGetter) getBlob(bucket, key string) blob { 92 | containerClient := bg.serviceClient.NewContainerClient(bucket) 93 | blobClient := containerClient.NewBlockBlobClient(key) 94 | return &azureBlob{ 95 | container: bucket, 96 | blob: key, 97 | blobClient: blobClient, 98 | serviceClient: bg.serviceClient, 99 | } 100 | } 101 | 102 | type blob interface { 103 | PutBlock(blockID string, chunk []byte, options *blockblob.StageBlockOptions) error 104 | PutBlockList(blocks []string, options *blockblob.CommitBlockListOptions) error 105 | Exists() (bool, error) 106 | Get(options *azblob.DownloadStreamOptions) (io.ReadCloser, error) 107 | Delete(options *azblob.DeleteBlobOptions) error 108 | GetSASURI(duration time.Duration, sharedKeyCredential *azblob.SharedKeyCredential) (string, error) 109 | } 110 | 111 | type azureBlob struct { 112 | container string 113 | blob string 114 | blobClient *blockblob.Client 115 | serviceClient *service.Client 116 | } 117 | 118 | type nopCloser struct { 119 | io.ReadSeeker 120 | } 121 | 122 | func (n nopCloser) Close() error { 123 | return nil 124 | } 125 | 126 | // NopCloser returns a ReadSeekCloser with a no-op close method wrapping the provided io.ReadSeeker. 127 | func NopCloser(rs io.ReadSeeker) io.ReadSeekCloser { 128 | return nopCloser{rs} 129 | } 130 | 131 | func (b *azureBlob) PutBlock(blockID string, chunk []byte, options *blockblob.StageBlockOptions) error { 132 | _, err := b.blobClient.StageBlock(context.TODO(), blockID, NopCloser(bytes.NewReader(chunk)), options) 133 | return err 134 | } 135 | 136 | func (b *azureBlob) PutBlockList(blocks []string, options *blockblob.CommitBlockListOptions) error { 137 | _, err := b.blobClient.CommitBlockList(context.TODO(), blocks, options) 138 | return err 139 | } 140 | 141 | func (b *azureBlob) Exists() (bool, error) { 142 | _, err := b.blobClient.GetProperties(context.TODO(), nil) 143 | if err == nil { 144 | return true, nil 145 | } 146 | if bloberror.HasCode(err, bloberror.ContainerNotFound, bloberror.BlobNotFound) { 147 | return false, nil 148 | } 149 | return false, err 150 | } 151 | 152 | func (b *azureBlob) Get(options *azblob.DownloadStreamOptions) (io.ReadCloser, error) { 153 | res, err := b.blobClient.BlobClient().DownloadStream(context.TODO(), options) 154 | if err != nil { 155 | return nil, errors.WithStack(err) 156 | } 157 | return res.Body, nil 158 | } 159 | 160 | func (b *azureBlob) Delete(options *azblob.DeleteBlobOptions) error { 161 | _, err := b.blobClient.Delete(context.TODO(), options) 162 | return err 163 | } 164 | 165 | // When the sharedKeyCredential is provided service SAS is used else delegation SAS is used 166 | func (b *azureBlob) GetSASURI(ttl time.Duration, sharedKeyCredential *azblob.SharedKeyCredential) (string, error) { 167 | var queryParam sas.QueryParameters 168 | var err error 169 | // because of clock skew it can happen that the token is not yet valid, so make it valid in the past 170 | startTime := time.Now().Add(-10 * time.Minute).UTC() 171 | expiryTime := time.Now().Add(ttl).UTC() 172 | blobSignatureValues := sas.BlobSignatureValues{ 173 | ContainerName: b.container, 174 | BlobName: b.blob, 175 | Protocol: sas.ProtocolHTTPS, 176 | StartTime: startTime, 177 | ExpiryTime: expiryTime, 178 | Permissions: to.Ptr(sas.BlobPermissions{Read: true}).String(), 179 | } 180 | 181 | if sharedKeyCredential == nil { 182 | var udc *service.UserDelegationCredential 183 | info := service.KeyInfo{ 184 | Start: to.Ptr(startTime.Format(sas.TimeFormat)), 185 | Expiry: to.Ptr(expiryTime.Format(sas.TimeFormat)), 186 | } 187 | udc, err = b.serviceClient.GetUserDelegationCredential(context.TODO(), info, nil) 188 | 189 | if err != nil { 190 | return "", err 191 | } 192 | queryParam, err = blobSignatureValues.SignWithUserDelegation(udc) 193 | } else { 194 | queryParam, err = blobSignatureValues.SignWithSharedKey(sharedKeyCredential) 195 | } 196 | if err != nil { 197 | return "", err 198 | } 199 | 200 | url := fmt.Sprintf("%s?%s", b.blobClient.URL(), queryParam.Encode()) 201 | return url, nil 202 | } 203 | 204 | type ObjectStore struct { 205 | log logrus.FieldLogger 206 | 207 | containerGetter containerGetter 208 | blobGetter blobGetter 209 | blockSize int 210 | // we need to keep the credential here to create the sas url 211 | sharedKeyCredential *azblob.SharedKeyCredential 212 | } 213 | 214 | func newObjectStore(logger logrus.FieldLogger) *ObjectStore { 215 | return &ObjectStore{log: logger} 216 | } 217 | 218 | // Init sets up the ObjectStore using the shared key or default azure credentials 219 | func (o *ObjectStore) Init(config map[string]string) error { 220 | if err := veleroplugin.ValidateObjectStoreConfigKeys(config, 221 | azure.BSLConfigResourceGroup, 222 | azure.BSLConfigStorageAccount, 223 | azure.BSLConfigSubscriptionID, 224 | blockSizeConfigKey, 225 | azure.BSLConfigActiveDirectoryAuthorityURI, 226 | azure.BSLConfigStorageAccountURI, 227 | azure.BSLConfigUseAAD, 228 | azure.BSLConfigStorageAccountAccessKeyName, 229 | credentialsFileConfigKey, 230 | ); err != nil { 231 | return err 232 | } 233 | 234 | client, cred, err := azure.NewStorageClient(o.log, config) 235 | if err != nil { 236 | return err 237 | } 238 | o.sharedKeyCredential = cred 239 | 240 | o.containerGetter = &azureContainerGetter{ 241 | serviceClient: client.ServiceClient(), 242 | } 243 | o.blobGetter = &azureBlobGetter{ 244 | serviceClient: client.ServiceClient(), 245 | } 246 | o.blockSize = getBlockSize(o.log, config) 247 | return nil 248 | } 249 | 250 | func getBlockSize(log logrus.FieldLogger, config map[string]string) int { 251 | val, ok := config[blockSizeConfigKey] 252 | if !ok { 253 | // no alternate block size specified in config, so return with the default 254 | return defaultBlockSize 255 | } 256 | 257 | blockSize, err := strconv.Atoi(val) 258 | if err != nil { 259 | log.WithError(err).Warnf("Error parsing config.blockSizeInBytes value %v, using default block size of %d", val, defaultBlockSize) 260 | return defaultBlockSize 261 | } 262 | 263 | if blockSize <= 0 { 264 | log.WithError(err).Warnf("Value provided for config.blockSizeInBytes (%d) is < 1, using default block size of %d", blockSize, defaultBlockSize) 265 | return defaultBlockSize 266 | } 267 | 268 | if blockSize > maxBlockSize { 269 | log.WithError(err).Warnf("Value provided for config.blockSizeInBytes (%d) is > the max size %d, using max block size of %d", blockSize, maxBlockSize, maxBlockSize) 270 | return maxBlockSize 271 | } 272 | 273 | return blockSize 274 | } 275 | 276 | func (o *ObjectStore) PutObject(bucket, key string, body io.Reader) error { 277 | blob := o.blobGetter.getBlob(bucket, key) 278 | // Azure requires a blob/object to be chunked if it's larger than 256MB. Since we 279 | // don't know ahead of time if the body is over this limit or not, and it would 280 | // require reading the entire object into memory to determine the size, we use the 281 | // chunking approach for all objects. 282 | var ( 283 | block = make([]byte, o.blockSize) 284 | blockIDs []string 285 | ) 286 | 287 | for { 288 | n, err := body.Read(block) 289 | if n > 0 { 290 | // blockID needs to be the same length for all blocks, so use a fixed width. 291 | // ref. https://docs.microsoft.com/en-us/rest/api/storageservices/put-block#uri-parameters 292 | blockID := fmt.Sprintf("%08d", len(blockIDs)) 293 | 294 | o.log.Debugf("Putting block (id=%s) of length %d", blockID, n) 295 | if putErr := blob.PutBlock(blockID, block[0:n], nil); putErr != nil { 296 | return errors.Wrapf(putErr, "error putting block %s", blockID) 297 | } 298 | 299 | blockIDs = append(blockIDs, blockID) 300 | } 301 | 302 | // got an io.EOF: we're done reading chunks from the body 303 | if err == io.EOF { 304 | break 305 | } 306 | // any other error: bubble it up 307 | if err != nil { 308 | return errors.Wrap(err, "error reading block from body") 309 | } 310 | } 311 | 312 | o.log.Debugf("Putting block list %v", blockIDs) 313 | if err := blob.PutBlockList(blockIDs, nil); err != nil { 314 | return errors.Wrap(err, "error putting block list") 315 | } 316 | 317 | return nil 318 | } 319 | 320 | func (o *ObjectStore) ObjectExists(bucket, key string) (bool, error) { 321 | blob := o.blobGetter.getBlob(bucket, key) 322 | exists, err := blob.Exists() 323 | if err != nil { 324 | return false, errors.WithStack(err) 325 | } 326 | 327 | return exists, nil 328 | } 329 | 330 | func (o *ObjectStore) GetObject(bucket, key string) (io.ReadCloser, error) { 331 | blob := o.blobGetter.getBlob(bucket, key) 332 | return blob.Get(nil) 333 | } 334 | 335 | func (o *ObjectStore) ListCommonPrefixes(bucket, prefix, delimiter string) ([]string, error) { 336 | container := o.containerGetter.getContainer(bucket) 337 | params := azcontainer.ListBlobsHierarchyOptions{ 338 | Prefix: &prefix, 339 | } 340 | 341 | var prefixes []string 342 | pager := container.ListBlobsHierarchy(delimiter, ¶ms) 343 | for pager.More() { 344 | page, err := pager.NextPage(context.TODO()) 345 | if err != nil { 346 | return nil, err 347 | } 348 | 349 | for _, prefix := range page.ListBlobsHierarchySegmentResponse.Segment.BlobPrefixes { 350 | prefixes = append(prefixes, *prefix.Name) 351 | } 352 | } 353 | 354 | return prefixes, nil 355 | } 356 | 357 | func (o *ObjectStore) ListObjects(bucket, prefix string) ([]string, error) { 358 | container := o.containerGetter.getContainer(bucket) 359 | params := azcontainer.ListBlobsFlatOptions{ 360 | Prefix: &prefix, 361 | } 362 | 363 | var objects []string 364 | pager := container.ListBlobs(¶ms) 365 | for pager.More() { 366 | page, err := pager.NextPage(context.TODO()) 367 | if err != nil { 368 | return nil, err 369 | } 370 | 371 | for _, blob := range page.ListBlobsFlatSegmentResponse.Segment.BlobItems { 372 | objects = append(objects, *blob.Name) 373 | } 374 | } 375 | return objects, nil 376 | } 377 | 378 | func (o *ObjectStore) DeleteObject(bucket string, key string) error { 379 | blob := o.blobGetter.getBlob(bucket, key) 380 | err := blob.Delete(nil) 381 | return errors.WithStack(err) 382 | } 383 | 384 | func (o *ObjectStore) CreateSignedURL(bucket, key string, ttl time.Duration) (string, error) { 385 | blob := o.blobGetter.getBlob(bucket, key) 386 | return blob.GetSASURI(ttl, o.sharedKeyCredential) 387 | } 388 | -------------------------------------------------------------------------------- /velero-plugin-for-microsoft-azure/volume_snapshotter.go: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright the Velero contributors. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"); 5 | you may not use this file except in compliance with the License. 6 | You may obtain a copy of the License at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | Unless required by applicable law or agreed to in writing, software 11 | distributed under the License is distributed on an "AS IS" BASIS, 12 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | See the License for the specific language governing permissions and 14 | limitations under the License. 15 | */ 16 | 17 | package main 18 | 19 | import ( 20 | "context" 21 | "fmt" 22 | "net/http" 23 | "regexp" 24 | "strconv" 25 | "strings" 26 | "time" 27 | 28 | "github.com/Azure/azure-sdk-for-go/sdk/azcore" 29 | "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm" 30 | azruntime "github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime" 31 | "github.com/Azure/azure-sdk-for-go/sdk/azcore/to" 32 | "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v4" 33 | uuid "github.com/gofrs/uuid" 34 | "github.com/pkg/errors" 35 | "github.com/sirupsen/logrus" 36 | veleroplugin "github.com/vmware-tanzu/velero/pkg/plugin/framework" 37 | "github.com/vmware-tanzu/velero/pkg/util/azure" 38 | v1 "k8s.io/api/core/v1" 39 | "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" 40 | "k8s.io/apimachinery/pkg/runtime" 41 | "sigs.k8s.io/azuredisk-csi-driver/pkg/util" 42 | ) 43 | 44 | const ( 45 | credentialsFileConfigKey = "credentialsFile" 46 | 47 | vslConfigKeyActiveDirectoryAuthorityURI = "activeDirectoryAuthorityURI" 48 | vslConfigKeySubscriptionID = "subscriptionId" 49 | vslConfigKeyResourceGroup = "resourceGroup" 50 | vslConfigKeyAPITimeout = "apiTimeout" 51 | vslConfigKeyIncremental = "incremental" 52 | vslConfigKeyTags = "tags" 53 | 54 | snapshotsResource = "snapshots" 55 | disksResource = "disks" 56 | 57 | diskCSIDriver = "disk.csi.azure.com" 58 | pollingDelay = 5 * time.Second 59 | ) 60 | 61 | type VolumeSnapshotter struct { 62 | log logrus.FieldLogger 63 | disks *armcompute.DisksClient 64 | snaps *armcompute.SnapshotsClient 65 | disksSubscription string 66 | snapsSubscription string 67 | disksResourceGroup string 68 | snapsResourceGroup string 69 | snapsIncremental *bool 70 | apiTimeout time.Duration 71 | snapsTags map[string]string 72 | } 73 | 74 | type snapshotIdentifier struct { 75 | subscription string 76 | resourceGroup string 77 | name string 78 | } 79 | 80 | func (si *snapshotIdentifier) String() string { 81 | return getComputeResourceName(si.subscription, si.resourceGroup, snapshotsResource, si.name) 82 | } 83 | 84 | func newVolumeSnapshotter(logger logrus.FieldLogger) *VolumeSnapshotter { 85 | return &VolumeSnapshotter{log: logger} 86 | } 87 | 88 | func (b *VolumeSnapshotter) Init(config map[string]string) error { 89 | if err := veleroplugin.ValidateVolumeSnapshotterConfigKeys(config, 90 | vslConfigKeyResourceGroup, 91 | vslConfigKeyAPITimeout, 92 | vslConfigKeySubscriptionID, 93 | vslConfigKeyIncremental, 94 | vslConfigKeyTags, 95 | credentialsFileConfigKey, 96 | ); err != nil { 97 | return err 98 | } 99 | 100 | creds, err := azure.LoadCredentials(config) 101 | if err != nil { 102 | return err 103 | } 104 | b.disksSubscription = creds[azure.CredentialKeySubscriptionID] 105 | if b.disksSubscription == "" { 106 | return errors.Errorf("%s is required in credential file", azure.CredentialKeySubscriptionID) 107 | } 108 | b.disksResourceGroup = creds[azure.CredentialKeyResourceGroup] 109 | if b.disksResourceGroup == "" { 110 | return errors.Errorf("%s is required in credential file", azure.CredentialKeyResourceGroup) 111 | } 112 | 113 | b.snapsSubscription = azure.GetFromLocationConfigOrCredential(config, creds, vslConfigKeySubscriptionID, azure.CredentialKeySubscriptionID) 114 | b.snapsResourceGroup = azure.GetFromLocationConfigOrCredential(config, creds, vslConfigKeyResourceGroup, azure.CredentialKeyResourceGroup) 115 | 116 | b.apiTimeout = 2 * time.Minute 117 | if val := config[vslConfigKeyAPITimeout]; val != "" { 118 | b.apiTimeout, err = time.ParseDuration(val) 119 | if err != nil { 120 | return errors.Wrapf(err, "unable to parse value %q for config key %q (expected a duration string)", val, vslConfigKeyAPITimeout) 121 | } 122 | } 123 | 124 | if val := config[vslConfigKeyIncremental]; val != "" { 125 | parseIncremental, err := strconv.ParseBool(val) 126 | if err != nil { 127 | return errors.Wrapf(err, "unable to parse value %q for config key %q (expected a boolean value)", val, vslConfigKeyIncremental) 128 | } 129 | b.snapsIncremental = &parseIncremental 130 | } 131 | 132 | if val := config[vslConfigKeyTags]; val != "" { 133 | b.snapsTags, err = util.ConvertTagsToMap(val) 134 | if err != nil { 135 | return errors.Wrapf(err, "unable to parse value %q for config key %q (the valid format is \"key1=value1,key2=value2\")", val, vslConfigKeyTags) 136 | } 137 | } 138 | 139 | clientOptions, err := azure.GetClientOptions(config, creds) 140 | if err != nil { 141 | return err 142 | } 143 | credential, err := azure.NewCredential(creds, clientOptions) 144 | if err != nil { 145 | return err 146 | } 147 | 148 | b.disks, err = armcompute.NewDisksClient(b.disksSubscription, credential, &arm.ClientOptions{ClientOptions: clientOptions}) 149 | if err != nil { 150 | return errors.Wrap(err, "error creating disk client") 151 | } 152 | 153 | b.snaps, err = armcompute.NewSnapshotsClient(b.snapsSubscription, credential, &arm.ClientOptions{ClientOptions: clientOptions}) 154 | if err != nil { 155 | return errors.Wrap(err, "error creating snapshot client") 156 | } 157 | 158 | return nil 159 | } 160 | 161 | func (b *VolumeSnapshotter) CreateVolumeFromSnapshot(snapshotID, volumeType, volumeAZ string, iops *int64) (string, error) { 162 | snapshotIdentifier, err := parseFullSnapshotName(snapshotID) 163 | diskStorageAccountType := armcompute.DiskStorageAccountTypes(volumeType) 164 | if err != nil { 165 | return "", err 166 | } 167 | 168 | // Lookup snapshot info for its Location & Tags so we can apply them to the volume 169 | snapshotInfo, err := b.snaps.Get(context.TODO(), snapshotIdentifier.resourceGroup, snapshotIdentifier.name, nil) 170 | if err != nil { 171 | return "", errors.WithStack(err) 172 | } 173 | 174 | uid, err := uuid.NewV4() 175 | if err != nil { 176 | return "", errors.WithStack(err) 177 | } 178 | diskName := "restore-" + uid.String() 179 | 180 | disk := armcompute.Disk{ 181 | Name: &diskName, 182 | Location: snapshotInfo.Location, 183 | Properties: &armcompute.DiskProperties{ 184 | CreationData: &armcompute.CreationData{ 185 | CreateOption: to.Ptr(armcompute.DiskCreateOptionCopy), 186 | SourceResourceID: to.Ptr(snapshotIdentifier.String()), 187 | }, 188 | }, 189 | SKU: &armcompute.DiskSKU{ 190 | Name: to.Ptr(diskStorageAccountType), 191 | }, 192 | Tags: snapshotInfo.Tags, 193 | } 194 | // If not a volume type 'zone redundant storage' restore the disk in the correct zone 195 | if diskStorageAccountType != armcompute.DiskStorageAccountTypesPremiumZRS && diskStorageAccountType != armcompute.DiskStorageAccountTypesStandardSSDZRS { 196 | regionParts := strings.Split(volumeAZ, "-") 197 | if len(regionParts) >= 2 { 198 | disk.Zones = []*string{to.Ptr(regionParts[len(regionParts)-1])} 199 | } 200 | } 201 | 202 | ctx, cancel := context.WithTimeout(context.Background(), b.apiTimeout) 203 | defer cancel() 204 | 205 | pollerResp, err := b.disks.BeginCreateOrUpdate(ctx, b.disksResourceGroup, *disk.Name, disk, nil) 206 | if err != nil { 207 | return "", errors.WithStack(err) 208 | } 209 | _, err = pollerResp.PollUntilDone(ctx, &azruntime.PollUntilDoneOptions{Frequency: pollingDelay}) 210 | if err != nil { 211 | return "", errors.WithStack(err) 212 | } 213 | return diskName, nil 214 | } 215 | 216 | func (b *VolumeSnapshotter) GetVolumeInfo(volumeID, volumeAZ string) (string, *int64, error) { 217 | res, err := b.disks.Get(context.TODO(), b.disksResourceGroup, volumeID, nil) 218 | if err != nil { 219 | return "", nil, errors.WithStack(err) 220 | } 221 | 222 | if res.SKU == nil { 223 | return "", nil, errors.New("disk has a nil SKU") 224 | } 225 | 226 | return string(*res.SKU.Name), nil, nil 227 | } 228 | 229 | func (b *VolumeSnapshotter) CreateSnapshot(volumeID, volumeAZ string, tags map[string]string) (string, error) { 230 | // Lookup disk info for its Location 231 | diskInfo, err := b.disks.Get(context.TODO(), b.disksResourceGroup, volumeID, nil) 232 | if err != nil { 233 | return "", errors.WithStack(err) 234 | } 235 | 236 | fullDiskName := getComputeResourceName(b.disksSubscription, b.disksResourceGroup, disksResource, volumeID) 237 | // snapshot names must be <= 80 characters long 238 | var snapshotName string 239 | uid, err := uuid.NewV4() 240 | if err != nil { 241 | return "", errors.WithStack(err) 242 | } 243 | suffix := "-" + uid.String() 244 | 245 | if len(volumeID) <= (80 - len(suffix)) { 246 | snapshotName = volumeID + suffix 247 | } else { 248 | snapshotName = volumeID[0:80-len(suffix)] + suffix 249 | } 250 | 251 | snap := armcompute.Snapshot{ 252 | Name: &snapshotName, 253 | Properties: &armcompute.SnapshotProperties{ 254 | CreationData: &armcompute.CreationData{ 255 | CreateOption: to.Ptr(armcompute.DiskCreateOptionCopy), 256 | SourceResourceID: &fullDiskName, 257 | }, 258 | Incremental: b.snapsIncremental, 259 | }, 260 | Tags: getSnapshotTags(tags, b.snapsTags, diskInfo.Tags), 261 | Location: diskInfo.Location, 262 | } 263 | 264 | ctx, cancel := context.WithTimeout(context.Background(), b.apiTimeout) 265 | defer cancel() 266 | 267 | pollerResp, err := b.snaps.BeginCreateOrUpdate(ctx, b.snapsResourceGroup, *snap.Name, snap, nil) 268 | if err != nil { 269 | return "", errors.WithStack(err) 270 | } 271 | _, err = pollerResp.PollUntilDone(ctx, &azruntime.PollUntilDoneOptions{Frequency: pollingDelay}) 272 | if err != nil { 273 | return "", errors.WithStack(err) 274 | } 275 | return getComputeResourceName(b.snapsSubscription, b.snapsResourceGroup, snapshotsResource, snapshotName), nil 276 | } 277 | 278 | func getSnapshotTags(veleroTags, snapsTags map[string]string, diskTags map[string]*string) map[string]*string { 279 | if diskTags == nil && len(veleroTags) == 0 && len(snapsTags) == 0 { 280 | return nil 281 | } 282 | 283 | snapshotTags := make(map[string]*string) 284 | 285 | // copy tags from disk to snapshot 286 | for k, v := range diskTags { 287 | snapshotTags[k] = stringPtr(*v) 288 | } 289 | 290 | // merge Velero-assigned tags with the disk's tags (note that we want current 291 | // Velero-assigned tags to overwrite any older versions of them that may exist 292 | // due to prior snapshots/restores) 293 | for k, v := range veleroTags { 294 | // Azure does not allow slashes in tag keys, so replace 295 | // with dash (inline with what Kubernetes does) 296 | key := strings.Replace(k, "/", "-", -1) 297 | snapshotTags[key] = stringPtr(v) 298 | } 299 | 300 | for k, v := range snapsTags { 301 | snapshotTags[k] = stringPtr(v) 302 | } 303 | 304 | return snapshotTags 305 | } 306 | 307 | func stringPtr(s string) *string { 308 | return &s 309 | } 310 | 311 | func (b *VolumeSnapshotter) DeleteSnapshot(snapshotID string) error { 312 | snapshotInfo, err := parseFullSnapshotName(snapshotID) 313 | if err != nil { 314 | return err 315 | } 316 | 317 | ctx, cancel := context.WithTimeout(context.Background(), b.apiTimeout) 318 | defer cancel() 319 | 320 | // we don't want to return an error if the snapshot doesn't exist, and 321 | // the Delete(..) call does not return a clear error if that's the case, 322 | // so first try to get it and return early if we get a 404. 323 | _, err = b.snaps.Get(ctx, snapshotInfo.resourceGroup, snapshotInfo.name, nil) 324 | if azureErr, ok := err.(*azcore.ResponseError); ok && azureErr.StatusCode == http.StatusNotFound { 325 | b.log.WithField("snapshotID", snapshotID).Debug("Snapshot not found") 326 | return nil 327 | } 328 | 329 | pollerResp, err := b.snaps.BeginDelete(ctx, snapshotInfo.resourceGroup, snapshotInfo.name, nil) 330 | if err != nil { 331 | return errors.WithStack(err) 332 | } 333 | _, err = pollerResp.PollUntilDone(ctx, &azruntime.PollUntilDoneOptions{Frequency: pollingDelay}) 334 | if err != nil { 335 | return errors.WithStack(err) 336 | } 337 | 338 | return nil 339 | } 340 | 341 | func getComputeResourceName(subscription, resourceGroup, resource, name string) string { 342 | return fmt.Sprintf("/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Compute/%s/%s", subscription, resourceGroup, resource, name) 343 | } 344 | 345 | var ( 346 | snapshotURIRegexp = regexp.MustCompile( 347 | `^\/subscriptions\/(?P.*)\/resourceGroups\/(?P.*)\/providers\/Microsoft.Compute\/snapshots\/(?P.*)$`) 348 | diskURIRegexp = regexp.MustCompile(`\/Microsoft.Compute\/disks\/.*$`) 349 | ) 350 | 351 | // parseFullSnapshotName takes a fully-qualified snapshot name and returns 352 | // a snapshot identifier or an error if the snapshot name does not match the 353 | // regexp. 354 | func parseFullSnapshotName(name string) (*snapshotIdentifier, error) { 355 | submatches := snapshotURIRegexp.FindStringSubmatch(name) 356 | if len(submatches) != len(snapshotURIRegexp.SubexpNames()) { 357 | return nil, errors.New("snapshot URI could not be parsed") 358 | } 359 | 360 | snapshotID := &snapshotIdentifier{} 361 | 362 | // capture names start at index 1 to line up with the corresponding indexes 363 | // of submatches (see godoc on SubexpNames()) 364 | for i, names := 1, snapshotURIRegexp.SubexpNames(); i < len(names); i++ { 365 | switch names[i] { 366 | case "subscription": 367 | snapshotID.subscription = submatches[i] 368 | case "resourceGroup": 369 | snapshotID.resourceGroup = submatches[i] 370 | case "snapshotName": 371 | snapshotID.name = submatches[i] 372 | default: 373 | return nil, errors.New("unexpected named capture from snapshot URI regex") 374 | } 375 | } 376 | 377 | return snapshotID, nil 378 | } 379 | 380 | func (b *VolumeSnapshotter) GetVolumeID(unstructuredPV runtime.Unstructured) (string, error) { 381 | pv := new(v1.PersistentVolume) 382 | if err := runtime.DefaultUnstructuredConverter.FromUnstructured(unstructuredPV.UnstructuredContent(), pv); err != nil { 383 | return "", errors.WithStack(err) 384 | } 385 | 386 | if pv.Spec.CSI != nil { 387 | if pv.Spec.CSI.Driver == diskCSIDriver { 388 | return strings.TrimPrefix(diskURIRegexp.FindString(pv.Spec.CSI.VolumeHandle), "/Microsoft.Compute/disks/"), nil 389 | } 390 | b.log.Infof("Unable to handle CSI driver: %s", pv.Spec.CSI.Driver) 391 | } 392 | 393 | if pv.Spec.AzureDisk == nil { 394 | return "", nil 395 | } 396 | 397 | if pv.Spec.AzureDisk.DiskName == "" { 398 | return "", errors.New("spec.azureDisk.diskName not found") 399 | } 400 | 401 | return pv.Spec.AzureDisk.DiskName, nil 402 | } 403 | 404 | func (b *VolumeSnapshotter) SetVolumeID(unstructuredPV runtime.Unstructured, volumeID string) (runtime.Unstructured, error) { 405 | pv := new(v1.PersistentVolume) 406 | if err := runtime.DefaultUnstructuredConverter.FromUnstructured(unstructuredPV.UnstructuredContent(), pv); err != nil { 407 | return nil, errors.WithStack(err) 408 | } 409 | 410 | if pv.Spec.CSI != nil { 411 | if pv.Spec.CSI.Driver == diskCSIDriver { 412 | pv.Spec.CSI.VolumeHandle = getComputeResourceName(b.disksSubscription, b.disksResourceGroup, disksResource, volumeID) 413 | } else { 414 | return nil, fmt.Errorf("unable to handle CSI driver: %s", pv.Spec.CSI.Driver) 415 | } 416 | 417 | } else if pv.Spec.AzureDisk != nil { 418 | pv.Spec.AzureDisk.DiskName = volumeID 419 | pv.Spec.AzureDisk.DataDiskURI = getComputeResourceName(b.disksSubscription, b.disksResourceGroup, disksResource, volumeID) 420 | } else { 421 | return nil, errors.New("spec.csi and spec.azureDisk not found") 422 | } 423 | 424 | res, err := runtime.DefaultUnstructuredConverter.ToUnstructured(pv) 425 | if err != nil { 426 | return nil, errors.WithStack(err) 427 | } 428 | 429 | return &unstructured.Unstructured{Object: res}, nil 430 | } 431 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/Azure/azure-sdk-for-go v67.2.0+incompatible h1:Uu/Ww6ernvPTrpq31kITVTIm/I5jlJ1wjtEH/bmSB2k= 2 | github.com/Azure/azure-sdk-for-go/sdk/azcore v1.18.1 h1:Wc1ml6QlJs2BHQ/9Bqu1jiyggbsSjramq2oUmp5WeIo= 3 | github.com/Azure/azure-sdk-for-go/sdk/azcore v1.18.1/go.mod h1:Ot/6aikWnKWi4l9QB7qVSwa8iMphQNqkWALMoNT3rzM= 4 | github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.10.1 h1:B+blDbyVIG3WaikNxPnhPiJ1MThR03b3vKGtER95TP4= 5 | github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.10.1/go.mod h1:JdM5psgjfBf5fo2uWOZhflPWyDBZ/O/CNAH9CtsuZE4= 6 | github.com/Azure/azure-sdk-for-go/sdk/azidentity/cache v0.3.2 h1:yz1bePFlP5Vws5+8ez6T3HWXPmwOK7Yvq8QxDBD3SKY= 7 | github.com/Azure/azure-sdk-for-go/sdk/azidentity/cache v0.3.2/go.mod h1:Pa9ZNPuoNu/GztvBSKk9J1cDJW6vk/n0zLtV4mgd8N8= 8 | github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.1 h1:FPKJS1T+clwv+OLGt13a8UjqeRuh0O4SJ3lUriThc+4= 9 | github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.1/go.mod h1:j2chePtV91HrC22tGoRX3sGY42uF13WzmmV80/OdVAA= 10 | github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v4 v4.2.1 h1:UPeCRD+XY7QlaGQte2EVI2iOcWvUYA2XY8w5T/8v0NQ= 11 | github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v4 v4.2.1/go.mod h1:oGV6NlB0cvi1ZbYRR2UN44QHxWFyGk+iylgD0qaMXjA= 12 | github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal v1.1.2 h1:mLY+pNLjCUeKhgnAJWAKhEUQM+RJQo2H1fuGSw1Ky1E= 13 | github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal v1.1.2/go.mod h1:FbdwsQ2EzwvXxOPcMFYO8ogEc9uMMIj3YkmCdXdAFmk= 14 | github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal/v3 v3.1.0 h1:2qsIIvxVT+uE6yrNldntJKlLRgxGbZ85kgtz5SNBhMw= 15 | github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal/v3 v3.1.0/go.mod h1:AW8VEadnhw9xox+VaVd9sP7NjzOAnaZBLRH6Tq3cJ38= 16 | github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork v1.0.0 h1:nBy98uKOIfun5z6wx6jwWLrULcM0+cjBalBFZlEZ7CA= 17 | github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork v1.0.0/go.mod h1:243D9iHbcQXoFUtgHJwL7gl2zx1aDuDMjvBZVGr2uW0= 18 | github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.2.0 h1:Dd+RhdJn0OTtVGaeDLZpcumkIVCtA/3/Fo42+eoYvVM= 19 | github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.2.0/go.mod h1:5kakwfW5CjC9KK+Q4wjXAg+ShuIm2mBMua0ZFj2C8PE= 20 | github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.0 h1:LR0kAX9ykz8G4YgLCaRDVJ3+n43R8MneB5dTy2konZo= 21 | github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.0/go.mod h1:DWAciXemNf++PQJLeXUB4HHH5OpsAh12HZnu2wXE1jA= 22 | github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.1 h1:lhZdRq7TIx0GJQvSyX2Si406vrYsov2FXGp/RnSEtcs= 23 | github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.1/go.mod h1:8cl44BDmi+effbARHMQjgOKA2AYvcohNm7KEt42mSV8= 24 | github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1 h1:WJTmL004Abzc5wDB5VtZG2PJk5ndYDgVacGqfirKxjM= 25 | github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1/go.mod h1:tCcJZ0uHAmvjsVYzEFivsRTN00oz5BEsRgQHu5JZ9WE= 26 | github.com/AzureAD/microsoft-authentication-library-for-go v1.4.2 h1:oygO0locgZJe7PpYPXT5A29ZkwJaPqcva7BVeemZOZs= 27 | github.com/AzureAD/microsoft-authentication-library-for-go v1.4.2/go.mod h1:wP83P5OoQ5p6ip3ScPr0BAq0BvuPAvacpEuSzyouqAI= 28 | github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= 29 | github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= 30 | github.com/bufbuild/protocompile v0.4.0 h1:LbFKd2XowZvQ/kajzguUp2DC9UEIQhIq77fZZlaQsNA= 31 | github.com/bufbuild/protocompile v0.4.0/go.mod h1:3v93+mbWn/v3xzN+31nwkJfrEpAUwp+BagBSZWx+TP8= 32 | github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= 33 | github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= 34 | github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= 35 | github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= 36 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 37 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 38 | github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= 39 | github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 40 | github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78= 41 | github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc= 42 | github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g= 43 | github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc= 44 | github.com/evanphx/json-patch v5.6.0+incompatible h1:jBYDEEiFBPxA0v50tFdvOzQQTCvpL6mnFh5mB2/l16U= 45 | github.com/evanphx/json-patch v5.6.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= 46 | github.com/evanphx/json-patch/v5 v5.9.0 h1:kcBlZQbplgElYIlo/n1hJbls2z/1awpXxpRi0/FOJfg= 47 | github.com/evanphx/json-patch/v5 v5.9.0/go.mod h1:VNkHZ/282BpEyt/tObQO8s5CMPmYYq14uClGH4abBuQ= 48 | github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk= 49 | github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM= 50 | github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU= 51 | github.com/fxamacker/cbor/v2 v2.7.0 h1:iM5WgngdRBanHcxugY4JySA0nk1wZorNOpTgCMedv5E= 52 | github.com/fxamacker/cbor/v2 v2.7.0/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXEeXQVeNoDQ= 53 | github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= 54 | github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= 55 | github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= 56 | github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= 57 | github.com/go-logr/zapr v1.3.0 h1:XGdV8XW8zdwFiwOA2Dryh1gj2KRQyOOoNmBy4EplIcQ= 58 | github.com/go-logr/zapr v1.3.0/go.mod h1:YKepepNBd1u/oyhd/yQmtjVXmm9uML4IXUgMOwR8/Gg= 59 | github.com/go-openapi/jsonpointer v0.19.6 h1:eCs3fxoIi3Wh6vtgmLTOjdhSpiqphQ+DaPn38N2ZdrE= 60 | github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs= 61 | github.com/go-openapi/jsonreference v0.20.2 h1:3sVjiK66+uXK/6oQ8xgcRKcFgQ5KXa2KvnJRumpMGbE= 62 | github.com/go-openapi/jsonreference v0.20.2/go.mod h1:Bl1zwGIM8/wsvqjsOQLJ/SH+En5Ap4rVB5KVcIDZG2k= 63 | github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14= 64 | github.com/go-openapi/swag v0.22.4 h1:QLMzNJnMGPRNDCbySlcj1x01tzU8/9LTTL9hZZZogBU= 65 | github.com/go-openapi/swag v0.22.4/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14= 66 | github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI= 67 | github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8= 68 | github.com/gobwas/glob v0.2.3 h1:A4xDbljILXROh+kObIiy5kIaPYD8e96x1tgBhUI5J+Y= 69 | github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJAkT8= 70 | github.com/gofrs/uuid v4.3.1+incompatible h1:0/KbAdpx3UXAx1kEOWHJeOkpbgRFGHVgv+CFIY7dBJI= 71 | github.com/gofrs/uuid v4.3.1+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM= 72 | github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= 73 | github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= 74 | github.com/golang-jwt/jwt/v5 v5.2.2 h1:Rl4B7itRWVtYIHFrSNd7vhTiz9UpLdi6gZhZ3wEeDy8= 75 | github.com/golang-jwt/jwt/v5 v5.2.2/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk= 76 | github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE= 77 | github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= 78 | github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= 79 | github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= 80 | github.com/google/gnostic-models v0.6.8 h1:yo/ABAfM5IMRsS1VnXjTBvUb61tFIHozhlYvRgGre9I= 81 | github.com/google/gnostic-models v0.6.8/go.mod h1:5n7qKqH0f5wFt+aWF8CW6pZLLNOfYuF5OpfBSENuI8U= 82 | github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= 83 | github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= 84 | github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= 85 | github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= 86 | github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0= 87 | github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= 88 | github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af h1:kmjWCqn2qkEml422C2Rrd27c3VGxi6a/6HNq8QmHRKM= 89 | github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af/go.mod h1:K1liHPHnj73Fdn/EKuT8nrFqBihUSKXoLYU0BuatOYo= 90 | github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= 91 | github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= 92 | github.com/hashicorp/go-hclog v1.4.0 h1:ctuWFGrhFha8BnnzxqeRGidlEcQkDyL5u8J8t5eA11I= 93 | github.com/hashicorp/go-hclog v1.4.0/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVHBcfoyhpF5M= 94 | github.com/hashicorp/go-plugin v1.6.0 h1:wgd4KxHJTVGGqWBq4QPB1i5BZNEx9BR8+OFmHDmTk8A= 95 | github.com/hashicorp/go-plugin v1.6.0/go.mod h1:lBS5MtSSBZk0SHc66KACcjjlU6WzEVP/8pwz68aMkCI= 96 | github.com/hashicorp/yamux v0.1.1 h1:yrQxtgseBDrq9Y652vSRDvsKCJKOUD+GzTS4Y0Y8pvE= 97 | github.com/hashicorp/yamux v0.1.1/go.mod h1:CtWFDAQgb7dxtzFs4tWbplKIe2jSi3+5vKbgIO0SLnQ= 98 | github.com/imdario/mergo v0.3.13 h1:lFzP57bqS/wsqKssCGmtLAb8A0wKjLGrve2q3PPVcBk= 99 | github.com/imdario/mergo v0.3.13/go.mod h1:4lJ1jqUDcsbIECGy0RUJAXNIhg+6ocWgb1ALK2O4oXg= 100 | github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= 101 | github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= 102 | github.com/jhump/protoreflect v1.15.1 h1:HUMERORf3I3ZdX05WaQ6MIpd/NJ434hTp5YiKgfCL6c= 103 | github.com/jhump/protoreflect v1.15.1/go.mod h1:jD/2GMKKE6OqX8qTjhADU1e6DShO+gavG9e0Q693nKo= 104 | github.com/joho/godotenv v1.4.0 h1:3l4+N6zfMWnkbPEXKng2o2/MR5mSwTrBih4ZEkkz1lg= 105 | github.com/joho/godotenv v1.4.0/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4= 106 | github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY= 107 | github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y= 108 | github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= 109 | github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= 110 | github.com/keybase/go-keychain v0.0.1 h1:way+bWYa6lDppZoZcgMbYsvC7GxljxrskdNInRtuthU= 111 | github.com/keybase/go-keychain v0.0.1/go.mod h1:PdEILRW3i9D8JcdM+FmY6RwkHGnhHxXwkPPMeUgOK1k= 112 | github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= 113 | github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= 114 | github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo= 115 | github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ= 116 | github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= 117 | github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= 118 | github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= 119 | github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= 120 | github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= 121 | github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= 122 | github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= 123 | github.com/kubernetes-csi/external-snapshotter/client/v8 v8.2.0 h1:Q3jQ1NkFqv5o+F8dMmHd8SfEmlcwNeo1immFApntEwE= 124 | github.com/kubernetes-csi/external-snapshotter/client/v8 v8.2.0/go.mod h1:E3vdYxHj2C2q6qo8/Da4g7P+IcwqRZyy3gJBzYybV9Y= 125 | github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= 126 | github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= 127 | github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= 128 | github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= 129 | github.com/mattn/go-colorable v0.1.9/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc= 130 | github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4= 131 | github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE= 132 | github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8= 133 | github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU= 134 | github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= 135 | github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= 136 | github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= 137 | github.com/mitchellh/go-testing-interface v1.14.1 h1:jrgshOhYAUVNMAJiKbEu7EqAwgJJ2JqpQmpLJOu07cU= 138 | github.com/mitchellh/go-testing-interface v1.14.1/go.mod h1:gfgS7OtZj6MA4U1UrDRp04twqAjfvlZyCfX3sDjEym8= 139 | github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= 140 | github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= 141 | github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= 142 | github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= 143 | github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= 144 | github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= 145 | github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= 146 | github.com/oklog/run v1.1.0 h1:GEenZ1cK0+q0+wsJew9qUg/DyD8k3JzYsZAi5gYi2mA= 147 | github.com/oklog/run v1.1.0/go.mod h1:sVPdnTZT1zYwAJeCMu2Th4T21pA3FPOQRfWjQlk7DVU= 148 | github.com/onsi/ginkgo/v2 v2.19.0 h1:9Cnnf7UHo57Hy3k6/m5k3dRfGTMXGvxhHFvkDTCTpvA= 149 | github.com/onsi/ginkgo/v2 v2.19.0/go.mod h1:rlwLi9PilAFJ8jCg9UE1QP6VBpd6/xj3SRC0d6TU0To= 150 | github.com/onsi/gomega v1.33.1 h1:dsYjIxxSR755MDmKVsaFQTE22ChNBcuuTWgkUDSubOk= 151 | github.com/onsi/gomega v1.33.1/go.mod h1:U4R44UsT+9eLIaYRB2a5qajjtQYn0hauxvRm16AVYg0= 152 | github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ= 153 | github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c/go.mod h1:7rwL4CYBLnjLxUqIJNnCWiEdr3bn6IUYi15bNlnbCCU= 154 | github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= 155 | github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 156 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 157 | github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= 158 | github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 159 | github.com/prometheus/client_golang v1.22.0 h1:rb93p9lokFEsctTys46VnV1kLCDpVZ0a/Y92Vm0Zc6Q= 160 | github.com/prometheus/client_golang v1.22.0/go.mod h1:R7ljNsLXhuQXYZYtw6GAE9AZg8Y7vEW5scdCXrWRXC0= 161 | github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk= 162 | github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE= 163 | github.com/prometheus/common v0.65.0 h1:QDwzd+G1twt//Kwj/Ww6E9FQq1iVMmODnILtW1t2VzE= 164 | github.com/prometheus/common v0.65.0/go.mod h1:0gZns+BLRQ3V6NdaerOhMbwwRbNh9hkGINtQAsP5GS8= 165 | github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc= 166 | github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk= 167 | github.com/redis/go-redis/v9 v9.8.0 h1:q3nRvjrlge/6UD7eTu/DSg2uYiU2mCL0G/uzBWqhicI= 168 | github.com/redis/go-redis/v9 v9.8.0/go.mod h1:huWgSWd8mW6+m0VPhJjSSQ+d6Nh1VICQ6Q5lHuCH/Iw= 169 | github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8= 170 | github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4= 171 | github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= 172 | github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ= 173 | github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= 174 | github.com/spf13/afero v1.10.0 h1:EaGW2JJh15aKOejeuJ+wpFSHnbd7GE6Wvp3TsNhb6LY= 175 | github.com/spf13/afero v1.10.0/go.mod h1:UBogFpq8E9Hx+xc5CNTTEpTnuHVmXDwZcZcE1eb/UhQ= 176 | github.com/spf13/cobra v1.8.1 h1:e5/vxKd/rZsfSJMUX1agtjeTDf+qv1/JdBF8gg5k9ZM= 177 | github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y= 178 | github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= 179 | github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= 180 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 181 | github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= 182 | github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= 183 | github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY= 184 | github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA= 185 | github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= 186 | github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 187 | github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 188 | github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1FQKckRals= 189 | github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= 190 | github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= 191 | github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA= 192 | github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= 193 | github.com/vmware-tanzu/velero v0.0.0-20250826085519-79b027577e6a h1:w2zh7uMwkBWhk6Abmos8uFtRNyMKgLtyAfz4mzbdvGA= 194 | github.com/vmware-tanzu/velero v0.0.0-20250826085519-79b027577e6a/go.mod h1:HsRTnCgxf41Eq1uZUQGsdMLrnYA1TPMLfEJ2JNYgEtk= 195 | github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM= 196 | github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg= 197 | github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= 198 | github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= 199 | go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA= 200 | go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A= 201 | go.opentelemetry.io/otel v1.37.0 h1:9zhNfelUvx0KBfu/gb+ZgeAfAgtWrfHJZcAqFC228wQ= 202 | go.opentelemetry.io/otel v1.37.0/go.mod h1:ehE/umFRLnuLa/vSccNq9oS1ErUlkkK71gMcN34UG8I= 203 | go.opentelemetry.io/otel/metric v1.37.0 h1:mvwbQS5m0tbmqML4NqK+e3aDiO02vsf/WgbsdpcPoZE= 204 | go.opentelemetry.io/otel/metric v1.37.0/go.mod h1:04wGrZurHYKOc+RKeye86GwKiTb9FKm1WHtO+4EVr2E= 205 | go.opentelemetry.io/otel/sdk v1.37.0 h1:ItB0QUqnjesGRvNcmAcU0LyvkVyGJ2xftD29bWdDvKI= 206 | go.opentelemetry.io/otel/sdk v1.37.0/go.mod h1:VredYzxUvuo2q3WRcDnKDjbdvmO0sCzOvVAiY+yUkAg= 207 | go.opentelemetry.io/otel/sdk/metric v1.36.0 h1:r0ntwwGosWGaa0CrSt8cuNuTcccMXERFwHX4dThiPis= 208 | go.opentelemetry.io/otel/sdk/metric v1.36.0/go.mod h1:qTNOhFDfKRwX0yXOqJYegL5WRaW376QbB7P4Pb0qva4= 209 | go.opentelemetry.io/otel/trace v1.37.0 h1:HLdcFNbRQBE2imdSEgm/kwqmQj1Or1l/7bW6mxVK7z4= 210 | go.opentelemetry.io/otel/trace v1.37.0/go.mod h1:TlgrlQ+PtQO5XFerSPUYG0JSgGyryXewPGyayAWSBS0= 211 | go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= 212 | go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= 213 | go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= 214 | go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= 215 | go.uber.org/zap v1.27.0 h1:aJMhYGrd5QSmlpLMr2MftRKl7t8J8PTZPA732ud/XR8= 216 | go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E= 217 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 218 | golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 219 | golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= 220 | golang.org/x/crypto v0.40.0 h1:r4x+VvoG5Fm+eJcxMaY8CQM7Lb0l1lsmjGBQ6s8BfKM= 221 | golang.org/x/crypto v0.40.0/go.mod h1:Qr1vMER5WyS2dfPHAlsOj01wgLbsyWtFn/aY+5+ZdxY= 222 | golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1 h1:k/i9J1pBpvlfR+9QsetwPyERsqu1GIbi967PQMq3Ivc= 223 | golang.org/x/exp v0.0.0-20230522175609-2e198f4a06a1/go.mod h1:V1LtkGg67GoY2N1AnLN78QLrzxkLyJw7RJb1gzOOz9w= 224 | golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 225 | golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 226 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 227 | golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 228 | golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 229 | golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= 230 | golang.org/x/net v0.42.0 h1:jzkYrhi3YQWD6MLBJcsklgQsoAcw89EcZbJw8Z614hs= 231 | golang.org/x/net v0.42.0/go.mod h1:FF1RA5d3u7nAYA4z2TkclSCKh68eSXtiFwcWQpPXdt8= 232 | golang.org/x/oauth2 v0.30.0 h1:dnDm7JmhM45NNpd8FDDeLhK6FwqbOf4MLCM9zb1BOHI= 233 | golang.org/x/oauth2 v0.30.0/go.mod h1:B++QgG3ZKulg6sRPGD/mqlHQs5rB3Ml9erfeDY7xKlU= 234 | golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 235 | golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 236 | golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 237 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 238 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 239 | golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 240 | golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 241 | golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 242 | golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 243 | golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 244 | golang.org/x/sys v0.0.0-20220503163025-988cb79eb6c6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 245 | golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 246 | golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 247 | golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 248 | golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA= 249 | golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= 250 | golang.org/x/term v0.33.0 h1:NuFncQrRcaRvVmgRkvM3j/F00gWIAlcmlB8ACEKmGIg= 251 | golang.org/x/term v0.33.0/go.mod h1:s18+ql9tYWp1IfpV9DmCtQDDSRBUjKaw9M1eAv5UeF0= 252 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 253 | golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= 254 | golang.org/x/text v0.27.0 h1:4fGWRpyh641NLlecmyl4LOe6yDdfaYNrGb2zdfo4JV4= 255 | golang.org/x/text v0.27.0/go.mod h1:1D28KMCvyooCX9hBiosv5Tz/+YLxj0j7XhWjpSUF7CU= 256 | golang.org/x/time v0.12.0 h1:ScB/8o8olJvc+CQPWrK3fPZNfh7qgwCrY0zJmoEQLSE= 257 | golang.org/x/time v0.12.0/go.mod h1:CDIdPxbZBQxdj6cxyCIdrNogrJKMJ7pr37NYpMcMDSg= 258 | golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 259 | golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 260 | golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= 261 | golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= 262 | golang.org/x/tools v0.34.0 h1:qIpSLOxeCYGg9TrcJokLBG4KFA6d795g0xkBkiESGlo= 263 | golang.org/x/tools v0.34.0/go.mod h1:pAP9OwEaY1CAW3HOmg3hLZC5Z0CCmzjAF2UQMSqNARg= 264 | golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 265 | golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 266 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 267 | golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 268 | gomodules.xyz/jsonpatch/v2 v2.4.0 h1:Ci3iUJyx9UeRx7CeFN8ARgGbkESwJK+KB9lLcWxY/Zw= 269 | gomodules.xyz/jsonpatch/v2 v2.4.0/go.mod h1:AH3dM2RI6uoBZxn3LVrfvJ3E0/9dG4cSrbuBJT4moAY= 270 | google.golang.org/genproto/googleapis/rpc v0.0.0-20250603155806-513f23925822 h1:fc6jSaCT0vBduLYZHYrBBNY4dsWuvgyff9noRNDdBeE= 271 | google.golang.org/genproto/googleapis/rpc v0.0.0-20250603155806-513f23925822/go.mod h1:qQ0YXyHHx3XkvlzUtpXDkS29lDSafHMZBAZDc03LQ3A= 272 | google.golang.org/grpc v1.73.0 h1:VIWSmpI2MegBtTuFt5/JWy2oXxtjJ/e89Z70ImfD2ok= 273 | google.golang.org/grpc v1.73.0/go.mod h1:50sbHOUqWoCQGI8V2HQLJM0B+LMlIUjNSZmow7EVBQc= 274 | google.golang.org/protobuf v1.36.6 h1:z1NpPI8ku2WgiWnf+t9wTPsn6eP1L7ksHUlkfLvd9xY= 275 | google.golang.org/protobuf v1.36.6/go.mod h1:jduwjTPXsFjZGTmRluh+L6NjiWu7pchiJ2/5YcXBHnY= 276 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 277 | gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= 278 | gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= 279 | gopkg.in/evanphx/json-patch.v4 v4.12.0 h1:n6jtcsulIzXPJaxegRbvFNNrZDjbij7ny3gmSPG+6V4= 280 | gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M= 281 | gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= 282 | gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= 283 | gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 284 | gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= 285 | gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= 286 | gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 287 | gopkg.in/yaml.v3 v3.0.0/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 288 | gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= 289 | gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 290 | k8s.io/api v0.31.3 h1:umzm5o8lFbdN/hIXbrK9oRpOproJO62CV1zqxXrLgk8= 291 | k8s.io/api v0.31.3/go.mod h1:UJrkIp9pnMOI9K2nlL6vwpxRzzEX5sWgn8kGQe92kCE= 292 | k8s.io/apiextensions-apiserver v0.31.3 h1:+GFGj2qFiU7rGCsA5o+p/rul1OQIq6oYpQw4+u+nciE= 293 | k8s.io/apiextensions-apiserver v0.31.3/go.mod h1:2DSpFhUZZJmn/cr/RweH1cEVVbzFw9YBu4T+U3mf1e4= 294 | k8s.io/apimachinery v0.31.3 h1:6l0WhcYgasZ/wk9ktLq5vLaoXJJr5ts6lkaQzgeYPq4= 295 | k8s.io/apimachinery v0.31.3/go.mod h1:rsPdaZJfTfLsNJSQzNHQvYoTmxhoOEofxtOsF3rtsMo= 296 | k8s.io/client-go v0.31.3 h1:CAlZuM+PH2cm+86LOBemaJI/lQ5linJ6UFxKX/SoG+4= 297 | k8s.io/client-go v0.31.3/go.mod h1:2CgjPUTpv3fE5dNygAr2NcM8nhHzXvxB8KL5gYc3kJs= 298 | k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk= 299 | k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE= 300 | k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 h1:BZqlfIlq5YbRMFko6/PM7FjZpUb45WallggurYhKGag= 301 | k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340/go.mod h1:yD4MZYeKMBwQKVht279WycxKyM84kkAx2DPrTXaeb98= 302 | k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 h1:pUdcCO1Lk/tbT5ztQWOBi5HBgbBP1J8+AsQnQCKsi8A= 303 | k8s.io/utils v0.0.0-20240711033017-18e509b52bc8/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= 304 | sigs.k8s.io/azuredisk-csi-driver v1.26.0 h1:61WFkqlpyuGjAYfr9Shl3D+nDaXVyPbxt2EWmhKL71A= 305 | sigs.k8s.io/azuredisk-csi-driver v1.26.0/go.mod h1:cKTnZgTH6U7hmtYJMjFa1j97zFCBd10zAhUIOEEO2U4= 306 | sigs.k8s.io/controller-runtime v0.19.3 h1:XO2GvC9OPftRst6xWCpTgBZO04S2cbp0Qqkj8bX1sPw= 307 | sigs.k8s.io/controller-runtime v0.19.3/go.mod h1:j4j87DqtsThvwTv5/Tc5NFRyyF/RF0ip4+62tbTSIUM= 308 | sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo= 309 | sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0= 310 | sigs.k8s.io/structured-merge-diff/v4 v4.4.1 h1:150L+0vs/8DA78h1u02ooW1/fFq/Lwr+sGiqlzvrtq4= 311 | sigs.k8s.io/structured-merge-diff/v4 v4.4.1/go.mod h1:N8hJocpFajUSSeSJ9bOZ77VzejKZaXsTtZo4/u7Io08= 312 | sigs.k8s.io/yaml v1.4.0 h1:Mk1wCc2gy/F0THH0TAp1QYyJNzRm2KCLy3o5ASXVI5E= 313 | sigs.k8s.io/yaml v1.4.0/go.mod h1:Ejl7/uTz7PSA4eKMyQCUTnhZYNmLIl+5c2lQPGR2BPY= 314 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | [![Build Status][101]][102] 3 | 4 | # Velero plugins for Microsoft Azure 5 | 6 | ## Overview 7 | 8 | This repository contains these plugins to support running Velero on Microsoft Azure: 9 | 10 | - An object store plugin for persisting and retrieving backups on Azure Blob Storage. Content of backup is kubernetes resources and metadata files for CSI objects, progress of async operations. It is also used to store the result data of backups and restores include log files, warning/error files, etc. 11 | 12 | - A volume snapshotter plugin for creating snapshots from volumes (during a backup) and volumes from snapshots (during a restore) on Azure Managed Disks. 13 | - Since v1.4.0 the snapshotter plugin can handle the volumes provisioned by CSI driver `disk.csi.azure.com`. 14 | - Since v1.5.0 the snapshotter plugin can handle the zone-redundant storage(ZRS) managed disks which can be used to support backup/restore across different available zones. 15 | 16 | ## Compatibility 17 | 18 | Below is a listing of plugin versions and respective Velero versions that are compatible. 19 | 20 | | Plugin Version | Velero Version | 21 | |----------------|----------------| 22 | | v1.13.x | v1.17.x | 23 | | v1.12.x | v1.16.x | 24 | | v1.11.x | v1.15.x | 25 | | v1.10.x | v1.14.x | 26 | | v1.9.x | v1.13.x | 27 | 28 | 29 | 30 | ## Filing issues 31 | 32 | If you would like to file a GitHub issue for the plugin, please open the issue on the [core Velero repo][103] 33 | 34 | 35 | ## Kubernetes cluster prerequisites 36 | 37 | Ensure that the VMs for your agent pool allow Managed Disks. If I/O performance is critical, 38 | consider using Premium Managed Disks, which are SSD backed. 39 | 40 | ## Setup 41 | 42 | To set up Velero on Azure, you: 43 | 44 | - [Create an Azure storage account and blob container][1] 45 | - [Get the resource group containing your VMs and disks][4] 46 | - [Set permissions for Velero][2] 47 | - [Install and start Velero][3] 48 | 49 | 50 | You can also use this plugin to create an additional [Backup Storage Location][12]. 51 | 52 | If you do not have the `az` Azure CLI 2.0 installed locally, follow the [install guide][21] to set it up. 53 | 54 | Run: 55 | 56 | ```bash 57 | az login 58 | ``` 59 | 60 | ## Setup Azure storage account and blob container 61 | 62 | ### (Optional) Change to the Azure subscription you want to create your backups in 63 | 64 | By default, Velero will store backups in the same Subscription as your VMs and disks and will 65 | not allow you to restore backups to a Resource Group in a different Subscription. To enable backups/restore 66 | across Subscriptions you will need to specify the Subscription ID to backup to. 67 | 68 | Use `az` to switch to the Subscription the backups should be created in. 69 | 70 | First, find the Subscription ID by name. 71 | 72 | ```bash 73 | AZURE_BACKUP_SUBSCRIPTION_NAME= 74 | AZURE_BACKUP_SUBSCRIPTION_ID=$(az account list --query="[?name=='$AZURE_BACKUP_SUBSCRIPTION_NAME'].id | [0]" -o tsv) 75 | ``` 76 | 77 | Second, change the Subscription. 78 | 79 | ```bash 80 | az account set -s $AZURE_BACKUP_SUBSCRIPTION_ID 81 | ``` 82 | 83 | Execute the next step – creating an storage account and blob container – using the active Subscription. 84 | 85 | ### Create Azure storage account and blob container 86 | 87 | Velero requires a storage account and blob container in which to store backups. 88 | 89 | The storage account can be created in the same Resource Group as your Kubernetes cluster or 90 | separated into its own Resource Group. The example below shows the storage account created in a 91 | separate `Velero_Backups` Resource Group. 92 | 93 | The storage account needs to be created with a globally unique id since this is used for dns. In 94 | the sample script below, we're generating a random name using `uuidgen`, but you can come up with 95 | this name however you'd like, following the [Azure naming rules for storage accounts][22]. The 96 | storage account is created with encryption at rest capabilities (Microsoft managed keys) and is 97 | configured to only allow access via https. 98 | 99 | Create a resource group for the backups storage account. Change the location as needed. 100 | 101 | ```bash 102 | AZURE_BACKUP_RESOURCE_GROUP=Velero_Backups 103 | az group create -n $AZURE_BACKUP_RESOURCE_GROUP --location WestUS 104 | ``` 105 | 106 | Create the storage account. 107 | 108 | ```bash 109 | AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')" 110 | az storage account create \ 111 | --name $AZURE_STORAGE_ACCOUNT_ID \ 112 | --resource-group $AZURE_BACKUP_RESOURCE_GROUP \ 113 | --sku Standard_GRS \ 114 | --encryption-services blob \ 115 | --https-only true \ 116 | --min-tls-version TLS1_2 \ 117 | --kind BlobStorage \ 118 | --access-tier Hot 119 | ``` 120 | 121 | Create the blob container named `velero`. Feel free to use a different name, preferably unique to a single Kubernetes cluster. See the [FAQ][11] for more details. 122 | 123 | ```bash 124 | BLOB_CONTAINER=velero 125 | az storage container create -n $BLOB_CONTAINER --public-access off --account-name $AZURE_STORAGE_ACCOUNT_ID 126 | ``` 127 | 128 | ## Get resource group containing your VMs and disks 129 | 130 | _(Optional) If you decided to backup to a different Subscription, make sure you change back to the Subscription 131 | of your cluster's resources before continuing._ 132 | 133 | 1. Set the name of the Resource Group that contains your Kubernetes cluster's virtual machines/disks. 134 | 135 | **WARNING**: If you're using [AKS][25], `AZURE_RESOURCE_GROUP` must be set to the name of the auto-generated resource group that is created 136 | when you provision your cluster in Azure, since this is the resource group that contains your cluster's virtual machines/disks. 137 | 138 | ```bash 139 | AZURE_RESOURCE_GROUP= 140 | ``` 141 | 142 | If you are unsure of the Resource Group name, run the following command to get a list that you can select from. Then set the `AZURE_RESOURCE_GROUP` environment variable to the appropriate value. 143 | 144 | ```bash 145 | az group list --query '[].{ ResourceGroup: name, Location:location }' 146 | ``` 147 | 148 | Get your cluster's Resource Group name from the `ResourceGroup` value in the response, and use it to set `$AZURE_RESOURCE_GROUP`. 149 | 150 | ## Set permissions for Velero 151 | 152 | There are several ways Velero can authenticate to Azure: (1) by using a Velero-specific [service principal][20] with secret-based authentication; (2) by using a Velero-specific [service principal][20] with certificate-based authentication; (3) by using [Azure AD Workload Identity][23]; or (4) by using a storage account access key. 153 | 154 | If you plan to use Velero to take Azure snapshots of your persistent volume managed disks, you **must** use the service principal or Azure AD Workload Identity method. 155 | 156 | If you don't plan to take Azure disk snapshots, any method is valid. 157 | 158 | ### Specify Role 159 | _**Note**: This is only required for (1) by using a Velero-specific service principal with secret-based authentication, (2) by using a Velero-specific service principal with certificate-based authentication and (3) by using Azure AD Workload Identity._ 160 | 161 | 1. Obtain your Azure Account Subscription ID: 162 | ``` 163 | AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv` 164 | ``` 165 | 166 | 2. Specify the role 167 | There are two ways to specify the role: use the built-in role or create a custom one. 168 | You can use the Azure built-in role `Contributor`: 169 | ``` 170 | AZURE_ROLE=Contributor 171 | ``` 172 | This will have subscription-wide access, so protect the credential generated with this role. 173 | 174 | It is always best practice to assign the minimum required permissions necessary for an application to do its work. 175 | 176 | > Note: With useAAD flag you will need to provide extra permissions `Storage Blob Data Contributor` covered in point 3 of section: [Create service principal](#create-service-principal) 177 | 178 | Here are the minimum required permissions needed by Velero to perform backups, restores, and deletions: 179 | - Storage Account 180 | > Back Compatability and Restic 181 | - Microsoft.Storage/storageAccounts/read 182 | - Microsoft.Storage/storageAccounts/listkeys/action 183 | - Microsoft.Storage/storageAccounts/regeneratekey/action 184 | > AAD Based Auth 185 | - Microsoft.Storage/storageAccounts/read 186 | - Microsoft.Storage/storageAccounts/blobServices/containers/delete 187 | - Microsoft.Storage/storageAccounts/blobServices/containers/read 188 | - Microsoft.Storage/storageAccounts/blobServices/containers/write 189 | - Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action 190 | > Data Actions for AAD auth 191 | - Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete 192 | - Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read 193 | - Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write 194 | - Microsoft.Storage/storageAccounts/blobServices/containers/blobs/move/action 195 | - Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action 196 | - Disk Management 197 | - Microsoft.Compute/disks/read 198 | - Microsoft.Compute/disks/write 199 | - Microsoft.Compute/disks/endGetAccess/action 200 | - Microsoft.Compute/disks/beginGetAccess/action 201 | - Snapshot Management 202 | - Microsoft.Compute/snapshots/read 203 | - Microsoft.Compute/snapshots/write 204 | - Microsoft.Compute/snapshots/delete 205 | - Microsoft.Compute/disks/beginGetAccess/action 206 | - Microsoft.Compute/disks/endGetAccess/action 207 | 208 | Use the following commands to create a custom role which has the minimum required permissions: 209 | ``` 210 | AZURE_ROLE=Velero 211 | az role definition create --role-definition '{ 212 | "Name": "'$AZURE_ROLE'", 213 | "Description": "Velero related permissions to perform backups, restores and deletions", 214 | "Actions": [ 215 | "Microsoft.Compute/disks/read", 216 | "Microsoft.Compute/disks/write", 217 | "Microsoft.Compute/disks/endGetAccess/action", 218 | "Microsoft.Compute/disks/beginGetAccess/action", 219 | "Microsoft.Compute/snapshots/read", 220 | "Microsoft.Compute/snapshots/write", 221 | "Microsoft.Compute/snapshots/delete", 222 | "Microsoft.Storage/storageAccounts/listkeys/action", 223 | "Microsoft.Storage/storageAccounts/regeneratekey/action", 224 | "Microsoft.Storage/storageAccounts/read", 225 | "Microsoft.Storage/storageAccounts/blobServices/containers/delete", 226 | "Microsoft.Storage/storageAccounts/blobServices/containers/read", 227 | "Microsoft.Storage/storageAccounts/blobServices/containers/write", 228 | "Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action" 229 | ], 230 | "DataActions" :[ 231 | "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete", 232 | "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read", 233 | "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write", 234 | "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/move/action", 235 | "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action" 236 | ], 237 | "AssignableScopes": ["/subscriptions/'$AZURE_SUBSCRIPTION_ID'"] 238 | }' 239 | ``` 240 | _(Optional) If you are using a different Subscription for backups and cluster resources, make sure to specify both subscriptions 241 | inside `AssignableScopes`._ 242 | 243 | ### Option 1: Create service principal with secret-based authentication 244 | 245 | 1. Obtain your Azure Account Tenant ID: 246 | 247 | ```bash 248 | AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv` 249 | ``` 250 | 251 | 2. Create a service principal with secet-based authentication. 252 | 253 | If you'll be using Velero to backup multiple clusters with multiple blob containers, it may be desirable to create a unique username per cluster rather than the default `velero`. 254 | 255 | Create service principal and let the CLI generate a password for you. Make sure to capture the password. 256 | 257 | _(Optional) If you are using a different Subscription for backups and cluster resources, make sure to specify both subscriptions 258 | in the `az` command using `--scopes`._ 259 | 260 | ```bash 261 | AZURE_CLIENT_SECRET=`az ad sp create-for-rbac --name "velero" --role $AZURE_ROLE --query 'password' -o tsv \ 262 | --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID[ /subscriptions/$AZURE_BACKUP_SUBSCRIPTION_ID]` 263 | ``` 264 | 265 | NOTE: Ensure that value for `--name` does not conflict with other service principals/app registrations. 266 | 267 | After creating the service principal, obtain the client id. 268 | 269 | ```bash 270 | AZURE_CLIENT_ID=`az ad sp list --display-name "velero" --query '[0].appId' -o tsv` 271 | ``` 272 | 3. (Optional)Assign additional permissions to the service principal (For useAAD=true with built-in role) 273 | 274 | If you use the custom role which has the blob data permissions, skip this step. 275 | 276 | If you chose the AAD route, this is an additional permissions required for the service principal to be able to access the storage account. 277 | ```bash 278 | az role assignment create --assignee $AZURE_CLIENT_ID --role "Storage Blob Data Contributor" --scope /subscriptions/$AZURE_SUBSCRIPTION_ID 279 | ``` 280 | 281 | Refer: [useAAD parameter in BackupStorageLocation.md](./backupstoragelocation.md#backup-storage-location) 282 | 283 | 4. Now you need to create a file that contains all the relevant environment variables. The command looks like the following: 284 | 285 | ```bash 286 | cat << EOF > ./credentials-velero 287 | AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID} 288 | AZURE_TENANT_ID=${AZURE_TENANT_ID} 289 | AZURE_CLIENT_ID=${AZURE_CLIENT_ID} 290 | AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET} 291 | AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP} 292 | AZURE_CLOUD_NAME=AzurePublicCloud 293 | EOF 294 | ``` 295 | 296 | > Available values for `AZURE_CLOUD_NAME`: `AzurePublicCloud`, `AzureUSGovernmentCloud`, `AzureChinaCloud` 297 | 298 | ### Option 2: Create service principal with certificate-based authentication 299 | 300 | 1. Obtain your Azure Account Tenant ID: 301 | 302 | ```bash 303 | AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv` 304 | ``` 305 | 306 | 2. Create a service principal with certificate-based authentication. 307 | 308 | If you'll be using Velero to backup multiple clusters with multiple blob containers, it may be desirable to create a unique username per cluster rather than the default `velero`. 309 | 310 | Create service principal and let the CLI creates a self-signed certificate for you. Make sure to capture the certificate. 311 | 312 | _(Optional) If you are using a different Subscription for backups and cluster resources, make sure to specify both subscriptions 313 | in the `az` command using `--scopes`._ 314 | 315 | ```bash 316 | AZURE_CLIENT_CERTIFICATE_PATH=`az ad sp create-for-rbac --name "velero" --role $AZURE_ROLE --query 'fileWithCertAndPrivateKey' -o tsv \ 317 | --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID[ /subscriptions/$AZURE_BACKUP_SUBSCRIPTION_ID] --create-cert` 318 | ``` 319 | 320 | NOTE: Ensure that value for `--name` does not conflict with other service principals/app registrations. 321 | 322 | After creating the service principal, obtain the client id. 323 | 324 | ```bash 325 | AZURE_CLIENT_ID=`az ad sp list --display-name "velero" --query '[0].appId' -o tsv` 326 | ``` 327 | 3. (Optional)Assign additional permissions to the service principal (For useAAD=true with built-in role) 328 | 329 | If you use the custom role which has the blob data permissions, skip this step. 330 | 331 | If you chose the AAD route, this is an additional permissions required for the service principal to be able to access the storage account. 332 | ```bash 333 | az role assignment create --assignee $AZURE_CLIENT_ID --role "Storage Blob Data Contributor" --scope /subscriptions/$AZURE_SUBSCRIPTION_ID 334 | ``` 335 | 336 | Refer: [useAAD parameter in BackupStorageLocation.md](./backupstoragelocation.md#backup-storage-location) 337 | 338 | 6. Now you need to create a file that contains all the relevant environment variables. The command looks like the following: 339 | 340 | ```bash 341 | cat << EOF > ./credentials-velero 342 | AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID} 343 | AZURE_TENANT_ID=${AZURE_TENANT_ID} 344 | AZURE_CLIENT_ID=${AZURE_CLIENT_ID} 345 | AZURE_CLIENT_CERTIFICATE=$(awk 'BEGIN {printf "\""} {sub(/\r/, ""); printf "%s\\n",$0;} END {printf "\""}' $AZURE_CLIENT_CERTIFICATE_PATH) 346 | AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP} 347 | AZURE_CLOUD_NAME=AzurePublicCloud 348 | EOF 349 | ``` 350 | 351 | > Available values for `AZURE_CLOUD_NAME`: `AzurePublicCloud`, `AzureUSGovernmentCloud`, `AzureChinaCloud` 352 | 353 | ### Option 3: Use Azure AD Workload Identity 354 | 355 | These instructions have been adapted from the [Azure AD Workload Identity Quick Start][24] documentation. 356 | 357 | Before proceeding, ensure that you have installed [workload identity mutating admission webhook][28] and [enabled the OIDC Issuer][29] for your cluster. 358 | 359 | 1. Create an identity for Velero: 360 | 361 | ```bash 362 | IDENTITY_NAME=velero 363 | 364 | az identity create \ 365 | --subscription $AZURE_SUBSCRIPTION_ID \ 366 | --resource-group $AZURE_RESOURCE_GROUP \ 367 | --name $IDENTITY_NAME 368 | 369 | IDENTITY_CLIENT_ID="$(az identity show -g $AZURE_RESOURCE_GROUP -n $IDENTITY_NAME --subscription $AZURE_SUBSCRIPTION_ID --query clientId -otsv)" 370 | ``` 371 | 372 | If you'll be using Velero to backup multiple clusters with multiple blob containers, it may be desirable to create a unique identity name per cluster rather than the default `velero`. 373 | 374 | 2. Assign the identity roles: 375 | 376 | ```bash 377 | az role assignment create --role $AZURE_ROLE --assignee $IDENTITY_CLIENT_ID --scope /subscriptions/$AZURE_SUBSCRIPTION_ID 378 | ``` 379 | 380 | (Optional)Assign additional permissions to the service principal (For useAAD=true with built-in role) 381 | 382 | If you use the custom role which has the blob data permissions, skip this step. 383 | 384 | If you chose the AAD route, this is an additional permissions required for the identity to be able to access the storage account. 385 | ```bash 386 | az role assignment create --assignee $IDENTITY_CLIENT_ID --role "Storage Blob Data Contributor" --scope /subscriptions/$AZURE_SUBSCRIPTION_ID 387 | ``` 388 | 389 | Refer: [useAAD parameter in BackupStorageLocation.md](./backupstoragelocation.md#backup-storage-location) 390 | 391 | 3. Create a service account for Velero 392 | 393 | ```bash 394 | # create namespace 395 | kubectl create namespace velero 396 | 397 | # create service account 398 | cat < 429 | ``` 430 | **WARNING**: If you're using [AKS][25], `CLUSTER_RESOURCE_GROUP` must be set to the name of the resource group where the cluster is created, not the auto-generated resource group that is created when you provision your cluster in Azure. 431 | 432 | ```bash 433 | CLUSTER_NAME=your_cluster_name 434 | 435 | SERVICE_ACCOUNT_ISSUER=$(az aks show --resource-group $CLUSTER_RESOURCE_GROUP --name $CLUSTER_NAME --query "oidcIssuerProfile.issuerUrl" -o tsv) 436 | ``` 437 | 438 | 5. Establish federated identity credential between the identity and the service account issuer & subject 439 | ```bash 440 | az identity federated-credential create \ 441 | --name "kubernetes-federated-credential" \ 442 | --identity-name "${IDENTITY_NAME}" \ 443 | --resource-group "${AZURE_RESOURCE_GROUP}" \ 444 | --issuer "${SERVICE_ACCOUNT_ISSUER}" \ 445 | --subject "system:serviceaccount:velero:velero" 446 | ``` 447 | 448 | 6. Create a file that contains all the relevant environment variables: 449 | 450 | ```bash 451 | cat << EOF > ./credentials-velero 452 | AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID} 453 | AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP} 454 | AZURE_CLOUD_NAME=AzurePublicCloud 455 | EOF 456 | ``` 457 | 458 | > Available values for `AZURE_CLOUD_NAME`: `AzurePublicCloud`, `AzureUSGovernmentCloud`, `AzureChinaCloud` 459 | 460 | 461 | ### Option 4: Use storage account access key 462 | 463 | _Note: this option is **not valid** if you are planning to take Azure snapshots of your managed disks with Velero._ 464 | 465 | 1. Obtain your Azure Storage account access key: 466 | 467 | ```bash 468 | AZURE_STORAGE_ACCOUNT_ACCESS_KEY=`az storage account keys list --account-name $AZURE_STORAGE_ACCOUNT_ID --query "[?keyName == 'key1'].value" -o tsv` 469 | ``` 470 | 471 | 1. Now you need to create a file that contains all the relevant environment variables. The command looks like the following: 472 | 473 | ```bash 474 | cat << EOF > ./credentials-velero 475 | AZURE_STORAGE_ACCOUNT_ACCESS_KEY=${AZURE_STORAGE_ACCOUNT_ACCESS_KEY} 476 | AZURE_CLOUD_NAME=AzurePublicCloud 477 | EOF 478 | ``` 479 | 480 | > Available values for `AZURE_CLOUD_NAME`: `AzurePublicCloud`, `AzureUSGovernmentCloud`, `AzureChinaCloud` 481 | 482 | ## Install and start Velero 483 | 484 | [Download][6] Velero 485 | 486 | Install Velero, including all prerequisites, into the cluster and start the deployment. This will create a namespace called `velero`, and place a deployment named `velero` in it. 487 | 488 | ### If using service principal: 489 | 490 | ```bash 491 | velero install \ 492 | --provider azure \ 493 | --plugins velero/velero-plugin-for-microsoft-azure:v1.13.0 \ 494 | --bucket $BLOB_CONTAINER \ 495 | --secret-file ./credentials-velero \ 496 | --backup-location-config useAAD="true",resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] \ 497 | --snapshot-location-config apiTimeout=[,resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] 498 | ``` 499 | 500 | ### If using Azure AD Workload Identity: 501 | 502 | ```bash 503 | velero install \ 504 | --provider azure \ 505 | --service-account-name velero \ 506 | --pod-labels azure.workload.identity/use=true \ 507 | --plugins velero/velero-plugin-for-microsoft-azure:v1.13.0 \ 508 | --bucket $BLOB_CONTAINER \ 509 | --secret-file ./credentials-velero \ 510 | --backup-location-config useAAD="true",resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] \ 511 | --snapshot-location-config apiTimeout=[,resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] 512 | ``` 513 | 514 | In plugin v1.8.0+, users can chose to use AAD route for velero to access storage account when using service principal or Azure AD Workload Identity. Earlier this was done using ListKeys on the storage account which is not a recommended practice. 515 | 516 | **Limitation:** The velero identity needs Reader permission alongside the "storage blob data contributor" role on the storage account. This is because the identity needs to be able to read the storage account properties to fetch the storage account's blob endpoint (azure storage accounts are no longer expected to follow the format of blob.core.windows.net with introduction of DNS zone storage accounts.). To circumvent this issue follow the steps below: 517 | 518 | **For users facing Storage Account Throttling issues** 519 | You can chose to provide the Storage Account's blob endpoint directly to Velero. This will help Velero to bypass the need to fetch the storage account properties and hence the need for Reader permission on the storage account. This can be done by providing the blob endpoint in the backup-location-config as shown below: 520 | 521 | ```bash 522 | velero install \ 523 | --provider azure \ 524 | --plugins velero/velero-plugin-for-microsoft-azure:v1.13.0 \ 525 | --bucket $BLOB_CONTAINER \ 526 | --secret-file ./credentials-velero \ 527 | --backup-location-config storageAccountURI="https://xxxxxx.blob.core.windows.net",useAAD="true",resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] \ 528 | --snapshot-location-config apiTimeout=[,resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] 529 | ``` 530 | 531 | Note: 532 | - If you have provided the storageAccountUri, providing the resourceGroup and storageAccount fields are optional. 533 | 534 | **Migrating from ListKeys to AAD route:** 535 | If you already had a velero setup using azure plugin < 1.8.0 it must be using the ListKeys approach which is not recommended. To migrate to the AAD route follow the steps below: 536 | 537 | You need to assign your velero identity the following permissions: 538 | 539 | ```bash 540 | az role assignment create --role "Storage Blob Data Contributor" --assignee $IDENTITY_CLIENT_ID --scope /subscriptions/$AZURE_SUBSCRIPTION_ID/resourceGroups/$AZURE_BACKUP_RESOURCE_GROUP/providers/Microsoft.Storage/storageAccounts/$AZURE_STORAGE_ACCOUNT_ID 541 | ``` 542 | 543 | After that update your velero BackupStorageLocation with the useAAD flag as shown below: 544 | 545 | ```bash 546 | velero backup-location set default --provider azure --bucket $BLOB_CONTAINER --config useAAD="true",resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] 547 | ``` 548 | 549 | Limitation: Listing storage account access key is still needed for Restic to work as expected on Azure. The useAAD route won't accrue to it and users using Restic should not remove the ListKeys permission from the velero identity. 550 | 551 | ### If using storage account access key and no Azure snapshots: 552 | 553 | ```bash 554 | velero install \ 555 | --provider azure \ 556 | --plugins velero/velero-plugin-for-microsoft-azure:v1.13.0 \ 557 | --bucket $BLOB_CONTAINER \ 558 | --secret-file ./credentials-velero \ 559 | --backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,storageAccountKeyEnvVar=AZURE_STORAGE_ACCOUNT_ACCESS_KEY[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] \ 560 | --use-volume-snapshots=false 561 | ``` 562 | 563 | Additionally, you can specify `--use-node-agent` to enable node agent support, and `--wait` to wait for the deployment to be ready. 564 | 565 | ### Optional installation steps 566 | 1. Specify [additional configurable parameters][7] for the `--backup-location-config` flag. 567 | 1. Specify [additional configurable parameters][8] for the `--snapshot-location-config` flag. 568 | 1. [Customize the Velero installation][9] further to meet your needs. 569 | 1. Velero does not officially [support for Windows containers][10]. If your cluster has both Windows and Linux agent pool, add a node selector to the `velero` deployment to run Velero only on the Linux nodes. This can be done using the below command. 570 | ```bash 571 | kubectl patch deploy velero --namespace velero --type merge --patch '{ \"spec\": { \"template\": { \"spec\": { \"nodeSelector\": { \"beta.kubernetes.io/os\": \"linux\"} } } } }' 572 | ``` 573 | 574 | 575 | For more complex installation needs, use either the Helm chart, or add `--dry-run -o yaml` options for generating the YAML representation for the installation. 576 | 577 | ## Create an additional Backup Storage Location 578 | 579 | If you are using Velero v1.6.0 or later, you can create additional Azure [Backup Storage Locations][13] that use their own credentials. 580 | These can also be created alongside Backup Storage Locations that use other providers. 581 | 582 | ### Limitations 583 | It is not possible to use different credentials for additional Backup Storage Locations if you are pod based authentication such as [Azure AD Workload Identity][23]. 584 | 585 | ### Prerequisites 586 | 587 | * Velero 1.6.0 or later 588 | * Azure plugin must be installed, either at install time, or by running `velero plugin add velero/velero-plugin-for-microsoft-azure:plugin-version`, replace the `plugin-version` with the corresponding value 589 | 590 | ### Configure the blob container and credentials 591 | 592 | To configure a new Backup Storage Location with its own credentials, it is necessary to follow the steps above to [create the storage account and blob container to use][1], and generate the credentials file to interact with that blob container. 593 | You can either [create a service principal][15] or [use a storage account access key][16] to create the credentials file. 594 | Once you have created the credentials file, create a [Kubernetes Secret][17] in the Velero namespace that contains these credentials: 595 | 596 | ```bash 597 | kubectl create secret generic -n velero bsl-credentials --from-file=azure= 598 | ``` 599 | 600 | This will create a secret named `bsl-credentials` with a single key (`azure`) which contains the contents of your credentials file. 601 | The name and key of this secret will be given to Velero when creating the Backup Storage Location, so it knows which secret data to use. 602 | 603 | ### Create Backup Storage Location 604 | 605 | Once the bucket and credentials have been configured, these can be used to create the new Backup Storage Location. 606 | 607 | If you are using a service principal, create the Backup Storage Location as follows: 608 | 609 | ```bash 610 | velero backup-location create \ 611 | --provider azure \ 612 | --bucket $BLOB_CONTAINER \ 613 | --config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] \ 614 | --credential=bsl-credentials=azure 615 | ``` 616 | 617 | Otherwise, use the following command if you are using a storage account access key: 618 | 619 | ```bash 620 | velero backup-location create \ 621 | --provider azure \ 622 | --bucket $BLOB_CONTAINER \ 623 | --config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,storageAccountKeyEnvVar=AZURE_STORAGE_ACCOUNT_ACCESS_KEY[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] \ 624 | --credential=bsl-credentials=azure 625 | ``` 626 | 627 | If you would like to customize the Storage Location and or the AAD URI associated with the backup location add the following to the `--config` argument: 628 | ```bash 629 | --config storageAccountURI='https://my-sa.blob.core.windows.net',activeDirectoryAuthorityURI='https://login.microsoftonline.us/' 630 | ``` 631 | 632 | The Backup Storage Location is ready to use when it has the phase `Available`. 633 | You can check this with the following command: 634 | 635 | ```bash 636 | velero backup-location get 637 | ``` 638 | 639 | To use this new Backup Storage Location when performing a backup, use the flag `--storage-location ` when running `velero backup create`. 640 | 641 | ## Extra security measures 642 | 643 | To improve security within Azure, it's good practice [to disable public traffic to your Azure Storage Account][26]. If your AKS cluster is in the same Azure Region as your storage account, access to your Azure Storage Account should be easily enabled by a [Virtual Network endpoint][27] on your VNet. 644 | 645 | ## Tips 646 | We recommend taking incremental snapshots of Azure Disks since they are more cost efficient and come with the following benefits (Read more: [Azure Docs][30]): 647 | 648 | > Incremental snapshots are point-in-time backups for managed disks that, when taken, consist only of the changes since the last snapshot. The first incremental snapshot is a full copy of the disk. The subsequent incremental snapshots occupy only delta changes to disks since the last snapshot. When you restore a disk from an incremental snapshot, the system reconstructs the full disk that represents the point in time backup of the disk when the incremental snapshot was taken. 649 | 650 | > If ZRS is available in the selected region, an incremental snapshot will use ZRS automatically. If ZRS isn't available in the region, then the snapshot will default to locally-redundant storage (LRS) 651 | 652 | **To enable Incremental snapshots, set `incremental` to `true` as part of `--snapshot-location-config`. Refer [additional configurable parameters][8] for the `--snapshot-location-config` flag.** 653 | 654 | [1]: #Create-Azure-storage-account-and-blob-container 655 | [2]: #Set-permissions-for-Velero 656 | [3]: #Install-and-start-Velero 657 | [4]: #Get-resource-group-containing-your-VMs-and-disks 658 | [6]: https://velero.io/docs/install-overview/ 659 | [7]: backupstoragelocation.md 660 | [8]: volumesnapshotlocation.md 661 | [9]: https://velero.io/docs/customize-installation/ 662 | [10]:https://velero.io/docs/v1.4/basic-install/#velero-on-windows 663 | [11]: https://velero.io/docs/faq/ 664 | [12]: #create-an-additional-backup-storage-location 665 | [13]: https://velero.io/docs/latest/api-types/backupstoragelocation/ 666 | [15]: #option-1-create-service-principal 667 | [16]: #option-3-use-storage-account-access-key 668 | [17]: https://kubernetes.io/docs/concepts/configuration/secret/ 669 | [20]: https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-application-objects 670 | [21]: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli 671 | [22]: https://docs.microsoft.com/en-us/azure/architecture/best-practices/naming-conventions#storage 672 | [23]: https://azure.github.io/azure-workload-identity/docs/introduction.html 673 | [24]: https://azure.github.io/azure-workload-identity/docs/quick-start.html 674 | [25]: https://azure.microsoft.com/en-us/services/kubernetes-service/ 675 | [26]: https://docs.microsoft.com/en-us/azure/storage/common/storage-network-security 676 | [27]: https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-service-endpoints-overview 677 | [28]: https://azure.github.io/azure-workload-identity/docs/installation/mutating-admission-webhook.html 678 | [29]: https://learn.microsoft.com/en-us/azure/aks/use-oidc-issuer#create-an-aks-cluster-with-oidc-issuer 679 | [30]: https://learn.microsoft.com/en-us/azure/virtual-machines/disks-incremental-snapshots 680 | [101]: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/workflows/Main%20CI/badge.svg 681 | [102]: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/actions?query=workflow%3A"Main+CI" 682 | [103]: https://github.com/vmware-tanzu/velero/issues/new/choose 683 | --------------------------------------------------------------------------------