├── .gitignore ├── LICENSE ├── NOTICE ├── README.md ├── ci ├── pipeline.yml ├── scripts │ └── export-director-metadata ├── secrets.sample.yml └── tasks │ └── repave │ ├── task.sh │ └── task.yml └── docker └── Dockerfile /.gitignore: -------------------------------------------------------------------------------- 1 | secrets.yml 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Redistribution and use in source and binary forms, with or without 2 | modification, are permitted provided that the following conditions are 3 | met: 4 | 5 | 1. Redistributions of source code must retain the above copyright 6 | notice, this list of conditions and the following disclaimer. 7 | 8 | 2. Redistributions in binary form must reproduce the above 9 | copyright notice, this list of conditions and the following 10 | disclaimer in the documentation and/or other materials provided 11 | with the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 14 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 15 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 16 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 17 | HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 18 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 19 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 20 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 21 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 22 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 23 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | Copyright 2022 VMware, Inc. 2 | 3 | This product is licensed to you under the BSD 2 clause (the "License"). You may not use this product except in compliance with the License. 4 | 5 | This product may include a number of subcomponents with separate copyright notices and license terms. Your use of these subcomponents is subject to the terms and conditions of the subcomponent's license, as noted in the LICENSE file. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Bosh Repave 2 | 3 | This is a Concourse pipeline sample that will perform the repave of VMs of selected bosh deployments in a PCF installation on a pre-defined schedule. 4 | 5 | The pipeline also allows for the selection of the VMs to be repaved within each deployment, either ALL or only ones of Non-Singleton instances. 6 | 7 | The pipeline scripts use `bosh recreate` as the mechanism to perform the repave of VMs. 8 | 9 | The scripts iterate through the list of selected deployments provided as a pipeline parameter (e.g. `cf, apm, ...`) and then issue a `bosh recreate` command for the targeted instances. 10 | 11 | 12 | ### Notes 13 | 14 | - The purpose of the pipeline is to provide a mechanism to selectively repave VMs of selected deployments on a regular basis, for organizations that require such procedure for any reason such as internal security regulations. 15 | 16 | - If you do not simply wish to target all VMs within one or more deployments, a typical pattern is to have a pipeline per deployment (only one deployment specified in each pipeline's `deployments` parameter). You can then set the `jobs` parameter for fine-grained control of what's repaved. 17 | 18 | - The use and the execution of this pipeline should be done with proper caution and planning, as the recreation of VMs that implement singleton jobs may cause outages to the platform. The pipeline provides a flag to skip VMs of singleton jobs. 19 | 20 | - This pipeline should be scheduled to run in a time-window that does not coincide with other procedures that may also affect the platform's availability or performance, such as backups, updates or upgrades. A scheduler (time resource) is provided by default in the pipeline implementation. 21 | 22 | - In addition to the scheduler resource, a "deployments lock" mechanism is implemented by the pipeline. Such mechanism should also be used in other pipelines such as the ones for backups and upgrades, so their execution will not collide with the repaving of VMs. 23 | 24 | 25 | ### How to use the pipeline 26 | 27 | 1. Clone this git project locally 28 | 1. Make a copy of `ci/secrets.sample.yml`: 29 | `cp ci/secrets.sample.yml ci/secrets.yml` 30 | 1. Edit the copy of the secrets file: 31 | `opsman-url`: Ops Manager URL. e.g. https://pcf.example.com 32 | `opsman-username`: Ops Manager admin user. e.g. `admin` 33 | `opsman-password`: Ops Manager user password 34 | `skip-ssl-validation`: skip SSL validation for Ops Manager login e.g. `true` or `false` 35 | `deployments`: comma-separated list of bosh deployments to repave. It has to contain the prefix of PCF deployed releases (from the output of `bosh deployments`, remove the `-XXXXXXX...` numeric suffix from the deployment names generated by PCF), e.g. `cf,apm` 36 | `jobs`: comma-separated list of jobs to repave. When using this, you should only specify a single deployment. This can be used to repave select VMs e.g. Diego cells within the `cf` deployment. See the related note above for a common pattern when using this feature. 37 | `repave-singleton-jobs`: flag to control which VM instances will be repaved. e.g. `true` repave all VMs of all jobs, `false` repave only jobs with more than one instance. 38 | `perform-dry-run-only`: for testing the repave action with a _dry-run_ of _bosh recreate_ without actually recreating any VMs. e.g. `true` perform dry-run, `false` execute repave 39 | `scheduler-time-window-start`: initial time for the scheduler's execution time window, e.g. `1:00 AM` 40 | `scheduler-time-window-stop`: end time for the scheduler's execution time window, e.g. `2:00 AM` 41 | `scheduler-time-location`: time zone for the scheduler. e.g. `America/Phoenix`. See [docs](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for accepted values. 42 | `lock-git-repo-uri`: The git repo URL for the pool resource used as a deployment lock. See [docs](https://github.com/concourse/pool-resource) for more information on how to bootstrap a pool resource repository. 43 | `lock-git-repo-branch`: git branch for the lock repository. e.g. `master` 44 | `lock-pool-name`: the pool name for the lock 45 | `lock-git-private-key`: private key for the pool resource repository 46 | 47 | 1. Create the pipeline with the `fly` command 48 | e.g. `fly -t sp -p repave -c ci/pipeline.yml -l ci/secrets.yml` 49 | 50 | 1. Unpause the pipeline to either run it manually or when the scheduler generates an event within the specified time-window. 51 | -------------------------------------------------------------------------------- /ci/pipeline.yml: -------------------------------------------------------------------------------- 1 | jobs: 2 | - name: repave 3 | serial: true 4 | plan: 5 | - put: lock 6 | params: {acquire: true} 7 | - get: bosh-repave-repo 8 | trigger: false 9 | - get: scheduler 10 | trigger: true 11 | - task: repave 12 | file: bosh-repave-repo/ci/tasks/repave/task.yml 13 | params: 14 | SKIP_SSL_VALIDATION: {{skip-ssl-validation}} 15 | OPSMAN_URL: {{opsman-url}} 16 | OPSMAN_USERNAME: {{opsman-username}} 17 | OPSMAN_PASSWORD: {{opsman-password}} 18 | DEPLOYMENTS: {{deployments}} 19 | JOBS: {{jobs}} 20 | REPAVE_SINGLETON_JOBS: {{repave-singleton-jobs}} 21 | PERFORM_DRY_RUN_ONLY: {{perform-dry-run-only}} 22 | ensure: 23 | put: lock 24 | params: 25 | release: lock 26 | 27 | resources: 28 | - name: bosh-repave-repo 29 | type: git 30 | source: 31 | uri: https://github.com/pivotalservices/bosh-repave.git 32 | branch: master 33 | - name: scheduler 34 | type: time 35 | source: 36 | start: {{scheduler-time-window-start}} 37 | stop: {{scheduler-time-window-stop}} 38 | location: {{scheduler-time-location}} 39 | - name: lock 40 | type: pool 41 | source: 42 | uri: {{lock-git-repo-uri}} 43 | branch: {{lock-git-repo-branch}} 44 | pool: {{lock-pool-name}} 45 | private_key: {{lock-git-private-key}} 46 | -------------------------------------------------------------------------------- /ci/scripts/export-director-metadata: -------------------------------------------------------------------------------- 1 | #!/bin/bash -eu 2 | 3 | shopt -s expand_aliases 4 | 5 | skip_ssl=${SKIP_SSL_VALIDATION:-false} 6 | skip_ssl_flag="" 7 | 8 | if ${skip_ssl}; then 9 | skip_ssl_flag="--skip-ssl-validation" 10 | fi 11 | 12 | alias om='om-linux ${skip_ssl_flag} --target "${OPSMAN_URL}" --username "${OPSMAN_USERNAME}" --password "${OPSMAN_PASSWORD}"' 13 | 14 | om curl -p /api/v0/deployed/director/manifest > director_manifest.json 15 | 16 | BOSH_CLIENT="ops_manager" 17 | BOSH_CLIENT_SECRET=$(jq -r '.jobs[] | select(.name == "bosh") | .properties.uaa.clients.ops_manager.secret' director_manifest.json) 18 | BOSH_ADDRESS=$(jq -r '.jobs[] | select(.name == "bosh") | .properties.director.address' director_manifest.json) 19 | 20 | BOSH_CA_CERT_PATH="${PWD}/bosh.crt" 21 | jq -r '.jobs[] | select(.name == "bosh") | .properties.director.config_server.ca_cert' director_manifest.json > "${BOSH_CA_CERT_PATH}" 22 | 23 | export BOSH_CLIENT 24 | export BOSH_CLIENT_SECRET 25 | export BOSH_CA_CERT_PATH 26 | export BOSH_ADDRESS 27 | -------------------------------------------------------------------------------- /ci/secrets.sample.yml: -------------------------------------------------------------------------------- 1 | opsman-url: https://pcf.example.com 2 | opsman-username: example-user 3 | opsman-password: example-password 4 | skip-ssl-validation: true 5 | deployments: cf,apm 6 | jobs: "" 7 | repave-singleton-jobs: false 8 | perform-dry-run-only: true 9 | scheduler-time-window-start: 1:00 AM 10 | scheduler-time-window-stop: 2:00 AM 11 | scheduler-time-location: America/Phoenix 12 | lock-git-repo-uri: 13 | lock-git-repo-branch: master 14 | lock-pool-name: 15 | lock-git-private-key: | 16 | -----BEGIN RSA PRIVATE KEY----- 17 | 18 | -----END RSA PRIVATE KEY----- 19 | -------------------------------------------------------------------------------- /ci/tasks/repave/task.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -eu 2 | 3 | source ./bosh-repave-repo/ci/scripts/export-director-metadata 4 | 5 | echo "Logging in to bosh $BOSH_ADDRESS" 6 | bosh alias-env bosh -e "${BOSH_ADDRESS}" --ca-cert "${BOSH_CA_CERT_PATH}" 7 | 8 | export BOSH_ENVIRONMENT=bosh 9 | 10 | om curl -p /api/v0/deployed/products > deployed_products.json 11 | 12 | # loop through deployments 13 | for deployment in $(echo $DEPLOYMENTS | sed "s/,/ /g") 14 | do 15 | DEPLOYMENT_NAME=$(jq --arg deployment "$deployment" -r '.[] | select( .type | contains($deployment)) | .guid' "deployed_products.json") 16 | 17 | jobIndexToCheck="1" 18 | if [ "${REPAVE_SINGLETON_JOBS,,}" == "true" ]; then 19 | jobIndexToCheck="0" 20 | fi 21 | 22 | dryRunStatement="" 23 | if [ "${PERFORM_DRY_RUN_ONLY,,}" == "true" ]; then 24 | dryRunStatement="--dry-run" 25 | fi 26 | 27 | # get list of vms 28 | if [ -z "$JOBS" ] 29 | then 30 | JOBS=$(bosh -d $DEPLOYMENT_NAME --json instances -i | jq --arg jobIndexToCheck "$jobIndexToCheck" -r '.Tables[].Rows[] | select(.index==$jobIndexToCheck) | .instance | split("/")[0]') 31 | else 32 | JOBS=$(echo $JOBS | sed "s/,/ /g") 33 | fi 34 | 35 | for job in $JOBS 36 | do 37 | echo "Recreating job [$job]" 38 | bosh -d $DEPLOYMENT_NAME -n recreate $dryRunStatement $job 39 | done 40 | 41 | done 42 | -------------------------------------------------------------------------------- /ci/tasks/repave/task.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | image_resource: 4 | type: docker-image 5 | source: 6 | repository: pivotalservices/bosh2 7 | tag: latest 8 | 9 | inputs: 10 | - name: bosh-repave-repo 11 | 12 | run: 13 | path: bosh-repave-repo/ci/tasks/repave/task.sh 14 | 15 | params: 16 | SKIP_SSL_VALIDATION: false 17 | OPSMAN_URL: 18 | OPSMAN_USERNAME: 19 | OPSMAN_PASSWORD: 20 | DEPLOYMENTS: 21 | JOBS: 22 | REPAVE_SINGLETON_JOBS: 23 | PERFORM_DRY_RUN_ONLY: 24 | -------------------------------------------------------------------------------- /docker/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM czero/cflinuxfs2:latest 2 | 3 | RUN \ 4 | apt-get update && \ 5 | wget -O bosh https://s3.amazonaws.com/bosh-cli-artifacts/bosh-cli-2.0.40-linux-amd64 && \ 6 | install bosh /usr/local/bin/bosh 7 | --------------------------------------------------------------------------------