├── .codeclimate.yml ├── .github ├── pull_request_template.md └── workflows │ └── security-considerations.yml ├── .gitignore ├── CODEOWNERS ├── CONTRIBUTING.md ├── LICENSE.md ├── README.md ├── SECURITY.md ├── bosh-errand.sh ├── bosh-errand.yml ├── bosh-logs-errand.sh ├── bosh-logs-errand.yml ├── decrypt.sh ├── decrypt.yml ├── diff-templates.rb ├── diff-templates.sh ├── diff-templates.yml ├── display.sh ├── display.yml ├── encrypt.sh ├── encrypt.yml ├── ensure-misbehaved-service.sh ├── ensure-misbehaved-service.yml ├── ensure-service.sh ├── ensure-service.yml ├── finalize-bosh-release.sh ├── finalize-bosh-release.yml ├── generate_key.sh ├── gh-release-notifier.sh ├── gh-release-notifier.yml ├── inflate.sh ├── inflate.yml ├── inspect.sh ├── inspect.yml ├── register-service-broker-and-set-plan-visibility.sh ├── register-service-broker-and-set-plan-visibility.yml ├── register-service-broker-exclude-sandboxes.yml ├── register-service-broker-only-sandboxes.yml ├── register-service-broker.sh ├── register-service-broker.yml ├── security-considerations ├── README.md ├── generate-pipeline.py ├── repos.txt ├── security-considerations.sh └── security-considerations.yml ├── set-plan-visibility.sh ├── set-plan-visibility.yml ├── terraform-apply.sh ├── terraform-apply.yml ├── terraform-destroy.sh ├── terraform-destroy.yml ├── terraform-state-to-yaml.rb ├── terraform-state-to-yaml.sh ├── terraform-state-to-yaml.yml ├── terraform12-state-to-yaml.rb ├── terraform12-state-to-yaml.sh ├── terraform12-state-to-yaml.yml ├── uaa-smoke-tests.py ├── uaa-smoke-tests.sh ├── uaa-smoke-tests.yml ├── upload-release.sh ├── upload-release.yml ├── write_file.sh └── write_file.yml /.codeclimate.yml: -------------------------------------------------------------------------------- 1 | engines: 2 | shellcheck: 3 | enabled: true 4 | 5 | ratings: 6 | paths: 7 | - "**.sh" 8 | -------------------------------------------------------------------------------- /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | ## Changes proposed in this pull request: 2 | - 3 | - 4 | - 5 | 6 | ## security considerations 7 | [Note the any security considerations here, or make note of why there are none] 8 | -------------------------------------------------------------------------------- /.github/workflows/security-considerations.yml: -------------------------------------------------------------------------------- 1 | name: Security Considerations 2 | 3 | on: 4 | pull_request: 5 | types: [opened, edited, reopened] 6 | branches: [main, master, develop] 7 | 8 | jobs: 9 | security-considerations: 10 | runs-on: ubuntu-latest 11 | steps: 12 | - uses: cloud-gov/security-considerations-action@main 13 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | tmp/ 2 | security-considerations/pipeline.yml 3 | security-considerations/cg-security-considerations.yml 4 | -------------------------------------------------------------------------------- /CODEOWNERS: -------------------------------------------------------------------------------- 1 | * @cloud-gov/platform-ops 2 | 3 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | ## Public domain 2 | 3 | This project is in the public domain within the United States, and 4 | copyright and related rights in the work worldwide are waived through 5 | the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/). 6 | 7 | All contributions to this project will be released under the CC0 8 | dedication. By submitting a pull request, you are agreeing to comply 9 | with this waiver of copyright interest. 10 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | As a work of the United States Government, this project is in the 2 | public domain within the United States. 3 | 4 | Additionally, we waive copyright and related rights in the work 5 | worldwide through the CC0 1.0 Universal public domain dedication. 6 | 7 | ## CC0 1.0 Universal Summary 8 | 9 | This is a human-readable summary of the 10 | [Legal Code (read the full text)](https://creativecommons.org/publicdomain/zero/1.0/legalcode). 11 | 12 | ### No Copyright 13 | 14 | The person who associated a work with this deed has dedicated the work to 15 | the public domain by waiving all of his or her rights to the work worldwide 16 | under copyright law, including all related and neighboring rights, to the 17 | extent allowed by law. 18 | 19 | You can copy, modify, distribute and perform the work, even for commercial 20 | purposes, all without asking permission. 21 | 22 | ### Other Information 23 | 24 | In no way are the patent or trademark rights of any person affected by CC0, 25 | nor are the rights that other persons may have in the work or in how the 26 | work is used, such as publicity or privacy rights. 27 | 28 | Unless expressly stated otherwise, the person who associated a work with 29 | this deed makes no warranties about the work, and disclaims liability for 30 | all uses of the work, to the fullest extent permitted by applicable law. 31 | When using or citing the work, you should not imply endorsement by the 32 | author or the affirmer. 33 | 34 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## cloud.gov common Concourse pipeline tasks 2 | 3 | This repo contains the source for some common concourse-ci.org pipeline tasks. 4 | 5 | To include in your pipeline, define a resource named `pipeline-tasks`: 6 | ```yaml 7 | resources: 8 | 9 | ... 10 | 11 | - name: pipeline-tasks 12 | type: git 13 | source: 14 | uri: ((pipeline-tasks-git-url)) 15 | branch: ((pipeline-tasks-git-branch)) 16 | ``` 17 | 18 | Add `pipeline-tasks-git-url` and `pipeline-tasks-git-branch` to your `credentials.yml`: 19 | 20 | ```yaml 21 | --- 22 | pipeline-tasks-git-url: https://github.com/18F/cg-pipeline-tasks.git 23 | pipeline-tasks-git-branch: master 24 | ``` 25 | 26 | ## Task usage 27 | ### bosh-errand 28 | ### decrypt 29 | ### display 30 | ### encrypt 31 | ### finalize-bosh-release 32 | ### inspect 33 | ### spiff-merge 34 | ### terraform-apply 35 | ### terraform-destroy 36 | ### terraform-state-to-yaml 37 | Reads a terraform state file to get the terraform `outputs` and converts to yaml for use in spiff merging into other BOSH templates. 38 | 39 | Requires: 40 | - STATE_FILE: The terraform state file 41 | 42 | Outputs: 43 | - terraform-yaml/state.yml 44 | 45 | ```yaml 46 | - get: my-other-manifest-bits 47 | - get: s3-terraform-state 48 | - get: pipeline-tasks 49 | - task: terraform-yaml 50 | file: pipeline-tasks/terraform-state-to-yaml.yml 51 | params: 52 | STATE_FILE: s3-terraform-state/terraform.tfstate 53 | - task: generate-manifest 54 | file: pipeline-tasks/spiff-merge.yml 55 | config: 56 | inputs: 57 | - name: pipeline-tasks 58 | - name: my-other-manifest-bits 59 | params: 60 | OUTPUT_FILE: spiff-merge/manifest.yml 61 | SOURCE_FILE: my-other-manifest-bits/manifest.yml 62 | MERGE_FILES: terraform-yaml/state.yml 63 | 64 | ``` 65 | ### upload-release 66 | ### write-file -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | 2 | **Reporting Security Issues** 3 | 4 | Please refrain from reporting security vulnerabilities through public GitHub issues. 5 | 6 | Instead, kindly report them via the information provided in [cloud.gov's security.txt](https://cloud.gov/.well-known/security.txt). 7 | 8 | When reporting, include the following details (as much as possible) to help us understand the nature and extent of the potential issue: 9 | 10 | - Type of issue (e.g., buffer overflow, SQL injection, cross-site scripting, etc.) 11 | - Full paths of related source file(s) 12 | - Location of affected source code (tag/branch/commit or direct URL) 13 | - Any special configuration required to reproduce the issue 14 | - Step-by-step instructions to reproduce the issue 15 | - Proof-of-concept or exploit code (if available) 16 | - Impact of the issue, including potential exploitation by attackers 17 | 18 | Providing this information will facilitate a quicker triage of your report. 19 | -------------------------------------------------------------------------------- /bosh-errand.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: set ft=sh 3 | 4 | set -eu 5 | 6 | args=("${BOSH_ERRAND}") 7 | args+=(${BOSH_FLAGS:-}) 8 | bosh -n run-errand "${args[@]}" 9 | -------------------------------------------------------------------------------- /bosh-errand.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | - name: common 16 | outputs: 17 | - name: bosh-errand 18 | 19 | run: 20 | path: pipeline-tasks/bosh-errand.sh 21 | 22 | params: 23 | BOSH_ENVIRONMENT: 24 | BOSH_CLIENT: 25 | BOSH_CLIENT_SECRET: 26 | BOSH_DEPLOYMENT: 27 | BOSH_ERRAND: 28 | BOSH_CA_CERT: 29 | BOSH_FLAGS: 30 | -------------------------------------------------------------------------------- /bosh-logs-errand.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: set ft=sh 3 | 4 | set -eu 5 | 6 | args=("${BOSH_ERRAND}") 7 | args+=(${BOSH_FLAGS:-}) 8 | bosh -n run-errand "${args[@]}" 9 | -------------------------------------------------------------------------------- /bosh-logs-errand.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | outputs: 16 | - name: bosh-errand 17 | 18 | run: 19 | path: pipeline-tasks/bosh-errand.sh 20 | 21 | params: 22 | BOSH_ENVIRONMENT: 23 | BOSH_CLIENT: 24 | BOSH_CLIENT_SECRET: 25 | BOSH_DEPLOYMENT: 26 | BOSH_ERRAND: 27 | BOSH_CA_CERT: 28 | BOSH_FLAGS: 29 | -------------------------------------------------------------------------------- /decrypt.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: set ft=sh 3 | 4 | set -e 5 | 6 | # 7 | # A little environment validation 8 | # 9 | if [ -z "$INPUT_FILE" ]; then 10 | echo "must specify \$INPUT_FILE" >&2 11 | exit 1 12 | fi 13 | if [ -z "$OUTPUT_FILE" ]; then 14 | echo "must specify \$OUTPUT_FILE" >&2 15 | exit 1 16 | fi 17 | if [ -z "$PASSPHRASE" ]; then 18 | echo "must specify \$PASSPHRASE" >&2 19 | exit 1 20 | fi 21 | 22 | # 23 | # Decrypt input file into output file given passphrase 24 | # 25 | openssl enc -aes-256-cbc -d -a -in "$INPUT_FILE" -out "$OUTPUT_FILE" -pass "pass:$PASSPHRASE" 2>&1 >/dev/null 26 | -------------------------------------------------------------------------------- /decrypt.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | - name: encrypt 16 | 17 | outputs: 18 | - name: decrypt 19 | 20 | run: 21 | path: pipeline-tasks/decrypt.sh 22 | 23 | params: 24 | INPUT_FILE: 25 | OUTPUT_FILE: 26 | PASSPHRASE: 27 | -------------------------------------------------------------------------------- /diff-templates.rb: -------------------------------------------------------------------------------- 1 | require 'yaml' 2 | 3 | l1 = ARGV[0] 4 | l2 = ARGV[1] 5 | first = YAML.load_file(l1) 6 | second = YAML.load_file(l2) 7 | 8 | def compare_yaml_hash(cf1, cf2, context = [], diff = "") 9 | cf1.each do |key, value| 10 | 11 | unless cf2.key?(key) 12 | diff << " MISSING - key : #{context.join(".")}.#{key}\n" 13 | next 14 | end 15 | 16 | value2 = cf2[key] 17 | if (value.class != value2.class) 18 | diff << " VERIFY - Key value type mismatch : #{context.join(".")}.#{key}\n" 19 | next 20 | end 21 | 22 | if value.is_a?(Hash) 23 | compare_yaml_hash(value, value2, (context + [key]), diff) 24 | next 25 | end 26 | 27 | if value.is_a?(Array) && value2.is_a?(Array) 28 | if value.length > value2.length 29 | diff << " CAUTION - Array length mismatch. Missing items? : #{context.join(".")}.#{key}\n" 30 | next 31 | end 32 | end 33 | 34 | end 35 | return diff 36 | end 37 | 38 | diff = compare_yaml_hash(first, second) 39 | if diff.to_s.strip.length != 0 40 | puts "Comparing #{l2} to upstream:" 41 | puts diff 42 | end 43 | 44 | -------------------------------------------------------------------------------- /diff-templates.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | set -e 4 | 5 | # 6 | # A little environment validation 7 | # 8 | if [ -z "$TEMPLATES_DIR" ]; then 9 | echo "must specify \$TEMPLATES_DIR" >&2 10 | exit 1 11 | fi 12 | 13 | SCRIPTPATH=$( cd $(dirname $0) ; pwd -P ) 14 | 15 | # Get the cf upstream templates 16 | UPSTREAM_DIR=$SCRIPTPATH/cf-release 17 | mkdir -p $UPSTREAM_DIR 18 | git -C $UPSTREAM_DIR init --quiet 19 | git -C $UPSTREAM_DIR remote add origin https://github.com/cloudfoundry/cf-release.git 20 | git -C $UPSTREAM_DIR config core.sparsecheckout true 21 | echo templates/ > $UPSTREAM_DIR/.git/info/sparse-checkout 22 | git -C $UPSTREAM_DIR pull origin master --quiet 23 | 24 | # Install required gem if not installed 25 | if [ -z $(gem list -i yaml) ]; then 26 | gem install yaml --no-ri --no-rdoc 2>&1 > /dev/null 27 | fi 28 | 29 | # Compare the templates with a ruby script 30 | nomatch () { case "$2" in $1) return 1 ;; *) return 0 ;; esac ; } 31 | for ours in `ls $TEMPLATES_DIR/*.yml` 32 | do 33 | # ignore any "secret" files 34 | if nomatch '*secret*' "$ours"; then 35 | # Get just the filename 36 | FILE=${ours##*/} 37 | if [ -f "$SCRIPTPATH/cf-release/templates/$FILE" ]; then 38 | ruby $SCRIPTPATH/diff-templates.rb $UPSTREAM_DIR/templates/$FILE $TEMPLATES_DIR/$FILE 39 | echo 40 | else 41 | echo "Not found in cf-release/templates: $FILE" 42 | echo 43 | fi 44 | fi 45 | done 46 | -------------------------------------------------------------------------------- /diff-templates.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | outputs: 16 | - name: diff-templates 17 | run: 18 | path: pipeline-tasks/diff-templates.sh 19 | 20 | params: 21 | TEMPLATES_DIR: 22 | -------------------------------------------------------------------------------- /display.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: set ft=sh 3 | 4 | set -e -x 5 | 6 | # 7 | # A little environment validation 8 | # 9 | if [ -z "$FILE" ]; then 10 | echo "must specify \$FILE" >&2 11 | exit 1 12 | fi 13 | 14 | # 15 | # Display the contents of a file 16 | # 17 | cat "$FILE" 18 | -------------------------------------------------------------------------------- /display.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | 16 | run: 17 | path: pipeline-tasks/display.sh 18 | 19 | params: 20 | FILE: 21 | -------------------------------------------------------------------------------- /encrypt.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: set ft=sh 3 | 4 | set -e 5 | 6 | # 7 | # A little environment validation 8 | # 9 | if [ -z "$INPUT_FILE" ]; then 10 | echo "must specify \$INPUT_FILE" >&2 11 | exit 1 12 | fi 13 | if [ -z "$OUTPUT_FILE" ]; then 14 | echo "must specify \$OUTPUT_FILE" >&2 15 | exit 1 16 | fi 17 | if [ -z "$PASSPHRASE" ]; then 18 | echo "must specify \$PASSPHRASE" >&2 19 | exit 1 20 | fi 21 | 22 | # 23 | # Encrypt input file into output file given passphrase 24 | # 25 | openssl enc -aes-256-cbc -a -in "$INPUT_FILE" -out "$OUTPUT_FILE" -pass "pass:$PASSPHRASE" 2>&1 >/dev/null 26 | -------------------------------------------------------------------------------- /encrypt.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | outputs: 16 | - name: encrypt 17 | run: 18 | path: pipeline-tasks/encrypt.sh 19 | 20 | params: 21 | INPUT_FILE: 22 | OUTPUT_FILE: 23 | PASSPHRASE: 24 | -------------------------------------------------------------------------------- /ensure-misbehaved-service.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # This is for services that don't correcty report their status via `cf 4 | # service` (the aws-broker being the prime offender). Instead, this script 5 | # blocks until it can create a service key. 6 | # 7 | # If you have a well-behaved service, use `ensure-service` instead. 8 | 9 | set -euo pipefail 10 | shopt -s inherit_errexit 11 | 12 | cf api "$CF_API_URL" 13 | (set +x; cf auth "$CF_USERNAME" "$CF_PASSWORD") 14 | cf target -o "$CF_ORGANIZATION" -s "$CF_SPACE" 15 | 16 | KEY="${SVC_NAME}-ci-key" 17 | 18 | if [[ -n "$(cf service "$SVC_NAME" --guid)" ]]; then 19 | cf create-service "$SVC_SERVICE" "$SVC_PLAN" "$SVC_NAME" 20 | fi 21 | 22 | echo -n "Waiting for service instance..." 23 | while ! cf create-service-key "$SVC_NAME" "$KEY" > /dev/null 2>&1; do 24 | echo -n "." 25 | sleep 5 26 | done 27 | echo 28 | cf delete-service-key -f "$SVC_NAME" "$KEY" || true 29 | -------------------------------------------------------------------------------- /ensure-misbehaved-service.yml: -------------------------------------------------------------------------------- 1 | # This is for services that don't correcty report their status via `cf 2 | # service` (the aws-broker being the prime offender). Instead, this script 3 | # blocks until it can create a service key. 4 | # 5 | # If you have a well-behaved service, use `ensure-service` instead. 6 | 7 | platform: linux 8 | 9 | image_resource: 10 | type: registry-image 11 | source: 12 | aws_access_key_id: ((ecr_aws_key)) 13 | aws_secret_access_key: ((ecr_aws_secret)) 14 | repository: general-task 15 | aws_region: us-gov-west-1 16 | tag: latest 17 | 18 | inputs: 19 | - name: pipeline-tasks 20 | 21 | run: 22 | path: pipeline-tasks/ensure-misbehaved-service.sh 23 | -------------------------------------------------------------------------------- /ensure-service.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -eux 4 | 5 | check_service() { 6 | instance="$1" 7 | timeout="${2:-300}" 8 | elapsed=0 9 | until [ "${elapsed}" -gt "${timeout}" ]; do 10 | status=$(cf service "${instance}") 11 | if echo "${status}" | grep "create succeeded"; then 12 | return 0 13 | elif echo "${status}" | grep "create failed"; then 14 | return 1 15 | fi 16 | let elapsed+=5 17 | sleep 5 18 | done 19 | return 1 20 | } 21 | 22 | # Authenticate 23 | cf api "${CF_API_URL}" 24 | (set +x; cf auth "${CF_USERNAME}" "${CF_PASSWORD}") 25 | cf target -o "${CF_ORGANIZATION}" -s "${CF_SPACE}" 26 | 27 | if ! cf service "${INSTANCE_NAME}"; then 28 | cf create-service "${SERVICE_NAME}" "${PLAN_NAME}" "${INSTANCE_NAME}" 29 | fi 30 | 31 | check_service "${INSTANCE_NAME}" "${TIMEOUT}" 32 | -------------------------------------------------------------------------------- /ensure-service.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | 16 | run: 17 | path: pipeline-tasks/ensure-service.sh 18 | -------------------------------------------------------------------------------- /finalize-bosh-release.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: set ft=sh 3 | 4 | set -e -u 5 | 6 | cd release-git-repo 7 | RELEASE_NAME=$(grep final_name config/final.yml | awk '{print $2}') 8 | 9 | tar -zxf "../final-builds-dir-tarball/final-builds-dir-${RELEASE_NAME}.tgz" 10 | tar -zxf "../releases-dir-tarball/releases-dir-${RELEASE_NAME}.tgz" 11 | cat < "config/private.yml" 12 | $PRIVATE_YML_CONTENT 13 | EOF 14 | 15 | if [ -n "$FINAL_YML_CONTENT" ]; then 16 | cat < "config/final.yml" 17 | $FINAL_YML_CONTENT 18 | EOF 19 | fi 20 | 21 | bosh-cli -n create-release --force --final --tarball="./${RELEASE_NAME}.tgz" 22 | latest_release=$(echo releases/"${RELEASE_NAME}"/"${RELEASE_NAME}"*.yml | grep -oe '[0-9.]\+.yml' | sed -e 's/\.yml$//' | sort -V | tail -1) 23 | mv "${RELEASE_NAME}.tgz" "../finalized-release/${RELEASE_NAME}-${latest_release}.tgz" 24 | 25 | tar -czhf "../finalized-release/final-builds-dir-${RELEASE_NAME}.tgz" .final_builds 26 | tar -czhf "../finalized-release/releases-dir-${RELEASE_NAME}.tgz" releases 27 | -------------------------------------------------------------------------------- /finalize-bosh-release.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | - name: release-git-repo 16 | - name: final-builds-dir-tarball 17 | - name: releases-dir-tarball 18 | outputs: 19 | - name: finalized-release 20 | run: 21 | path: pipeline-tasks/finalize-bosh-release.sh 22 | 23 | params: 24 | PRIVATE_YML_CONTENT: 25 | FINAL_YML_CONTENT: 26 | -------------------------------------------------------------------------------- /generate_key.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: set ft=sh 3 | 4 | set -e 5 | 6 | # 7 | # A little environment validation 8 | # 9 | if [ -z "$PASSPHRASE" ]; then 10 | echo "must specify \$PASSPHRASE" >&2 11 | exit 1 12 | fi 13 | 14 | if [ -z "$BUCKET" ]; then 15 | echo "must specify \$BUCKET" >&2 16 | exit 1 17 | fi 18 | 19 | # 20 | # Generate the key 21 | # 22 | TIME="$(date +"%s")" 23 | NAME="bosh-$TIME" 24 | 25 | openssl genrsa -out $NAME.pem 4096 26 | 27 | openssl rsa -in $NAME.pem -pubout > $NAME.pub 28 | 29 | # 30 | # Upload to EC2 as an access key 31 | # 32 | aws ec2 import-key-pair --key-name $NAME --public-key-material `sed '1d;$d' $NAME.pub | tr -d '\n'` 33 | 34 | # 35 | # Encrypt it and upload it to S3 36 | # 37 | INPUT_FILE=$NAME.pem OUTPUT_FILE=$NAME.enc.pem ./encrypt.sh 38 | 39 | aws s3 cp $NAME.enc.pem s3://$BUCKET/$NAME.enc.pem 40 | -------------------------------------------------------------------------------- /gh-release-notifier.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Creates a new issue based on a github-release Concourse resource. Optionally 4 | # closes old issues created by this task. 5 | # 6 | # Note: It is possible this task could close unintended tasks. However, this should be highly unlikely 7 | # as we use a label that starts with `dependency-` and only find issues created by cg-ci-bot. To 8 | # avoid issues, don't use a label that starts with `dependency-`. 9 | 10 | if [[ -z "$DEPENDENCY_NAME" ]] || [[ -z "$GH_REPO" ]] || [[ -z "$GH_TOKEN" ]] || [[ -z "$PROJECT_NAME" ]] || [[ -z "$SQUAD_LABEL" ]]; then 11 | cat <&2 11 | exit 1 12 | fi 13 | 14 | # 15 | # Temporary file for storing compressed directory 16 | # 17 | NOW=`date +"%m-%d-%Y-%H-%M-%S"` 18 | TEMP_FILE="inflate-$NOW.tar.gz" 19 | 20 | # 21 | # Inflate a given directory into a new one with symbolic links dereferenced 22 | # 23 | tar -C "$INPUT_DIR" -czhf "$TEMP_FILE" . 24 | tar -C inflate -xzf "$TEMP_FILE" 25 | 26 | # 27 | # Cleaning up the temp file 28 | # 29 | rm -f "$TEMP_FILE" 30 | -------------------------------------------------------------------------------- /inflate.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | outputs: 16 | - name: inflate 17 | run: 18 | path: pipeline-tasks/inflate.sh 19 | 20 | params: 21 | INPUT_DIR: 22 | -------------------------------------------------------------------------------- /inspect.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: set ft=sh 3 | 4 | set -e -x 5 | 6 | # 7 | # A little environment validation 8 | # 9 | if [ -z "$RESOURCE" ]; then 10 | echo "must specify \$RESOURCE" >&2 11 | exit 1 12 | fi 13 | 14 | # 15 | # Decrypt input file into output file given passphrase 16 | # 17 | ls -al $RESOURCE 18 | -------------------------------------------------------------------------------- /inspect.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | 16 | run: 17 | path: pipeline-tasks/inspect.sh 18 | 19 | params: 20 | RESOURCE: 21 | -------------------------------------------------------------------------------- /register-service-broker-and-set-plan-visibility.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -eux 4 | 5 | here=$(dirname $0) 6 | pushd $(dirname "$0") 7 | ./register-service-broker.sh 8 | ./set-plan-visibility.sh "$@" 9 | popd 10 | -------------------------------------------------------------------------------- /register-service-broker-and-set-plan-visibility.yml: -------------------------------------------------------------------------------- 1 | --- 2 | ###### 3 | # Register a service broker and set identical plan availabilities for all the broker's plans 4 | # In the future, services should call this as `register-service-broker-and-set-plan-visibility.yml` 5 | # then we should unlink `register-service-broker.yml` for uses without messing with plan visibility 6 | ###### 7 | platform: linux 8 | 9 | image_resource: 10 | type: registry-image 11 | source: 12 | aws_access_key_id: ((ecr_aws_key)) 13 | aws_secret_access_key: ((ecr_aws_secret)) 14 | repository: general-task 15 | aws_region: us-gov-west-1 16 | tag: latest 17 | 18 | inputs: 19 | - name: pipeline-tasks 20 | 21 | run: 22 | path: pipeline-tasks/register-service-broker-and-set-plan-visibility.sh 23 | 24 | params: 25 | CF_API_URL: 26 | CF_USERNAME: 27 | CF_PASSWORD: 28 | CF_ORGANIZATION: 29 | CF_SPACE: 30 | BROKER_NAME: 31 | AUTH_USER: 32 | AUTH_PASS: 33 | SERVICES: 34 | -------------------------------------------------------------------------------- /register-service-broker-exclude-sandboxes.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | 16 | run: 17 | path: pipeline-tasks/register-service-broker-and-set-plan-visibility.sh 18 | args: 19 | - -s 20 | -------------------------------------------------------------------------------- /register-service-broker-only-sandboxes.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | 16 | run: 17 | path: pipeline-tasks/register-service-broker-and-set-plan-visibility.sh 18 | args: 19 | - -o 20 | -------------------------------------------------------------------------------- /register-service-broker.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | ############# 4 | # Register a service broker with CF 5 | # environment config 6 | # CF_API_URL API to auth against 7 | # CF_USERNAME User to auth as. Needs to have cf admin permissions 8 | # CF_PASSWORD Password for user 9 | # CF_SPACE Space to target - if the broker is a CF app in $CF_SPACE and $BROKER_URL is unset, we can discover the broker's url this way 10 | # CF_ORGANIZATION org to target - if the broker is a CF app in $CF_ORGANIZATION and $BROKER_URL is unset, we can discover the broker's url this way 11 | # BROKER_NAME Name to register broker as, should also match the broker's app name to leverage BROKER_URL discovery 12 | # BROKER_URL URL where CAPI can reach the service broker 13 | # AUTH_USER Username for CAPI to talk to the broker 14 | # AUTH_PASS Password for CAPI to talk to the broker 15 | # 16 | ############# 17 | 18 | # Authenticate 19 | cf api "${CF_API_URL}" 20 | (set +x; cf auth "${CF_USERNAME}" "${CF_PASSWORD}") 21 | cf target -o "${CF_ORGANIZATION}" -s "${CF_SPACE}" 22 | 23 | # Get service broker URL 24 | if [ -z "${BROKER_URL:-}" ]; then 25 | BROKER_URL=https://$(cf app "${BROKER_NAME}" | grep routes: | awk '{print $2}') 26 | fi 27 | 28 | # Create or update service broker 29 | if ! cf create-service-broker "${BROKER_NAME}" "${AUTH_USER}" "${AUTH_PASS}" "${BROKER_URL}"; then 30 | cf update-service-broker "${BROKER_NAME}" "${AUTH_USER}" "${AUTH_PASS}" "${BROKER_URL}" 31 | fi 32 | -------------------------------------------------------------------------------- /register-service-broker.yml: -------------------------------------------------------------------------------- 1 | register-service-broker-and-set-plan-visibility.yml -------------------------------------------------------------------------------- /security-considerations/README.md: -------------------------------------------------------------------------------- 1 | # Security Considerations job 2 | 3 | This is a concourse pipeline that checks to see that every pull 4 | request has a Security Considerations section. 5 | 6 | 7 | # Updating the job for repo changes 8 | 1. add/remove the repos as necessary in repos.txt 9 | 2. run `python generate-pipeline.py` 10 | 3. get the credentials file from our secrets store. Call it security-considerations.yml 11 | 4. run `fly -t fr sp -p security-considerations-check -c ./pipeline.yml -l cg-security-considerations.yml` 12 | 13 | ## making sure your PR passes 14 | 1. consider the security impacts of your changes 15 | 2. create a section in your PR that looks like this: 16 | ``` 17 | # Security Considerations 18 | 19 | ``` 20 | The "security considerations" must be a heading, using the # syntax, but it can be any level of heading, and any capitalization you'd like 21 | -------------------------------------------------------------------------------- /security-considerations/generate-pipeline.py: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | import os 3 | 4 | 5 | def main(): 6 | repos_file = find_repos_file() 7 | repos = [] 8 | context = "pr-message-includes-security-considerations" 9 | with open(repos_file, "r") as f: 10 | for line in f: 11 | line = line.strip() 12 | if line and not line.startswith('#'): 13 | repos.append(line) 14 | with open(repos_file.parent / "pipeline.yml", "w") as f: 15 | 16 | f.write("---\njobs:") 17 | for repo in repos: 18 | resource_name = pr_resource_name(repo) 19 | f.write(f""" 20 | - name: pull-status-check-{repo_name(repo)} 21 | plan: 22 | - get: pipeline-tasks 23 | - get: {resource_name} 24 | version: every 25 | trigger: true 26 | - put: {resource_name} 27 | params: 28 | path: {resource_name} 29 | status: pending 30 | context: {context} 31 | - task: build 32 | input_mapping: 33 | pull-request: {resource_name} 34 | file: pipeline-tasks/security-considerations/security-considerations.yml 35 | on_success: 36 | put: {resource_name} 37 | params: 38 | path: {resource_name} 39 | status: success 40 | context: {context} 41 | on_failure: 42 | put: {resource_name} 43 | params: 44 | path: {resource_name} 45 | status: failure 46 | context: {context} 47 | """) 48 | f.write(""" 49 | resources: 50 | - name: pipeline-tasks 51 | type: git 52 | source: 53 | uri: ((pipeline-tasks-git-url)) 54 | branch: ((pipeline-tasks-git-branch)) 55 | """) 56 | for repo in repos: 57 | resource_name = pr_resource_name(repo) 58 | f.write(f""" 59 | - name: {resource_name} 60 | type: pull-request 61 | source: 62 | repo: {repo} 63 | access_token: ((status-access-token)) 64 | every: true 65 | """) 66 | f.write(""" 67 | resource_types: 68 | - name: pull-request 69 | type: docker-image 70 | source: 71 | repository: jtarchie/pr 72 | """) 73 | 74 | 75 | def repo_name(repo): 76 | """make a resource name from a repo name""" 77 | return repo.split("/")[1] 78 | 79 | def pr_resource_name(repo): 80 | return f"pull-request-{repo_name(repo)}" 81 | 82 | def find_repos_file(): 83 | script_dir = os.path.dirname(os.path.realpath(__file__)) 84 | repos_file = Path(script_dir) / "repos.txt" 85 | if repos_file.exists(): 86 | return repos_file 87 | else: 88 | raise Exception("Can't find repos.txt") 89 | 90 | if __name__ == "__main__": 91 | main() 92 | -------------------------------------------------------------------------------- /security-considerations/repos.txt: -------------------------------------------------------------------------------- 1 | # each line should contain a github repo name, in the form organization/repo, e.g. cloud-gov/cg-pipeline-tasks 2 | # lines where the first non-whitespace character is a # are ignored 3 | cloud-gov/aws-broker 4 | cloud-gov/cf-byo-broker 5 | cloud-gov/cf-cdn-service-broker 6 | cloud-gov/cf-domain-broker 7 | cloud-gov/cf-domain-broker-alb 8 | cloud-gov/cf-domain-broker-alb-boshrelease 9 | cloud-gov/cf-ex-drupal8 10 | cloud-gov/cf-hello-worlds 11 | cloud-gov/cf-limit-check 12 | cloud-gov/cg-awslogs-boshrelease 13 | cloud-gov/cg-billing 14 | cloud-gov/cg-bosh-dependencies 15 | cloud-gov/cg-buildpack-notify 16 | cloud-gov/cg-cert-check 17 | cloud-gov/cg-clamav-boshrelease 18 | cloud-gov/cg-deploy-admin-ui 19 | cloud-gov/cg-deploy-aws-servicebroker 20 | cloud-gov/cg-deploy-bosh 21 | cloud-gov/cg-deploy-cf 22 | cloud-gov/cg-deploy-concourse 23 | cloud-gov/cg-deploy-concourse-docker-image 24 | cloud-gov/cg-deploy-elasticache-broker 25 | cloud-gov/cg-deploy-elb-log-ingestor 26 | cloud-gov/cg-deploy-kubernetes 27 | cloud-gov/cg-deploy-logsearch 28 | cloud-gov/cg-deploy-nessus-manager 29 | cloud-gov/cg-deploy-opslogin 30 | cloud-gov/cg-deploy-postfix 31 | cloud-gov/cg-deploy-powerdns 32 | cloud-gov/cg-deploy-prometheus 33 | cloud-gov/cg-deploy-rds-broker 34 | cloud-gov/cg-deploy-riemann-firehose-nozzle 35 | cloud-gov/cg-deploy-s3-broker 36 | cloud-gov/cg-deploy-s3-logstash 37 | cloud-gov/cg-deploy-shibboleth 38 | cloud-gov/cg-deploy-stratos 39 | cloud-gov/cg-deploy-volume-services 40 | cloud-gov/cg-diagrams 41 | cloud-gov/cg-django-uaa 42 | cloud-gov/docker-registry-mirror 43 | cloud-gov/cg-elb-log-ingestor 44 | cloud-gov/cg-elb-log-ingestor-boshrelease 45 | cloud-gov/cg-harden-boshrelease 46 | cloud-gov/cg-laboratory 47 | cloud-gov/cg-migrate-db 48 | cloud-gov/cg-nessus-agent-boshrelease 49 | cloud-gov/cg-nessus-manager-boshrelease 50 | cloud-gov/cg-pipeline-tasks 51 | cloud-gov/cg-product 52 | cloud-gov/cg-provision 53 | cloud-gov/cg-python3-boshrelease 54 | cloud-gov/cg-quotas-db 55 | cloud-gov/cg-report-federalist-s3 56 | cloud-gov/cg-sandbox 57 | cloud-gov/cg-sandbox-bot 58 | cloud-gov/cg-scripts 59 | cloud-gov/cg-secret-rotation 60 | cloud-gov/cg-secureproxy-boshrelease 61 | cloud-gov/cg-site 62 | cloud-gov/cg-snort-boshrelease 63 | cloud-gov/cg-style 64 | cloud-gov/cg-tripwire-boshrelease 65 | cloud-gov/cg-uaa-extras 66 | cloud-gov/cg-windows-stemcell-builder 67 | cloud-gov/concourse-http-resource 68 | cloud-gov/deploy-credhub 69 | cloud-gov/docker-boshrelease 70 | cloud-gov/elastalert-boshrelease 71 | cloud-gov/external-domain-broker 72 | cloud-gov/external-domain-broker-migrator 73 | cloud-gov/finalized-release 74 | cloud-gov/kibana-cf-auth-proxy 75 | cloud-gov/kubernetes-broker 76 | cloud-gov/kubernetes-broker-exporter 77 | cloud-gov/kubernetes-release 78 | cloud-gov/legacy-domain-certificate-renewer 79 | cloud-gov/log-spammer 80 | cloud-gov/logsearch-boshrelease 81 | cloud-gov/logsearch-for-cloudfoundry 82 | cloud-gov/nfs-volume-release 83 | cloud-gov/oauth2-proxy-boshrelease 84 | cloud-gov/oauth2_proxy 85 | cloud-gov/pdns-release 86 | cloud-gov/postfix-boshrelease 87 | cloud-gov/s3-resource 88 | cloud-gov/s3-resource-simple 89 | cloud-gov/shibboleth-boshrelease 90 | cloud-gov/uaa-customized-boshrelease 91 | cloud-gov/uaa-bot 92 | cloud-gov/cg-uaa-extras-redis-migrator 93 | cloud-gov/container-scanning 94 | cloud-gov/cvd-sync 95 | 96 | -------------------------------------------------------------------------------- /security-considerations/security-considerations.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -x 4 | 5 | # look for a security considerations header and the two lines after it. drop the heading drop the instructions drop any empty lines 6 | lines=$(grep -iEA 2 '^#+ *security considerations' pull-request/.git/body | grep -Ev '^#+ *security considerations' | grep -Ev '^ *\[Note' | grep -Ev '^ *$') 7 | 8 | if [[ -z ${lines} ]]; then 9 | echo "Didn't find Security Considerations section in PR" 10 | exit 1 11 | fi 12 | 13 | -------------------------------------------------------------------------------- /security-considerations/security-considerations.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: target 12 | 13 | inputs: 14 | - name: pull-request 15 | - name: pipeline-tasks 16 | 17 | run: 18 | path: pipeline-tasks/security-considerations/security-considerations.sh 19 | -------------------------------------------------------------------------------- /set-plan-visibility.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ############# 4 | # Make plans available for a service broker 5 | # cli flags: 6 | # -s enable for all orgs EXCEPT sandbox and smoke tests 7 | # -o enable for ONLY sandbox and smoke tests orgs 8 | # environment config 9 | # CF_API_URL API to auth against 10 | # CF_USERNAME User to auth as. Needs to have cf admin permissions 11 | # CF_PASSWORD Password for user 12 | # CF_SPACE Space to target when authenticating - probably doesn't matter? 13 | # CF_ORGANIZATION Org to target when authenticating - probably doesn't matter? 14 | # SERVICES Set of "$name" or "$name:$plan" values, such as "fooservice barservice:bazplan barservice:quuxplan" 15 | # BROKER_NAME Registered name of the broker 16 | # SERVICE_ORGANIZATION Organizations that should have access to the plan, in the form "org1 org2 org3" 17 | # SERVICE_ORGANIZATION_DENYLIST Organizations that should not have access to the plan, in the form "org1 org2 org3" 18 | # Only one of [`SERVICE_ORGANIZATION`, `SERVICE_ORGANIZATION_BLACKLIST`, `-s`] may be specified 19 | # 20 | ############# 21 | 22 | set -eux 23 | 24 | # Authenticate 25 | cf api "${CF_API_URL}" 26 | (set +x; cf auth "${CF_USERNAME}" "${CF_PASSWORD}") 27 | cf target -o "${CF_ORGANIZATION}" -s "${CF_SPACE}" 28 | 29 | if [ -n "${SERVICE_ORGANIZATION_BLACKLIST:-}" ]; then 30 | echo "Use of SERVICE_ORGANIZATION_BLACKLIST is deprecated - use SERVICE_ORGANIZATION_DENYLIST instead" >&2 31 | : "${SERVICE_ORGANIZATION_DENYLIST:${SERVICE_ORGANIZATION_BLACKLIST}}" 32 | fi 33 | 34 | 35 | while getopts ":so" opt; do 36 | case $opt in 37 | s) 38 | # Set plan visibility to EXCLUDE sandboxes if -s flag is set 39 | ORGLIST="" 40 | for org in $(cf orgs | grep 'sandbox\|SMOKE\|CATS'); do 41 | ORGLIST+=${org}" " 42 | done 43 | export SERVICE_ORGANIZATION_DENYLIST=${ORGLIST} 44 | ;; 45 | o) 46 | # Set plan visibility to ONLY sandboxes if -o flag is set 47 | ORGLIST="" 48 | for org in $(cf orgs | grep 'sandbox\|SMOKE\|CATS'); do 49 | ORGLIST+=${org}" " 50 | done 51 | export SERVICE_ORGANIZATION=${ORGLIST} 52 | ;; 53 | \?) 54 | echo "Invalid option: -$OPTARG" >&2 55 | ;; 56 | esac 57 | done 58 | 59 | 60 | 61 | if [ -n "${SERVICE_ORGANIZATION:-}" ] && [ -n "${SERVICE_ORGANIZATION_DENYLIST:-}" ]; then 62 | echo "You may set SERVICE_ORGANIZATION or SERVICE_ORGANIZATION_DENYLIST but not both" 63 | exit 1; 64 | fi 65 | 66 | if [ -z "${SERVICES:-}" ]; then 67 | echo "SERVICES environment variable must be set" 68 | exit 1; 69 | fi 70 | 71 | # Enable access to service plans 72 | # Services should be a set of "$name" or "$name:$plan" values, such as 73 | # "redis28-multinode mongodb30-multinode:persistent" 74 | for SERVICE in $SERVICES; do 75 | SERVICE_NAME=$(echo "${SERVICE}:" | cut -d ':' -f1) 76 | SERVICE_PLAN=$(echo "${SERVICE}:" | cut -d ':' -f2) 77 | ARGS=("${SERVICE_NAME}") 78 | if [ -n "${SERVICE_PLAN}" ]; then ARGS+=("-p" "${SERVICE_PLAN}"); fi 79 | if [ -n "${BROKER_NAME}" ]; then ARGS+=("-b" "${BROKER_NAME}"); fi 80 | 81 | # if we have a denylist, then we enable for all organizations EXCEPT those 82 | # since CF doesn't suport this; enumerate all organizations, and filter out those on the denylist 83 | # and enable for each remaining org 84 | if [ -n "${SERVICE_ORGANIZATION_DENYLIST:-}" ]; then 85 | 86 | for org in $(cf orgs | tail -n +4 | grep -Fvxf <(echo "$SERVICE_ORGANIZATION_DENYLIST" | tr " " "\n")); do 87 | cf enable-service-access "${ARGS[@]}" -o "${org}" 88 | done 89 | 90 | else 91 | # if we don't have a denylist, but do have an allowlist, then iterate over that list 92 | # and enable for each of those orgs 93 | if [ -n "${SERVICE_ORGANIZATION:-}" ]; then 94 | 95 | for org in $(echo "${SERVICE_ORGANIZATION}" | tr " " "\n"); do 96 | cf enable-service-access "${ARGS[@]}" -o "${org}" 97 | done 98 | 99 | # if we don't have any kind of list, enable for all orgs 100 | else 101 | 102 | cf enable-service-access "${ARGS[@]}" 103 | 104 | fi 105 | fi 106 | done 107 | -------------------------------------------------------------------------------- /set-plan-visibility.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | 16 | run: 17 | path: pipeline-tasks/set-plan-visibility.sh 18 | 19 | params: 20 | CF_API_URL: 21 | CF_USERNAME: 22 | CF_PASSWORD: 23 | CF_ORGANIZATION: 24 | CF_SPACE: 25 | BROKER_NAME: 26 | AUTH_USER: 27 | AUTH_PASS: 28 | SERVICES: 29 | -------------------------------------------------------------------------------- /terraform-apply.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: set ft=sh 3 | 4 | set -euxo pipefail 5 | 6 | if [ "$TERRAFORM_ACTION" != "plan" ] && \ 7 | [ "$TERRAFORM_ACTION" != "apply" ]; then 8 | echo 'must set $TERRAFORM_ACTION to "plan" or "apply"' >&2 9 | exit 1 10 | fi 11 | 12 | TERRAFORM="${TERRAFORM_BIN:-terraform}" 13 | BASE="$(pwd)" 14 | 15 | DIR="terraform-templates" 16 | 17 | set +x 18 | # unset TF_VARs that are empty strings 19 | for tfvar in "${!TF_VAR_@}"; do 20 | if [[ -z "${!tfvar}" ]]; then 21 | unset ${tfvar} 22 | fi 23 | done 24 | set -x 25 | 26 | if [ -n "${TEMPLATE_SUBDIR:-}" ]; then 27 | DIR="${DIR}/${TEMPLATE_SUBDIR}" 28 | fi 29 | 30 | ${TERRAFORM} -chdir=${DIR} get -update > terraform-get-output.txt 31 | 32 | init_args=( 33 | "-backend=true" 34 | "-backend-config=encrypt=true" 35 | "-backend-config=bucket=${S3_TFSTATE_BUCKET}" 36 | "-backend-config=key=${STACK_NAME}/terraform.tfstate" 37 | ) 38 | set +x 39 | if [ -n "${TF_VAR_aws_region:-}" ]; then 40 | init_args+=("-backend-config=region=${TF_VAR_aws_region}") 41 | fi 42 | if [ -n "${TF_VAR_aws_access_key:-}" ]; then 43 | init_args+=("-backend-config=access_key=${TF_VAR_aws_access_key}") 44 | fi 45 | if [ -n "${TF_VAR_aws_secret_key:-}" ]; then 46 | init_args+=("-backend-config=secret_key=${TF_VAR_aws_secret_key}") 47 | fi 48 | 49 | ${TERRAFORM} -chdir="${DIR}" init \ 50 | "${init_args[@]}" 51 | set -x 52 | 53 | if [ "${TERRAFORM_ACTION}" = "plan" ]; then 54 | ${TERRAFORM} -chdir="${DIR}" "${TERRAFORM_ACTION}" \ 55 | -refresh=true \ 56 | -input=false \ 57 | -out=${BASE}/terraform-state/terraform.tfplan \ 58 | | tee ${BASE}/terraform-state/terraform-plan-output.txt \ 59 | | grep -v --line-buffered --extended-regexp "Reading\.\.\.|Read complete after|Refreshing state\.\.\." 60 | 61 | # Write a sentinel value; pipelines can alert to slack if set using `text_file` 62 | # Ensure that slack notification resource detects text file 63 | touch ${BASE}/terraform-state/message.txt 64 | if ! cat ${BASE}/terraform-state/terraform-plan-output.txt | grep 'No changes.' ; then 65 | echo "sentinel" > ${BASE}/terraform-state/message.txt 66 | fi 67 | else 68 | ${TERRAFORM} -chdir="${DIR}" "${TERRAFORM_ACTION}" \ 69 | -refresh=true \ 70 | -input=false \ 71 | -auto-approve \ 72 | | grep -v --line-buffered --extended-regexp "Reading\.\.\.|Read complete after|Refreshing state\.\.\." 73 | 74 | if [ -n "${TF_VAR_aws_region:-}" ]; then 75 | export AWS_DEFAULT_REGION="${TF_VAR_aws_region}" 76 | fi 77 | if [ -n "${TF_VAR_aws_access_key:-}" ]; then 78 | export AWS_ACCESS_KEY_ID="${TF_VAR_aws_access_key}" 79 | fi 80 | if [ -n "${TF_VAR_aws_secret_key:-}" ]; then 81 | export AWS_SECRET_ACCESS_KEY="${TF_VAR_aws_secret_key}" 82 | fi 83 | aws s3 cp "s3://${S3_TFSTATE_BUCKET}/${STACK_NAME}/terraform.tfstate" terraform-state 84 | fi 85 | -------------------------------------------------------------------------------- /terraform-apply.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | - name: terraform-templates 16 | 17 | outputs: 18 | - name: terraform-state 19 | 20 | run: 21 | path: pipeline-tasks/terraform-apply.sh 22 | 23 | params: 24 | TERRAFORM_ACTION: 25 | STACK_NAME: 26 | TEMPLATE_SUBDIR: 27 | S3_TFSTATE_BUCKET: 28 | AWS_ACCESS_KEY_ID: 29 | AWS_SECRET_ACCESS_KEY: 30 | AWS_DEFAULT_REGION: 31 | TF_VAR_stack_description: 32 | TF_VAR_aws_default_region: 33 | TF_VAR_cdn_broker_username: 34 | TF_VAR_cdn_broker_bucket: 35 | TF_VAR_cdn_broker_cloudfront_prefix: 36 | TF_VAR_cdn_broker_hosted_zone: 37 | TF_VAR_lets_encrypt_hosted_zone: 38 | -------------------------------------------------------------------------------- /terraform-destroy.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: set ft=sh 3 | 4 | set -eux 5 | 6 | TERRAFORM="${TERRAFORM_BIN:-terraform}" 7 | 8 | DIR="terraform-templates" 9 | 10 | if [ -n "$TEMPLATE_SUBDIR" ]; then 11 | DIR="$DIR/$TEMPLATE_SUBDIR" 12 | fi 13 | 14 | # Hack: Disable `prevent_destroy` if requested 15 | # See https://github.com/hashicorp/terraform/issues/3874 16 | if [ "${PREVENT_PREVENT_DESTROY:-}" == "true" ]; then 17 | find terraform-templates -type f -name "*.tf" -exec sed -i 's/prevent_destroy = true/prevent_destroy = false/g' {} + 18 | fi 19 | 20 | ${TERRAFORM} -chdir=${DIR} get \ 21 | -update 22 | 23 | ${TERRAFORM} -chdir=${DIR} init \ 24 | -backend=true \ 25 | -backend-config="encrypt=true" \ 26 | -backend-config="bucket=${S3_TFSTATE_BUCKET}" \ 27 | -backend-config="key=${STACK_NAME}/terraform.tfstate" 28 | 29 | terraform -chdir=${DIR} destroy \ 30 | -refresh=true \ 31 | -force 32 | 33 | aws s3 cp "s3://${S3_TFSTATE_BUCKET}/${STACK_NAME}/terraform.tfstate" terraform-state 34 | -------------------------------------------------------------------------------- /terraform-destroy.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | - name: terraform-templates 16 | outputs: 17 | - name: terraform-state 18 | 19 | run: 20 | path: pipeline-tasks/terraform-destroy.sh 21 | -------------------------------------------------------------------------------- /terraform-state-to-yaml.rb: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | require 'json' 4 | require 'yaml' 5 | 6 | outputs = JSON.load($stdin) 7 | 8 | terraform_outputs = { 'terraform_outputs' => {} } 9 | outputs['modules'][0]['outputs'].each {|k, v| 10 | terraform_outputs['terraform_outputs'][k] = v.fetch("value") 11 | } 12 | 13 | puts YAML.dump(terraform_outputs) 14 | -------------------------------------------------------------------------------- /terraform-state-to-yaml.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: set ft=sh 3 | 4 | set -e 5 | 6 | if [ -z "$STATE_FILE" ]; then 7 | echo "must specify \$STATE_FILE" >&2 8 | exit 1 9 | fi 10 | 11 | SCRIPTPATH=$( cd $(dirname $0) ; pwd -P ) 12 | 13 | # Install required gems if not installed 14 | if [ -z $(gem list -i yaml) ]; then 15 | gem install yaml --no-ri --no-rdoc 2>&1 > /dev/null 16 | fi 17 | if [ -z $(gem list -i json) ]; then 18 | gem install json --no-ri --no-rdoc 2>&1 > /dev/null 19 | fi 20 | 21 | cat terraform-state/$STATE_FILE | $SCRIPTPATH/terraform-state-to-yaml.rb > terraform-yaml/state.yml 22 | -------------------------------------------------------------------------------- /terraform-state-to-yaml.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | - name: terraform-state 16 | outputs: 17 | - name: terraform-yaml 18 | 19 | run: 20 | path: pipeline-tasks/terraform-state-to-yaml.sh 21 | 22 | params: 23 | STATE_FILE: 24 | -------------------------------------------------------------------------------- /terraform12-state-to-yaml.rb: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | require 'json' 4 | require 'yaml' 5 | 6 | outputs = JSON.load($stdin) 7 | 8 | terraform_outputs = { 'terraform_outputs' => {} } 9 | outputs['outputs'].each {|k, v| 10 | terraform_outputs['terraform_outputs'][k] = v.fetch("value") 11 | } 12 | 13 | puts YAML.dump(terraform_outputs) 14 | -------------------------------------------------------------------------------- /terraform12-state-to-yaml.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: set ft=sh 3 | 4 | set -e 5 | 6 | if [ -z "$STATE_FILE" ]; then 7 | echo "must specify \$STATE_FILE" >&2 8 | exit 1 9 | fi 10 | 11 | SCRIPTPATH=$( cd $(dirname $0) ; pwd -P ) 12 | 13 | # Install required gems if not installed 14 | if [ -z $(gem list -i yaml) ]; then 15 | gem install yaml --no-ri --no-rdoc 2>&1 > /dev/null 16 | fi 17 | if [ -z $(gem list -i json) ]; then 18 | gem install json --no-ri --no-rdoc 2>&1 > /dev/null 19 | fi 20 | 21 | cat terraform-state/$STATE_FILE | $SCRIPTPATH/terraform12-state-to-yaml.rb > terraform-yaml/state.yml 22 | -------------------------------------------------------------------------------- /terraform12-state-to-yaml.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | - name: terraform-state 16 | outputs: 17 | - name: terraform-yaml 18 | 19 | run: 20 | path: pipeline-tasks/terraform12-state-to-yaml.sh 21 | 22 | params: 23 | STATE_FILE: 24 | -------------------------------------------------------------------------------- /uaa-smoke-tests.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | import os 5 | 6 | import requests 7 | from bs4 import BeautifulSoup 8 | 9 | 10 | def test_login(): 11 | resp = requests.get(os.getenv("BASE_URL")) 12 | 13 | # Assert OK 14 | assert resp.status_code == 200 15 | 16 | page = BeautifulSoup(resp.content, features="html.parser") 17 | 18 | # Assert SAML link with GSA IDP 19 | links = page.find_all("a", class_="saml-login-link") 20 | next(each for each in links if "gsa" in each.attrs["href"]) 21 | 22 | assert len(page.find_all("input", class_="island-button")) == 1 23 | assert len(page.find_all("div", class_="modal-body")) == 1 24 | 25 | 26 | if __name__ == "__main__": 27 | test_login() 28 | -------------------------------------------------------------------------------- /uaa-smoke-tests.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | set -e 4 | 5 | SCRIPTPATH=$(cd $(dirname $0); pwd -P) 6 | pip install bs4 requests 7 | "${SCRIPTPATH}"/uaa-smoke-tests.py 8 | -------------------------------------------------------------------------------- /uaa-smoke-tests.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | 16 | run: 17 | path: pipeline-tasks/uaa-smoke-tests.sh 18 | -------------------------------------------------------------------------------- /upload-release.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e -u 4 | 5 | BOSH_CERT="${BOSH_CERT:-}" 6 | if [ -n "${BOSH_CERT}" ]; then 7 | bosh-cli -n -e "${BOSH_TARGET}" --ca-cert "certificate/${BOSH_CERT}" alias-env env 8 | else 9 | bosh-cli -n -e "${BOSH_TARGET}" alias-env env 10 | fi 11 | 12 | if [ -n "${BOSH_USERNAME:-}" ]; then 13 | # Hack: Add trailing newline to skip OTP prompt 14 | bosh-cli -e env log-in </dev/null 15 | ${BOSH_USERNAME} 16 | ${BOSH_PASSWORD} 17 | 18 | EOF 19 | fi 20 | 21 | for r in release/*.tgz; do 22 | bosh-cli -e env upload-release "${r}" 23 | done 24 | -------------------------------------------------------------------------------- /upload-release.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | - name: release 16 | - name: certificate 17 | run: 18 | path: pipeline-tasks/upload-release.sh 19 | 20 | params: 21 | BOSH_USERNAME: 22 | BOSH_PASSWORD: 23 | BOSH_TARGET: 24 | -------------------------------------------------------------------------------- /write_file.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: set ft=sh 3 | 4 | set -e 5 | 6 | # 7 | # A little environment validation 8 | # 9 | if [ -z "$CONTENT" ]; then 10 | echo "must specify \$CONTENT" >&2 11 | exit 1 12 | fi 13 | if [ -z "$FILE_NAME" ]; then 14 | echo "must specify \$FILE_NAME" >&2 15 | exit 1 16 | fi 17 | 18 | # 19 | # Render content into a file 20 | # 21 | cat < "$FILE_NAME" 22 | $CONTENT 23 | EOF 24 | 25 | -------------------------------------------------------------------------------- /write_file.yml: -------------------------------------------------------------------------------- 1 | --- 2 | platform: linux 3 | 4 | image_resource: 5 | type: registry-image 6 | source: 7 | aws_access_key_id: ((ecr_aws_key)) 8 | aws_secret_access_key: ((ecr_aws_secret)) 9 | repository: general-task 10 | aws_region: us-gov-west-1 11 | tag: latest 12 | 13 | inputs: 14 | - name: pipeline-tasks 15 | outputs: 16 | - name: write_file 17 | run: 18 | path: pipeline-tasks/write_file.sh 19 | 20 | params: 21 | CONTENT: 22 | FILE_NAME: 23 | --------------------------------------------------------------------------------