├── .circleci └── config.yml ├── .gitignore ├── .shellcheckrc ├── Dockerfile ├── LICENSE ├── README.md ├── lefthook.yml ├── scripts ├── colors.sh ├── orb_utils.sh ├── publish_orb.sh ├── publish_orbs.sh ├── validate_orb.sh └── validate_orbs.sh ├── setup.sh └── src ├── auto ├── README.md └── auto.yml ├── hokusai ├── README.md └── hokusai.yml ├── orb-tools └── orb-tools.yml ├── release └── release.yml ├── remote-docker ├── README.md └── remote-docker.yml ├── skip-wip-ci ├── README.md └── skip-wip-ci.yml └── yarn ├── README.md └── yarn.yml /.circleci/config.yml: -------------------------------------------------------------------------------- 1 | version: 2.1 2 | 3 | executors: 4 | cli: 5 | docker: 6 | - image: circleci/circleci-cli:0.1.25725 7 | 8 | jobs: 9 | validate_orbs: 10 | executor: cli 11 | steps: 12 | - checkout 13 | - run: 14 | name: Validate orbs 15 | command: NAMESPACE=artsy scripts/validate_orbs.sh 16 | 17 | publish_orbs: 18 | executor: cli 19 | steps: 20 | - checkout 21 | - run: 22 | name: Install slack notifier 23 | command: | 24 | curl --location --output ./slack \ 25 | https://github.com/cloudposse/slack-notifier/releases/download/0.2.0/slack-notifier_linux_amd64 26 | chmod +x ./slack 27 | - run: 28 | name: Publish orbs 29 | command: NAMESPACE=artsy scripts/publish_orbs.sh 30 | 31 | workflows: 32 | build: 33 | jobs: 34 | - validate_orbs: 35 | context: circleci-api 36 | filters: 37 | branches: 38 | ignore: 39 | - main 40 | - publish_orbs: 41 | context: circleci-api 42 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | test.yml -------------------------------------------------------------------------------- /.shellcheckrc: -------------------------------------------------------------------------------- 1 | disable=SC2005,SC2155,SC2086,SC1091 -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM alpine:latest 2 | COPY ./scripts /tmp/orb-scripts 3 | RUN apk add --no-cache bash git curl jq 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Artsy 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Orbs 2 | 3 | CircleCI orbs used at Artsy 4 | 5 | ⚠️ [`artsy/yarn`](https://github.com/artsy/orbs/tree/main/src/yarn) has now been deprecated in favor of [`circleci/node`](https://circleci.com/developer/orbs/orb/circleci/node). Please use that for all Node.js-based CI needs. 6 | 7 | ## What's an Orb? 8 | 9 | > Orbs are packages of CircleCI configuration that can be shared across projects. Orbs allow you to make a single bundle of jobs, commands, and executors that can reference each other and can be imported into a CircleCI build configuration and invoked in their own namespace. Orbs are registered with CircleCI, with revisions expressed using the semver pattern. 10 | 11 | For more info on Orbs, checkout their [docs](https://circleci.com/docs/orb-intro/)! 12 | 13 | (The TLDR is there's an `orb` yml configuration file that's used to share things like [executors][orb-executors], [commands][orb-commands], and [jobs][orb-jobs] across your CircleCi builds.) 14 | 15 | ## Getting Started 16 | 17 | Orb files are stored in `src//.yml`. The nested directory is so that every orb can have associated documentation beside it. 18 | 19 | To make it easier to perform changes locally, it's recommended that you run `setup.sh` in the root. This will setup the tools you need to be able to run circle commands locally along with some helpful pre-commit hooks. 20 | 21 | ## Versioning 22 | 23 | Every orb has a comment like `# Orb Version 1.2.3` on the first line of the file. This comment is significant in that it's used to determine which version of the orb should be deployed (which will be discussed in the next section). Orbs in `main` will have a comment representing the currently deployed production version. 24 | 25 | When you make a change to an orb file you _must_ update the version. CI checks will fail otherwise. 26 | 27 | ## Deploying 28 | 29 | There are two types of deployments that happen in this repo. 30 | 31 | 1. Canary deployments that happen on every PR change 32 | 2. Production deployments that happen when a PR is merged to main. 33 | 34 | When you make a change to an orb (and update the version) a canary version will be published. Check the build logs for the version name. This canary build can be used (before your PR is merged to main) to test orb changes in other projects. It's _highly_ recommended that you utilize this canary system to test changes that may impact many projects. 35 | 36 | Upon merging a PR, CI will publish the changed orbs to CircleCI's public registry. Artsy also has [renovate configuration][reno-config] to update orb changes across Artsy's GitHub org. 37 | 38 | The deployment process is driven by a set of bash scripts in the `scripts` directory. `publish_orbs.sh` is responsible for publishing both the canary and release version of the orbs. It's heavily commented and you're encouraged to read through it if you're interested in how the process works. 39 | 40 | [orb-executors]: https://circleci.com/docs/2.0/reusing-config/#authoring-reusable-executors 41 | [orb-commands]: https://circleci.com/docs/2.0/reusing-config/#authoring-reusable-commands 42 | [orb-jobs]: https://circleci.com/docs/2.0/reusing-config/#authoring-reusable-commands 43 | [reno-config]: https://github.com/artsy/renovate-config/blob/1210eeba081c4aeb1369ed9257cbe7b1e76276e0/lib/config.js 44 | -------------------------------------------------------------------------------- /lefthook.yml: -------------------------------------------------------------------------------- 1 | # For more information on configuring lefthook, see the guide: 2 | # https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md 3 | 4 | pre-commit: 5 | parallel: true 6 | commands: 7 | validate-orbs: 8 | glob: "src/**/*.yml" 9 | run: NAMESPACE=artsy scripts/validate_orb.sh {staged_files} 10 | validate-circle-config: 11 | glob: ".circleci/config.yml" 12 | run: circleci config validate 13 | validate-shell-scripts: 14 | glob: "*.sh" 15 | run: shellcheck {staged_files} 16 | 17 | lint-all-scripts: 18 | commands: 19 | lint-scripts: 20 | glob: "*.sh" 21 | run: shellcheck {all_files} 22 | 23 | fix-scripts: 24 | commands: 25 | fix-scripts: 26 | glob: "*.sh" 27 | run: shellcheck -f diff {all_files} | git apply 28 | -------------------------------------------------------------------------------- /scripts/colors.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | if [ -z "$TERM" ] || [ "$TERM" = "dumb" ]; then 4 | _red="\e[31m" 5 | _green="\e[32m" 6 | _yellow="\e[33m" 7 | _reset="\e[0m" 8 | else 9 | _red=$(tput -T "$TERM" setaf 1) 10 | _green=$(tput -T "$TERM" setaf 2) 11 | _yellow=$(tput -T "$TERM" setaf 3) 12 | _reset=$(tput -T "$TERM" sgr0) 13 | fi 14 | 15 | COLOR() { 16 | echo -e "$1$2${_reset}" 17 | } 18 | 19 | RED() { 20 | echo "$(COLOR "$_red" "$1")" 21 | } 22 | 23 | GREEN() { 24 | echo "$(COLOR "$_green" "$1")" 25 | } 26 | 27 | YELLOW() { 28 | echo "$(COLOR "$_yellow" "$1")" 29 | } 30 | -------------------------------------------------------------------------------- /scripts/orb_utils.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | VERSION_REGEX="[0-9]*\.[0-9]*\.[0-9]*" 4 | 5 | check_for_namespace() { 6 | NAMESPACE=${NAMESPACE:-""} 7 | if [ -z "$NAMESPACE" ]; then 8 | echo "An env variable NAMESPACE must be provided that matches your CircleCI orb namespace" 9 | exit 1 10 | fi 11 | } 12 | 13 | get_orb_path() { 14 | local ORB="$1" 15 | local YML_PATH="./src/$ORB/$ORB.yml" 16 | echo "$YML_PATH" 17 | } 18 | 19 | get_orb_from_path() { 20 | local filename="$(basename "$1")" 21 | echo "${filename%.*}" 22 | } 23 | 24 | get_orb_version() { 25 | local YML_PATH=$(get_orb_path "$1") 26 | 27 | VERSION_COMMENT=$(head -n 1 "$YML_PATH") 28 | VERSION=$(echo "$VERSION_COMMENT" | grep -o "$VERSION_REGEX") 29 | 30 | echo "$VERSION" 31 | } 32 | 33 | is_orb_changed() { 34 | check_for_namespace 35 | 36 | local ORB="$1" 37 | local ORB_PATH="$(get_orb_path "$ORB")" 38 | local CHANGED="$(git diff --name-only origin/main "$ORB_PATH")" 39 | 40 | if [ -n "$CHANGED" ]; then 41 | echo "true" 42 | fi 43 | } 44 | 45 | is_orb_created() { 46 | check_for_namespace 47 | local CREATED=$(circleci orb list "$NAMESPACE" | grep -w "$NAMESPACE/$1") 48 | if [ -n "$CREATED" ]; then 49 | echo "true" 50 | fi 51 | } 52 | 53 | is_orb_published() { 54 | check_for_namespace 55 | local PUBLISHED=$( 56 | circleci orb info "$NAMESPACE/$1" >/dev/null 2>&1 57 | echo $? 58 | ) 59 | if [ "$PUBLISHED" -eq "0" ]; then 60 | echo "true" 61 | fi 62 | } 63 | 64 | get_published_orb_version() { 65 | check_for_namespace 66 | local LAST_PUBLISHED=$(circleci orb info "$NAMESPACE/$1" | grep -i latest | grep -o "$VERSION_REGEX") 67 | echo "$LAST_PUBLISHED" 68 | } 69 | 70 | compare_version() { 71 | local GREATER=">" 72 | local LESS="<" 73 | local EQUAL="=" 74 | 75 | IFS='.' read -ra VERSION1 <<<"$1" 76 | IFS='.' read -ra VERSION2 <<<"$2" 77 | 78 | for ((i = 0; i < ${#VERSION1[@]}; ++i)); do 79 | if [ "${VERSION1[i]}" -gt "${VERSION2[i]}" ]; then 80 | echo $GREATER 81 | return 82 | elif [ "${VERSION1[i]}" -lt "${VERSION2[i]}" ]; then 83 | echo $LESS 84 | return 85 | fi 86 | done 87 | 88 | echo $EQUAL 89 | } 90 | -------------------------------------------------------------------------------- /scripts/publish_orb.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -euo pipefail 3 | 4 | # This script is called from publish_orbs.sh 5 | # 6 | # Usage: 7 | # publish_orb.sh 8 | # 9 | # When $CI isn't set or $DRY_RUN is set this script will 10 | # skip any actual publishing steps. This means it _shouldn't_ 11 | # do any publishing when you're testing locally. 12 | 13 | # "import" some utility functions 14 | . ./scripts/orb_utils.sh 15 | . ./scripts/colors.sh 16 | 17 | check_for_namespace 18 | 19 | # Grab the current git branch 20 | BRANCH=$(git branch | grep "\*" | cut -d ' ' -f2) 21 | ORB="$1" 22 | IS_CHANGED=$(is_orb_changed "$ORB") 23 | 24 | if [ "$BRANCH" != "main" ] && [ -z "$IS_CHANGED" ]; then 25 | echo "$(YELLOW "[skipped]") Publish for $NAMESPACE/$ORB because there are no changes" 26 | exit 0 27 | fi 28 | 29 | echo "" 30 | echo "----- Begin publish of $NAMESPACE/$1 orb -----" 31 | echo "" 32 | 33 | # Make sure used variables are defined 34 | DRY_RUN=${DRY_RUN:-""} 35 | CI=${CI:-""} 36 | CIRCLE_PULL_REQUEST=${CIRCLE_PULL_REQUEST:-""} 37 | CIRCLE_SHA1=${CIRCLE_SHA1:-$(git rev-parse HEAD)} 38 | SLACK_WEBHOOK_URL=${SLACK_WEBHOOK_URL:-""} 39 | 40 | # Set a dry-run mode 41 | if [ -n "$DRY_RUN" ] || [ -z "$CI" ]; then 42 | DRY_RUN="true" 43 | echo "$(YELLOW "[Running in dry-run mode]")" 44 | else 45 | DRY_RUN="" 46 | fi 47 | 48 | # Build CircleCI token argument 49 | TOKEN="" 50 | if [ -n "${CIRCLE_TOKEN:-}" ]; then 51 | TOKEN="--token $CIRCLE_TOKEN" 52 | elif [ -z "$DRY_RUN" ]; then 53 | echo "$(RED "Must provide CIRCLE_TOKEN env var")" 54 | echo "" 55 | exit 1 56 | fi 57 | 58 | # Build the dev version prefix. When not on the main branch this will be 59 | # used to publish a dev version of the orb. That can be pulled in using 60 | # $NAMESPACE/@dev:. This is useful for testing purposes. 61 | # 62 | # This will be referred to as "dev mode" in later comments 63 | DEV="" 64 | VERSION_POSTFIX="" 65 | if [ "$BRANCH" != "main" ]; then 66 | DEV="dev:" 67 | echo "$(YELLOW "[Running in dev mode]")" 68 | 69 | # Build the version postfix which should be unique per branch 70 | VERSION_POSTFIX="$(echo "$BRANCH" | md5sum | awk '{ print $1 }')" 71 | VERSION_POSTFIX="$VERSION_POSTFIX" 72 | fi 73 | 74 | # When in dev mode 75 | if [ -n "$DEV" ]; then 76 | echo "" 77 | echo "This will be a dev deployment (prefixed with dev:)" 78 | fi 79 | 80 | ORB_PATH=$(get_orb_path "$ORB") 81 | VERSION=$(get_orb_version "$ORB") 82 | IS_PUBLISHED=$(is_orb_published "$ORB") 83 | IS_CREATED=$(is_orb_created "$ORB") 84 | 85 | # Ensure the orb is valid 86 | circleci orb validate "$ORB_PATH" 87 | 88 | if [ -n "$DEV" ]; then 89 | FULL_VERSION="$DEV$VERSION_POSTFIX" 90 | else 91 | FULL_VERSION="$VERSION" 92 | fi 93 | 94 | # If the orb has been previously published (i.e. it already exists in circle's registry) 95 | if [ -n "$IS_PUBLISHED" ]; then 96 | 97 | LAST_PUBLISHED=$(get_published_orb_version "$ORB") 98 | 99 | case $(compare_version "$VERSION" "$LAST_PUBLISHED") in 100 | "=") 101 | # When not in dev mode 102 | if [ -z "$DEV" ]; then 103 | echo "$NAMESPACE/$ORB@$VERSION is the latest, skipping publish" 104 | exit 0 105 | fi 106 | ;; 107 | "<") 108 | echo "$(RED "$NAMESPACE/$ORB@$LAST_PUBLISHED is the latest, cannot publish older version $VERSION")" 109 | echo "$(RED "Please update $ORB_PATH to have a version greater than $LAST_PUBLISHED")" 110 | exit 1 111 | ;; 112 | ">") 113 | # when not in dev mode 114 | if [ -z "$DEV" ]; then 115 | echo "Preparing to bump $NAMESPACE/$ORB from $LAST_PUBLISHED to $VERSION" 116 | fi 117 | ;; 118 | *) 119 | echo "$(RED "Version comparison for $NAMESPACE/$ORB failed.")" 120 | echo "$(RED "Current version: $VERSION")" 121 | echo "$(RED "Published version: $LAST_PUBLISHED")" 122 | exit 1 123 | ;; 124 | esac 125 | 126 | elif [ -z "$IS_CREATED" ]; then 127 | echo "Orb $NAMESPACE/$ORB isn't in the registry. Creating its registry entry..." 128 | circleci orb create "$NAMESPACE/$ORB" $TOKEN --no-prompt 129 | fi 130 | 131 | # Publish to CircleCI (when it's not a dry run) 132 | if [ -z "$DRY_RUN" ]; then 133 | echo "Preparing to publish dev orb $NAMESPACE/$ORB@$FULL_VERSION" 134 | circleci orb publish "$ORB_PATH" "$NAMESPACE/$ORB@$FULL_VERSION" $TOKEN 135 | else 136 | echo "$(YELLOW "[skipped]") circleci orb publish $ORB_PATH $NAMESPACE/$ORB@$FULL_VERSION" 137 | fi 138 | 139 | # Publish to slack (when it's neither a dry run or dev mode) 140 | if [ -z "$DRY_RUN" ] && [ -z "$DEV" ] && [ -n "$SLACK_WEBHOOK_URL" ]; then 141 | ./slack \ 142 | -color "good" \ 143 | -title "Circle CI $ORB orb v$VERSION published!" \ 144 | -title_link "${CIRCLE_BUILD_URL:-https://circleci.com/gh/$NAMESPACE/orbs/tree/main}" \ 145 | -user_name "artsyit" \ 146 | -icon_emoji ":crystal_ball:" 147 | 148 | elif [ -z "$SLACK_WEBHOOK_URL" ]; then 149 | echo "$(YELLOW "[skipped]") Post to slack: No SLACK_WEBHOOK_URL environment variable set" 150 | 151 | # When it's a dry run but not in dev mode 152 | elif [ -z "$DRY_RUN" ]; then 153 | echo "$(YELLOW "[skipped]") Post to slack: Circle CI $ORB orb v$VERSION published" 154 | fi 155 | 156 | echo "" 157 | echo "----- End publish of $NAMESPACE/$1 orb -----" 158 | echo "" 159 | -------------------------------------------------------------------------------- /scripts/publish_orbs.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -euo pipefail 3 | 4 | # shellcheck disable=SC2045 5 | for orb in $(ls ./src); do 6 | ./scripts/publish_orb.sh "$orb" 7 | done 8 | -------------------------------------------------------------------------------- /scripts/validate_orb.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -euo pipefail 3 | 4 | . ./scripts/orb_utils.sh 5 | 6 | check_for_namespace 7 | 8 | ORB="$1" 9 | 10 | if [ -f "$ORB" ]; then 11 | ORB_PATH="$ORB" 12 | ORB=$(get_orb_from_path "$ORB") 13 | else 14 | ORB_PATH=$(get_orb_path "$ORB") 15 | fi 16 | 17 | echo "" 18 | echo "Validating $NAMESPACE/$ORB orb" 19 | 20 | if [ ! -f "$ORB_PATH" ]; then 21 | echo "No orb exists at $ORB_PATH" 22 | exit 1 23 | fi 24 | 25 | VERSION=$(get_orb_version "$ORB") 26 | VERSION_COMMENT=$(head -n 1 "$ORB_PATH") 27 | IS_PUBLISHED=$(is_orb_published "$ORB") 28 | IS_CREATED=$(is_orb_created "$ORB") 29 | 30 | # Ensure the version is defined and that the version comment actually is a comment... 31 | if [ -z "$VERSION" ] || [ ! "${VERSION_COMMENT:0:1}" == "#" ]; then 32 | echo "" 33 | echo "Orb at $ORB_PATH does not have a version comment" 34 | echo "Add something like '# Orb Version 1.0.0' at the top of the file" 35 | echo "That version will be used as the published version" 36 | return 37 | fi 38 | 39 | if [ -n "$IS_CREATED" ] && [ -n "$IS_PUBLISHED" ]; then 40 | 41 | PUBLISHED_VERSION=$(get_published_orb_version "$ORB") 42 | BRANCH=$(git rev-parse --abbrev-ref HEAD) 43 | if [ "$BRANCH" != "main" ]; then 44 | 45 | CHANGED_FILES="$(git diff --name-only HEAD..origin/main)" 46 | UPDATED_FILES="$(git status -s | cut -c4-)" 47 | #shellcheck disable=SC2206 48 | ALL_CHANGES=(${CHANGED_FILES[@]} ${UPDATED_FILES[@]}) 49 | for file in "${ALL_CHANGES[@]}"; do 50 | if [[ "$ORB_PATH" == *"$file" ]] && [[ "$VERSION" == "$PUBLISHED_VERSION" ]]; then 51 | echo "" 52 | echo "$NAMESPACE/$ORB has been updated since main but hasn't had its version bumped." 53 | echo "Update its version in $ORB_PATH" 54 | exit 1 55 | fi 56 | done 57 | 58 | fi 59 | 60 | fi 61 | 62 | circleci orb validate "$ORB_PATH" 63 | 64 | echo "" 65 | -------------------------------------------------------------------------------- /scripts/validate_orbs.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -euo pipefail 3 | 4 | # shellcheck disable=SC2045 5 | for orb in $(ls ./src); do 6 | ./scripts/validate_orb.sh "$orb" 7 | done 8 | -------------------------------------------------------------------------------- /setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Sets up development dependencies 4 | 5 | echo -n "Checking for cicleci... " 6 | if ! [ -x "$(command -v circleci)" ]; then 7 | echo "not found, installing" 8 | brew install circleci 9 | echo "You'll need to create a personal circle api key to use the cli. Opening that page now..." 10 | sleep 2 11 | open https://circleci.com/account/api 12 | echo "Once your done creating a token, finish the circle setup..." 13 | sleep 3 14 | circleci setup 15 | else 16 | echo "found, skipping." 17 | fi 18 | 19 | echo -n "Checking for lefthook... " 20 | if ! [ -x "$(command -v lefthook)" ]; then 21 | echo "not found, installing" 22 | brew install Arkweid/lefthook/lefthook 23 | lefthook install 24 | else 25 | echo "found, skipping." 26 | fi 27 | 28 | echo -n "Checking for spellcheck... " 29 | if ! [ -x "$(command -v shellcheck)" ]; then 30 | echo "not found, installing" 31 | brew install shellcheck 32 | else 33 | echo "found, skipping." 34 | fi 35 | -------------------------------------------------------------------------------- /src/auto/README.md: -------------------------------------------------------------------------------- 1 | # artsy/auto 2 | 3 | This orb controls Artsy's package deployment process. It builds off of [a simple community orb](https://github.com/auto-it/orbs/blob/master/src/release/release.yml) that's itself a simple wrapper for [intuit's auto](https://github.com/intuit/auto). 4 | 5 | If you're making a generic change about how `auto` executes, it's recommended you do that [upstream](https://github.com/auto-it/orbs/blob/master/src/release/release.yml). If you're actually changing something specific to how Artsy deploys, then this is the right place for it. 6 | -------------------------------------------------------------------------------- /src/auto/auto.yml: -------------------------------------------------------------------------------- 1 | # Orb Version 2.2.0 2 | 3 | version: 2.1 4 | description: Publish NPM packages and canary deployments with Intuit's Auto 5 | orbs: 6 | yarn: artsy/yarn@5.1.3 7 | auto: auto/release@0.2.3 8 | utils: artsy/orb-tools@0.5.0 9 | executors: 10 | node: 11 | parameters: 12 | node-version: 13 | type: string 14 | default: "14.18.1" 15 | docker: 16 | - image: cimg/node:<< parameters.node-version >> 17 | jobs: 18 | publish: 19 | parameters: 20 | node-version: 21 | type: string 22 | default: "14.18.1" 23 | version: 24 | type: string 25 | default: v10.36.5 26 | args: 27 | type: string 28 | default: "" 29 | executor: 30 | name: node 31 | node-version: << parameters.node-version >> 32 | environment: 33 | AUTO_VERSION: << parameters.version >> 34 | steps: 35 | - yarn/pre-release 36 | - auto/shipit: 37 | arguments: << parameters.args >> 38 | publish-canary: 39 | parameters: 40 | node-version: 41 | type: string 42 | default: "14.18.1" 43 | version: 44 | type: string 45 | default: v10.36.5 46 | args: 47 | type: string 48 | default: "" 49 | executor: 50 | name: node 51 | node-version: << parameters.node-version >> 52 | environment: 53 | AUTO_VERSION: << parameters.version >> 54 | steps: 55 | - utils/skip-if-fork-or-not-pr: 56 | pr-skip-message: Skipping, only deploys canaries on PR builds 57 | fork-skip-message: Skipping, can't deploy canaries from a fork 58 | - yarn/pre-release 59 | - auto/canary: 60 | arguments: << parameters.args >> 61 | -------------------------------------------------------------------------------- /src/hokusai/README.md: -------------------------------------------------------------------------------- 1 | # artsy/hokusai 2 | 3 | This orb is built to share hokusai configuration across many CircleCI setups. It currently provides CircleCI workflow steps for `test`, `deploy-staging` and `deploy-production`, using a PR-based release process. Use the latest version of this orb in your app to ensure that hokusai and all related libs are up to date, and that deployments use the latest recommended workflow. 4 | 5 | Enabling orbs requires CircleCI 2.1, which is enabled for an app in 2 steps: 6 | - In CircleCI UI > App "Build Settings" > "Advanced Settings", turn the "Enable pipelines" radio to `true`. 7 | - In the app's `.circleci/config.yml`, set `version: 2.1` at the top of the file. 8 | 9 | If not already configured, a read+write Github key is required for CircleCI. Find [instructions for creating and saving this key here](https://github.com/artsy/README/blob/main/playbooks/deployments.md#recommendations). 10 | 11 | To use the orb, within your app's `.circleci/config.yml`, use the hokusai orb for one or all workflow steps. It is recommended to use the orb for all steps, but implementation will depend on a particular app's needs. 12 | 13 | See these example PR's for implementation: 14 | - [Use Orb for all steps](https://github.com/artsy/metaphysics/pull/1713/files) 15 | - [Use Orb for some steps](https://github.com/artsy/positron/pull/2014/files ) 16 | -------------------------------------------------------------------------------- /src/hokusai/hokusai.yml: -------------------------------------------------------------------------------- 1 | # Orb Version 0.23.0 2 | 3 | 4 | version: 2.1 5 | description: Reusable hokusai tasks for managing deployments 6 | 7 | executors: 8 | deploy: 9 | docker: 10 | - image: artsy/hokusai:latest 11 | auth: 12 | username: $DOCKERHUB_USER 13 | password: $DOCKERHUB_PASSWORD 14 | beta: 15 | docker: 16 | - image: artsy/hokusai:beta 17 | auth: 18 | username: $DOCKERHUB_USER 19 | password: $DOCKERHUB_PASSWORD 20 | 21 | commands: 22 | setup: 23 | steps: 24 | - add_ssh_keys 25 | - checkout 26 | 27 | setup-docker: 28 | parameters: 29 | remote_docker_version: 30 | type: string 31 | default: docker23 32 | steps: 33 | - setup 34 | - setup_remote_docker: 35 | version: << parameters.remote_docker_version >> 36 | 37 | install-aws-iam-authenticator: 38 | parameters: 39 | uri: 40 | type: string 41 | default: "https://artsy-provisioning-public.s3.amazonaws.com/aws-iam-authenticator_0.4.0_linux_amd64" 42 | steps: 43 | - run: 44 | name: Install AWS IAM Authenticator 45 | command: | 46 | curl -L -o aws-iam-authenticator << parameters.uri >> 47 | chmod +x ./aws-iam-authenticator 48 | mv aws-iam-authenticator /usr/local/bin/ 49 | 50 | configure-hokusai: 51 | parameters: 52 | configUri: 53 | type: string 54 | default: "s3://artsy-provisioning-public/hokusai/hokusai-ci.yml" 55 | steps: 56 | - run: 57 | name: Configure Hokusai 58 | command: | 59 | HOKUSAI_GLOBAL_CONFIG=<< parameters.configUri >> hokusai configure 60 | 61 | push-image: 62 | parameters: 63 | remote_docker_version: 64 | type: string 65 | default: docker23 66 | steps: 67 | - setup-docker: 68 | remote_docker_version: << parameters.remote_docker_version >> 69 | - run: 70 | name: Push 71 | command: | 72 | set +euo pipefail 73 | 74 | if hokusai registry images --tag-exists "$CIRCLE_SHA1" >/dev/null 75 | then 76 | echo "Skipping push as the tag $CIRCLE_SHA1 already exists in the Docker registry" 77 | else 78 | hokusai registry push --tag $CIRCLE_SHA1 79 | fi 80 | 81 | run-tests: 82 | parameters: 83 | filename: 84 | type: string 85 | default: "" 86 | description: The docker-compose yaml file to use 87 | flags: 88 | type: string 89 | default: "" 90 | description: Optional hokusai flags 91 | steps: 92 | - run: 93 | name: Test 94 | command: | 95 | if [ "<< parameters.filename >>" != "" ]; then 96 | hokusai test -f << parameters.filename >> << parameters.flags >> 97 | else 98 | hokusai test << parameters.flags >> 99 | fi 100 | 101 | git-push-staging-branch-command: 102 | parameters: 103 | project-name: 104 | type: string 105 | description: The name of the project as it appears on github 106 | steps: 107 | - run: 108 | name: Git Push Staging Branch 109 | command: git push git@github.com:artsy/<< parameters.project-name >>.git $CIRCLE_SHA1:refs/heads/staging --force 110 | 111 | jobs: 112 | test: 113 | executor: << parameters.executor >> 114 | parameters: 115 | executor: 116 | type: executor 117 | default: deploy 118 | filename: 119 | type: string 120 | default: "" 121 | description: The docker-compose yaml file to use 122 | flags: 123 | type: string 124 | default: "" 125 | description: Optional hokusai flags 126 | remote_docker_version: 127 | type: string 128 | default: docker23 129 | description: specify circleci remote docker version 130 | steps: 131 | - setup-docker: 132 | remote_docker_version: << parameters.remote_docker_version >> 133 | - run-tests: 134 | filename: << parameters.filename >> 135 | flags: << parameters.flags >> 136 | 137 | push: 138 | executor: << parameters.executor >> 139 | parameters: 140 | executor: 141 | type: executor 142 | default: deploy 143 | remote_docker_version: 144 | type: string 145 | default: docker23 146 | steps: 147 | - push-image: 148 | remote_docker_version: << parameters.remote_docker_version >> 149 | 150 | deploy-staging: 151 | executor: << parameters.executor >> 152 | parameters: 153 | executor: 154 | type: executor 155 | default: deploy 156 | project-name: 157 | type: string 158 | description: The name of the project as it appears on github 159 | time-out: 160 | type: string 161 | description: How long to wait for shell output before timing out 162 | default: 20m 163 | steps: 164 | - setup 165 | - install-aws-iam-authenticator 166 | - configure-hokusai 167 | - run: 168 | name: Validate Kubernetes Yaml 169 | command: hokusai staging update --skip-checks --dry-run 170 | - run: 171 | name: Deploy 172 | command: hokusai staging deploy $CIRCLE_SHA1 --update-config 173 | no_output_timeout: << parameters.time-out >> 174 | - git-push-staging-branch-command: 175 | project-name: << parameters.project-name >> 176 | 177 | git-push-staging-branch: 178 | executor: << parameters.executor >> 179 | parameters: 180 | executor: 181 | type: executor 182 | default: deploy 183 | project-name: 184 | type: string 185 | description: The name of the project as it appears on github 186 | steps: 187 | - setup 188 | - git-push-staging-branch-command: 189 | project-name: << parameters.project-name >> 190 | 191 | retag-staging: 192 | executor: << parameters.executor >> 193 | parameters: 194 | executor: 195 | type: executor 196 | default: deploy 197 | steps: 198 | - setup 199 | - run: 200 | name: retag staging 201 | command: hokusai registry retag staging $CIRCLE_SHA1 202 | 203 | deploy-production: 204 | executor: << parameters.executor >> 205 | parameters: 206 | executor: 207 | type: executor 208 | default: deploy 209 | time-out: 210 | type: string 211 | description: How long to wait for shell output before timing out 212 | default: 20m 213 | steps: 214 | - setup 215 | - install-aws-iam-authenticator 216 | - configure-hokusai 217 | - run: 218 | name: Validate Kubernetes Yaml 219 | command: hokusai production update --skip-checks --dry-run 220 | - run: 221 | name: What's being deployed 222 | command: hokusai pipeline gitcompare --org-name artsy || true 223 | - run: 224 | name: Changes with migrations 225 | command: hokusai pipeline gitlog | grep migration || true 226 | - run: 227 | name: Deploy 228 | command: hokusai pipeline promote --update-config 229 | no_output_timeout: << parameters.time-out >> 230 | -------------------------------------------------------------------------------- /src/orb-tools/orb-tools.yml: -------------------------------------------------------------------------------- 1 | # Orb Version 0.5.1 2 | 3 | version: 2.1 4 | description: A simple set of tools for managing orbs at Artsy. 5 | 6 | executors: 7 | orb-scripts: 8 | docker: 9 | - image: artsy/orb-scripts:latest 10 | 11 | commands: 12 | check-for-env-vars: 13 | parameters: 14 | env-vars: 15 | description: Comma separated list of environment variables to check for. No spaces. 16 | type: string 17 | steps: 18 | - run: 19 | name: Check for environment variables 20 | command: | 21 | # Skip Forks 22 | if [ -n "$CIRCLE_PR_NUMBER" ]; then 23 | echo "Skipping, forks won't have the right env vars anyway." 24 | exit 0 25 | fi 26 | #Print the split string 27 | for env in << parameters.env-vars >>; do 28 | if [ -z "$(printenv $env)" ]; then 29 | echo "The env var '$env' was expected to be set, but isn't. Set it and then try again." 30 | exit 1 31 | fi 32 | done 33 | skip-if-fork-or-not-pr: 34 | parameters: 35 | pr-skip-message: 36 | type: string 37 | default: Skipping because this isn't a pr build 38 | fork-skip-message: 39 | type: string 40 | default: Skipping because this is a fork 41 | steps: 42 | - run: 43 | name: Don't deploy canary if it's not a PR or if it's a fork 44 | command: | 45 | if [ -z "$CIRCLE_PULL_REQUEST" ]; then 46 | echo "<< parameters.pr-skip-message >>" 47 | circleci-agent step halt 48 | fi 49 | 50 | if [ -n "$CIRCLE_PR_NUMBER" ]; then 51 | echo "<< parameters.fork-skip-message >>" 52 | circleci-agent step halt 53 | fi 54 | setup-paths: 55 | steps: 56 | - run: 57 | name: Set orb scripts path 58 | command: | 59 | if [ ! -d "./scripts" ]; then 60 | ln -s /tmp/orb-scripts scripts 61 | fi 62 | 63 | jobs: 64 | validate: 65 | executor: orb-scripts 66 | parameters: 67 | namespace: 68 | description: CircleCI orb namespace 69 | type: string 70 | steps: 71 | - checkout 72 | - setup-paths 73 | - run: 74 | name: Validate orbs 75 | command: NAMESPACE=<< parameters.namespace >> scripts/validate_orbs.sh 76 | publish: 77 | executor: orb-scripts 78 | parameters: 79 | namespace: 80 | description: CircleCI orb namespace 81 | type: string 82 | steps: 83 | - checkout 84 | - setup-paths 85 | - run: 86 | name: Install slack notifier 87 | command: | 88 | curl --location --output ./slack \ 89 | https://github.com/cloudposse/slack-notifier/releases/download/0.2.0/slack-notifier_linux_amd64 90 | chmod +x ./slack 91 | - run: 92 | name: Publish orbs 93 | command: NAMESPACE=<< parameters.namespace >> scripts/publish_orbs.sh 94 | -------------------------------------------------------------------------------- /src/release/release.yml: -------------------------------------------------------------------------------- 1 | # Orb Version 0.0.1 2 | 3 | version: 2.1 4 | description: Wrapper for Artsy's release management tools 5 | 6 | executors: 7 | scripts: 8 | docker: 9 | - image: artsy/orb-scripts 10 | 11 | jobs: 12 | block: 13 | executor: scripts 14 | parameters: 15 | project_id: 16 | type: integer 17 | description: The primary key of the project object in horizon 18 | steps: 19 | - run: 20 | name: Check for deploy blocks 21 | description: Check horizon to see if there's an unresolved deployment block 22 | command: | 23 | set -eo pipefail 24 | 25 | RESULT=$(curl --max-time 5 -s -f -u ${HORIZON_USER}:${HORIZON_PASS} "https://releases.artsy.net/api/deploy_blocks?project_id=${TEST_DEPLOY_BLOCK_PROJECT_ID:-<< parameters.project_id >>}&resolved=false") 26 | if [ "$RESULT" == "[]" ]; then 27 | exit 0 28 | else 29 | echo "Deployment currently blocked, check https://releases.artsy.net" 30 | echo "$RESULT" | jq 31 | exit 5 32 | fi 33 | 34 | # Below is used to test this orb. Just uncomment the workflows setting and run the following command 35 | 36 | # circleci config process src/release/release.yml > test.yml && circleci local execute --job block -c ./test.yml -e HORIZON_USER= -e HORIZON_PASS= -e TEST_DEPLOY_BLOCK_PROJECT_ID=8 37 | 38 | # workflows: 39 | # test: 40 | # jobs: 41 | # - block: 42 | # project_id: -1 43 | -------------------------------------------------------------------------------- /src/remote-docker/README.md: -------------------------------------------------------------------------------- 1 | # artsy/remote-docker 2 | 3 | This orb establishes a connection with an Artsy-managed remote Docker daemon in 4 | order to run a Docker build in an environment optimized to retain cached Docker 5 | layers. 6 | 7 | If the orb is unable to establish a connection with the Artsy-managed remote 8 | Docker daemon, the Docker build will fallback to Circle CI's infrastructure. 9 | 10 | This orb requires the following environment variables to be available within the 11 | environment: 12 | 13 | - `AWS_ACCESS_KEY_ID` 14 | - `AWS_SECRET_ACCESS_KEY` 15 | 16 | ```yaml 17 | # In your project's .circleci/config.yml 18 | 19 | # Using the volatile label is _not_ recommended. 20 | # Use the version in the comment at the top of node.yml instead. 21 | 22 | orbs: 23 | artsy-remote-docker: artsy/remote-docker@volatile 24 | 25 | workflows: 26 | default: 27 | jobs: 28 | - artsy-remote-docker/build 29 | ``` 30 | -------------------------------------------------------------------------------- /src/remote-docker/remote-docker.yml: -------------------------------------------------------------------------------- 1 | # Orb Version 0.1.24 2 | 3 | version: 2.1 4 | description: > 5 | Execute Docker build via Artsy-managed Docker daemon with a Circle CI fallback 6 | 7 | executors: 8 | deploy: 9 | docker: 10 | - image: artsy/hokusai:latest 11 | auth: 12 | username: $DOCKERHUB_USER 13 | password: $DOCKERHUB_PASSWORD 14 | beta: 15 | docker: 16 | - image: artsy/hokusai:beta 17 | auth: 18 | username: $DOCKERHUB_USER 19 | password: $DOCKERHUB_PASSWORD 20 | 21 | commands: 22 | setup-artsy-remote-docker: 23 | parameters: 24 | artsy_docker_host: 25 | type: string 26 | artsy_docker_port: 27 | type: integer 28 | artsy_s3_path_root: 29 | type: string 30 | steps: 31 | - run: 32 | name: Setup Artsy Remote Docker Connection 33 | command: | 34 | if [[ -n "$AWS_ACCESS_KEY_ID" && -n "$AWS_SECRET_ACCESS_KEY" ]]; then 35 | printf "%s Setting up remote docker connection...\n" "$(TZ=UTC date)" 36 | mkdir ~/.docker 37 | aws s3 cp s3://<< parameters.artsy_s3_path_root >>/ca.pem ~/.docker/ca.pem 38 | aws s3 cp s3://<< parameters.artsy_s3_path_root >>/cert.pem ~/.docker/cert.pem 39 | aws s3 cp s3://<< parameters.artsy_s3_path_root >>/key.pem ~/.docker/key.pem 40 | 41 | echo 'export DOCKER_HOST="tcp://<< parameters.artsy_docker_host >>:<< parameters.artsy_docker_port >>"' >> "$BASH_ENV" 42 | echo 'export DOCKER_TLS_VERIFY="1"' >> "$BASH_ENV" 43 | source "$BASH_ENV" 44 | 45 | printf "Checking remote docker connection...\n" 46 | if docker ps --last 1 --quiet; then 47 | printf "Remote docker connection established.\n" 48 | else 49 | printf "Remote docker daemon unavailable. Reverting back to Circle CI docker.\n" 50 | rm $BASH_ENV 51 | exit 0 52 | fi 53 | else 54 | printf 'Required environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` unavailable. Reverting back to Circle CI docker.\n' 55 | exit 0 56 | fi 57 | 58 | buildkit-build-image-via-artsy: 59 | steps: 60 | - run: 61 | name: Build via Artsy Remote Docker Connection (Buildkit) 62 | no_output_timeout: 15m 63 | command: | 64 | if test -f "$BASH_ENV"; then 65 | source $BASH_ENV 66 | 67 | export DOCKER_BUILDKIT=1 68 | export BUILDKIT_PROGRESS=plain 69 | export COMPOSE_DOCKER_CLI_BUILD=1 70 | export HOKUSAI_ALWAYS_VERBOSE=1 71 | 72 | export TAG_LABEL="" 73 | if [ ! -z "${BUILD_TARGET}" ]; then 74 | TAG_LABEL="-${BUILD_TARGET}" 75 | fi 76 | export BUILD_TAG="${CIRCLE_SHA1}${TAG_LABEL}" 77 | 78 | printf "%s Building image...\n" "$(TZ=UTC date)" 79 | BUILD_TARGET="$BUILD_TARGET" BUILD_TAG="${BUILD_TAG}" hokusai build 80 | printf "%s Image built.\n" "$(TZ=UTC date)" 81 | 82 | printf "Skipping local docker build fallback...\n" 83 | fi 84 | circleci step halt 85 | 86 | buildkit-push-image-via-artsy: 87 | steps: 88 | - run: 89 | name: Push via Artsy Remote Docker Connection (Buildkit) 90 | no_output_timeout: 15m 91 | command: | 92 | if test -f "$BASH_ENV"; then 93 | source $BASH_ENV 94 | 95 | export DOCKER_BUILDKIT=1 96 | export BUILDKIT_PROGRESS=plain 97 | export COMPOSE_DOCKER_CLI_BUILD=1 98 | export HOKUSAI_ALWAYS_VERBOSE=1 99 | 100 | export TAG_LABEL="" 101 | if [ ! -z "${BUILD_TARGET}" ]; then 102 | TAG_LABEL="-${BUILD_TARGET}" 103 | fi 104 | 105 | export BUILD_TAG="${CIRCLE_SHA1}${TAG_LABEL}" 106 | 107 | if hokusai registry images --tag-exists "$BUILD_TAG" >/dev/null 108 | then 109 | echo "Skipping push as the tag $BUILD_TAG already exists in the Docker registry" 110 | else 111 | printf "%s Pushing image...\n" "$(TZ=UTC date)" 112 | hokusai registry push \ 113 | --no-build \ 114 | --local-tag="${BUILD_TAG}" \ 115 | --tag="${BUILD_TAG}" \ 116 | --skip-latest 117 | printf "%s Image pushed.\n" "$(TZ=UTC date)" 118 | fi 119 | 120 | printf "Skipping local docker build fallback...\n" 121 | fi 122 | circleci step halt 123 | 124 | build-image-via-artsy: 125 | steps: 126 | - run: 127 | name: Build & Push via Artsy Remote Docker Connection 128 | no_output_timeout: 15m 129 | command: | 130 | if test -f "$BASH_ENV"; then 131 | source $BASH_ENV 132 | 133 | printf "%s Building image...\n" "$(TZ=UTC date)" 134 | export BUILD_TARGET="$BUILD_TARGET" 135 | export BUILD_TAG="$CIRCLE_SHA1" 136 | hokusai build 137 | printf "%s Image built.\n" "$(TZ=UTC date)" 138 | 139 | if hokusai registry images --tag-exists "$BUILD_TAG" >/dev/null 140 | then 141 | echo "Skipping push as the tag $BUILD_TAG already exists in the Docker registry" 142 | else 143 | printf "%s Pushing image...\n" "$(TZ=UTC date)" 144 | hokusai registry push \ 145 | --no-build \ 146 | --local-tag="$BUILD_TAG" \ 147 | --tag="$BUILD_TAG" \ 148 | --skip-latest 149 | printf "%s Image pushed.\n" "$(TZ=UTC date)" 150 | fi 151 | 152 | printf "Skipping local docker build fallback...\n" 153 | circleci step halt 154 | else 155 | printf "Remote docker build unavailable. Reverting back to Circle CI docker.\n" 156 | fi 157 | 158 | build-image-via-circle: 159 | steps: 160 | - run: 161 | name: Build & Push via Circle CI Fallback 162 | no_output_timeout: 15m 163 | command: | 164 | printf "%s Building image...\n" "$(TZ=UTC date)" 165 | export BUILD_TARGET="$BUILD_TARGET" 166 | export BUILD_TAG="$CIRCLE_SHA1" 167 | hokusai build 168 | printf "%s Image built.\n" "$(TZ=UTC date)" 169 | 170 | if hokusai registry images --tag-exists "$BUILD_TAG" >/dev/null 171 | then 172 | echo "Skipping push as the tag $BUILD_TAG already exists in the Docker registry" 173 | else 174 | printf "%s Pushing image...\n" "$(TZ=UTC date)" 175 | hokusai registry push \ 176 | --no-build \ 177 | --local-tag="$BUILD_TAG" \ 178 | --tag="$BUILD_TAG" \ 179 | --skip-latest 180 | printf "%s Image pushed.\n" "$(TZ=UTC date)" 181 | fi 182 | 183 | test: 184 | steps: 185 | - run: hokusai registry pull --tag "$CIRCLE_SHA1" 186 | - run: 187 | name: Test 188 | command: hokusai test --no-build 189 | no_output_timeout: 3600s 190 | 191 | jobs: 192 | test: 193 | executor: << parameters.executor >> 194 | parameters: 195 | executor: 196 | type: executor 197 | default: deploy 198 | steps: 199 | - add_ssh_keys 200 | - checkout 201 | - setup_remote_docker: 202 | version: 20.10.24 203 | - test 204 | 205 | build: 206 | executor: << parameters.executor >> 207 | parameters: 208 | executor: 209 | type: executor 210 | default: deploy 211 | artsy_docker_host: 212 | type: string 213 | default: docker.artsy.net 214 | artsy_docker_port: 215 | type: integer 216 | default: 2376 217 | artsy_s3_path_root: 218 | type: string 219 | default: artsy-dockerd 220 | steps: 221 | - add_ssh_keys 222 | - checkout 223 | - setup-artsy-remote-docker: 224 | artsy_docker_host: << parameters.artsy_docker_host >> 225 | artsy_docker_port: << parameters.artsy_docker_port >> 226 | artsy_s3_path_root: << parameters.artsy_s3_path_root >> 227 | - build-image-via-artsy 228 | - setup_remote_docker: 229 | version: 20.10.24 230 | - build-image-via-circle 231 | 232 | buildkit-build: 233 | executor: << parameters.executor >> 234 | parameters: 235 | executor: 236 | type: executor 237 | default: deploy 238 | artsy_docker_host: 239 | type: string 240 | default: docker.artsy.net 241 | artsy_docker_port: 242 | type: integer 243 | default: 2376 244 | artsy_s3_path_root: 245 | type: string 246 | default: artsy-dockerd 247 | steps: 248 | - add_ssh_keys 249 | - checkout 250 | - setup-artsy-remote-docker: 251 | artsy_docker_host: << parameters.artsy_docker_host >> 252 | artsy_docker_port: << parameters.artsy_docker_port >> 253 | artsy_s3_path_root: << parameters.artsy_s3_path_root >> 254 | - buildkit-build-image-via-artsy 255 | 256 | buildkit-push: 257 | executor: << parameters.executor >> 258 | parameters: 259 | executor: 260 | type: executor 261 | default: deploy 262 | artsy_docker_host: 263 | type: string 264 | default: docker.artsy.net 265 | artsy_docker_port: 266 | type: integer 267 | default: 2376 268 | artsy_s3_path_root: 269 | type: string 270 | default: artsy-dockerd 271 | steps: 272 | - add_ssh_keys 273 | - checkout 274 | - setup-artsy-remote-docker: 275 | artsy_docker_host: << parameters.artsy_docker_host >> 276 | artsy_docker_port: << parameters.artsy_docker_port >> 277 | artsy_s3_path_root: << parameters.artsy_s3_path_root >> 278 | - buildkit-push-image-via-artsy 279 | -------------------------------------------------------------------------------- /src/skip-wip-ci/README.md: -------------------------------------------------------------------------------- 1 | # artsy/skip-wip-ci 2 | 3 | This node contains shared commands and jobs that hit Github's API and skips draft PRs or PRs with "\[wip\]", "\[skip ci\]", or "\[ci skip\]" in title or commit message 4 | 5 | ## Usage example 6 | 7 | To use the `check-skippable-ci` job 8 | 9 | ```yaml 10 | # In your project's .circleci/config.yml 11 | 12 | # Using the volatile label is _not_ recommended. 13 | # Use the version in the comment at the top of node.yml instead. 14 | 15 | orbs: 16 | skip-wip-ci: artsy/skip-wip-ci@volatile 17 | 18 | workflows: 19 | default: 20 | jobs: 21 | - skip-wip-ci/check-skippable-ci 22 | - test: 23 | <<: *not_staging_release 24 | ``` 25 | 26 | skip-wip-ci will run in parallel to test and cancel the test job if criteria is met 27 | -------------------------------------------------------------------------------- /src/skip-wip-ci/skip-wip-ci.yml: -------------------------------------------------------------------------------- 1 | # Orb Version 0.3.0 2 | 3 | version: 2.1 4 | description: 'Use to skip CI when PR title contains "[wip]", "[skip ci]", "[ci skip]" or is draft' 5 | 6 | commands: 7 | check-skippable-pr: 8 | steps: 9 | - run: apk add --no-cache bash curl jq 10 | - run: 11 | shell: /bin/bash 12 | name: Check skippable PR 13 | command: | 14 | required_env_vars=( 15 | "GITHUB_TOKEN" 16 | "CIRCLE_PROJECT_USERNAME" 17 | "CIRCLE_PR_REPONAME" 18 | "CIRCLE_PR_NUMBER" 19 | "CIRCLE_TOKEN" 20 | "CIRCLE_BUILD_NUM" 21 | ) 22 | 23 | for required_env_var in ${required_env_vars[@]}; do 24 | if [[ -z "${!required_env_var}" ]]; then 25 | printf "${required_env_var} not provided, but that doesn't mean we should skip CI.\n" 26 | exit 0 27 | fi 28 | done 29 | 30 | # Since we're piggybacking off of an existing OAuth var, tweak the var for our uses 31 | token=$(printf "${GITHUB_TOKEN}" | cut -d':' -f1) 32 | 33 | headers="Authorization: token $token" 34 | api_endpoint="https://api.github.com/repos/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PR_REPONAME}/pulls/${CIRCLE_PR_NUMBER}" 35 | 36 | # Fetch PR metadata from Github's API and parse fields from json 37 | github_res=$(curl --silent --header "${headers}" "${api_endpoint}" | jq '{mergeable_state: .mergeable_state, title: .title}') 38 | mergeable_state=$(printf "${github_res}" | jq '.mergeable_state') 39 | title=$(printf "${github_res}" | jq '.title' | tr '[:upper:]' '[:lower:]') 40 | echo "${title}" 41 | 42 | if [[ "${title}" == "null" && "${mergeable_state}" == "null" ]]; then 43 | printf "Couldn't fetch info on PR, but that doesn't mean we should skip CI.\n" 44 | exit 0 45 | fi 46 | 47 | cancel_running_jobs=0 48 | 49 | if [[ "${mergeable_state}" == "\"draft\"" ]]; then 50 | printf "PR is a draft, skipping CI!\n" 51 | cancel_running_jobs=1 52 | fi 53 | 54 | for skip_token in '[skip ci]' '[ci skip]' '[wip]'; do 55 | if [[ ${title} == *"${skip_token}"* ]]; then 56 | printf "Found \"${skip_token}\" in PR title, skipping CI!\n" 57 | cancel_running_jobs=1 58 | fi 59 | done 60 | 61 | if [[ "${cancel_running_jobs}" == 1 ]]; then 62 | printf "Attempting to cancel any running jobs" 63 | CIRCLE_API_BASE_URL="https://circleci.com/api/v1.1/project/github/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}" 64 | AUTH_PARAMS="circle-token=${CIRCLE_TOKEN}" 65 | SELF_BUILD_NUM="${CIRCLE_BUILD_NUM}" 66 | 67 | all_jobs=$(curl --silent --show-error "${CIRCLE_API_BASE_URL}/tree/${CIRCLE_BRANCH}?${AUTH_PARAMS}") 68 | 69 | running_jobs=$(echo "${all_jobs}" | jq "map(if .status == \"running\" or .status == \"not_running\" then .build_num else \"None\" end) - [${SELF_BUILD_NUM}] - [\"None\"] | .[]") 70 | 71 | for buildNum in $running_jobs; do 72 | printf "Canceling ${buildNum}" 73 | curl --silent --show-error --request POST "${CIRCLE_API_BASE_URL}/${buildNum}/cancel?${AUTH_PARAMS}" > /dev/null 74 | done 75 | else 76 | printf "No reason to skip CI, let's go!" 77 | fi 78 | 79 | exit 0 80 | 81 | check-skippable-commit: 82 | steps: 83 | - run: apk add --no-cache bash curl jq 84 | - run: 85 | shell: /bin/bash 86 | name: Check skippable commit 87 | command: | 88 | 89 | required_env_vars=( 90 | "GITHUB_TOKEN" 91 | "CIRCLE_PROJECT_USERNAME" 92 | "CIRCLE_PR_REPONAME" 93 | "CIRCLE_SHA1" 94 | "CIRCLE_TOKEN" 95 | "CIRCLE_BUILD_NUM" 96 | ) 97 | 98 | for required_env_var in ${required_env_vars[@]}; do 99 | if [[ -z "${!required_env_var}" ]]; then 100 | printf "${required_env_var} not provided, but that doesn't mean we should skip CI.\n" 101 | exit 0 102 | fi 103 | done 104 | 105 | token=$(printf "${GITHUB_TOKEN}" | cut -d':' -f1) 106 | 107 | headers="Authorization: token $token" 108 | api_endpoint="https://api.github.com/repos/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PR_REPONAME}/git/commits/${CIRCLE_SHA1}" 109 | 110 | # Fetch last commit message from github using git SHA 111 | last_commit_message=$(curl --silent --header "${headers}" "${api_endpoint}" | jq '.message') 112 | 113 | for skip_token in '[skip ci]' '[ci skip]' '[wip]'; do 114 | if [[ ${last_commit_message} == *"${skip_token}"* ]]; then 115 | printf "Found \"${skip_token}\" in head commit message, skipping CI!\n" 116 | cancel_running_jobs=1 117 | fi 118 | done 119 | 120 | if [[ "${cancel_running_jobs}" == 1 ]]; then 121 | printf "Attempting to cancel any running jobs" 122 | CIRCLE_API_BASE_URL="https://circleci.com/api/v1.1/project/github/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}" 123 | AUTH_PARAMS="circle-token=${CIRCLE_TOKEN}" 124 | SELF_BUILD_NUM="${CIRCLE_BUILD_NUM}" 125 | 126 | all_jobs=$(curl --silent --show-error "${CIRCLE_API_BASE_URL}/tree/${CIRCLE_BRANCH}?${AUTH_PARAMS}") 127 | 128 | running_jobs=$(echo "${all_jobs}" | jq "map(if .status == \"running\" or .status == \"not_running\" then .build_num else \"None\" end) - [${SELF_BUILD_NUM}] - [\"None\"] | .[]") 129 | 130 | for buildNum in $running_jobs; do 131 | printf "Canceling ${buildNum}" 132 | curl --silent --show-error --request POST "${CIRCLE_API_BASE_URL}/${buildNum}/cancel?${AUTH_PARAMS}" > /dev/null 133 | done 134 | else 135 | printf "No reason to skip CI, let's go!" 136 | fi 137 | 138 | exit 0 139 | 140 | jobs: 141 | check-skippable-pr: 142 | docker: 143 | - image: alpine:3.7 144 | steps: 145 | - check-skippable-pr 146 | 147 | check-skippable-commit: 148 | docker: 149 | - image: alpine:3.7 150 | steps: 151 | - check-skippable-commit 152 | 153 | check-skippable-ci: 154 | docker: 155 | - image: alpine:3.7 156 | steps: 157 | - check-skippable-commit 158 | - check-skippable-pr 159 | -------------------------------------------------------------------------------- /src/yarn/README.md: -------------------------------------------------------------------------------- 1 | # artsy/yarn 2 | 3 | ⚠️ [`artsy/yarn`](https://github.com/artsy/orbs/tree/main/src/yarn) has now been deprecated in favor of [`circleci/node`](https://circleci.com/developer/orbs/orb/circleci/node). Please use that for all Node.js-based CI needs. 4 | 5 | This orb contains the bulk of Artsy's commonly used [yarn](https://yarnpkg.com/en/) commands that power the builds for our frontend and Node projects. 6 | -------------------------------------------------------------------------------- /src/yarn/yarn.yml: -------------------------------------------------------------------------------- 1 | # Orb Version 6.5.0 2 | 3 | version: 2.1 4 | description: Common yarn commands 5 | 6 | orbs: 7 | queue: eddiewebb/queue@1.0.110 8 | slack: circleci/slack@3.4.2 9 | utils: artsy/orb-tools@0.5.0 10 | 11 | executors: 12 | node: 13 | docker: 14 | - image: cimg/node:18.15.0 15 | 16 | commands: 17 | # https://circleci.com/docs/2.0/caching/#basic-example-of-dependency-caching 18 | save_dependencies: 19 | steps: 20 | - save_cache: 21 | key: yarn-deps-v1-{{ checksum "yarn.lock" }} 22 | paths: 23 | - ./node_modules 24 | 25 | # If there isn't a match to the first key, it'll do a partial match of the 26 | # second. That means after the first cache save there will always be a cache 27 | # hit, but it might be an older version of the cache 28 | # 29 | # https://circleci.com/docs/2.0/caching/#restoring-cache 30 | load_dependencies: 31 | steps: 32 | - restore_cache: 33 | keys: 34 | - yarn-deps-v1-{{ checksum "yarn.lock" }} 35 | 36 | # By default when yarn runs it does not check the filesystem to ensure the 37 | # packages it expects to be installed are actually installed. Using the 38 | # --check-files flag ensures that any packages or files missing or out of date 39 | # on the file system (i.e. those which might be restored from cache) match what 40 | # the yarn.lock file specifies 41 | install: 42 | steps: 43 | - run: yarn 44 | 45 | setup: 46 | steps: 47 | - add_ssh_keys 48 | - checkout 49 | - load_dependencies 50 | - install 51 | 52 | update_dependencies: 53 | steps: 54 | - setup 55 | - save_dependencies 56 | 57 | run-script: 58 | parameters: 59 | script: 60 | type: string 61 | steps: 62 | - setup 63 | - run: yarn << parameters.script >> 64 | 65 | pre-release: 66 | steps: 67 | - setup 68 | # Setup the .npmrc with the proper registry and auth token to publish 69 | - run: 70 | name: Setup npmrc 71 | command: echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" >> ~/.npmrc 72 | 73 | run-release: 74 | parameters: 75 | script: 76 | type: string 77 | default: yarn release 78 | steps: 79 | - pre-release 80 | - run: << parameters.script >> 81 | 82 | jobs: 83 | run: 84 | executor: node 85 | parameters: 86 | script: 87 | type: string 88 | steps: 89 | - run-script: 90 | script: << parameters.script >> 91 | 92 | lint: 93 | executor: node 94 | steps: 95 | - run-script: 96 | script: lint 97 | 98 | relay: 99 | executor: node 100 | steps: 101 | - run-script: 102 | script: relay 103 | 104 | type-check: 105 | executor: node 106 | steps: 107 | - run-script: 108 | script: type-check 109 | 110 | test: 111 | executor: node 112 | parameters: 113 | args: 114 | type: string 115 | default: "" 116 | steps: 117 | - run-script: 118 | script: test << parameters.args >> 119 | 120 | jest: 121 | executor: node 122 | environment: 123 | JEST_JUNIT_OUTPUT_DIR: "reports/junit/js-test-results.xml" 124 | parameters: 125 | args: 126 | description: Arguments to be passed directly to jest 127 | type: string 128 | default: -w 4 129 | run_all_tests_if_these_files_change: 130 | type: string 131 | default: "" 132 | description: | 133 | If any of these files have changed, re-run the complete test suite instead of doing incremental tests. 134 | This is used in conjunction with `only_test_changed` to determine what global resources may alter test 135 | outcomes that jest may not be able to pick up on. 136 | only_test_changed: 137 | type: boolean 138 | default: false 139 | notify_slack_on_failure: 140 | description: | 141 | If a slack message should be sent out if building off of main branch fails. Requires SLACK_WEBHOOK environment 142 | variable to be set. 143 | type: boolean 144 | default: false 145 | steps: 146 | - when: 147 | condition: << parameters.notify_slack_on_failure >> 148 | steps: 149 | - utils/check-for-env-vars: 150 | env-vars: SLACK_WEBHOOK 151 | - setup 152 | - when: 153 | condition: << parameters.only_test_changed >> 154 | steps: 155 | - run: 156 | name: jest 157 | command: | 158 | important_files_modified() { 159 | FILES="yarn.lock << parameters.run_all_tests_if_these_files_change >>" 160 | for file in ${FILES[@]}; do 161 | if ! git diff origin/main -s --exit-code $file; then 162 | return 0 163 | fi 164 | done 165 | return 1 166 | } 167 | 168 | if [[ "$(git rev-parse --abbrev-ref HEAD)" == "main" ]] || important_files_modified; then 169 | yarn jest --coverage --reporters=default --reporters=jest-junit << parameters.args >> 170 | else 171 | yarn jest --coverage --changedSince=origin/main --reporters=default --reporters=jest-junit << parameters.args >> 172 | fi 173 | - unless: 174 | condition: << parameters.only_test_changed >> 175 | steps: 176 | - run: 177 | name: jest 178 | command: yarn jest --coverage --reporters=default --reporters=jest-junit << parameters.args >> 179 | - store_test_results: 180 | path: reports/junit 181 | - store_artifacts: 182 | path: reports/junit 183 | - when: 184 | condition: << parameters.notify_slack_on_failure >> 185 | steps: 186 | - utils/skip-if-fork-or-not-pr 187 | - slack/status: 188 | fail_only: true 189 | only_for_branches: main 190 | failure_message: $CIRCLE_PROJECT_REPONAME's tests failed for main 191 | 192 | update-cache: 193 | executor: node 194 | steps: 195 | - update_dependencies 196 | 197 | # A job responsible for ensuring only 1 main build runs at a time so that 198 | # there are no deployment race conditions 199 | workflow-queue: 200 | executor: node 201 | steps: 202 | - queue/until_front_of_line: 203 | time: "2" # how long a queue will wait until the job exits 204 | only-on-branch: main # restrict queueing to a specific branch (default *) 205 | consider-job: false # block whole workflow if any job still running 206 | 207 | release: 208 | executor: node 209 | steps: 210 | - run-release 211 | --------------------------------------------------------------------------------