├── README.md ├── LICENSE ├── vault-list.sh ├── vault-logout.sh ├── vault-pwgen.sh ├── vault-unwrap.sh ├── file2configmap.sh ├── vault-read.sh ├── vault-info.sh ├── vault-list-tree.sh ├── vault-write.sh ├── vault-list-users.sh ├── vault-login.sh ├── vault-token-inventory.sh ├── functions.sh └── vault-secrets-the-simple-way.md /README.md: -------------------------------------------------------------------------------- 1 | # vault-scripts 2 | A collection of scripts to help operate Hashicorp Vault. 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Xu Wang 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /vault-list.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -e 2 | 3 | # MIT License 4 | # 5 | # Copyright (c) 2018 Xu Wang 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy 8 | # of this software and associated documentation files (the "Software"), to deal 9 | # in the Software without restriction, including without limitation the rights 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | # copies of the Software, and to permit persons to whom the Software is 12 | # furnished to do so, subject to the following conditions: 13 | # 14 | # The above copyright notice and this permission notice shall be included in all 15 | # copies or substantial portions of the Software. 16 | # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | # SOFTWARE. 24 | # Usage: 25 | # vault-list 26 | 27 | # include functions 28 | THIS_DIR=$(dirname "$0") 29 | 30 | set -u 31 | VAULT_ADDR=${VAULT_ADDR} 32 | path=$1 33 | 34 | vault kv list $path 35 | -------------------------------------------------------------------------------- /vault-logout.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # MIT License 4 | # 5 | # Copyright (c) 2018 Xu Wang 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy 8 | # of this software and associated documentation files (the "Software"), to deal 9 | # in the Software without restriction, including without limitation the rights 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | # copies of the Software, and to permit persons to whom the Software is 12 | # furnished to do so, subject to the following conditions: 13 | # 14 | # The above copyright notice and this permission notice shall be included in all 15 | # copies or substantial portions of the Software. 16 | # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | # SOFTWARE. 24 | # Revoke login token and all its children 25 | # and cleanup local cache 26 | 27 | THIS_DIR=$(dirname "$0") 28 | 29 | set -u 30 | export VAULT_ADDR=${VAULT_ADDR} 31 | VAULT_CACHE=${VAULT_CACHE:-$HOME/.vault-local} 32 | 33 | echo "Logout ${VAULT_ADDR}" 34 | vault token revoke -self &> /dev/null 35 | rm -f ${HOME}/.vault-token 36 | 37 | # Cleanup local cache 38 | rm -rf ${VAULT_CACHE} 39 | -------------------------------------------------------------------------------- /vault-pwgen.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # MIT License 4 | # 5 | # Copyright (c) 2018 Xu Wang 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy 8 | # of this software and associated documentation files (the "Software"), to deal 9 | # in the Software without restriction, including without limitation the rights 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | # copies of the Software, and to permit persons to whom the Software is 12 | # furnished to do so, subject to the following conditions: 13 | # 14 | # The above copyright notice and this permission notice shall be included in all 15 | # copies or substantial portions of the Software. 16 | # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | # SOFTWARE. 24 | 25 | if [ -z "$1" ]; then 26 | echo "Usage: $0 []" 27 | echo "You must have the write privilege to " 28 | exit 1 29 | fi 30 | 31 | length=${2:-20} 32 | 33 | if vault-read.sh $1 &> /dev/null 34 | then 35 | echo "ERROR: $1 has value." 36 | else 37 | echo Generate a new password and save to the $1 38 | pwd=$(pwgen $length 1 | tr -d '\n') 39 | vault-write.sh $1 $pwd 40 | fi 41 | -------------------------------------------------------------------------------- /vault-unwrap.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # MIT License 4 | # 5 | # Copyright (c) 2018 Xu Wang 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy 8 | # of this software and associated documentation files (the "Software"), to deal 9 | # in the Software without restriction, including without limitation the rights 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | # copies of the Software, and to permit persons to whom the Software is 12 | # furnished to do so, subject to the following conditions: 13 | # 14 | # The above copyright notice and this permission notice shall be included in all 15 | # copies or substantial portions of the Software. 16 | # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | # SOFTWARE. 24 | # Get the secret wrapped by a vault token 25 | # Usage: 26 | # vault-unwrap.sh 27 | 28 | if ! which vault > /dev/null 29 | then 30 | echo vault cli is missing, please install it from https://www.vaultproject.io/downloads.html 31 | exit 1 32 | fi 33 | 34 | set -u 35 | VAULT_ADDR=${VAULT_ADDR} 36 | VAULT_TOKEN=$1 37 | 38 | shift 39 | vault unwrap $@ 40 | 41 | # api version 42 | # curl -sSL --header "X-Vault-Token: $VAULT_TOKEN" --request POST $VAULT_ADDR/v1/sys/wrapping/unwrap 43 | -------------------------------------------------------------------------------- /file2configmap.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # MIT License 4 | # 5 | # Copyright (c) 2019 Xu Wang 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy 8 | # of this software and associated documentation files (the "Software"), to deal 9 | # in the Software without restriction, including without limitation the rights 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | # copies of the Software, and to permit persons to whom the Software is 12 | # furnished to do so, subject to the following conditions: 13 | # 14 | # The above copyright notice and this permission notice shall be included in all 15 | # copies or substantial portions of the Software. 16 | # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | # SOFTWARE. 24 | # 25 | # Usage: 26 | # file2configmap 27 | # Read content from and convert it to Kubernetes configmap resource. 28 | 29 | OLDIFS=$IFS 30 | IFS='' 31 | 32 | if [ "$#" -lt 3 ]; then 33 | echo "Usage: $(basenae $0) []" 34 | exit 1 35 | fi 36 | # Required arguments 37 | set -u 38 | APP_NAMESPACE=$1 39 | MAP_NAME=$2 40 | KEY=$3 41 | set +u 42 | 43 | cat < 27 | # 28 | # Assumption: a secret object has two fields: format and value. 29 | # "format" can be text or base64, "value" is the actual data. 30 | # The code will output value either in text or base64 decoded value. 31 | 32 | SCRIPT_NAME=$(basename "$0") 33 | if [ "$#" != "1" ]; then 34 | echo usage "$SCRIPT_NAME: " 35 | exit 1 36 | else 37 | path=$1 38 | fi 39 | 40 | VAULT_ADDR=${VAULT_ADDR:-https://vault.example.com} 41 | 42 | # Get the kv secret from path in json 43 | data=$(vault kv get -format=json $path) 44 | if [ "$?" != "0" ]; then 45 | exit 1 46 | fi 47 | 48 | f=$(echo "$data" | jq -r '.data.format//.data.data.format' 2> /dev/null) 49 | v=$(echo "$data" | jq -r '.data.value//.data.data.value') 50 | 51 | # if value format is base64, decode it 52 | if [ "base64" == "$f" ] 53 | then 54 | echo -n "$v" | base64 --decode 55 | else 56 | echo -n "$v" 57 | fi 58 | -------------------------------------------------------------------------------- /vault-info.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # MIT License 4 | # 5 | # Copyright (c) 2018 Xu Wang 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy 8 | # of this software and associated documentation files (the "Software"), to deal 9 | # in the Software without restriction, including without limitation the rights 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | # copies of the Software, and to permit persons to whom the Software is 12 | # furnished to do so, subject to the following conditions: 13 | # 14 | # The above copyright notice and this permission notice shall be included in all 15 | # copies or substantial portions of the Software. 16 | # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | # SOFTWARE. 24 | THIS_DIR=$(dirname "$0") 25 | 26 | # include functions 27 | source $THIS_DIR/functions.sh 28 | 29 | vault_info() { 30 | export VAULT_ADDR=${VAULT_ADDR} 31 | VAULT_AUTH_METHOD=${VAULT_AUTH_METHOD:-ldap} 32 | SEC_PATH=${SEC_PATH:-auth/token/lookup-self} 33 | 34 | echo "VAULT SERVER: $VAULT_ADDR" 35 | # seal status (0 unsealed, 2 sealed, 1 error) 36 | vault status 37 | [[ $? -eq 1 ]] && die "Error checking vault status." 38 | if vault-list.sh ${SEC_PATH} 2>&1 >/dev/null | grep 'missing client token' 2>&1 >/dev/null 39 | then 40 | echo "You are not logged in VAULT" 41 | elif vault-list.sh ${SEC_PATH} 2>&1 >/dev/null | grep 'permission denied' 2>&1 >/dev/null 42 | then 43 | echo "You are logged in VAULT but you don't have permissions to access ${SEC_PATH}" 44 | else 45 | echo "You are logged in VAULT and has the access to ${SEC_PATH}" 46 | vault read auth/token/lookup-self 47 | fi 48 | } 49 | 50 | vault_info 51 | -------------------------------------------------------------------------------- /vault-list-tree.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -e 2 | 3 | # MIT License 4 | # 5 | # Copyright (c) 2018 Xu Wang 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy 8 | # of this software and associated documentation files (the "Software"), to deal 9 | # in the Software without restriction, including without limitation the rights 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | # copies of the Software, and to permit persons to whom the Software is 12 | # furnished to do so, subject to the following conditions: 13 | # 14 | # The above copyright notice and this permission notice shall be included in all 15 | # copies or substantial portions of the Software. 16 | # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | # SOFTWARE. 24 | # 25 | # List all kv path recursively from a given path 26 | # Usage: 27 | # $0 28 | 29 | export VAULT_ADDR=${VAULT_ADDR:-http://127.0.0.1:8200} 30 | 31 | function get_children() { 32 | if children=$( $vault_list -format=json $1 2>/dev/null) 33 | then 34 | echo $children | jq -r '.[]' 35 | fi 36 | } 37 | 38 | function list() { 39 | echo $1 40 | for child in $(get_children $1) 41 | do 42 | list "${1}${child}" 43 | done 44 | } 45 | 46 | function abort() { 47 | echo "$1" 48 | echo && echo "Usage: $0 " && echo 49 | exit 1 50 | } 51 | 52 | ###### 53 | # Main 54 | ###### 55 | 56 | if [[ -z "$1" ]] || [[ "$1" =~ ^- ]] 57 | then 58 | abort "" 59 | else 60 | root=$1 61 | root=${root%/} # Remove trailing slash if any. Will add it back later 62 | fi 63 | 64 | if ! msg=$(vault status 2>&1) 65 | then 66 | abort "$msg." 67 | fi 68 | 69 | # For kv2 backend only 70 | vault_list="vault kv list" 71 | 72 | if msg=$($vault_list $root 2>&1) 73 | then 74 | list $root/ 75 | else 76 | abort "Error: $msg." 77 | fi 78 | -------------------------------------------------------------------------------- /vault-write.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # MIT License 4 | # 5 | # Copyright (c) 2018 Xu Wang 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy 8 | # of this software and associated documentation files (the "Software"), to deal 9 | # in the Software without restriction, including without limitation the rights 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | # copies of the Software, and to permit persons to whom the Software is 12 | # furnished to do so, subject to the following conditions: 13 | # 14 | # The above copyright notice and this permission notice shall be included in all 15 | # copies or substantial portions of the Software. 16 | # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | # SOFTWARE. 24 | # Usage: 25 | # vault-write [ | @] 26 | # 27 | # The script will write a secret object with two field: format and value. 28 | # "format" can be text or binary based on the input data. If the data is binary, the code will 29 | # base64 encode it and store in vault in "value" field. 30 | # 31 | # See vault-read.sh to read back data. 32 | 33 | #!/bin/bash 34 | # 35 | # Usage: vault-write ["secret strings>" | @] 36 | 37 | SCRIPT_NAME=$(basename "$0") 38 | 39 | VAULT_ADDR=${VAULT_ADDR:-https://vault.example.com} 40 | 41 | if [ "$#" != "2" ]; then 42 | echo usage "$SCRIPT_NAME: [\"\" | @]" 43 | exit 1 44 | fi 45 | 46 | path=$1 47 | 48 | if [[ "$2" =~ ^@ ]]; 49 | then 50 | # if the data is in a file 51 | src=$(echo $2 | cut -c 2-) 52 | if file -b --mime-encoding $src | grep -s binary > /dev/null 53 | then 54 | # if data is binary, base64 encode it and set format=base64 55 | cat $src | base64 | vault kv put $path value=- format="base64" 56 | else 57 | # otherwise set format=text 58 | cat $src | vault kv put $path value=- format="text" 59 | fi 60 | else 61 | vault kv put $path value="$2" format="text" 62 | fi 63 | 64 | -------------------------------------------------------------------------------- /vault-list-users.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -e 2 | 3 | # MIT License 4 | # 5 | # Copyright (c) 2018 Xu Wang 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy 8 | # of this software and associated documentation files (the "Software"), to deal 9 | # in the Software without restriction, including without limitation the rights 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | # copies of the Software, and to permit persons to whom the Software is 12 | # furnished to do so, subject to the following conditions: 13 | # 14 | # The above copyright notice and this permission notice shall be included in all 15 | # copies or substantial portions of the Software. 16 | # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | # SOFTWARE. 24 | # Usage: 25 | # vault-list-users.sh [] 26 | 27 | # include functions 28 | THIS_DIR=$(dirname "$0") 29 | source $THIS_DIR/functions.sh 30 | 31 | user=$1 32 | tmpfile=$(mktemp) 33 | 34 | function list_aliases() { 35 | vault list -format json identity/entity-alias/id | jq -r '.[]' > $tmpfile 36 | } 37 | 38 | function list_all_users() { 39 | for i in `cat $tmpfile` 40 | do 41 | vault read -format=json identity/entity-alias/id/$i | jq -r '.data.name' 42 | done 43 | } 44 | 45 | function get_user() { 46 | for i in `cat $tmpfile` 47 | do 48 | vault read -format=json identity/entity-alias/id/$i | \ 49 | jq -r ".data | select(.name==\"$user\")" > $tmpfile.$user 50 | if [ -s "$tmpfile.$user" ]; then 51 | cat $tmpfile.$user 52 | break 53 | fi 54 | done 55 | } 56 | 57 | # MAIN 58 | 59 | if vault token lookup > /dev/null 2>&1 ; then 60 | admin=$(vault token lookup -format=json | jq -r '.data.display_name') 61 | echo "Using $admin token to lookup vault users." 62 | else 63 | echo "Valid vault token is required. Please run vault login." 64 | exit 1 65 | fi 66 | 67 | list_aliases 68 | 69 | if [ ! -z "$user" ]; then 70 | get_user 71 | else 72 | list_all_users 73 | fi 74 | 75 | rm -rf $tmpfile $tmpfile.$user 76 | 77 | -------------------------------------------------------------------------------- /vault-login.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # MIT License 4 | # 5 | # Copyright (c) 2018 Xu Wang 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy 8 | # of this software and associated documentation files (the "Software"), to deal 9 | # in the Software without restriction, including without limitation the rights 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | # copies of the Software, and to permit persons to whom the Software is 12 | # furnished to do so, subject to the following conditions: 13 | # 14 | # The above copyright notice and this permission notice shall be included in all 15 | # copies or substantial portions of the Software. 16 | # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | # SOFTWARE. 24 | 25 | ############################################################################### 26 | # login to vault 27 | ############################################################################### 28 | 29 | THIS_DIR=$(dirname "$0") 30 | 31 | # include functions 32 | source $THIS_DIR/functions.sh 33 | 34 | trap_errors 35 | 36 | set +u # optional vars 37 | export VAULT_AUTH_PATH=${VAULT_AUTH_PATH:-ldap} 38 | export VAULT_AUTH_METHOD=${VAULT_AUTH_METHOD:-ldap} 39 | export VAULT_USER=${VAULT_USER:-$USER} 40 | export VAULT_ROLE_ID=${VAULT_ROLE_ID} 41 | set -u 42 | export VAULT_ADDR=${VAULT_ADDR} 43 | 44 | echo "VAULT SERVER: $VAULT_ADDR" 45 | 46 | if ! [ -z $VAULT_ROLE_ID ]; then 47 | echo "attempting to login in w/ approle provided in env vars VAULT_ROLE_ID & VAULT_SECRET_ID" 48 | vault write -format json auth/approle/login \ 49 | role_id=$VAULT_ROLE_ID \ 50 | secret_id=$VAULT_SECRET_ID \ 51 | | jq -er .auth.client_token > ~/.vault-token 52 | fi 53 | 54 | if ! vault token lookup > /dev/null 2>&1; then 55 | if ! [ -z $VAULT_ROLE_ID ]; then 56 | >&2 "ERROR: VAULT_TOKEN does not exist or is not valid" && false 57 | fi 58 | 59 | echo "Please login VAULT as vault user ${VAULT_USER} with DUO device ready:" 60 | vault login -method=${VAULT_AUTH_METHOD} -path=${VAULT_AUTH_PATH} username=${VAULT_USER} 61 | else 62 | echo "You are logged in VAULT" 63 | fi 64 | 65 | -------------------------------------------------------------------------------- /vault-token-inventory.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -e 2 | 3 | # MIT License 4 | # 5 | # Copyright (c) 2018 Xu Wang 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy 8 | # of this software and associated documentation files (the "Software"), to deal 9 | # in the Software without restriction, including without limitation the rights 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | # copies of the Software, and to permit persons to whom the Software is 12 | # furnished to do so, subject to the following conditions: 13 | # 14 | # The above copyright notice and this permission notice shall be included in all 15 | # copies or substantial portions of the Software. 16 | # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | # SOFTWARE. 24 | 25 | ############################################################################### 26 | # List current tokens's properties using token accessors. 27 | ############################################################################### 28 | 29 | set +u # optional vars 30 | export VAULT_SEC_PATH=${VAULT_SEC_PATH:-auth/token/lookup-self} 31 | export VAULT_AUTH_PATH=${VAULT_AUTH_PATH:-ldap} 32 | export VAULT_AUTH_METHOD=${VAULT_AUTH_METHOD:-ldap} 33 | export VAULT_USER=${LOGNAME:-$USER} 34 | 35 | set -u 36 | export VAULT_ADDR=${VAULT_ADDR:-https} 37 | 38 | echo "VAULT ADDR: $VAULT_ADDR" 39 | 40 | if ! vault token lookup > /dev/null 2>&1; then 41 | echo "Hey $VAULT_USER: Please login VAULT with DUO device ready. Waiting...:" 42 | vault login -method=${VAULT_AUTH_METHOD} -path=${VAULT_AUTH_PATH} 43 | fi 44 | 45 | if vault token capabilities ${VAULT_SEC_PATH}/* | grep -q root; then 46 | echo "You are logged in VAULT with root token!" 47 | fi 48 | 49 | if vault token capabilities auth/tokens/accessors | grep -q deny; then 50 | >&2 "ERROR: no permission to list auth/tokens/accessors" && false 51 | else 52 | vault list -format=json auth/token/accessors | jq -r '.[]' | tee /tmp/token-accessors.$$ 53 | for i in `cat /tmp/token-accessors.$$` 54 | do 55 | echo 56 | echo Look up with token accessor $i 57 | vault token lookup --format=json --accessor $i 2>/dev/null >> /tmp/token-$$.json 58 | done 59 | fi 60 | rm -rf /tmp/token-accessors.$$ 61 | echo Token inventory is saved in /tmp/token-$$.json 62 | 63 | echo Usefull tricks: 64 | echo "cat /tmp/token-$$.json | jq -r [.data] | jq '.[] | select(.meta.username==\"xuwang\")'" 65 | echo "cat /tmp/token-$$.json | jq -r [.data] | jq '.[] | select(.meta.username==\"xuwang\").policies'" 66 | 67 | echo Get all the root info: 68 | echo "cat /tmp/token-$$.json | jq -r '[.data][] | select(.policies[] | contains(\"root\"))'" 69 | echo "cat /tmp/token-$$.json | jq -r '[.data][] | select(.policies[] | contains(\"root\")) | .accessor'" 70 | -------------------------------------------------------------------------------- /functions.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # MIT License 4 | # 5 | # Copyright (c) 2018 Xu Wang 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining a copy 8 | # of this software and associated documentation files (the "Software"), to deal 9 | # in the Software without restriction, including without limitation the rights 10 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | # copies of the Software, and to permit persons to whom the Software is 12 | # furnished to do so, subject to the following conditions: 13 | # 14 | # The above copyright notice and this permission notice shall be included in all 15 | # copies or substantial portions of the Software. 16 | # 17 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | # SOFTWARE. 24 | 25 | ############################################################################### 26 | # Helper functions 27 | ############################################################################### 28 | 29 | # colors 30 | yellow='\[\033[0;33m\]' 31 | red='\[\033[0;31m\]' 32 | reset='\[\033[0m\]' 33 | 34 | # http://stackoverflow.com/a/25515370 35 | yell() { echo "$0: $*" >&2; } 36 | die() { yell "$*"; exit 111; } 37 | try() { "$@" || die "cannot $*"; } 38 | 39 | err() { 40 | echo 41 | echo ${red}ERROR: ${@}$reset 42 | echo 43 | exit 1 44 | } 45 | 46 | err_report() { 47 | echo "$1: error on line $2" 48 | } 49 | 50 | trap_errors() { 51 | if [ "$debug_scripts" = "true" ]; then 52 | set -x 53 | fi 54 | 55 | set -eeuo pipefail 56 | trap 'err_report $BASH_SOURCE $LINENO' err 57 | export shellopts 58 | } 59 | 60 | # check if a variable is set. useful for set -u 61 | is_set() { 62 | declare -p $1 &> /dev/null 63 | } 64 | 65 | # is the variable set and have length? 66 | not_empty_var() { 67 | is_set $1 && eval val=\$$1 && [[ "$val" ]] 68 | } 69 | 70 | # is the variable unset or zero length? useful for set -u 71 | empty_var() { 72 | ! not_empty_var $1 73 | } 74 | 75 | is_null() { 76 | [[ $1 == "null" ]] 77 | } 78 | 79 | is_null_or_empty() { 80 | [[ $1 == "null" ]] || [[ -z "${1// }" ]] 81 | } 82 | 83 | confirm() { 84 | ## Ask to confirm an action 85 | echo "$*" 86 | echo "CONTINUE? [Y/N]: "; read ANSWER 87 | [[ $ANSWER == "Y" ]] 88 | } 89 | 90 | get_root_dir() { 91 | local f 92 | if [[ $1 == /* ]]; then f=2; else f=1; fi 93 | echo "$1" | cut -d "/" -f$f 94 | } 95 | 96 | # test if a path is a KV v2 path 97 | is_kv() { 98 | local kv_mount=$(get_root_dir $1) 99 | vault secrets list -format=json \ 100 | | jq -re --arg v "$kv_mount/" '.[$v] | select(((.type=="kv") or (.type=="generic")) and (.options.version=="2"))' \ 101 | | grep version &>/dev/null 102 | } 103 | 104 | vault_read_cmd() { 105 | echo "vault kv get" 106 | } 107 | 108 | vault_write_cmd() { 109 | echo "vault kv put" 110 | } 111 | 112 | vault_list_cmd() { 113 | echo "vault kv list" 114 | } 115 | 116 | urlencode() { 117 | # urlencode 118 | old_lc_collate=$LC_COLLATE 119 | LC_COLLATE=C 120 | 121 | local length="${#1}" 122 | for (( i = 0; i < length; i++ )); do 123 | local c="${1:i:1}" 124 | case $c in 125 | [a-zA-Z0-9.~_-]) printf "$c" ;; 126 | *) printf '%%%02X' "'$c" ;; 127 | esac 128 | done 129 | 130 | LC_COLLATE=$old_lc_collate 131 | } 132 | 133 | urldecode() { 134 | # urldecode 135 | 136 | local url_encoded="${1//+/ }" 137 | printf '%b' "${url_encoded//%/\\x}" 138 | } 139 | 140 | export -f yell die try err err_report trap_errors is_set not_empty_var empty_var confirm is_null is_null_or_empty get_root_dir 141 | export -f is_kv vault_read_cmd vault_write_cmd vault_list_cmd 142 | export -f urlencode urldecode 143 | -------------------------------------------------------------------------------- /vault-secrets-the-simple-way.md: -------------------------------------------------------------------------------- 1 | # Vault Secrets the Simple Way 2 | 3 | This article introduces a simple way of working with [Hashicorp Vault](https://github.com/hashicorp/vault) key-value secrets engine. 4 | 5 | ## Prerequisites 6 | 7 | You should have a running Hashicorp/Vault service and [Vault](https://www.vaultproject.io/downloads/) command available. 8 | 9 | Here is a quick way to install a [Vault dev server](https://learn.hashicorp.com/vault/getting-started/dev-server). 10 | 11 | ## Challenge 12 | 13 | The basic form of using key-value secret backend is simple: 14 | 15 | * Put a secret in vault 16 | 17 | ```console 18 | 19 | $ vault kv put secret/test-me username=me password=changeme 20 | Key Value 21 | --- ----- 22 | created_time 2020-02-21T19:10:19.074035078Z 23 | deletion_time n/a 24 | destroyed false 25 | version 1 26 | ``` 27 | 28 | * Get secret from vault 29 | 30 | ```console 31 | $ vault kv get -format=json secret/test-me 32 | Handling connection for 8443 33 | { 34 | ... 35 | "renewable": false, 36 | "data": { 37 | "data": { 38 | "password": "changeme", 39 | "username": "me" 40 | }, 41 | ... 42 | } 43 | ``` 44 | 45 | The data in key-value store can be of any type, such as SSL certificates, application configurations containing secrets, binary data, database connection strings with credentials. 46 | 47 | The data input can come from command line, from a program, from Vault secret plugin provider, etc. etc. 48 | 49 | Vault users usually design their vault secret input key field based on their application, get the secret with the same key field, 50 | and process the data based on type of content. For example, database admins may put secret with *db-user=\ password=\*, 51 | an authentication app may use *login=\ password=\*, or another app may do *key_id=\ key_secret=\*. 52 | 53 | In addition to all kinds of input fields, for binary data, you need to base64-encode secret first before putting it in Vault, and base64-decode it after reading 54 | back from Vault. 55 | 56 | This presents a challenge in a team environment that secrets are treated like pets, handled differently by different tools that need to know how the data is put into Vault. 57 | 58 | For example, for the *username=me password=changeme* secret we added, we can get value back with *-field=username and -field=password*: 59 | 60 | ```console 61 | $ vault kv get -field=username secret/test-me 62 | me 63 | $ vault kv get -field=password secret/test-me 64 | changeme 65 | ``` 66 | 67 | Another secret might use different field name: 68 | 69 | ```console 70 | $ vault kv get -field=login secret/another-test-me 71 | another-me 72 | $ vault kv get -field=secret secret/another-test-me 73 | changeme-too 74 | ``` 75 | 76 | This can quickly get out of control when you have many secrets, many people, and many applications need to access 77 | the secretes. 78 | 79 | ## Solution 80 | 81 | Although each secret is unique, we can **standardize input field name** with a team-agreed convention so 82 | the same field can be used to get the content of a secret, or to test the type of a secret, no matter what kind of secret it is. 83 | 84 | ## The Field Naming Convention 85 | 86 | In our environment, we require all secrets contain two *key=value* fields, namely: 87 | 88 | * **format**: this indicates the type of data. Can be "text" or "base64". 89 | * **value**: this is the actual secret content. Can be quoted strings, a text file, or a binary file. 90 | 91 | Using this convention, let's put a second version of the *test-me* secret with content from a json file */tmp/my-secret.json*: 92 | 93 | ```console 94 | { 95 | "username": "me", 96 | "password": "changeme", 97 | } 98 | ``` 99 | 100 | Then write to vault with *format=text* and *value=@/tmp/my-secret* field: 101 | 102 | ``` 103 | $ vault kv put secret/test-me format=text value=@/tmp/my-secret 104 | Key Value 105 | --- ----- 106 | created_time 2020-02-21T22:06:15.801274628Z 107 | deletion_time n/a 108 | destroyed false 109 | version 2 110 | ``` 111 | 112 | Get secret from vault with *value* and *format* field: 113 | 114 | ``` 115 | $ vault kv get -field=value secret/test-me 116 | { 117 | "username": "me", 118 | "password": "changeme", 119 | } 120 | 121 | $ vault kv get -field=format secret/test-me 122 | text 123 | ``` 124 | 125 | Note we get the data with common *field=value* convention. We also can use *field=format* to decide if we need base64 decode. 126 | 127 | We have developed with two simple scripts **vault-write.sh** and **vault-read.sh** to make it even easier to use based on this convention. 128 | 129 | ## Vault-write.sh 130 | 131 | ```console 132 | #!/bin/bash 133 | # 134 | # Usage: vault-write ["secret strings>" | @] 135 | 136 | SCRIPT_NAME=$(basename "$0") 137 | VAULT_ADDR=${VAULT_ADDR:-https://vault.example.com} 138 | if [ "$#" != "2" ]; then 139 | echo usage "$SCRIPT_NAME: [\"\" | @]" 140 | exit 1 141 | fi 142 | 143 | path=$1 144 | 145 | if [[ "$2" =~ ^@ ]]; 146 | then 147 | # if the data is in a file 148 | src=$(echo $2 | cut -c 2-) 149 | if file -b --mime-encoding $src | grep -s binary > /dev/null 150 | then 151 | # if data is binary, base64 encode it and set format=base64 152 | cat $src | base64 | vault kv put $path value=- format="base64" 153 | else 154 | # otherwise set format=text 155 | cat $src | vault kv put $path value=- format="text" 156 | fi 157 | else 158 | vault kv put $path value="$2" format="text" 159 | fi 160 | ``` 161 | 162 | The *vault-write.sh* accepts a secret path and a secret value either as string or a file. It decides 163 | if the data needs base64 encoding based on the content of the secret. 164 | 165 | Let's now write a binary file to *secret/test-me* path. 166 | 167 | ```console 168 | $ vault-write.sh secret/test-me @/tmp/gitlab.png 169 | Key Value 170 | --- ----- 171 | created_time 2020-02-21T22:47:57.918585831Z 172 | deletion_time n/a 173 | destroyed false 174 | version 3 175 | 176 | $ vault kv get --format=json secret/test-me 177 | { 178 | ... 179 | "data": { 180 | "data": { 181 | "format": "base64", 182 | "value": "iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAIAAADYYG7QAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAAgY0hSTQAAeiYAAICEAAD6AAAAgOgAAHUwAADqYAAAOpgAABdwnLpRPAAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAAOxAAADsQBlSsOGwAACdpJREFUWMPNWVtsXNUVXXufe+fOy/OyPc7TCXGABiUkIZDgQhCUFiSC+tMWqrZAkVrgB6kf/auqVlW/+tNSVWorKhQIoCBKQiXUp1TxVGl4hAbHCk4cYjsPe/yYp+dx7z1n98Mz4xnbSbAT1B5daaR7ztl7nbX23ufcMyQiWLIZAwDMuOrtkpbpooD+R+2iBOSf/U3hxd9/Hi4LB/+Qf+api/UuYkgERABOdhA52DwlrS+vqDWMnOohU8J1s0tbXsSQMQDKb/6NU6AIKu++DgBirgIzYgBU33+HQlCdKP/ztaa7SwJSCkDp0PMcDnMoVDr8PACwugqAWAEovfoCBRzuiBYPHWi6a23tkjUIPHNTmkNhEQPP2/DehSW5XSY99ekjX+yFMWQHTD638djMYsvtDM2x+t7benySIlGOxrxz49Wj7za7rggQ4J445o+OcUecwhE9la28+ffFltsBsQJQPPw8hRQxEysOqtIct1eoGhGA4qEDsEDKIiKKBIqHl7DcIlmT1T1rAeJYAgSTzwLY8O9zV0E1YPSOPqmUOdkJETNbknJp40fTCyy3MCQGQG3gQ3/sPMcSEANjOJbwz553Bz+6ItWMBuANn/BOn+Z4EsZAhKMxnZmpHnlzgeUWQMQAin/cTw5TI/hJWeRw4aVnmgNWpBfX9VIgy268YwpbxZf3L1CtKZkABGD0to3iuZxINXcck8tC8YZ/jV2hamN3b9G5GdWZniMMxKZUkFpl44eTrZYbizYCwP1kwBsZ4Vhyvl4Zw/GEP3a2odryNz5jAPhnR9yTJ1QiVUcDQAx3xPT4VPXIW62qNQDVs+A5skCW1ca3ssim4ivPNoetoM0lFNmBdiUVhazioedaVWsrjGN3Xa+LeZXqaqvozCY7ReFo7xunVoYGwNn7dvoXxlT36nmG5iwXchBpzWJusuqNDLsnh1Qzelo453jKOz3sjQw3By9LLz01URv4SCW72tDMWe5I+GfP144frQNqzbLSK/uJQEpBNMS0PJqUIqB06NnmNIhc/gHEGAClwwegQba9hGUmslF69UAzHuYlO33bVp2fVp3pJThg1tOTqrN70xv/WYFeZ/bd7g0PqVVroPXCPmKdn2G2+o5+CgACC2JALLNnu9YV1LY+RAJzGdc+DfDjVMvnfvFDtwxF3mdJNgK04UA8lO4ty6brYNtYPI0I1bgePyXZYUr2AaaRUJ8+Hbszi4+jcAtoz7I54GAg5vkfvDzzxrlAFJDLpxsRaiWzqn9NeLOBay09xdcwEdxdkU9/jeRTIKpLpl/YKjxGhV3yzhjiVI+tNgvEUqPEqtH3NPmzbFsQwcVQCUAkvq8p3LvbosKElgBIFo0R5DTt6ZWuY1SJqO+OoO64MinTx8lLYW2GvrCaCh6xEAmh9TEiTDrvxFmXa/Bd8T3xLvL4nviuP1tzYoqlaAyITLs1IRYquNTXgw1T5MYlNyq5oTogc/IgDGBFkZ/FthJ6ulFyF9dAIQW3Gu1mI9Zlv1UIMKIiaQtuRWhRDBCh7CLVhZ0uCgWoMAjyyQt1QDL0IoUigIYJoJzFrTYCMdQWYiIicU045nMoKP5lqpHxNTnBcMyFa2jB2ojgekAE/UFUp6AdiKZwQk4fAsDQNWQ+QKgbxoAA14ZcwK1x1ALQ/gKOjFGM2WBnUHtCl9hGiIwrwZRj8azWjAXZZTQqFvq7YF1A1QYBohHsxPRxVDIsJw+K8aBC9WlEKCskM7ixGwVZYEtIwatGupVZ7KZNL9GaI2kbflVILehD3qcbutGdQQngxqo4IERy8iX143sq5BZhBVvzCVUX64BqCpk8gi2HFSI2nhUNFSaIjEsX+RwWrcUKdm9WXC0JWgKICEUX61bjpgpyJZDdBlXZUh5nTH8MMSDVsmIBBZAvYIeHWArl1mASrVlZFaczrF2z9OZPZFwTTIUtq6w1zZslQtVFOIFdgsIMJNDmkRUEmB5gvvegzE6hcAbcilegA3Cn0B8GIvC8pm8hBbcc6WJjFC2l2pxe4S6GWxG06OX78IPoj8FMwHfayhLbKI5K/gJ/5QDTmr3WE1mEV0lmsL6OejABFQvB89idwiw3N2oiRs1EYh4FHOPrxYDm8iua9OHqFk0FJcLNXYiex6xqVNT6KVEygwh2Wo+do437GACchHp4mHc8IRNDcAtgu46JGUVg9Qxt7UXea1rRmiy76qRCZrFqRMYzTjJkWRXtt6wt7+HaHvRmUTCNs5iAbbhFmRjiG7+nHjmDyBo0jh8CgO/8rXrgH1K6gNzJhnwCspEryQ1FrFmLYq2+IFLwytG0rTUvUK2eX90KfkMvIpRcpNPYXmsEstRlyp2S0nn1jb/yl56eh9HIRQCg9V+2nhR07ZDxQYgGMWAgQRQnsMdCMIlaDUREjJofTvjkBKX9RCHakO1E4h5qmpjrNdCKYrdCeQomABgQQ7SMD6Jzq/Wkod57W0rCUvdD6oEjfMfPZXIY5QzYBhm4DmQM/UnUnLlqqTXbVtlJhtpyjUi7JhB3bKdqNAGA9lG20N8JawI1GyRgG5WMTA3z7T9RD36ARfvzYkACgG/+kXp0EDAyPQhSUIwSIX6edq5BwYCMQMEvR9KW9udVI4j2KdLjwK8aUSBBUWh7D1ITKAKKQUqmB2E89fAx3v3TprtLA2rIl9yivp/lTV+V8RPwK7CCyNekL4tN65GvkWLUdCShKeCINi16BSJJjZpPilGoYV23XF9EoQoVhF+V8RN0zT71WIE6ty1wdwlA7VTt+5O6/znJjaI4AhVBPoudFSS6UXa1ZtsuB5Ih7WoQgUh7xo45AbtsDKPiIt6JWwSFLCiC0qhkR9R9z6j7X1uSmM8CqJ60dP1D1hNZhHskcxzagTdXLcPiGehKNB2YU40gxqdI2oGpGs9AB7EnDH8KviOTxxFMWY9P0JZHG2hoBYBa+HQS6qFh3vUDyQwhN4vQJG5OomKh6oYTPtkB0UaMiLIjKYOai1kLe7oQmUK+LJkh3v64emQEofTFZPrsgNrl2/tL9c3Xxc3K6Amsy9OWtJ7WAaccSIa1Z4xv7I6gEyybaZ+2pLE2h7EhKY+rr/+F7/rdpWVq42AF99T6lb0y+jb1bcM7UTV7Lltdnz0+CUL8uq7OjnM6tAp7XTl1lNbsUg8cWe6dyXJvWASA+tpbfNev5MzHuHEKnIiEq7BsgYpEa+A4tmfl9FG+7WfqwfdXcIOzAoYaIZk7pf98C00YHrl2ZMA3vrlmu2OuGTZdrrrnLUrvbBv8uTGEeQeJzepbWbnlHqQHHPhBEqQHZEe/9Z1SAw2WiwYAZOXN1H9nDhYe7sl/Oy2Z/Qu7lt+u8M+XhiLlC9A+OtavTKY2/v/f/g36L/vYxfPvUMQ9AAAAJXRFWHRkYXRlOmNyZWF0ZQAyMDE2LTExLTE4VDEwOjU2OjA3LTA4OjAwebQiUgAAACV0RVh0ZGF0ZTptb2RpZnkAMjAxNi0xMS0xOFQxMDo1NjowNy0wODowMAjpmu4AAAAASUVORK5CYII=\n" 183 | }, 184 | } 185 | ``` 186 | Our 3rd version of *test-me* is base64 encoded and stored in Vault with **format=base64 value=\** by *vault-write.sh* automatically. 187 | 188 | We can use *vault kv get -field=value secret/test-me | base64 --decode > /tmp/gitlab1.png* to get the binary file back or use our next script **vault-read.sh**. 189 | 190 | ## Vault-read.sh 191 | 192 | ```console 193 | #!/bin/bash 194 | # Usage: vault-read 195 | 196 | SCRIPT_NAME=$(basename "$0") 197 | if [ "$#" != "1" ]; then 198 | echo usage "$SCRIPT_NAME: " 199 | exit 1 200 | else 201 | path=$1 202 | fi 203 | export VAULT_ADDR=${VAULT_ADDR:-https://vault.example.com} 204 | 205 | # Get the kv secret from path in json 206 | data=$(vault kv get -format=json $path) 207 | if [ "$?" != "0" ]; then 208 | exit 1 209 | fi 210 | 211 | f=$(echo "$data" | jq -r '.data.format//.data.data.format' 2> /dev/null) 212 | v=$(echo "$data" | jq -r '.data.value//.data.data.value') 213 | 214 | # if value format is base64, decode it 215 | if [ "base64" == "$f" ] 216 | then 217 | echo -n "$v" | base64 --decode 218 | else 219 | echo -n "$v" 220 | fi 221 | ``` 222 | 223 | This code reads from key-value store and if the *format* field is base64, it will run base64 decode to decode the content. Since the 3rd version of the secret 224 | is a png binary file, let's save it to a file instead of output to standard out. 225 | 226 | ```console 227 | $ vault-read.sh secret/test-me > /tmp/gitlab2.png 228 | ``` 229 | 230 | You can compare /tmp/gitlab.png (original file), /tmp/gitlab1.png and /tmp/gitlab2.png, they are all the same png binary format file. 231 | 232 | ## Cleanup 233 | 234 | To delete all versions created in this exercise: 235 | 236 | ```console 237 | $ vault kv metadata delete secret/test-me 238 | Success! Data deleted (if it existed) at: secret/metadata/test-me 239 | $ vault kv get secret/test-me 240 | No value found at secret/data/test-me 241 | ``` 242 | 243 | ## Summary 244 | 245 | We defined a convention so that all vault secret objects have two common fields: **format=[text|base64]** and **value=[strings|@file]**. 246 | 247 | Now we can get any secret's content by **value** field, and use the content of **format** field to decide if we need to run base64 decode. 248 | 249 | We also introduced simple [vault-read.sh](./vault-read.sh) and [vault-write.sh](./vault-write.sh) scripts which not only make it easy for operators and applications alike to store and get secrets from vault with auto-type detecting, base64-encoding and base64-decoding, but also help us to enforce the field naming convention. 250 | --------------------------------------------------------------------------------