├── LICENSE ├── Makefile ├── README.md └── bin ├── aws-secrets-get ├── aws-secrets-init-resources ├── aws-secrets-purge-resources ├── aws-secrets-run-in-env └── aws-secrets-send /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2017 PromptWorks, LLC 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a 4 | copy of this software and associated documentation files (the "Software"), 5 | to deal in the Software without restriction, including without limitation 6 | the rights to use, copy, modify, merge, publish, distribute, sublicense, 7 | and/or sell copies of the Software, and to permit persons to whom the 8 | Software is furnished to do so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included 11 | in all copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL 16 | THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR 17 | OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 18 | ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 19 | OTHER DEALINGS IN THE SOFTWARE. 20 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | install: 2 | install bin/* /usr/local/bin 3 | 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | aws-secrets 2 | =========== 3 | Manage secrets on AWS instances with KMS encryption, IAM roles and S3 storage. 4 | 5 | Synopsis 6 | ======== 7 | 8 | aws-secrets requires [aws-cli](https://aws.amazon.com/cli/) version 1.8 or later. 9 | 10 | Installation: 11 | ``` 12 | git clone -o github https://github.com/promptworks/aws-secrets 13 | cd aws-secrets 14 | make install 15 | # (or just copy `bin/*` to somewhere in your PATH) 16 | ``` 17 | 18 | Set up AWS resources for an application named quizzo: 19 | ``` 20 | aws-secrets-init-resources quizzo 21 | ``` 22 | 23 | Make some secrets, send them to the cloud and the AWS S3 bucket: 24 | ``` 25 | echo "SECRET=xyzzy" > quizzo-env 26 | aws-secrets-send quizzo quizzo-env 27 | ``` 28 | 29 | Each send overwrites the existing secrets in the store. 30 | 31 | Retrieve the secrets and print them to `STDOUT`: 32 | 33 | ``` 34 | aws-secrets-get quizzo 35 | ``` 36 | 37 | The last one can be run by: 38 | - users in the `quizzo-manage-secrets` group 39 | - programs on EC2 instances which have been started with the `quizzo-secrets` IAM profile 40 | 41 | To start an EC2 instance with the quizzo-secrets IAM profile from the CLI: 42 | 43 | `aws ec2 run-instances ...--iam-instance-profile Name=quizzo-secrets` 44 | 45 | To start an ECS cluster with the `quizzo` IAM profile, select `quizzo-secrets-instances` from the 46 | Container Instance IAM Role selection on the Create Cluster screen. Or you could also start an ECS task with the `quizzo` IAM role by selecting it in your Task Definition. 47 | 48 | Description 49 | =========== 50 | 51 | This repository contains a handful of scripts: 52 | 53 | - `aws-secrets-init-resources` 54 | - `aws-secrets-send` 55 | - `aws-secrets-get` 56 | - `aws-secrets-run-in-env` 57 | - `aws-secrets-purge-resources` 58 | 59 | They can be used to set up and maintain a file containing environment 60 | variables which can then be used by an application running on an Amazon EC2 61 | instance. They can also be used when running an application in a 62 | docker container on an EC2 instance. 63 | 64 | *`aws-secrets-init-resources`* creates the following AWS resources: 65 | 66 | - A customer master key (CMK). 67 | - An alias for the key. 68 | - An S3 bucket. 69 | - A few roles to be used by an instance profile: one for S3 access, one for decryption with the CMK. 70 | - A group with access policies to get/put to S3 and encrypt/decrypt with the CMK. 71 | 72 | *`aws-secrets-send`* takes an app name and a filename as input and uses 73 | the CMK to encrypt it, then sends it to an object in the S3 bucket. 74 | 75 | *`aws-secrets-get`* take an app name as input, and uses it to 76 | construct the name of the S3 bucket and object. It then retrieves 77 | and decrypts the file and prints it to stdout. 78 | 79 | If the file contains lines of the form: 80 | 81 | ``` 82 | X=yyyy 83 | ``` 84 | then exporting the output will put those 85 | variables into the current environment. i.e. 86 | 87 | ``` 88 | export `aws-secrets-get quizzo` 89 | ``` 90 | 91 | *`aws-secrets-run-in-env`* is a short script that does the above and 92 | then executes another program, with its arguments. 93 | 94 | *`aws-secrets-purge-resources`* removes the resources associated with this 95 | app which were created by `aws-secrets-init-resources`. 96 | 97 | Examples 98 | ======= 99 | To use this in a docker file, add a line like this: 100 | ``` 101 | CMD ["aws-secrets-run-in-env", "quizzo", "start-quizzo"] 102 | ``` 103 | where "quizzo" is the name of your app, and "start-quizzo" 104 | is the script that starts the app. 105 | 106 | Notes 107 | ====== 108 | 109 | - These scripts depend on having the AWS CLI installed. (See references below) 110 | 111 | - Changing AWS_DEFAULT_REGION (or the aws-cli configuration) will effect the region used for API calls. 112 | 113 | - Changing AWS_SECRETS_BUCKET_REGION will specify the region in which the S3 bucket is created. 114 | 115 | References 116 | ========== 117 | 118 | - https://www.promptworks.com/blog/handling-environment-secrets-in-docker-on-the-aws-container-service 119 | - http://docs.aws.amazon.com/cli/latest/userguide/installing.html 120 | - https://gist.github.com/themoxman/1d137b9a1729ba8722e4 121 | -------------------------------------------------------------------------------- /bin/aws-secrets-get: -------------------------------------------------------------------------------- 1 | #!/bin/sh -e 2 | 3 | # aws-secrets-get 4 | # Retrieve an encrypted secrets file from s3 and print it to stdout. 5 | 6 | die() { 7 | echo "$@" 8 | exit 9 | } 10 | app=$1 11 | [ -z "$app" ] && die "Missing app. Usage: $0 "; 12 | 13 | s3_key=$app-aws-secrets 14 | s3_bucket=$app-secrets 15 | kms_alias=$app-secrets 16 | 17 | tmp=`mktemp -d` 18 | 19 | aws s3api get-object --bucket $s3_bucket --key $s3_key $tmp/out > /tmp/errs 20 | 21 | aws kms decrypt --ciphertext-blob fileb://$tmp/out --output text --query Plaintext | base64 --decode 22 | 23 | rm -rf $tmp 24 | -------------------------------------------------------------------------------- /bin/aws-secrets-init-resources: -------------------------------------------------------------------------------- 1 | #!/bin/bash -e 2 | 3 | # aws-secrets-init-resources 4 | # 5 | # Initialize a set of AWS resources to manage secret keys for an application. 6 | # 7 | die() { 8 | echo "$@" 9 | exit 10 | } 11 | 12 | app=$1 13 | 14 | [ -z "$app" ] && die "Usage: $0 " 15 | 16 | [[ "$app" =~ ^[a-z0-9\-]+$ ]] || die "The application name should consist only of a-z, 0-9 or - charaters." 17 | 18 | which aws >/dev/null || die "aws-cli not found." 19 | 20 | major=`aws --version 2>&1 | awk '{print $1}' | sed 's|aws-cli/1.\([0-9]*\).*$|\1|'` 21 | [ ! -z "$major" ] && [ $major -lt 8 ] && die "aws-cli version needs to be >= 1.8." 22 | 23 | echo "Initializing resources for $app."; 24 | 25 | # KMS alias 26 | alias=$app-secrets 27 | found=`aws kms list-aliases --query "Aliases[?AliasName == 'alias/$alias'] | [0].TargetKeyId" --output text` 28 | if [ "$found" == "None" ]; then 29 | echo 'Making new key' 30 | key_id=`aws kms create-key --query 'KeyMetadata.KeyId' --output text` 31 | echo "Making new alias $alias" 32 | aws kms create-alias --target-key-id $key_id --alias-name "alias/$alias" 33 | else 34 | echo "Found existing alias $alias, key $found" 35 | key_id=$found 36 | fi 37 | key_arn=`aws kms list-keys --query "Keys[?KeyId == '$key_id' ] | [0].KeyArn" --output text` 38 | 39 | # S3 bucket 40 | bucket=$app-secrets 41 | found=`aws s3api list-buckets --query "Buckets[?Name == '$bucket'] | [0].Name" --output text` 42 | if [ "$found" == "None" ]; then 43 | echo "Creating bucket $bucket" 44 | if [ ! -z "$AWS_SECRETS_BUCKET_REGION" ]; then 45 | BUCKET_CONSTRAINT="--create-bucket-configuration LocationConstraint=$AWS_SECRETS_BUCKET_REGION" 46 | BUCKET_REGION="--region $AWS_SECRETS_BUCKET_REGION" 47 | fi 48 | made=`aws s3api create-bucket --bucket $bucket --acl private $BUCKET_REGION $BUCKET_CONSTRAINT --output text` 49 | else 50 | echo "Found existing bucket $bucket" 51 | fi 52 | 53 | # IAM Role for instance profile 54 | instancerole=$app-secrets-instances 55 | instance_access_policy=$app-s3-read-secrets 56 | instance_decrypt_policy=$app-s3-decrypt-secrets 57 | found=`aws iam list-roles --query "Roles[?RoleName == '$instancerole'] | [0].RoleId" --output text` 58 | if [ "$found" == "None" ]; then 59 | echo "Creating new role for instances: $instancerole" 60 | trustpolicy=`mktemp` 61 | cat <<-TRUSTPOLICY >>$trustpolicy 62 | { 63 | "Id": "key-$app-secrets-instance-trust-policy", 64 | "Version": "2012-10-17", 65 | "Statement": [ 66 | { 67 | "Effect": "Allow", 68 | "Principal": { 69 | "Service" : [ 70 | "ecs-tasks.amazonaws.com", 71 | "ec2.amazonaws.com" 72 | ] 73 | }, 74 | "Action": "sts:AssumeRole" 75 | } 76 | ] 77 | } 78 | TRUSTPOLICY 79 | response=`aws iam create-role --role-name $instancerole --assume-role-policy-document file://$trustpolicy` 80 | rm $trustpolicy 81 | accesspolicy=`mktemp` 82 | cat <<-ACCESSPOLICY >> $accesspolicy 83 | { 84 | "Id": "key-$app-secrets-instance-access-policy", 85 | "Version": "2012-10-17", 86 | "Statement": [ 87 | { 88 | "Effect" : "Allow", 89 | "Action" : "s3:GetObject", 90 | "Resource" : "arn:aws:s3:::$bucket/*" 91 | } 92 | ] 93 | } 94 | ACCESSPOLICY 95 | aws iam put-role-policy --role-name $instancerole --policy-name $instance_access_policy \ 96 | --policy-document file://$accesspolicy 97 | rm $accesspolicy 98 | decryptpolicy=`mktemp` 99 | cat <<-DECRYPTPOLICY >> $decryptpolicy 100 | { 101 | "Version": "2012-10-17", 102 | "Id": "key-$app-secrets-instance-decrypt-policy", 103 | "Statement": [ 104 | { 105 | "Effect": "Allow", 106 | "Action": "kms:Decrypt", 107 | "Resource": "$key_arn" 108 | } 109 | ] 110 | } 111 | DECRYPTPOLICY 112 | aws iam put-role-policy --role-name $instancerole --policy-name $instance_decrypt_policy \ 113 | --policy-document file://$decryptpolicy 114 | rm $decryptpolicy 115 | else 116 | echo "Found existing role $instancerole ($found)" 117 | fi 118 | 119 | # Instance profile for instances 120 | instanceprofile=$app-secrets 121 | found=`aws iam list-instance-profiles \ 122 | --query "InstanceProfiles[?InstanceProfileName == '$instanceprofile'] | [0].InstanceProfileId" --output text` 123 | if [ "$found" == "None" ]; then 124 | echo "Creating new instance profile $instanceprofile." 125 | response=`aws iam create-instance-profile --instance-profile-name $instanceprofile` 126 | aws iam add-role-to-instance-profile --instance-profile-name $instanceprofile --role-name $instancerole 127 | else 128 | echo "Found existing instance profile ($found)" 129 | fi 130 | 131 | # Group for managing secrets 132 | group=$app-manage-secrets 133 | group_policy=$app-secrets-s3-read-write 134 | group_encrypt_policy=$app-encrypt-secrets 135 | found=`aws iam list-groups --query "Groups[?GroupName == '$group'] | [0].GroupName" --output text` 136 | if [ "$found" == "None" ]; then 137 | echo "Creating group $group." 138 | response=`aws iam create-group --group-name $group` 139 | accesspolicy=`mktemp` 140 | cat <<-ACCESSPOLICY >> $accesspolicy 141 | { 142 | "Id": "key-$app-secrets-group-access-policy", 143 | "Version": "2012-10-17", 144 | "Statement": [ 145 | { 146 | "Effect" : "Allow", 147 | "Action" : [ 148 | "s3:GetObject", 149 | "s3:PutObject" 150 | ], 151 | "Resource" : "arn:aws:s3:::$bucket/*" 152 | } 153 | ] 154 | } 155 | ACCESSPOLICY 156 | aws iam put-group-policy --group-name $group --policy-name $group_policy \ 157 | --policy-document file://$accesspolicy 158 | encryptpolicy=`mktemp` 159 | cat <<-ENCRYPTPOLICY >> $encryptpolicy 160 | { 161 | "Version": "2012-10-17", 162 | "Id": "key-$app-secrets-encrypt-policy", 163 | "Statement": [ 164 | { 165 | "Effect": "Allow", 166 | "Action": [ "kms:Encrypt", "kms:Decrypt" ], 167 | "Resource": "$key_arn" 168 | } 169 | ] 170 | } 171 | ENCRYPTPOLICY 172 | aws iam put-group-policy --group-name $group --policy-name $group_encrypt_policy \ 173 | --policy-document file://$encryptpolicy 174 | rm $encryptpolicy 175 | 176 | else 177 | echo "Group $group exists." 178 | fi 179 | 180 | echo "Done!" 181 | echo 182 | echo 183 | echo "Now what?" 184 | echo 185 | echo "# List users, add them to the group $group:" 186 | echo aws iam list-users --query 'Users[0].UserName' 187 | echo "aws iam add-user-to-group --group-name $group --user-name some_user" 188 | echo "aws iam get-group --group-name $group" 189 | echo 190 | echo "# Create a file with an environment containing secret keys:" 191 | echo "echo 'foo=bar$RANDOM' > aws-secrets" 192 | echo 193 | echo "# Encrypt and send it to s3 :" 194 | echo "aws-secrets-send $app aws-secrets" 195 | echo 196 | echo "# Retrieve it:" 197 | echo "aws-secrets-get $app" 198 | echo 199 | echo "# Start an EC2 Instance" 200 | echo "## Start an instance that can access the secrets:" 201 | echo "aws ec2 run-instances --image-id ami-2d39803a --instance-type t2.nano --key-name some_aws_key \ 202 | --iam-instance-profile Name=$instanceprofile" 203 | echo 204 | echo "## View AWS credentials on the instance : " 205 | echo "ssh ...instance ip..." 206 | echo "curl -L http://169.254.169.254/latest/meta-data/iam/security-credentials/$instancerole" 207 | echo 208 | echo "## Copy aws-secrets-get to your instance and run it there to retrieve the secrets" 209 | echo "scp aws-secrets-send ..instance ip.." 210 | echo 211 | echo "# Start an ECS Cluster" 212 | echo "## Create your cluster using the $instancerole IAM role on the container instance" 213 | 214 | -------------------------------------------------------------------------------- /bin/aws-secrets-purge-resources: -------------------------------------------------------------------------------- 1 | #!/bin/sh -e 2 | 3 | die() { 4 | echo "$@" 5 | exit 6 | } 7 | 8 | app=$1 9 | 10 | [ -z "$app" ] && die "Usage: $0 " 11 | 12 | [[ "$app" =~ ^[a-z0-9\-]+$ ]] || die "The application name should consist only of a-z, 0-9 or - charaters." 13 | 14 | echo "Purging resources for $app."; 15 | 16 | 17 | alias=$app-secrets 18 | bucket=$app-secrets 19 | instancerole=$app-secrets-instances 20 | instance_access_policy=$app-s3-read-secrets 21 | instance_decrypt_policy=$app-s3-decrypt-secrets 22 | instanceprofile=$app-secrets 23 | group=$app-manage-secrets 24 | group_policy=$app-secrets-s3-read-write 25 | group_encrypt_policy=$app-encrypt-secrets 26 | s3_key=$app-aws-secrets 27 | 28 | echo "Deleting group policy" 29 | aws iam delete-group-policy --group-name $group --policy-name=$group_policy 30 | aws iam delete-group-policy --group-name $group --policy-name=$group_encrypt_policy 31 | 32 | users=`aws iam get-group --group-name $group --query 'Users[*].UserName' --output text` 33 | for u in $users; do 34 | echo "removing $u from group $group" 35 | aws iam remove-user-from-group --user-name $u --group-name $group 36 | done 37 | 38 | echo "Deleting group $group" 39 | aws iam delete-group --group-name $group 40 | 41 | echo "Deleting role policies" 42 | aws iam delete-role-policy --role-name $instancerole --policy-name=$instance_access_policy 43 | aws iam delete-role-policy --role-name $instancerole --policy-name=$instance_decrypt_policy 44 | 45 | echo "Removing roles from instance profile" 46 | aws iam remove-role-from-instance-profile --instance-profile-name $instanceprofile --role-name $instancerole 47 | 48 | echo "Deleting instance profile" 49 | aws iam delete-instance-profile --instance-profile-name $instanceprofile 50 | 51 | echo "Deleting role $instancerole" 52 | aws iam delete-role --role-name $instancerole 53 | 54 | echo "Deleting bucket" 55 | aws s3api delete-object --bucket $bucket --key $s3_key 56 | aws s3api delete-bucket --bucket $bucket 57 | 58 | echo "Scheduling key deletion" 59 | key_id=`aws kms list-aliases --query "Aliases[?AliasName == 'alias/$alias'] | [0].TargetKeyId" --output text` 60 | aws kms schedule-key-deletion --key-id $key_id --pending-window-in-days 7 61 | 62 | echo "Deleting Key alias" 63 | aws kms delete-alias --alias-name "alias/$alias" 64 | 65 | -------------------------------------------------------------------------------- /bin/aws-secrets-run-in-env: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | app=$1 4 | shift 5 | 6 | export `aws-secrets-get $app` 7 | 8 | exec "$@" 9 | -------------------------------------------------------------------------------- /bin/aws-secrets-send: -------------------------------------------------------------------------------- 1 | #!/bin/sh -e 2 | 3 | # aws-secrets-send 4 | # Encrypt a file using a KMS key alias, then send it to an s3 bucket. 5 | 6 | die() { 7 | echo "$@" 8 | exit 9 | } 10 | app=$1 11 | secrets_file=$2 12 | [ -z "$app" ] && die "Missing app. Usage: $0 "; 13 | [ -z "$secrets_file" ] && die "Missing filename. Usage: $0 "; 14 | 15 | src=$secrets_file 16 | s3_bucket=$app-secrets 17 | s3_key=$app-aws-secrets 18 | kms_alias=$app-secrets 19 | 20 | tmp=`mktemp -d` 21 | encrypted=$tmp/data.enc 22 | 23 | key_id=`aws kms list-aliases --output text --query "Aliases[?AliasName=='alias/$kms_alias'].TargetKeyId | [0]"` 24 | 25 | aws kms encrypt \ 26 | --key-id $key_id \ 27 | --plaintext fileb://$src \ 28 | --query CiphertextBlob \ 29 | --output text \ 30 | | base64 --decode \ 31 | > $encrypted 32 | 33 | aws s3api put-object \ 34 | --bucket $s3_bucket \ 35 | --key $s3_key \ 36 | --acl private \ 37 | --body $encrypted \ 38 | --output text \ 39 | --query 'None' \ 40 | | egrep -v '^None$' || true 41 | 42 | rm -rf $tmp 43 | --------------------------------------------------------------------------------