├── images └── bottlerocket-image-cache.png ├── CODE_OF_CONDUCT.md ├── .gitignore ├── LICENSE ├── CONTRIBUTING.md ├── ebs-snapshot-instance.yaml ├── README.md └── snapshot.sh /images/bottlerocket-image-cache.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/bottlerocket-images-cache/HEAD/images/bottlerocket-image-cache.png -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Created by https://www.toptal.com/developers/gitignore/api/visualstudiocode 2 | # Edit at https://www.toptal.com/developers/gitignore?templates=visualstudiocode 3 | 4 | ### VisualStudioCode ### 5 | .vscode/* 6 | !.vscode/tasks.json 7 | !.vscode/launch.json 8 | !.vscode/extensions.json 9 | !.vscode/*.code-snippets 10 | 11 | # Local History for Visual Studio Code 12 | .history/ 13 | 14 | # Built Visual Studio Code Extensions 15 | *.vsix 16 | 17 | ### VisualStudioCode Patch ### 18 | # Ignore all local history of files 19 | .history 20 | .ionide 21 | 22 | # End of https://www.toptal.com/developers/gitignore/api/visualstudiocode -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT No Attribution 2 | 3 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of 6 | this software and associated documentation files (the "Software"), to deal in 7 | the Software without restriction, including without limitation the rights to 8 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 9 | the Software, and to permit persons to whom the Software is furnished to do so. 10 | 11 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 12 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 13 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 14 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 15 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 16 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *main* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. -------------------------------------------------------------------------------- /ebs-snapshot-instance.yaml: -------------------------------------------------------------------------------- 1 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | AWSTemplateFormatVersion: 2010-09-09 4 | Description: Bottlerocket instance to snapshot data volume with configurable network settings. 5 | 6 | Parameters: 7 | AmiID: 8 | Type: AWS::SSM::Parameter::Value 9 | Description: "The ID of the AMI." 10 | Default: /aws/service/bottlerocket/aws-k8s-1.27/x86_64/latest/image_id 11 | InstanceType: 12 | Type: String 13 | Description: "EC2 instance type to launch" 14 | Default: m5.large 15 | InstanceRole: 16 | Type: String 17 | Description: "Name of IAM Role used in instance" 18 | Default: NONE 19 | Encrypt: 20 | Type: String 21 | Description: "Encrypt the EBS volumes" 22 | Default: NONE 23 | KMSId: 24 | Type: String 25 | Description: "Id of the KMS Key used for the snapshot" 26 | Default: NONE 27 | SnapshotSize: 28 | Type: Number 29 | Description: "Size of the target snapshot" 30 | Default: 50 31 | SecurityGroupId: 32 | Type: String 33 | Description: "Optional Security Group ID. If not provided, the default VPC security group will be used." 34 | Default: NONE 35 | SubnetId: 36 | Type: String 37 | Description: "Optional Subnet ID. If not provided, a subnet from the default VPC will be used." 38 | Default: NONE 39 | AssociatePublicIpAddress: 40 | Type: String 41 | Description: "Whether to associate a public IP address to the instance" 42 | Default: "true" 43 | AllowedValues: 44 | - "true" 45 | - "false" 46 | 47 | Conditions: 48 | CreateNewIAMRole: !Equals [!Ref InstanceRole, NONE] 49 | UseCustomKMSId: !Not [!Equals [!Ref KMSId, NONE]] 50 | Encrypt: !Not [!Equals [!Ref Encrypt, NONE]] 51 | UseCustomSecurityGroup: !Not [!Equals [!Ref SecurityGroupId, NONE]] 52 | UseCustomSubnet: !Not [!Equals [!Ref SubnetId, NONE]] 53 | 54 | Resources: 55 | BottlerocketNodeRole: 56 | Type: "AWS::IAM::Role" 57 | Condition: CreateNewIAMRole 58 | Properties: 59 | Path: / 60 | AssumeRolePolicyDocument: 61 | Version: "2012-10-17" 62 | Statement: 63 | - Effect: Allow 64 | Principal: 65 | Service: 66 | !Sub "ec2.${AWS::URLSuffix}" 67 | Action: 68 | - "sts:AssumeRole" 69 | ManagedPolicyArns: 70 | - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly" 71 | - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonSSMManagedInstanceCore" 72 | - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy" 73 | 74 | BottlerocketNodeInstanceProfile: 75 | Type: "AWS::IAM::InstanceProfile" 76 | Properties: 77 | Path: "/" 78 | Roles: 79 | - !If [CreateNewIAMRole, !Ref BottlerocketNodeRole, !Ref InstanceRole] 80 | 81 | BottlerocketLaunchTemplate: 82 | Type: AWS::EC2::LaunchTemplate 83 | Properties: 84 | LaunchTemplateData: 85 | ImageId: !Ref AmiID 86 | InstanceType: !Ref InstanceType 87 | IamInstanceProfile: 88 | Name: !Ref BottlerocketNodeInstanceProfile 89 | EbsOptimized: true 90 | UserData: 91 | Fn::Base64: | 92 | [settings.host-containers.admin] 93 | enabled = true 94 | BlockDeviceMappings: 95 | - DeviceName: /dev/xvda 96 | Ebs: 97 | VolumeSize: 10 98 | VolumeType: gp3 99 | DeleteOnTermination: true 100 | - DeviceName: /dev/xvdb 101 | Ebs: 102 | VolumeSize: !Ref SnapshotSize 103 | VolumeType: gp3 104 | Encrypted: 105 | Fn::If: 106 | - Encrypt 107 | - true 108 | - false 109 | KmsKeyId: 110 | Fn::If: 111 | - UseCustomKMSId 112 | - !Ref KMSId 113 | - !Ref AWS::NoValue 114 | Throughput: 1000 115 | Iops: 4000 116 | DeleteOnTermination: true 117 | NetworkInterfaces: 118 | - AssociatePublicIpAddress: !Ref AssociatePublicIpAddress 119 | DeviceIndex: "0" 120 | Groups: 121 | - !If [UseCustomSecurityGroup, !Ref SecurityGroupId, !Ref 'AWS::NoValue'] 122 | SubnetId: !If [UseCustomSubnet, !Ref SubnetId, !Ref 'AWS::NoValue'] 123 | 124 | BottlerocketInstance: 125 | Type: AWS::EC2::Instance 126 | Properties: 127 | LaunchTemplate: 128 | LaunchTemplateId: !Ref BottlerocketLaunchTemplate 129 | Version: !GetAtt BottlerocketLaunchTemplate.LatestVersionNumber 130 | 131 | Outputs: 132 | InstanceId: 133 | Value: !Ref BottlerocketInstance 134 | Description: Instance Id -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Caching Container Images for AWS Bottlerocket Instances 2 | The purpose of this sample is to reduce the boot time of containers with large images by caching the images in the data volume of Bottlerocket OS. 3 | 4 | Data analytics and machine learning workloads often require large container images (usually measured by Gigabytes), which can take several minutes to pull and extract from Amazon ECR or other image registry. Reduce image pulling time is the key of improving efficiency of launching these containers. 5 | 6 | [Bottlerocket OS](https://github.com/bottlerocket-os/bottlerocket) is a Linux-based open-source operating system built by AWS specifically for running containers. It has two volumes, an OS volume and a data volume, with the latter used for storing artifacts and container images. This sample will leverage the data volume to pull images and take snapshots for later usage. 7 | 8 | To demonstrate the process of caching images in EBS snapshots and launching them in an EKS cluster, this sample will use Amazon EKS optimized Bottlerocket AMIs. 9 | 10 | ## How this script works 11 | 12 | ![bottlerocket-image-cache drawio](images/bottlerocket-image-cache.png) 13 | 14 | 1. Launch an EC2 instance with Bottlerocket for EKS AMI, 15 | 2. Access to instance via Amazon System Manager 16 | 3. Pull images to be cached in this EC2 using Amazon System Manager Run Command. 17 | 4. Shut down the instance, build the EBS snapshot for the data volume. 18 | 5. Terminate the instance. 19 | 20 | ## Build EBS snapshot with cached container image 21 | 1. Set up [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) 22 | 2. Run the following command to clone this projects in your local environment. 23 | ```bash 24 | git clone https://github.com/aws-samples/bottlerocket-images-cache/ 25 | cd bottlerocket-images-cache/ 26 | ``` 27 | 28 | 3. Run `snapshot.sh` to build the EBS snapshot. Replace `us-west-2` to your region, and replace `public.ecr.aws/eks-distro/kubernetes/pause:3.2` to a comma seperated list of container images. 29 | ```bash 30 | ./snapshot.sh -r us-west-2 public.ecr.aws/eks-distro/kubernetes/pause:3.2 31 | ``` 32 | 33 | ## Command-line Parameters 34 | 35 | ```bash 36 | $ ./snapshot.sh -h 37 | usage: ./snapshot.sh [options] 38 | Build EBS snapshot for Bottlerocket data volume with cached container images 39 | Options: 40 | -h,--help Print this help. 41 | -r,--region Set AWS region to build the EBS snapshot. (default: use environment variable of AWS_DEFAULT_REGION or IMDS) 42 | -a,--ami Set SSM Parameter path for Bottlerocket ID. (default: /aws/service/bottlerocket/aws-k8s-1.27/x86_64/latest/image_id) 43 | -i,--instance-type Set EC2 instance type to build this snapshot. (default: m5.large) 44 | -e,--encrypt Encrypt the generated snapshot. (default: false) 45 | -k,--kms-id Use a specific KMS Key Id to encrypt this snapshot, should use together with -e 46 | -s,--snapshot-size Use a specific volume size (in GiB) for this snapshot. (default: 50) 47 | -R,--instance-role Name of existing IAM role for created EC2 instance. (default: Create on launching) 48 | -q,--quiet Redirect output to stderr and output generated snapshot ID to stdout only. (default: false) 49 | -sg,--security-group-id Set a specific Security Group ID for the instance. (default: use default VPC security group) 50 | -sn,--subnet-id Set a specific Subnet ID for the instance. (default: use default VPC subnet) 51 | -op,--output-parameter-name Set the SSM parameter name to store the generated snapshot ID. (default: NONE) 52 | -p,--public-ip Associate a public IP address with the instance. (default: true) 53 | ``` 54 | 55 | ## Required IAM Policy 56 | 57 | This script requires the following IAM policies: 58 | 59 | ``` 60 | "cloudformation:*", 61 | "iam:GetInstanceProfile", 62 | "iam:CreateInstanceProfile", 63 | "iam:AddRoleToInstanceProfile", 64 | "iam:DeleteInstanceProfile", 65 | "iam:RemoveRoleFromInstanceProfile", 66 | "iam:PassRole", 67 | "ec2:Describe*", 68 | "ec2:RunInstances", 69 | "ec2:TerminateInstances", 70 | "ec2:StopInstances", 71 | "ec2:StartInstances", 72 | "ec2:CreateVolume", 73 | "ec2:CreateTags", 74 | "ec2:AssociateIamInstanceProfile", 75 | "ec2:ReplaceIamInstanceProfileAssociation", 76 | "ec2:DisassociateIamInstanceProfile", 77 | "ec2:CreateLaunchTemplate", 78 | "ec2:CreateLaunchTemplateVersion", 79 | "ec2:ModifyLaunchTemplate", 80 | "ec2:DeleteLaunchTemplate", 81 | "ec2:CreateSnapshot", 82 | "ssm:CancelCommand", 83 | "ssm:SendCommand", 84 | "ssm:ListCommands", 85 | "ssm:DescribeInstanceInformation", 86 | "ssm:ListCommandInvocations", 87 | "ssm:GetCommandInvocation", 88 | "ssm:DescribeInstanceProperties", 89 | "ssm:GetParameters" 90 | ``` 91 | 92 | If you choose to encrypt the snapshot with KMS using the `--encrypt` and/or `--kms-id` option, the following IAM policies is required: 93 | 94 | ``` 95 | "kms:RetireGrant", 96 | "kms:CreateGrant", 97 | "kms:ReEncrypt*", 98 | "kms:GenerateDataKey*", 99 | "kms:Encrypt", 100 | "kms:DescribeKey", 101 | "kms:Decrypt" 102 | ``` 103 | 104 | If you let the script create required IAM role for you, the following IAM policies is required: 105 | 106 | ``` 107 | "iam:AttachRolePolicy", 108 | "iam:CreateRole" 109 | "iam:DeleteRole" 110 | "iam:DetachRolePolicy" 111 | "iam:ListRoles" 112 | "iam:ListRolePolicies" 113 | "iam:ListPolicies" 114 | ``` 115 | 116 | If you choose to write the snapshot ID to SSM parameter store, the following IAM policies is required: 117 | 118 | ``` 119 | "ssm:PutParameter" 120 | ``` 121 | 122 | ## Using snapshot with Amazon EKS 123 | 124 | There are 3 approaches to provision Amazon EC2 nodes for Amazon EKS cluster: 125 | * EKS Managed Node Group 126 | * Self managed nodes 127 | * EC2 Fleet managed by [Karpenter](https://karpenter.sh/) 128 | 129 | You can use EBS snapshot created by the script with nodes created by all the approaches. 130 | 131 | ### With Managed Node Group or Self managed nodes 132 | 133 | You can use a launch template to create volume from snapshot. When creating launch template, specify snapshot ID on volume with **device name** `/dev/xvdb` only. For detail, please refer to [Customizing managed nodes with launch templates](https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html) on Amazon EKS documentation. 134 | 135 | ### With Karpenter 136 | 137 | You can specify snapshot ID in a Karpenter node template. You should also specify AMI used when provisioning node is `BottleRocket`. Add the content on `EC2NodeClass` (or `AWSNodeTemplate` on older release of Karpenter): 138 | 139 | `v1beta1` API: 140 | ```yaml 141 | apiVersion: karpenter.k8s.aws/v1beta1 142 | kind: EC2NodeClass 143 | metadata: 144 | name: default 145 | spec: 146 | amiFamily: Bottlerocket # Ensure OS is BottleRocket 147 | blockDeviceMappings: 148 | - deviceName: /dev/xvdb 149 | ebs: 150 | volumeSize: 50Gi 151 | volumeType: gp3 152 | kmsKeyID: "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab" # Specify KMS ID if you use custom KMS key 153 | snapshotID: snap-0123456789 # Specify your snapshot ID here 154 | ``` 155 | 156 | `v1alpha1` API: 157 | ```yaml 158 | apiVersion: karpenter.k8s.aws/v1alpha1 159 | kind: AWSNodeTemplate 160 | spec: 161 | amiFamily: Bottlerocket # Ensure OS is BottleRocket 162 | blockDeviceMappings: 163 | - deviceName: /dev/xvdb # Make sure device name is /dev/xvdb 164 | ebs: 165 | volumeSize: 50Gi 166 | volumeType: gp3 167 | snapshotID: snap-0123456789 # Specify your snapshot ID here 168 | kmsKeyID: "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab" # Specify KMS ID if you use custom KMS key 169 | ``` 170 | 171 | ## Security 172 | 173 | See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information. 174 | 175 | ## License 176 | 177 | This library is licensed under the MIT-0 License. See the LICENSE file. 178 | -------------------------------------------------------------------------------- /snapshot.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 4 | # SPDX-License-Identifier: MIT-0 5 | 6 | set -e 7 | 8 | function print_help { 9 | echo "usage: $0 [options] " 10 | echo "Build EBS snapshot for Bottlerocket data volume with cached container images" 11 | echo "Options:" 12 | echo "-h,--help Print this help." 13 | echo "-A, --arch Set image architectures to pull (comma-separated). (default: amd64)" 14 | echo "-r,--region Set AWS region to build the EBS snapshot. (default: use environment variable of AWS_DEFAULT_REGION or IMDS)" 15 | echo "-a,--ami Set SSM Parameter path for Bottlerocket ID. (default: /aws/service/bottlerocket/aws-k8s-1.27/x86_64/latest/image_id)" 16 | echo "-i,--instance-type Set EC2 instance type to build this snapshot. (default: m5.large)" 17 | echo "-e,--encrypt Encrypt the generated snapshot. (default: false)" 18 | echo "-k,--kms-id Use a specific KMS Key Id to encrypt this snapshot, should use together with -e" 19 | echo "-s,--snapshot-size Use a specific volume size (in GiB) for this snapshot. (default: 50)" 20 | echo "-R,--instance-role Name of existing IAM role for created EC2 instance. (default: Create on launching)" 21 | echo "-q,--quiet Redirect output to stderr and output generated snapshot ID to stdout only. (default: false)" 22 | echo "-sg,--security-group-id Set a specific Security Group ID for the instance. (default: use default VPC security group)" 23 | echo "-sn,--subnet-id Set a specific Subnet ID for the instance. (default: use default VPC subnet)" 24 | echo "-op,--output-parameter-name Set the SSM parameter name to store the generated snapshot ID. (default: NONE)" 25 | echo "-p,--public-ip Associate a public IP address with the instance. (default: true)" 26 | } 27 | 28 | QUIET=false 29 | ASSOCIATE_PUBLIC_IP=true 30 | 31 | function log() { 32 | datestring=$(date +"%Y-%m-%d %H:%M:%S") 33 | if [ "$QUIET" = false ]; then 34 | echo -e "$datestring I - $*" 35 | else 36 | echo -e "$datestring I - $*" >&2 37 | fi 38 | } 39 | 40 | function logerror() { 41 | datestring=$(date +"%Y-%m-%d %H:%M:%S") 42 | echo -e "$datestring E - $*" >&2 43 | } 44 | 45 | function cleanup() { 46 | log "Cleaning up stack $1..." 47 | if aws cloudformation describe-stacks --stack-name "$1" &> /dev/null; then 48 | aws cloudformation delete-stack --stack-name "$1" 49 | log "Stack deletion initiated." 50 | else 51 | log "Stack $1 not found or already deleted." 52 | fi 53 | } 54 | 55 | while [[ $# -gt 0 ]]; do 56 | key="$1" 57 | case $key in 58 | -h|--help) 59 | print_help 60 | exit 1 61 | ;; 62 | -r|--region) 63 | AWS_DEFAULT_REGION="$2" 64 | shift 65 | shift 66 | ;; 67 | -a|--ami) 68 | AMI_ID="$2" 69 | shift 70 | shift 71 | ;; 72 | -i|--instance-type) 73 | INSTANCE_TYPE="$2" 74 | shift 75 | shift 76 | ;; 77 | -e|--encrypt) 78 | ENCRYPT=true 79 | shift 80 | ;; 81 | -k|--kms-id) 82 | if [ -n "$ENCRYPT" ] && [[ $ENCRYPT == true ]]; then 83 | KMS_ID="$2" 84 | else 85 | logerror "KMS Key should only be specified when snapshot is encrypted. (-e)" 86 | exit 2 87 | fi 88 | shift 89 | shift 90 | ;; 91 | -s|--snapshot-size) 92 | SNAPSHOT_SIZE="$2" 93 | shift 94 | shift 95 | ;; 96 | -R|--instance-role) 97 | INSTANCE_ROLE="$2" 98 | shift 99 | shift 100 | ;; 101 | -q|--quiet) 102 | QUIET=true 103 | shift 104 | ;; 105 | -sg|--security-group-id) 106 | SECURITY_GROUP_ID="$2" 107 | shift 108 | shift 109 | ;; 110 | -sn|--subnet-id) 111 | SUBNET_ID="$2" 112 | shift 113 | shift 114 | ;; 115 | -p|--public-ip) 116 | ASSOCIATE_PUBLIC_IP="$2" 117 | shift 118 | shift 119 | ;; 120 | -op|--output-parameter-name) 121 | OUTPUT_PARAMETER_NAME="$2" 122 | shift 123 | shift 124 | ;; 125 | -A|--arch) 126 | ARCHITECTURES="$2" 127 | shift 128 | shift 129 | ;; 130 | *) 131 | POSITIONAL+=("$1") # save it in an array for later 132 | shift # past argument 133 | ;; 134 | esac 135 | done 136 | 137 | set +u 138 | set -- "${POSITIONAL[@]}" # restore positional parameters 139 | IMAGES="$1" 140 | set -u 141 | 142 | AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-$(aws ec2 describe-availability-zones --output text --query 'AvailabilityZones[0].[RegionName]')} 143 | AMI_ID=${AMI_ID:-/aws/service/bottlerocket/aws-k8s-1.27/x86_64/latest/image_id} 144 | INSTANCE_TYPE=${INSTANCE_TYPE:-m5.large} 145 | INSTANCE_ROLE=${INSTANCE_ROLE:-NONE} 146 | ENCRYPT=${ENCRYPT:-NONE} 147 | KMS_ID=${KMS_ID:-NONE} 148 | SNAPSHOT_SIZE=${SNAPSHOT_SIZE:-50} 149 | SECURITY_GROUP_ID=${SECURITY_GROUP_ID:-NONE} 150 | SUBNET_ID=${SUBNET_ID:-NONE} 151 | ASSOCIATE_PUBLIC_IP=${ASSOCIATE_PUBLIC_IP:-true} 152 | OUTPUT_PARAMETER_NAME=${OUTPUT_PARAMETER_NAME:-NONE} 153 | ARCHITECTURES=${ARCHITECTURES:-amd64} 154 | SCRIPTPATH=$(dirname "$0") 155 | CTR_CMD="apiclient exec admin sheltie ctr -a /run/containerd/containerd.sock -n k8s.io" 156 | 157 | if [ -z "${AWS_DEFAULT_REGION}" ]; then 158 | logerror "Please set AWS region" 159 | exit 1 160 | fi 161 | 162 | if [ -z "${IMAGES}" ]; then 163 | logerror "Please set images list" 164 | exit 1 165 | fi 166 | 167 | # Validate image names to prevent injection 168 | for img in $(echo "$IMAGES" | tr ',' ' '); do 169 | if [[ ! "$img" =~ ^[a-zA-Z0-9._/-]+:[a-zA-Z0-9._-]+$ ]] && [[ ! "$img" =~ ^[a-zA-Z0-9._/-]+(:[a-zA-Z0-9._-]+)?@sha256:[a-f0-9]{64}$ ]]; then 170 | logerror "Invalid image format: $img" 171 | exit 1 172 | fi 173 | done 174 | 175 | # Validate architectures 176 | for arch in $(echo "$ARCHITECTURES" | tr ',' ' '); do 177 | if [[ ! "$arch" =~ ^(amd64|arm64|386|arm)$ ]]; then 178 | logerror "Invalid architecture: $arch" 179 | exit 1 180 | fi 181 | done 182 | 183 | # Use read -a to create arrays from comma-separated strings 184 | IFS=',' read -r -a IMAGES_LIST <<< "$IMAGES" 185 | IFS=',' read -r -a ARCH_LIST <<< "$ARCHITECTURES" 186 | # Validate AWS CLI is available and configured 187 | if ! command -v aws &> /dev/null; then 188 | logerror "AWS CLI is not installed or not in PATH" 189 | exit 1 190 | fi 191 | 192 | # Test AWS credentials 193 | if ! aws sts get-caller-identity &> /dev/null; then 194 | logerror "AWS credentials not configured or invalid" 195 | exit 1 196 | fi 197 | 198 | export AWS_DEFAULT_REGION="${AWS_DEFAULT_REGION}" 199 | 200 | ############################################################################################## 201 | export AWS_PAGER="" 202 | 203 | # launch EC2 204 | RAND=$(od -An -N2 -i /dev/urandom | tr -d ' ' | cut -c1-4) 205 | CFN_STACK_NAME="Bottlerocket-ebs-snapshot-$RAND" 206 | log "[1/8] Deploying EC2 CFN stack $CFN_STACK_NAME ..." 207 | CFN_PARAMS="AmiID=$AMI_ID InstanceType=$INSTANCE_TYPE InstanceRole=$INSTANCE_ROLE Encrypt=$ENCRYPT KMSId=$KMS_ID SnapshotSize=$SNAPSHOT_SIZE SecurityGroupId=$SECURITY_GROUP_ID SubnetId=$SUBNET_ID AssociatePublicIpAddress=$ASSOCIATE_PUBLIC_IP" 208 | 209 | # log $CFN_PARAMS 210 | 211 | if ! aws cloudformation deploy \ 212 | --stack-name "$CFN_STACK_NAME" \ 213 | --template-file "$SCRIPTPATH/ebs-snapshot-instance.yaml" \ 214 | --capabilities CAPABILITY_NAMED_IAM \ 215 | --parameter-overrides $CFN_PARAMS > /dev/null; then 216 | logerror "Failed to deploy CloudFormation stack" 217 | exit 1 218 | fi 219 | 220 | INSTANCE_ID=$(aws cloudformation describe-stacks --stack-name "$CFN_STACK_NAME" --query "Stacks[0].Outputs[?OutputKey=='InstanceId'].OutputValue" --output text) 221 | 222 | # wait for SSM ready 223 | log "[2/8] Launching SSM ." 224 | while [[ $(aws ssm describe-instance-information --filters "Key=InstanceIds,Values=$INSTANCE_ID" --query "InstanceInformationList[0].PingStatus" --output text) != "Online" ]] 225 | do 226 | sleep 5 227 | done 228 | log "SSM launched in instance $INSTANCE_ID." 229 | 230 | # stop kubelet.service 231 | log "[3/8] Stopping kubelet.service .." 232 | CMDID=$(aws ssm send-command --instance-ids "$INSTANCE_ID" \ 233 | --document-name "AWS-RunShellScript" --comment "Stop kubelet" \ 234 | --parameters commands="apiclient exec admin sheltie systemctl stop kubelet" \ 235 | --query "Command.CommandId" --output text) 236 | aws ssm wait command-executed --command-id "$CMDID" --instance-id "$INSTANCE_ID" > /dev/null 237 | log "Kubelet service stopped." 238 | 239 | # cleanup existing images 240 | log "[4/8] Cleanup existing images .." 241 | CMDID=$(aws ssm send-command --instance-ids "$INSTANCE_ID" \ 242 | --document-name "AWS-RunShellScript" --comment "Cleanup existing images" \ 243 | --parameters commands="$CTR_CMD images rm \$($CTR_CMD images ls -q)" \ 244 | --query "Command.CommandId" --output text) 245 | aws ssm wait command-executed --command-id "$CMDID" --instance-id "$INSTANCE_ID" > /dev/null 246 | log "Existing images cleaned" 247 | 248 | # pull images 249 | log "[5/8] Pulling images:" 250 | for IMG in "${IMAGES_LIST[@]}" 251 | do 252 | ECR_REGION=$(echo "$IMG" | sed -n "s/^[0-9]*\.dkr\.ecr\.\([a-z1-9-]*\)\.amazonaws\.com.*$/\1/p") 253 | [ -n "$ECR_REGION" ] && ECRPWD="--u AWS:$(aws ecr get-login-password --region "$ECR_REGION")" || ECRPWD="" 254 | for PLATFORM in "${ARCH_LIST[@]}" 255 | do 256 | log "Pulling $IMG - $PLATFORM ... " 257 | COMMAND="$CTR_CMD images pull --label io.cri-containerd.image=managed --platform $PLATFORM $ECRPWD $IMG" 258 | CMDID=$(aws ssm send-command --instance-ids "$INSTANCE_ID" \ 259 | --document-name "AWS-RunShellScript" --comment "Pull Image ${IMG:0:75} - $PLATFORM" \ 260 | --parameters commands="$COMMAND" \ 261 | --query "Command.CommandId" --output text) 262 | # Wait with timeout to prevent infinite loops 263 | WAIT_COUNT=0 264 | MAX_WAIT=60 # 5 minutes max wait 265 | until aws ssm wait command-executed --command-id "$CMDID" --instance-id "$INSTANCE_ID" &> /dev/null && log "$IMG - $PLATFORM pulled. " 266 | do 267 | sleep 5 268 | WAIT_COUNT=$((WAIT_COUNT + 1)) 269 | if [ $WAIT_COUNT -gt $MAX_WAIT ]; then 270 | logerror "Timeout waiting for image $IMG to pull" 271 | cleanup "$CFN_STACK_NAME" 272 | exit 1 273 | fi 274 | if [ "$(aws ssm get-command-invocation --command-id "$CMDID" --instance-id "$INSTANCE_ID" --output text --query Status)" == "Failed" ]; then 275 | REASON=$(aws ssm get-command-invocation --command-id "$CMDID" --instance-id "$INSTANCE_ID" --output text --query StandardOutputContent) 276 | logerror "Image $IMG pulling failed with following output: " 277 | logerror "$REASON" 278 | cleanup "$CFN_STACK_NAME" 279 | exit 1 280 | fi 281 | done 282 | done 283 | done 284 | 285 | # stop EC2 286 | log "[6/8] Stopping instance ... " 287 | aws ec2 stop-instances --instance-ids "$INSTANCE_ID" --output text > /dev/null 288 | aws ec2 wait instance-stopped --instance-ids "$INSTANCE_ID" > /dev/null && log "Instance $INSTANCE_ID stopped" 289 | 290 | # create EBS snapshot 291 | log "[7/8] Creating snapshot ... " 292 | DATA_VOLUME_ID=$(aws ec2 describe-instances --instance-id "$INSTANCE_ID" --query "Reservations[0].Instances[0].BlockDeviceMappings[?DeviceName=='/dev/xvdb'].Ebs.VolumeId" --output text) 293 | SNAPSHOT_ID=$(aws ec2 create-snapshot --volume-id "$DATA_VOLUME_ID" --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Name,Value=Bottlerocket Data Volume}]' --description "Bottlerocket Data Volume snapshot with ${IMAGES:0:200}" --query "SnapshotId" --output text) 294 | until aws ec2 wait snapshot-completed --snapshot-ids "$SNAPSHOT_ID" &> /dev/null && log "Snapshot $SNAPSHOT_ID generated." 295 | do 296 | sleep 5 297 | done 298 | 299 | # destroy temporary instance 300 | log "[8/8] Cleanup." 301 | cleanup "$CFN_STACK_NAME" 302 | 303 | # write snapshot-id to parameter store 304 | if [ "$OUTPUT_PARAMETER_NAME" != "NONE" ]; then 305 | log "Updating SSM parameter $OUTPUT_PARAMETER_NAME" 306 | aws ssm put-parameter --name "$OUTPUT_PARAMETER_NAME" --value "$SNAPSHOT_ID" --type String --overwrite 307 | fi 308 | 309 | # done! 310 | log "--------------------------------------------------" 311 | log "All done! Created snapshot in $AWS_DEFAULT_REGION: $SNAPSHOT_ID" 312 | if [ $QUIET = true ]; then 313 | echo "$SNAPSHOT_ID" 314 | fi 315 | --------------------------------------------------------------------------------