├── images ├── 01.png ├── 02.png ├── 03.png ├── 04.png ├── 05.png ├── 06.png ├── 07.png ├── 11.png ├── 12.png └── amazon-eks-serverless-drainer.png ├── .gitignore ├── .github └── PULL_REQUEST_TEMPLATE.md ├── libs.sh ├── CODE_OF_CONDUCT.md ├── bootstrap ├── LICENSE ├── sam-sar.yaml ├── main.sh ├── Makefile ├── CONTRIBUTING.md ├── sam.yaml └── README.md /images/01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/amazon-eks-serverless-drainer/master/images/01.png -------------------------------------------------------------------------------- /images/02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/amazon-eks-serverless-drainer/master/images/02.png -------------------------------------------------------------------------------- /images/03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/amazon-eks-serverless-drainer/master/images/03.png -------------------------------------------------------------------------------- /images/04.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/amazon-eks-serverless-drainer/master/images/04.png -------------------------------------------------------------------------------- /images/05.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/amazon-eks-serverless-drainer/master/images/05.png -------------------------------------------------------------------------------- /images/06.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/amazon-eks-serverless-drainer/master/images/06.png -------------------------------------------------------------------------------- /images/07.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/amazon-eks-serverless-drainer/master/images/07.png -------------------------------------------------------------------------------- /images/11.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/amazon-eks-serverless-drainer/master/images/11.png -------------------------------------------------------------------------------- /images/12.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/amazon-eks-serverless-drainer/master/images/12.png -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | main 2 | main.zip 3 | vendor/ 4 | sam-packaged.yaml 5 | .DS_Store 6 | packaged.yaml 7 | custom.mk 8 | func.d/ 9 | -------------------------------------------------------------------------------- /images/amazon-eks-serverless-drainer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/amazon-eks-serverless-drainer/master/images/amazon-eks-serverless-drainer.png -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. 7 | -------------------------------------------------------------------------------- /libs.sh: -------------------------------------------------------------------------------- 1 | export KUBECONFIG=/tmp/kubeconfig 2 | 3 | update_kubeconfig(){ 4 | aws eks update-kubeconfig --name "$1" --kubeconfig /tmp/kubeconfig 5 | } 6 | 7 | get_nodes(){ 8 | kubectl get no 9 | } 10 | 11 | get_pods(){ 12 | kubectl get po 13 | } 14 | 15 | get_all(){ 16 | kubectl get all 17 | } 18 | 19 | 20 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /bootstrap: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | set -euo pipefail 4 | 5 | # Initialization - load function handler 6 | #source $LAMBDA_TASK_ROOT/"$(echo $_HANDLER | cut -d. -f1).sh" 7 | 8 | export PATH=$PATH:/opt/awscli:/opt/kubectl 9 | 10 | 11 | # Processing 12 | while true 13 | do 14 | HEADERS="$(mktemp)" 15 | # Get an event 16 | EVENT_DATA=$(curl -sS -LD "$HEADERS" -X GET "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next") 17 | REQUEST_ID=$(grep -Fi Lambda-Runtime-Aws-Request-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2) 18 | 19 | # Execute the handler function from the script 20 | RESPONSE=$(./$(echo "$_HANDLER" | cut -d. -f2).sh "$EVENT_DATA") 21 | 22 | 23 | echo "=========[RESPONSE]=======" 24 | echo "$RESPONSE" 25 | echo "=========[/RESPONSE]=======" 26 | 27 | # Send the response 28 | curl -s -X POST "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$REQUEST_ID/response" -d "$RESPONSE" 29 | done 30 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | -------------------------------------------------------------------------------- /sam-sar.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | 3 | Metadata: 4 | AWS::ServerlessRepo::Application: 5 | Name: eks-lambda-drainer 6 | Description: "Lambda handler for node draining on Amazon EKS spot instances" 7 | Author: Pahud Hsieh 8 | SpdxLicenseId: Apache-2.0 9 | LicenseUrl: LICENSE 10 | ReadmeUrl: README.md 11 | Labels: ['eks','lambda','drain','taint', 'autoscaling', 'lifecyclehook'] 12 | HomePageUrl: https://github.com/pahud/eks-lambda-drainer 13 | SemanticVersion: 1.0.1-beta 14 | SourceCodeUrl: https://github.com/pahud/eks-lambda-drainer 15 | 16 | Parameters: 17 | ClusterName: 18 | Type: String 19 | Default: default 20 | FunctionName: 21 | Type: String 22 | Default: defaultFunc 23 | FunctionRoleArn: 24 | Type: String 25 | Default: "" 26 | 27 | Transform: AWS::Serverless-2016-10-31 28 | Description: eks-lambda-drainer Func 29 | Resources: 30 | Func: 31 | Type: AWS::Serverless::Application 32 | Properties: 33 | Location: 34 | # serverless app from all regoins should be able to import this ApplicationId from 'us-east-1' across accounts. 35 | ApplicationId: arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer 36 | SemanticVersion: 1.0.1-beta 37 | Parameters: 38 | ClusterName: !Ref ClusterName 39 | FunctionName: !Ref FunctionName 40 | FunctionRoleArn: !Ref FunctionRoleArn 41 | 42 | Outputs: 43 | Func: 44 | Description: "Lambda function Arn" 45 | Value: !GetAtt Func.Outputs.FuncArn 46 | FuncIamRole: 47 | Description: "Lambda function IAM role Arn" 48 | Value: !GetAtt Func.Outputs.FuncIamRole 49 | 50 | -------------------------------------------------------------------------------- /main.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # set -euo pipefail 3 | # include the common-used shortcuts 4 | source libs.sh 5 | 6 | echo $1 7 | 8 | taintNode(){ 9 | kubectl taint nodes "$1" SpotTerminating=true:NoExecute 10 | } 11 | 12 | drainNode(){ 13 | kubectl drain "$1" --ignore-daemonsets --delete-local-data 14 | } 15 | 16 | getNodeNameByInstanceId(){ 17 | aws ec2 describe-instances --instance-id "$1" --query 'Reservations[0].Instances[0].NetworkInterfaces[0].PrivateDnsName' --output text 18 | } 19 | 20 | getClusterNameFromTags(){ 21 | x=$(aws ec2 describe-tags --filters Name=resource-id,Values="$1" Name=value,Values=owned Name=resource-type,Values=instance --query "Tags[0].Key" --output text) 22 | echo ${x##*/} 23 | } 24 | 25 | update_kubeconfig(){ 26 | aws eks update-kubeconfig --name "$1" --kubeconfig /tmp/kubeconfig 27 | } 28 | 29 | 30 | detailType=$(echo $1 | jq -r '.["detail-type"] | select(type == "string")') 31 | instanceId=$(echo $1 | jq -r '.detail.EC2InstanceId | select(type == "string")') 32 | autoScalingGroupName=$(echo $1 | jq -r '.detail.AutoScalingGroupName | select(type == "string")') 33 | lifecycleActionToken=$(echo $1 | jq -r '.detail.LifecycleActionToken | select(type == "string")') 34 | lifecycleHookName=$(echo $1 | jq -r '.detail.LifecycleHookName | select(type == "string")') 35 | lifecycleTransition=$(echo $1 | jq -r '.detail.LifecycleTransition | select(type == "string")') 36 | 37 | # always get the cluster_name from EC2 Tag 38 | input_cluster_name=$(getClusterNameFromTags $instanceId) 39 | 40 | # always update kubeconfig 41 | update_kubeconfig "$cluster_name" 42 | 43 | # drain the node immediately 44 | echo "[INFO] start the node draining now" 45 | nodeName=$(getNodeNameByInstanceId $instanceId) 46 | if [ "${drain_type}" == "taint" ]; then 47 | echo "[INFO] start taint ${nodeName}" 48 | taintNode "$nodeName" 49 | else 50 | echo "[INFO] start drain ${nodeName}" 51 | drainNode "$nodeName" 52 | fi 53 | echo "[INFO] sleep a while before we callback the hook so the pods have enough time for resheduling" 54 | sleep 10 55 | echo "[INFO] OK. let's kubectl descirbe node/${nodeName}" 56 | kubectl describe node/${nodeName} 57 | 58 | 59 | if [ "$detailType"=="EC2 Instance-terminate Lifecycle Action" ]; then 60 | echo "start autoscaling group complete-lifecycle-actiopn callback" 61 | aws autoscaling complete-lifecycle-action \ 62 | --lifecycle-hook-name $lifecycleHookName \ 63 | --auto-scaling-group-name $autoScalingGroupName \ 64 | --instance-id $instanceId \ 65 | --lifecycle-action-token $lifecycleActionToken \ 66 | --lifecycle-action-result "CONTINUE" 67 | fi 68 | 69 | exit 0 -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | SECRETS_FILE ?= secrets.mk 2 | ifeq ($(shell test -e $(SECRETS_FILE) && echo -n yes),yes) 3 | include $(SECRETS_FILE) 4 | endif 5 | CUSTOM_FILE ?= custom.mk 6 | ifeq ($(shell test -e $(CUSTOM_FILE) && echo -n yes),yes) 7 | include $(CUSTOM_FILE) 8 | endif 9 | ROOT ?= $(shell pwd) 10 | AWS_ACCOUNT_ID := $(shell aws sts get-caller-identity --query 'Account' --output text) 11 | # modify this as your own S3 temp bucket. Make sure your locak IAM user have read/write access 12 | S3BUCKET ?= pahud-tmp-ap-northeast-1 13 | LAMBDA_REGION ?= ap-northeast-1 14 | STACKNAME ?= eks-lambda-drainer2 15 | LAMBDA_FUNC_NAME ?= $(STACKNAME) 16 | 17 | # Your Amazon EKS cluster name 18 | CLUSTER_NAME ?= eksdemo 19 | 20 | 21 | .PHONY: all 22 | all: func-prep sam-package sam-deploy 23 | 24 | .PHONY: func-prep 25 | func-prep: 26 | @rm -rf ./func.d; mkdir ./func.d 27 | @cp main.sh bootstrap libs.sh func.d/ && chmod +x ./func.d/bootstrap ./func.d/main.sh 28 | 29 | .PHONY: sam-package 30 | sam-package: 31 | @docker run -ti \ 32 | -v $(PWD):/home/samcli/workdir \ 33 | -v $(HOME)/.aws:/home/samcli/.aws \ 34 | -w /home/samcli/workdir \ 35 | -e AWS_DEFAULT_REGION=$(LAMBDA_REGION) \ 36 | pahud/aws-sam-cli:latest sam package --template-file sam.yaml --s3-bucket $(S3BUCKET) --output-template-file packaged.yaml 37 | 38 | .PHONY: sam-sar-package 39 | sam-sar-package: 40 | @docker run -ti \ 41 | -v $(PWD):/home/samcli/workdir \ 42 | -v $(HOME)/.aws:/home/samcli/.aws \ 43 | -w /home/samcli/workdir \ 44 | -e AWS_DEFAULT_REGION=$(LAMBDA_REGION) \ 45 | pahud/aws-sam-cli:latest sam package --template-file sam-sar.yaml --s3-bucket $(S3BUCKET) --output-template-file packaged.yaml 46 | 47 | 48 | .PHONY: sam-package-from-sar 49 | sam-package-from-sar: sam-sar-package 50 | 51 | 52 | .PHONY: sam-publish 53 | sam-publish: 54 | @docker run -ti \ 55 | -v $(PWD):/home/samcli/workdir \ 56 | -v $(HOME)/.aws:/home/samcli/.aws \ 57 | -w /home/samcli/workdir \ 58 | -e AWS_DEFAULT_REGION=$(LAMBDA_REGION) \ 59 | pahud/aws-sam-cli:latest sam publish --region $(LAMBDA_REGION) --template packaged.yaml 60 | 61 | 62 | .PHONY: sam-deploy 63 | sam-deploy: 64 | @aws --region $(LAMBDA_REGION) cloudformation deploy \ 65 | --parameter-overrides FunctionName=$(LAMBDA_FUNC_NAME) ClusterName=$(CLUSTER_NAME) FunctionRoleArn=$(FunctionRoleArn) \ 66 | --template-file ./packaged.yaml --stack-name "$(LAMBDA_FUNC_NAME)" --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM 67 | # print the cloudformation stack outputs 68 | @aws --region $(LAMBDA_REGION) cloudformation describe-stacks --stack-name "$(LAMBDA_FUNC_NAME)" --query 'Stacks[0].Outputs' 69 | 70 | .PHONY: sam-logs-tail 71 | sam-logs-tail: 72 | sam logs --name $(LAMBDA_FUNC_NAME) --tail 73 | # @docker run -ti \ 74 | # -v $(PWD):/home/samcli/workdir \ 75 | # -v $(HOME)/.aws:/home/samcli/.aws \ 76 | # -w /home/samcli/workdir \ 77 | # -e AWS_DEFAULT_REGION=$(LAMBDA_REGION) \ 78 | # -e AWS_REGION=$(LAMBDA_REGION) \ 79 | # pahud/aws-sam-cli:latest sam logs --name eks-lambda-drainer --tail --debug 80 | 81 | .PHONY: sam-destroy 82 | sam-destroy: 83 | # destroy the stack now 84 | @aws --region $(LAMBDA_REGION) cloudformation delete-stack --stack-name "$(LAMBDA_FUNC_NAME)" 85 | # deleting the stack. check your cloudformaion console to make sure stack is completely deleted 86 | 87 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check [existing open](https://github.com/awslabs/amazon-eks-serverless-drainer/issues), or [recently closed](https://github.com/awslabs/amazon-eks-serverless-drainer/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/awslabs/amazon-eks-serverless-drainer/labels/help%20wanted) issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](https://github.com/awslabs/amazon-eks-serverless-drainer/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /sam.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | 3 | Metadata: 4 | AWS::ServerlessRepo::Application: 5 | Name: eks-lambda-drainer 6 | Description: "Lambda handler for node draining on Amazon EKS spot instances" 7 | Author: Pahud Hsieh 8 | SpdxLicenseId: Apache-2.0 9 | LicenseUrl: LICENSE 10 | ReadmeUrl: README.md 11 | Labels: ['eks','lambda','drain','taint', 'autoscaling', 'lifecyclehook'] 12 | HomePageUrl: https://github.com/pahud/eks-lambda-drainer 13 | SemanticVersion: 1.0.1-beta 14 | SourceCodeUrl: https://github.com/pahud/eks-lambda-drainer 15 | 16 | Mappings: 17 | LayerArn: 18 | ap-northeast-1: 19 | kubectl: 'arn:aws:lambda:ap-northeast-1:903779448426:layer:eks-kubectl-layer:30' 20 | ap-northeast-2: 21 | kubectl: 'arn:aws:lambda:ap-northeast-2:903779448426:layer:eks-kubectl-layer:2' 22 | ap-southeast-1: 23 | kubectl: 'arn:aws:lambda:ap-southeast-1:903779448426:layer:eks-kubectl-layer:2' 24 | ap-southeast-2: 25 | kubectl: 'arn:aws:lambda:ap-southeast-2:903779448426:layer:eks-kubectl-layer:2' 26 | ca-central-1: 27 | kubectl: 'arn:aws:lambda:ca-central-1:903779448426:layer:eks-kubectl-layer:1' 28 | us-east-1: 29 | kubectl: 'arn:aws:lambda:us-east-1:903779448426:layer:eks-kubectl-layer:2' 30 | us-west-1: 31 | kubectl: 'arn:aws:lambda:us-west-1:903779448426:layer:eks-kubectl-layer:1' 32 | us-west-2: 33 | kubectl: 'arn:aws:lambda:us-west-2:903779448426:layer:eks-kubectl-layer:2' 34 | us-east-2: 35 | kubectl: 'arn:aws:lambda:us-east-2:903779448426:layer:eks-kubectl-layer:3' 36 | eu-central-1: 37 | kubectl: 'arn:aws:lambda:eu-central-1:903779448426:layer:eks-kubectl-layer:2' 38 | eu-west-1: 39 | kubectl: 'arn:aws:lambda:eu-west-1:903779448426:layer:eks-kubectl-layer:2' 40 | eu-north-1: 41 | kubectl: 'arn:aws:lambda:eu-north-1:903779448426:layer:eks-kubectl-layer:1' 42 | sa-east-1: 43 | kubectl: 'arn:aws:lambda:sa-east-1:903779448426:layer:eks-kubectl-layer:1' 44 | cn-north-1: 45 | kubectl: 'arn:aws-cn:lambda:cn-north-1:937788672844:layer:eks-kubectl-layer:2' 46 | cn-northwest-1: 47 | kubectl: 'arn:aws-cn:lambda:cn-northwest-1:937788672844:layer:eks-kubectl-layer:2' 48 | 49 | Parameters: 50 | ClusterName: 51 | Type: String 52 | Default: default 53 | FunctionName: 54 | Type: String 55 | Default: defaultFunc 56 | FunctionRoleArn: 57 | Type: String 58 | Default: "" 59 | 60 | Conditions: 61 | UseExistingLambdaFuncRole: !Not [!Equals [ !Ref FunctionRoleArn, "" ]] 62 | NotUseExistingLambdaFuncRole: !Equals [ !Ref FunctionRoleArn, "" ] 63 | 64 | Transform: AWS::Serverless-2016-10-31 65 | Description: eks-lambda-drainer Func 66 | Resources: 67 | Func: 68 | Type: AWS::Serverless::Function 69 | Properties: 70 | FunctionName: !Ref FunctionName 71 | Description: github.com/pahud/eks-lambda-drainer 72 | Handler: main 73 | CodeUri: ./func.d 74 | Runtime: provided 75 | Layers: 76 | # - !Sub "arn:aws:lambda:ap-northeast-1:${AWS::AccountId}:layer:layer-eks-kubectl-layer-stack:2" 77 | - !FindInMap 78 | - LayerArn 79 | - !Ref 'AWS::Region' 80 | - kubectl 81 | MemorySize: 512 82 | Environment: 83 | Variables: 84 | cluster_name: !Ref ClusterName 85 | Role: 86 | !If 87 | - UseExistingLambdaFuncRole 88 | - !Ref FunctionRoleArn 89 | - !GetAtt MyLambdaRole.Arn 90 | Timeout: 50 91 | Events: 92 | CWE1: 93 | Type: CloudWatchEvent 94 | Properties: 95 | Pattern: 96 | source: 97 | - aws.ec2 98 | detail-type: 99 | - "EC2 Spot Instance Interruption Warning" 100 | CWE2: 101 | Type: CloudWatchEvent 102 | Properties: 103 | Pattern: 104 | source: 105 | - aws.autoscaling 106 | detail-type: 107 | - "EC2 Instance-terminate Lifecycle Action" 108 | 109 | MyLambdaRole: 110 | Condition: NotUseExistingLambdaFuncRole 111 | Type: AWS::IAM::Role 112 | Properties: 113 | AssumeRolePolicyDocument: 114 | Version: "2012-10-17" 115 | Statement: 116 | - 117 | Effect: "Allow" 118 | Principal: 119 | Service: 120 | - "lambda.amazonaws.com" 121 | Action: 122 | - "sts:AssumeRole" 123 | ManagedPolicyArns: 124 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 125 | # MaxSessionDuration: Integer 126 | Path: / 127 | # PermissionsBoundary: String 128 | Policies: 129 | # - AWSLambdaBasicExecutionRole # Managed Policy 130 | - 131 | PolicyName: Root 132 | PolicyDocument: 133 | Version: '2012-10-17' # Policy Document 134 | Statement: 135 | - Effect: Allow 136 | Action: 137 | - ec2:DescribeInstances 138 | - ec2:DescribeTags 139 | - eks:DescribeCluster 140 | - autoscaling:CompleteLifecycleAction 141 | Resource: '*' 142 | RoleName: !Sub "${AWS::StackName}-FuncRole" 143 | 144 | 145 | 146 | # Func2: 147 | # Condition: NotUseExistingLambdaFuncRole 148 | # Type: AWS::Serverless::Function 149 | # Properties: 150 | # FunctionName: !Ref FunctionName 151 | # Description: github.com/pahud/eks-lambda-drainer 152 | # Handler: main 153 | # CodeUri: ./func.d 154 | # Runtime: provided 155 | # Layers: 156 | # # - !Sub "arn:aws:lambda:ap-northeast-1:${AWS::AccountId}:layer:layer-eks-kubectl-layer-stack:2" 157 | # - !FindInMap 158 | # - LayerArn 159 | # - !Ref 'AWS::Region' 160 | # - kubectl 161 | # MemorySize: 512 162 | # Environment: 163 | # Variables: 164 | # cluster_name: !Ref ClusterName 165 | # Policies: 166 | # - AWSLambdaBasicExecutionRole # Managed Policy 167 | # - Version: '2012-10-17' # Policy Document 168 | # Statement: 169 | # - Effect: Allow 170 | # Action: 171 | # - ec2:DescribeInstances 172 | # - ec2:DescribeTags 173 | # - eks:DescribeCluster 174 | # - autoscaling:CompleteLifecycleAction 175 | # Resource: '*' 176 | # Timeout: 50 177 | # Events: 178 | # CWE1: 179 | # Type: CloudWatchEvent 180 | # Properties: 181 | # Pattern: 182 | # source: 183 | # - aws.ec2 184 | # detail-type: 185 | # - "EC2 Spot Instance Interruption Warning" 186 | # CWE2: 187 | # Type: CloudWatchEvent 188 | # Properties: 189 | # Pattern: 190 | # source: 191 | # - aws.autoscaling 192 | # detail-type: 193 | # - "EC2 Instance-terminate Lifecycle Action" 194 | 195 | 196 | 197 | Outputs: 198 | FuncArn: 199 | Description: "Lambda function Arn" 200 | Value: !GetAtt Func.Arn 201 | 202 | FuncIamRole: 203 | Description: "Lambda function IAM role Arn" 204 | Value: 205 | !If 206 | - UseExistingLambdaFuncRole 207 | - !Ref FunctionRoleArn 208 | - !GetAtt MyLambdaRole.Arn 209 | 210 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Amazon EKS Serverless Drainer 2 | 3 | Amazon EKS node drainer with AWS Lambda. 4 | 5 | [![](https://img.shields.io/badge/Available-serverless%20app%20repository-blue.svg)](https://serverlessrepo.aws.amazon.com/#/applications/arn:aws:serverlessrepo:us-east-1:903779448426:applications~eks-lambda-drainer) 6 | 7 | **amazon-eks-serverless-drainer** is an Amazon EKS node drainer with AWS Lambda. If you provision spot instances or spotfleet in your Amazon EKS nodegroup, you can listen to the spot termination signal from **CloudWatch Events** 120 seconds in prior to the final termination process. By configuring this Lambda function as the CloudWatch Event target, **amazon-eks-serverless-drainer** will drain the terminating node and all the pods without relative toleration will be evicted and rescheduled to another node - your workload will get very minimal impact on the spot instance termination. 8 | 9 | ![](images/amazon-eks-serverless-drainer.png) 10 | 11 | 12 | 13 | ## Implementations 14 | 15 | Previously this project has a native `golang` implementation with `client-go`. 16 | However, as AWS [announced](https://amzn.to/2SUlcv3) `Lambda layer` and `Lambda custom runtime`, thanks to the [aws-samples/aws-lambda-layer-kubectl](https://github.com/aws-samples/aws-lambda-layer-kubectl) project, 17 | it's very easy to implement this with a few lines of bash script in Lambda([tweet](https://twitter.com/pahudnet/status/1095369690556162049)) whilst the code size could be reduced from `11MB` to just `2.4KB`. 18 | So we will stick to `bash` implementation in this branch. We believe this will eliminate the complexity to help people develop similar projects in the future. 19 | 20 | 21 | 22 | # Option 1: Deployt from SAR(Serverless App Repository) 23 | 24 | The most simple way to build this stack is creating from SAR: 25 | 26 | Edit `Makefile` and then 27 | 28 | ```bash 29 | $ make sam-package-from-sar sam-deploy 30 | ``` 31 | Or just click the button to deploy 32 | 33 | 34 | | Region | Click and Deploy | 35 | | :----------------: | :----------------------------------------------------------: | 36 | | **us-east-1** | [![](https://img.shields.io/badge/SAR-Deploy%20Now-yellow.svg)](https://deploy.serverlessrepo.app/us-east-1/?app=arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer) | 37 | | **us-east-2** | [![](https://img.shields.io/badge/SAR-Deploy%20Now-yellow.svg)](https://deploy.serverlessrepo.app/us-east-2/?app=arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer) | 38 | | **us-west-1** | [![](https://img.shields.io/badge/SAR-Deploy%20Now-yellow.svg)](https://deploy.serverlessrepo.app/us-west-1/?app=arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer) | 39 | | **us-west-2** | [![](https://img.shields.io/badge/SAR-Deploy%20Now-yellow.svg)](https://deploy.serverlessrepo.app/us-west-2/?app=arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer) | 40 | | **ap-northeast-1** | [![](https://img.shields.io/badge/SAR-Deploy%20Now-yellow.svg)](https://deploy.serverlessrepo.app/ap-northeast-1/?app=arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer) | 41 | | **ap-northeast-2** | [![](https://img.shields.io/badge/SAR-Deploy%20Now-yellow.svg)](https://deploy.serverlessrepo.app/ap-northeast-2/?app=arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer) | 42 | | **ap-southeast-1** | [![](https://img.shields.io/badge/SAR-Deploy%20Now-yellow.svg)](https://deploy.serverlessrepo.app/ap-southeast-1/?app=arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer) | 43 | | **ap-southeast-2** | [![](https://img.shields.io/badge/SAR-Deploy%20Now-yellow.svg)](https://deploy.serverlessrepo.app/ap-southeast-2/?app=arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer) | 44 | | **eu-central-1** | [![](https://img.shields.io/badge/SAR-Deploy%20Now-yellow.svg)](https://deploy.serverlessrepo.app/eu-central-1/?app=arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer) | 45 | | **eu-west-1** | [![](https://img.shields.io/badge/SAR-Deploy%20Now-yellow.svg)](https://deploy.serverlessrepo.app/eu-west-1/?app=arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer) | 46 | | **eu-west-2** | [![](https://img.shields.io/badge/SAR-Deploy%20Now-yellow.svg)](https://deploy.serverlessrepo.app/eu-west-2/?app=arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer) | 47 | | **eu-west-3** | [![](https://img.shields.io/badge/SAR-Deploy%20Now-yellow.svg)](https://deploy.serverlessrepo.app/eu-west-3/?app=arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer) | 48 | | **eu-north-1** | [![](https://img.shields.io/badge/SAR-Deploy%20Now-yellow.svg)](https://deploy.serverlessrepo.app/eu-north-1/?app=arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer) | 49 | | **sa-east-1** | [![](https://img.shields.io/badge/SAR-Deploy%20Now-yellow.svg)](https://deploy.serverlessrepo.app/sa-east-1/?app=arn:aws:serverlessrepo:us-east-1:903779448426:applications/eks-lambda-drainer) | 50 | 51 | 52 | 53 | This will provision the whole **amazon-eks-serverless-drainer** stack from SAR including `aws-lambda-layer-kubectl` lambda layer out-of-the-box. The benefit is you don't have to build the layer yourself. 54 | 55 | 56 | 57 | # Option 2: Building from scratch 58 | 59 | If you want to build it from scratch including the `aws-lambda-layer-kubectl` 60 | 61 | 62 | ## Prepare your Layer 63 | 64 | Follow the [instructions](https://github.com/aws-samples/aws-lambda-layer-kubectl) to build and publish your `aws-lambda-layer-kubectl` Lambda Layer. 65 | Copy the layer ARN(e.g. `arn:aws:lambda:ap-northeast-1:${AWS::AccountId}:layer:layer-eks-kubectl-layer-stack:2`) 66 | 67 | 68 | 69 | ## Edit the sam.yaml 70 | 71 | Set the value of `Layers` to the layer arn in the previous step. 72 | 73 | ``` 74 | Layers: 75 | - !Sub "arn:aws:lambda:ap-northeast-1:${AWS::AccountId}:layer:layer-eks-kubectl-layer-stack:2" 76 | 77 | ``` 78 | 79 | 80 | 81 | # update Makefile 82 | 83 | edit `Makefile` and update **S3BUCKET** variable: 84 | 85 | modify this to your private S3 bucket you have read/write access to 86 | ``` 87 | S3BUCKET ?= pahud-temp-ap-northeast-1 88 | ``` 89 | 90 | set the AWS region you are deploying to 91 | ``` 92 | LAMBDA_REGION ?= ap-northeast-1 93 | ``` 94 | 95 | 96 | 97 | ## package and deploy with `SAM` 98 | 99 | ``` 100 | $ make func-prep sam-package sam-deploy 101 | ``` 102 | (`SAM` will deplly a cloudformation stack for you in your `{LAMBDA_REGION}` and register cloudwatch events as the Lambda source event) 103 | ``` 104 | Uploading to 032ea7f22f8fedab0d016ed22f2bdea4 11594869 / 11594869.0 (100.00%) 105 | Successfully packaged artifacts and wrote output template to file packaged.yaml. 106 | Execute the following command to deploy the packaged template 107 | aws cloudformation deploy --template-file /home/samcli/workdir/packaged.yaml --stack-name 108 | 109 | Waiting for changeset to be created.. 110 | Waiting for stack create/update to complete 111 | Successfully created/updated stack - eks-lambda-drainer 112 | # print the cloudformation stack outputs 113 | aws --region ap-northeast-1 cloudformation describe-stacks --stack-name "eks-lambda-drainer" --query 'Stacks[0].Outputs' 114 | [ 115 | { 116 | "Description": "Lambda function Arn", 117 | "OutputKey": "Func", 118 | "OutputValue": "arn:aws:lambda:ap-northeast-1:xxxxxxxx:function:eks-lambda-drainer-Func-1P5RHJ50KEVND" 119 | }, 120 | { 121 | "Description": "Lambda function IAM role Arn", 122 | "OutputKey": "FuncIamRole", 123 | "OutputValue": "arn:aws:iam::xxxxxxxx:role/eks-lambda-drainer-FuncRole-TCZVVLEG1HKD" 124 | } 125 | ] 126 | ``` 127 | 128 | 129 | 130 | 131 | 132 | 133 | # Add Lambda Role into ConfigMap 134 | 135 | `eks-lambda-drainer` will run with provided lambda role or with exactly the role arn you specified in the parameter. Make sure you have added the role into `aws-auth` ConfigMap. 136 | 137 | Read Amazon EKS [document](https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html) about how to add an IAM Role to the `aws-auth` ConfigMap. 138 | 139 | Edit the `aws-auth` ConfigMap by 140 | 141 | ``` 142 | kubectl edit -n kube-system configmap/aws-auth 143 | ``` 144 | 145 | And insert `rolearn`, `groups` and `username` into the `mapRoles`, make sure the groups contain `system:masters` 146 | 147 | For eample 148 | 149 | ``` 150 | apiVersion: v1 151 | kind: ConfigMap 152 | metadata: 153 | name: aws-auth 154 | namespace: kube-system 155 | data: 156 | mapRoles: | 157 | - rolearn: arn:aws:iam::xxxxxxxx:role/eksdemo-NG-1RPL723W45VT5-NodeInstanceRole-1D4S7IF32IDU1 158 | username: system:node:{{EC2PrivateDNSName}} 159 | groups: 160 | - system:bootstrappers 161 | - system:nodes 162 | - rolearn: arn:aws:iam::xxxxxxxx:role/eks-lambda-drainer-FuncRole-TCZVVLEG1HKD 163 | username: EKSForLambda 164 | groups: 165 | - system:masters 166 | ``` 167 | The first `rolearn` is your Amazon EKS NodeInstanceRole and the 2nd `rolearn` would be your Lambda Role. 168 | 169 | 170 | # Validation 171 | 172 | You may decrease the `desired capacity` of your autoscaling group for Amazon EKS nodegroup. Behind the scene, on 173 | instance termination from auoscaling group, the node will first enter the **Terminating:Wait** state and after a pre-defined graceful period of time(default: 10 seconds), 174 | **eks-lambda-drainer** will be invoked through the CloudWatch Event and perform `kubectl drain` on the node and immediately 175 | put **CompleteLifecycleAction** back to the hook and the autoscaling group then move on to the 176 | **Terminaing:Proceed** phase to execute the last termination process. The Pods in the terminating node will be rescheduled to other node(s) before the termination 177 | Your service will have almost zero impact. 178 | 179 | 180 | # In Actions 181 | 182 | Live tail the log 183 | 184 | ``` 185 | $ make sam-logs-tail 186 | ``` 187 | 188 | ![](images/11.png) 189 | 190 | 191 | 192 | ![](images/12.png) 193 | 194 | 195 | 196 | # kubectl drain or kubectl taint 197 | 198 | By default, `eks-lambda-drainer` will `kubectl drain` the node, however, if you specify Lambda environment variable `drain_type=taint` then it will `kubectl taint` the node.([details](https://github.com/pahud/eks-lambda-drainer/blob/c36e3aab1590177719e1eb389f077829ec238504/main.sh#L46-L52)) 199 | 200 | 201 | 202 | # cluster name auto discovery 203 | 204 | You don't have to specify the Amazon EKS cluster name, by default `eks-lambda-drainer` will determine the EC2 Tag of the terminating node: 205 | 206 | ``` 207 | kubernetes.io/cluster/{cluster_name} = owned 208 | ``` 209 | 210 | For example, `kubernetes.io/cluster/eksdemo = owned` will make the `cluster_name=eksdemo`. 211 | 212 | 213 | 214 | # clean up 215 | 216 | ``` 217 | $ make sam-destroy 218 | ``` 219 | (this will destroy the cloudformation stack and all resources in it) 220 | 221 | 222 | ## License Summary 223 | 224 | This sample code is made available under the MIT-0 license. See the LICENSE file. 225 | --------------------------------------------------------------------------------