├── .gitignore ├── Makefile ├── Procfile ├── README.md ├── docker └── buildkite-agent-ecs │ └── Dockerfile ├── go.mod ├── go.sum ├── lambdas ├── ecs-service-scaler │ └── main.go └── ecs-spotfleet-scaler │ └── main.go ├── scripts ├── create.sh ├── develop.sh └── update.sh └── templates ├── agent ├── README.md └── template.yaml ├── compute └── spotfleet │ ├── README.md │ └── template.yaml └── vpc ├── README.md └── template.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | /lambdas/*/handler 2 | /*.zip 3 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .PHONY: sync lint validate docker 2 | VERSION=$(shell git describe --tags --candidates=1 --dirty 2>/dev/null \ 3 | || printf "dev-%s" "$$(git rev-parse --short HEAD)") 4 | FLAGS=-s -w -X main.Version=$(VERSION) 5 | 6 | LAMBDA_S3_BUCKET := buildkite-aws-stack-ecs-dev 7 | LAMBDA_S3_BUCKET_PATH := / 8 | LAMBDAS = ecs-service-scaler.zip ecs-spotfleet-scaler.zip 9 | 10 | DOCKER_AGENT_TAG := buildkite/agent-ecs 11 | DOCKER_SOCKGUARD_TAG := buildkite/sockguard-ecs 12 | 13 | build: $(LAMBDAS) 14 | 15 | clean: 16 | -rm $(LAMBDAS) 17 | -rm lambdas/ecs-service-scaler/handler 18 | -rm lambdas/ecs-spotfleet-scaler/handler 19 | 20 | %.zip: lambdas/%/handler 21 | zip -9 -v -j $@ "$<" 22 | 23 | lambdas/ecs-service-scaler/handler: lambdas/ecs-service-scaler/main.go 24 | docker run \ 25 | --volume go-module-cache:/go/pkg/mod \ 26 | --volume $(PWD):/code \ 27 | --workdir /code \ 28 | --rm golang:1.11 \ 29 | go build -ldflags="$(FLAGS)" -o ./lambdas/ecs-service-scaler/handler ./lambdas/ecs-service-scaler 30 | chmod +x lambdas/ecs-service-scaler/handler 31 | 32 | lambdas/ecs-spotfleet-scaler/handler: lambdas/ecs-spotfleet-scaler/main.go 33 | docker run \ 34 | --volume go-module-cache:/go/pkg/mod \ 35 | --volume $(PWD):/code \ 36 | --workdir /code \ 37 | --rm golang:1.11 \ 38 | go build -ldflags="$(FLAGS)" -o ./lambdas/ecs-spotfleet-scaler/handler ./lambdas/ecs-spotfleet-scaler 39 | chmod +x lambdas/ecs-spotfleet-scaler/handler 40 | 41 | lambda-sync: $(LAMBDAS) 42 | aws s3 sync \ 43 | --acl public-read \ 44 | --exclude '*' --include '*.zip' \ 45 | . s3://$(LAMBDA_S3_BUCKET)$(LAMBDA_S3_BUCKET_PATH) 46 | 47 | lambda-versions: 48 | aws s3api head-object \ 49 | --bucket ${LAMBDA_S3_BUCKET} \ 50 | --key ecs-spotfleet-scaler.zip --query "VersionId" --output text 51 | aws s3api head-object \ 52 | --bucket ${LAMBDA_S3_BUCKET} \ 53 | --key ecs-service-scaler.zip --query "VersionId" --output text 54 | 55 | docker: docker-agent docker-sockguard 56 | 57 | docker-agent: 58 | docker build --tag "$(DOCKER_AGENT_TAG)" ./docker/buildkite-agent-ecs 59 | 60 | docker-sockguard: 61 | cp $(GOPATH)/src/github.com/buildkite/sockguard/build/sockguard-linux-amd64 ./docker/sockguard-ecs/sockguard 62 | docker build --tag "$(DOCKER_SOCKGUARD_TAG)" ./docker/sockguard-ecs 63 | 64 | docker-push: 65 | docker push $(DOCKER_AGENT_TAG) 66 | docker push $(DOCKER_SOCKGUARD_TAG) 67 | 68 | lint: 69 | find templates -name '*.yaml' | xargs -n1 cfn-lint 70 | -------------------------------------------------------------------------------- /Procfile: -------------------------------------------------------------------------------- 1 | ecs-service-scaler: go run ./lambdas/ecs-service-scaler 2 | ecs-spotfleet-scaler: go run ./lambdas/ecs-spotfleet-scaler 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Elastic CI Stack for AWS: ECS Edition (2 elastic 2 stack) 2 | 3 | This was an **experimental version** of our main [AWS stack](https://github.com/buildkite/elastic-ci-stack-for-aws) that makes use of ECS and Spot Fleets. Due to low uptake, and new directions, we're now pursuing other ideas. 4 | 5 | If you are using this stack and cannot use our main AWS stack, please reach out: support@buildkite.com 6 | 7 | ## Design Goals 8 | 9 | * Agents/Queues that each have their own IAM Role 10 | * Docker-based isolation for Jobs 11 | * Shared underlying compute infrastructure via Spotfleet 12 | * Fast auto-scaling 13 | 14 | ## How is isolation currently provided? 15 | 16 | Agents are running in docker containers on ECS instances, each with their own [Task IAM Roles](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html). The ECS Agent uses firewall rules to prevent containers from accessing the Instance Roles and also prevents usage of certain docker features like `host` networking. 17 | 18 | ## Caveats ☣️🚨🦑 19 | 20 | * Agent session tokens (`BUILDKITE_AGENT_ACCESS_TOKEN`) are exposed to builds and are valid for the duration of the agent uptime. Exposing this token to third-party pull requests would be disasterous. 21 | 22 | ## Stacks 23 | 24 | ### VPC 25 | 26 | The [VPC Stack](templates/vpc/README.md) provides an underlying VPC that will handle as many subnets as you have available. 27 | 28 | ### Spotfleet Compute 29 | 30 | The [Spotfleet Stack](templates/compute/spotfleet/README.md) provides an ECS Cluster and an AWS Spotfleet that powers it. It auto-scales based on the needs of ECS Services in the Cluster. 31 | 32 | ### Agent 33 | 34 | The [Agent Stack](templates/agent/README.md) provides an ECS Service that runs a Buildkite Agent as an ECS Task. Each Agent has it's own [Task IAM Roles](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html), independent of the IAM permissions that the host that it's running on has. 35 | 36 | ## Installation 37 | 38 | Clone this repository and create each stack from the templates mentioned above. 39 | -------------------------------------------------------------------------------- /docker/buildkite-agent-ecs/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM buildkite/agent:3 2 | 3 | RUN apk -v --update add \ 4 | python \ 5 | py-pip \ 6 | groff \ 7 | less \ 8 | mailcap \ 9 | && \ 10 | pip install --upgrade awscli s3cmd==2.0.1 python-magic && \ 11 | apk -v --purge del py-pip && \ 12 | rm /var/cache/apk/* 13 | 14 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/buildkite/elastic-ci-stack-for-aws-ecs 2 | 3 | require ( 4 | github.com/aws/aws-lambda-go v1.7.0 5 | github.com/aws/aws-sdk-go v1.15.82 6 | github.com/davecgh/go-spew v1.1.1 // indirect 7 | github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af // indirect 8 | github.com/pmezard/go-difflib v1.0.0 // indirect 9 | github.com/stretchr/testify v1.2.2 // indirect 10 | golang.org/x/net v0.0.0-20181114220301-adae6a3d119a // indirect 11 | golang.org/x/text v0.3.0 // indirect 12 | ) 13 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/aws/aws-lambda-go v1.7.0 h1:g3Ad7aw27B2lhQLIuK7Aha+cWSaHr7ZNlngveHkhZyo= 2 | github.com/aws/aws-lambda-go v1.7.0/go.mod h1:zUsUQhAUjYzR8AuduJPCfhBuKWUaDbQiPOG+ouzmE1A= 3 | github.com/aws/aws-sdk-go v1.15.82 h1:tvOP/hcmpiUqtqJnU/IwJkqTEfnbsgja0xbPjvZuzbI= 4 | github.com/aws/aws-sdk-go v1.15.82/go.mod h1:E3/ieXAlvM0XWO57iftYVDLLvQ824smPP3ATZkfNZeM= 5 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 6 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 7 | github.com/jmespath/go-jmespath v0.0.0-20160202185014-0b12d6b521d8 h1:12VvqtR6Aowv3l/EQUlocDHW2Cp4G9WJVH7uyH8QFJE= 8 | github.com/jmespath/go-jmespath v0.0.0-20160202185014-0b12d6b521d8/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k= 9 | github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af h1:pmfjZENx5imkbgOkpRUYLnmbU7UEFbjtDA2hxJ1ichM= 10 | github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k= 11 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 12 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 13 | github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w= 14 | github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= 15 | golang.org/x/net v0.0.0-20181114220301-adae6a3d119a h1:gOpx8G595UYyvj8UK4+OFyY4rx037g3fmfhe5SasG3U= 16 | golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 17 | golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= 18 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 19 | -------------------------------------------------------------------------------- /lambdas/ecs-service-scaler/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "context" 5 | "encoding/json" 6 | "fmt" 7 | "log" 8 | "net/http" 9 | "net/url" 10 | "os" 11 | "time" 12 | 13 | "github.com/aws/aws-lambda-go/lambda" 14 | "github.com/aws/aws-sdk-go/aws" 15 | "github.com/aws/aws-sdk-go/aws/session" 16 | "github.com/aws/aws-sdk-go/service/ecs" 17 | ) 18 | 19 | const ( 20 | defaultMetricsEndpoint = "https://agent.buildkite.com/v3" 21 | ) 22 | 23 | var ( 24 | Version string = "dev" 25 | ) 26 | 27 | func main() { 28 | if os.Getenv(`DEBUG`) != "" { 29 | _, err := Handler(context.Background(), json.RawMessage([]byte{})) 30 | if err != nil { 31 | log.Fatal(err) 32 | } 33 | } else { 34 | lambda.Start(Handler) 35 | } 36 | } 37 | 38 | func Handler(ctx context.Context, evt json.RawMessage) (string, error) { 39 | log.Printf("ecs-spotfleet-scaler version %s", Version) 40 | 41 | var timeout <-chan time.Time = make(chan time.Time) 42 | var interval time.Duration = 10 * time.Second 43 | 44 | if intervalStr := os.Getenv(`LAMBDA_INTERVAL`); intervalStr != "" { 45 | var err error 46 | interval, err = time.ParseDuration(intervalStr) 47 | if err != nil { 48 | return "", err 49 | } 50 | } 51 | 52 | if timeoutStr := os.Getenv(`LAMBDA_TIMEOUT`); timeoutStr != "" { 53 | timeoutDuration, err := time.ParseDuration(timeoutStr) 54 | if err != nil { 55 | return "", err 56 | } 57 | timeout = time.After(timeoutDuration) 58 | } 59 | 60 | var mustGetEnv = func(env string) string { 61 | val := os.Getenv(env) 62 | if val == "" { 63 | panic(fmt.Sprintf("Env %q not set", env)) 64 | } 65 | return val 66 | } 67 | 68 | sess := session.New() 69 | conf := config{ 70 | BuildkiteToken: mustGetEnv(`BUILDKITE_TOKEN`), 71 | BuildkiteQueue: mustGetEnv(`BUILDKITE_QUEUE`), 72 | ECSCluster: mustGetEnv(`ECS_CLUSTER`), 73 | ECSService: mustGetEnv(`ECS_SERVICE`), 74 | } 75 | 76 | for { 77 | select { 78 | case <-timeout: 79 | return "", nil 80 | default: 81 | err := scaleECSServiceCapacity(sess, conf) 82 | if err != nil { 83 | log.Printf("Err: %#v", err.Error()) 84 | return "", nil 85 | } 86 | 87 | log.Printf("Sleeping for %v", interval) 88 | time.Sleep(interval) 89 | } 90 | } 91 | } 92 | 93 | type config struct { 94 | BuildkiteToken string 95 | BuildkiteQueue string 96 | ECSCluster string 97 | ECSService string 98 | } 99 | 100 | func scaleECSServiceCapacity(sess *session.Session, config config) error { 101 | client := newBuildkiteClient(config.BuildkiteToken) 102 | count, err := client.GetScheduledJobCount(config.BuildkiteQueue) 103 | if err != nil { 104 | return err 105 | } 106 | 107 | svc := ecs.New(sess) 108 | 109 | result, err := svc.DescribeServices(&ecs.DescribeServicesInput{ 110 | Cluster: aws.String(config.ECSCluster), 111 | Services: []*string{ 112 | aws.String(config.ECSService), 113 | }, 114 | }) 115 | if err != nil { 116 | return err 117 | } 118 | 119 | if len(result.Services[0].Deployments) > 1 { 120 | log.Printf("Deployment in progress, waiting") 121 | return nil 122 | } 123 | 124 | log.Printf("Modifying service %s, setting count=%d", config.ECSService, count) 125 | _, err = svc.UpdateService(&ecs.UpdateServiceInput{ 126 | Cluster: aws.String(config.ECSCluster), 127 | Service: aws.String(config.ECSService), 128 | DesiredCount: aws.Int64(count), 129 | }) 130 | if err != nil { 131 | return err 132 | } 133 | 134 | return nil 135 | } 136 | 137 | type buildkiteClient struct { 138 | Endpoint string 139 | AgentToken string 140 | UserAgent string 141 | Queue string 142 | } 143 | 144 | func newBuildkiteClient(agentToken string) *buildkiteClient { 145 | return &buildkiteClient{ 146 | Endpoint: defaultMetricsEndpoint, 147 | UserAgent: fmt.Sprintf("elastic-ci-stack-for-aws-ecs/ecs-service-scaler/%s", Version), 148 | AgentToken: agentToken, 149 | } 150 | } 151 | 152 | func (c *buildkiteClient) GetScheduledJobCount(queue string) (int64, error) { 153 | log.Printf("Collecting agent metrics for queue %q", queue) 154 | t := time.Now() 155 | 156 | endpoint, err := url.Parse(c.Endpoint) 157 | if err != nil { 158 | return 0, err 159 | } 160 | 161 | endpoint.Path += "/metrics" 162 | 163 | req, err := http.NewRequest("GET", endpoint.String(), nil) 164 | if err != nil { 165 | return 0, err 166 | } 167 | 168 | req.Header.Set("User-Agent", c.UserAgent) 169 | req.Header.Set("Authorization", fmt.Sprintf("Token %s", c.AgentToken)) 170 | 171 | res, err := http.DefaultClient.Do(req) 172 | if err != nil { 173 | return 0, err 174 | } 175 | 176 | var resp struct { 177 | Jobs struct { 178 | Queues map[string]struct { 179 | Total int64 `json:"total"` 180 | } `json:"queues"` 181 | } `json:"jobs"` 182 | } 183 | 184 | defer res.Body.Close() 185 | err = json.NewDecoder(res.Body).Decode(&resp) 186 | if err != nil { 187 | return 0, err 188 | } 189 | 190 | var count int64 191 | 192 | if queue, exists := resp.Jobs.Queues[queue]; exists { 193 | count = queue.Total 194 | } 195 | 196 | log.Printf("↳ Got %d total jobs (took %v)", count, time.Now().Sub(t)) 197 | return count, nil 198 | } 199 | -------------------------------------------------------------------------------- /lambdas/ecs-spotfleet-scaler/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "context" 5 | "encoding/json" 6 | "fmt" 7 | "log" 8 | "os" 9 | "strconv" 10 | "time" 11 | "math" 12 | 13 | "github.com/aws/aws-lambda-go/lambda" 14 | "github.com/aws/aws-sdk-go/aws" 15 | "github.com/aws/aws-sdk-go/aws/session" 16 | "github.com/aws/aws-sdk-go/service/ec2" 17 | "github.com/aws/aws-sdk-go/service/ecs" 18 | ) 19 | 20 | const ( 21 | // Converts cpu/memory needed into a capacity figure for spotfleet 22 | cpuDivisor = 1024 23 | memoryDivisor = 2048 24 | ) 25 | 26 | var ( 27 | Version string = "dev" 28 | ) 29 | 30 | func main() { 31 | if os.Getenv(`DEBUG`) != "" { 32 | _, err := Handler(context.Background(), json.RawMessage([]byte{})) 33 | if err != nil { 34 | log.Fatal(err) 35 | } 36 | } else { 37 | lambda.Start(Handler) 38 | } 39 | } 40 | 41 | func Handler(ctx context.Context, evt json.RawMessage) (string, error) { 42 | log.Printf("ecs-spotfleet-scaler version %s", Version) 43 | 44 | var timeout <-chan time.Time = make(chan time.Time) 45 | var interval time.Duration = 10 * time.Second 46 | 47 | if intervalStr := os.Getenv(`LAMBDA_INTERVAL`); intervalStr != "" { 48 | var err error 49 | interval, err = time.ParseDuration(intervalStr) 50 | if err != nil { 51 | return "", err 52 | } 53 | } 54 | 55 | if timeoutStr := os.Getenv(`LAMBDA_TIMEOUT`); timeoutStr != "" { 56 | timeoutDuration, err := time.ParseDuration(timeoutStr) 57 | if err != nil { 58 | return "", err 59 | } 60 | timeout = time.After(timeoutDuration) 61 | } 62 | 63 | var mustGetEnv = func(env string) string { 64 | val := os.Getenv(env) 65 | if val == "" { 66 | panic(fmt.Sprintf("Env %q not set", env)) 67 | } 68 | return val 69 | } 70 | 71 | var conf = config { 72 | ECSCluster: mustGetEnv(`ECS_CLUSTER`), 73 | SpotFleetRequestId: mustGetEnv(`SPOT_FLEET`), 74 | } 75 | 76 | if ms := os.Getenv(`MIN_SIZE`); ms != "" { 77 | var err error 78 | conf.MinSize, err = strconv.ParseInt(ms, 10, 32) 79 | if err != nil { 80 | return "", fmt.Errorf("failed to parse MIN_SIZE: %v", err) 81 | } 82 | } 83 | 84 | if ms := os.Getenv(`MAX_SIZE`); ms != "" { 85 | var err error 86 | conf.MaxSize, err = strconv.ParseInt(ms, 10, 32) 87 | if err != nil { 88 | return "", fmt.Errorf("failed to parse MAX_SIZE: %v", err) 89 | } 90 | } 91 | 92 | sess := session.New() 93 | 94 | for { 95 | select { 96 | case <-timeout: 97 | return "", nil 98 | default: 99 | err := scaleSpotFleetCapacity(sess, conf) 100 | if err != nil { 101 | log.Printf("Err: %#v", err.Error()) 102 | return "", nil 103 | } 104 | 105 | log.Printf("Sleeping for %v", interval) 106 | time.Sleep(interval) 107 | } 108 | } 109 | } 110 | 111 | type config struct { 112 | ECSCluster string 113 | SpotFleetRequestId string 114 | MinSize, MaxSize int64 115 | } 116 | 117 | func scaleSpotFleetCapacity(sess *session.Session, config config) error { 118 | svc := ecs.New(sess) 119 | listServicesOutput, err := svc.ListServices(&ecs.ListServicesInput{ 120 | Cluster: aws.String(config.ECSCluster), 121 | }) 122 | if err != nil { 123 | return err 124 | } 125 | 126 | describeServicesOutput, err := svc.DescribeServices(&ecs.DescribeServicesInput{ 127 | Cluster: aws.String(config.ECSCluster), 128 | Services: listServicesOutput.ServiceArns, 129 | }) 130 | if err != nil { 131 | return err 132 | } 133 | 134 | var cpuRequired int64 135 | var memoryRequired int64 136 | 137 | if svcLen := len(describeServicesOutput.Services); svcLen == 0 { 138 | log.Printf("No services defined") 139 | return nil 140 | } 141 | 142 | for _, service := range describeServicesOutput.Services { 143 | log.Printf("Service %s has desired=%d, running=%d, pending=%d", 144 | *service.ServiceName, *service.DesiredCount, *service.RunningCount, *service.PendingCount) 145 | 146 | describeTaskDefinitionResult, err := svc.DescribeTaskDefinition(&ecs.DescribeTaskDefinitionInput{ 147 | TaskDefinition: service.TaskDefinition, 148 | }) 149 | if err != nil { 150 | return err 151 | } 152 | 153 | taskCPURequired, err := strconv.ParseInt(*describeTaskDefinitionResult.TaskDefinition.Cpu, 10, 64) 154 | if err != nil { 155 | return err 156 | } 157 | 158 | cpuRequired += (taskCPURequired * *service.DesiredCount) 159 | 160 | taskMemoryRequired, err := strconv.ParseInt(*describeTaskDefinitionResult.TaskDefinition.Memory, 10, 64) 161 | if err != nil { 162 | return err 163 | } 164 | 165 | memoryRequired += (taskMemoryRequired * *service.DesiredCount) 166 | } 167 | 168 | log.Printf("Total needed CPU is %d, total needed memory is %d", cpuRequired, memoryRequired) 169 | 170 | // do maths in floats to handle fractions 171 | var required float64 = float64(cpuRequired) / float64(cpuDivisor) 172 | if float64(memoryRequired)/float64(memoryDivisor) > required { 173 | required = float64(memoryRequired) / float64(memoryDivisor) 174 | } 175 | 176 | var requiredInt int64 = int64(math.Round(required)) 177 | 178 | ec2Svc := ec2.New(sess) 179 | 180 | describeSpotFleetOutput, err := ec2Svc.DescribeSpotFleetRequests(&ec2.DescribeSpotFleetRequestsInput{ 181 | SpotFleetRequestIds: []*string{ 182 | aws.String(config.SpotFleetRequestId), 183 | }, 184 | }) 185 | if err != nil { 186 | return err 187 | } 188 | 189 | if len(describeSpotFleetOutput.SpotFleetRequestConfigs) == 0 { 190 | return fmt.Errorf("No spot fleet found for %s", config.SpotFleetRequestId) 191 | } 192 | 193 | spotFleetConfig := describeSpotFleetOutput.SpotFleetRequestConfigs[0] 194 | 195 | log.Printf("Spotfleet %s has target=%d", 196 | config.SpotFleetRequestId, 197 | *spotFleetConfig.SpotFleetRequestConfig.TargetCapacity, 198 | ) 199 | 200 | // Spot fleet can't be modified whilst in "modifying" 201 | if *spotFleetConfig.SpotFleetRequestState == "modifying" { 202 | log.Printf("Spot fleet is presently in %q state", *spotFleetConfig.SpotFleetRequestState) 203 | return nil 204 | } 205 | 206 | if requiredInt < config.MinSize { 207 | log.Printf("Adjusting count to maintain minimum size of %d, would have been %d", 208 | config.MinSize, requiredInt) 209 | requiredInt = config.MinSize 210 | } 211 | 212 | // Don't change spot fleet if it's already at TargetCapacity 213 | if *spotFleetConfig.SpotFleetRequestConfig.TargetCapacity == requiredInt { 214 | log.Printf("TargetCapacity is already at correct count of %d", requiredInt) 215 | return nil 216 | } 217 | 218 | log.Printf("Modifying spotfleet %s, setting TargetCapacity=%d", config.SpotFleetRequestId, requiredInt) 219 | 220 | _, err = ec2Svc.ModifySpotFleetRequest(&ec2.ModifySpotFleetRequestInput{ 221 | SpotFleetRequestId: aws.String(config.SpotFleetRequestId), 222 | TargetCapacity: aws.Int64(requiredInt), 223 | }) 224 | if err != nil { 225 | return err 226 | } 227 | 228 | return nil 229 | } 230 | -------------------------------------------------------------------------------- /scripts/create.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -euo pipefail 3 | 4 | LAMBDA_BUCKET=buildkite-aws-stack-ecs-dev 5 | DOCKER_IMAGE=lox24/buildkite-agent-ecs 6 | QUEUE=dev 7 | VPC_STACK=${VPC_STACK:-buildkite-vpc} 8 | SPOT_FLEET_STACK=${SPOT_FLEET_STACK:-buildkite-spotfleet} 9 | AGENT_STACK=${AGENT_STACK:-"buildkite-agent-$QUEUE"} 10 | 11 | if ! aws cloudformation describe-stacks --stack-name "${VPC_STACK}" &> /dev/null ; then 12 | ## Figure out what availability zones are available 13 | EC2_AVAILABILITY_ZONES=$(aws ec2 describe-availability-zones \ 14 | --query 'AvailabilityZones[?State==`available`].ZoneName' \ 15 | --output text | sed -E -e 's/[[:blank:]]+/,/g') 16 | 17 | EC2_AVAILABILITY_ZONES_COUNT=$(awk -F, '{print NF-1}' <<< "$EC2_AVAILABILITY_ZONES") 18 | 19 | ## Create a VPC stack 20 | echo "~~~ Creating ${VPC_STACK}" 21 | parfait create-stack \ 22 | -t templates/vpc/template.yaml \ 23 | "$VPC_STACK" \ 24 | "AvailabilityZones=${EC2_AVAILABILITY_ZONES}" \ 25 | "SubnetConfiguration=${EC2_AVAILABILITY_ZONES_COUNT} public subnets" 26 | fi 27 | 28 | ## Get Private Subnets and Vpc from VPC stack 29 | EC2_VPC_ID="$(aws cloudformation describe-stacks \ 30 | --stack-name "$VPC_STACK" \ 31 | --query 'Stacks[0].Outputs[?OutputKey==`Vpc`].OutputValue' \ 32 | --output text)" 33 | 34 | EC2_VPC_SUBNETS="$(aws cloudformation describe-stacks \ 35 | --stack-name "$VPC_STACK" \ 36 | --query 'Stacks[0].Outputs[?OutputKey==`PublicSubnets`].OutputValue' \ 37 | --output text)" 38 | 39 | if ! aws cloudformation describe-stacks --stack-name "${SPOT_FLEET_STACK}" &> /dev/null ; then 40 | echo "~~~ Creating ${SPOT_FLEET_STACK}" 41 | parfait create-stack \ 42 | -t templates/compute/spotfleet/template.yaml \ 43 | "${SPOT_FLEET_STACK}" \ 44 | "VPC=${EC2_VPC_ID?}" \ 45 | "Subnets=${EC2_VPC_SUBNETS}" \ 46 | "LambdaBucket=${LAMBDA_BUCKET}" 47 | fi 48 | 49 | if ! aws cloudformation describe-stacks --stack-name "${AGENT_STACK}" &> /dev/null ; then 50 | echo "~~~ Creating ${AGENT_STACK}" 51 | parfait create-stack \ 52 | -t templates/agent/template.yaml \ 53 | "${AGENT_STACK}" \ 54 | "ECSCluster=${SPOT_FLEET_STACK}" 55 | "AgentDockerImage=${DOCKER_IMAGE}" \ 56 | "BuildkiteQueue=${QUEUE}" \ 57 | "LambdaBucket=${LAMBDA_BUCKET}" 58 | fi 59 | 60 | -------------------------------------------------------------------------------- /scripts/develop.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -euo pipefail 3 | 4 | # Used in development of the stack. Updates the docker image and updates 5 | # stacks locally 6 | 7 | DOCKER_AGENT_IMAGE=lox24/buildkite-agent-ecs 8 | DOCKER_SOCKGUARD_IMAGE=lox24/sockguard-ecs 9 | 10 | SPOT_FLEET_STACK=${SPOT_FLEET_STACK:-buildkite-spotfleet-dev} 11 | AGENT_STACK=${AGENT_STACK:-"buildkite-agent-dev"} 12 | 13 | printf -- '\n--- Updating docker images\n' 14 | 15 | make docker docker-push \ 16 | "DOCKER_AGENT_TAG=${DOCKER_AGENT_IMAGE}" \ 17 | "DOCKER_SOCKGUARD_TAG=${DOCKER_SOCKGUARD_IMAGE}" 18 | 19 | docker_image="$(docker inspect \ 20 | --format='{{index .RepoDigests 0}}' \ 21 | ${DOCKER_AGENT_IMAGE}:latest)" 22 | 23 | printf -- '\n--- Updating spotfleet stack\n' 24 | 25 | parfait update-stack \ 26 | -t templates/compute/spotfleet/template.yaml \ 27 | "$SPOT_FLEET_STACK" \ 28 | "MinSize=1" \ 29 | "LambdaScheduleState=DISABLED" 30 | 31 | printf -- '\n--- Updating agent stack' 32 | 33 | parfait update-stack \ 34 | -t templates/agent/template.yaml \ 35 | "$AGENT_STACK" \ 36 | "AgentDockerImage=${docker_image}" \ 37 | "LambdaScheduleState=DISABLED" 38 | -------------------------------------------------------------------------------- /scripts/update.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -euo pipefail 3 | 4 | SPOT_FLEET_STACK=${SPOT_FLEET_STACK:-buildkite-spotfleet} 5 | AGENT_STACK=${AGENT_STACK:-"buildkite-agent-default"} 6 | 7 | echo '--- Updating spotfleet stack' 8 | parfait update-stack \ 9 | -t templates/compute/spotfleet/template.yaml \ 10 | "${SPOT_FLEET_STACK}" 11 | 12 | echo '--- Updating agent stack' 13 | parfait update-stack \ 14 | -t templates/agent/template.yaml \ 15 | "${AGENT_STACK}" 16 | -------------------------------------------------------------------------------- /templates/agent/README.md: -------------------------------------------------------------------------------- 1 | # Agent Stack 2 | 3 | ## Creating 4 | 5 | ```bash 6 | # Set an agent registration token in SSM 7 | aws ssm put-parameter --name "/buildkite/agent_token" --type String --value "xxx" 8 | 9 | # Create the stack 10 | aws cloudformation create-stack \ 11 | --output text \ 12 | --stack-name buildkite-agent-default \ 13 | --template-body "file://$PWD/templates/agent/template.yaml" \ 14 | --capabilities CAPABILITY_IAM \ 15 | --parameters \ 16 | "ParameterKey=LambdaBucket,ParameterValue=buildkite-aws-stack-ecs-dev" 17 | ``` 18 | -------------------------------------------------------------------------------- /templates/agent/template.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | Description: Agent Service for Buildkite Elastic Stack 3 | 4 | Parameters: 5 | BuildkiteAgentToken: 6 | Description: Buildkite Agent Registration token 7 | Type: 'AWS::SSM::Parameter::Value' 8 | Default: /buildkite/agent_token 9 | 10 | BuildkiteQueue: 11 | Description: Queue name that agents will use, targeted in pipeline steps using "queue={value}" 12 | Type: String 13 | Default: default 14 | MinLength: 1 15 | 16 | AgentDockerImage: 17 | Type: String 18 | Default: "lox24/buildkite-agent-ecs" 19 | 20 | DockerImage: 21 | Type: String 22 | Default: "docker:stable-dind" 23 | 24 | ECSCluster: 25 | Description: The name of the ECS cluster to create a service on 26 | Type: String 27 | 28 | LambdaBucket: 29 | Type: String 30 | Default: "buildkite-aws-stack-ecs-dev" 31 | 32 | LambdaObjectVersion: 33 | Type: String 34 | Default: "naqG4Q5Li7Q03hH_OGEKAHc7pKCn2g3A" 35 | 36 | LambdaScheduleState: 37 | Type: String 38 | Default: "ENABLED" 39 | 40 | Resources: 41 | ECSExecutionRole: 42 | Type: AWS::IAM::Role 43 | Properties: 44 | Path: / 45 | AssumeRolePolicyDocument: 46 | Version: 2012-10-17 47 | Statement: 48 | - Effect: Allow 49 | Principal: 50 | Service: ecs-tasks.amazonaws.com 51 | Action: sts:AssumeRole 52 | ManagedPolicyArns: 53 | - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy 54 | 55 | ECSLogGroup: 56 | Type: "AWS::Logs::LogGroup" 57 | Properties: 58 | RetentionInDays: 7 59 | 60 | ECSAgentTaskDefinition: 61 | Type: "AWS::ECS::TaskDefinition" 62 | Properties: 63 | NetworkMode: bridge 64 | ExecutionRoleArn: !Ref ECSExecutionRole 65 | Cpu: "512" 66 | Memory: "256" 67 | ContainerDefinitions: 68 | - Name: buildkite-agent 69 | Image: !Ref AgentDockerImage 70 | Essential: true 71 | Command: ["start"] 72 | Environment: 73 | - Name: BUILDKITE_AGENT_TOKEN 74 | Value: !Ref BuildkiteAgentToken 75 | - Name: BUILDKITE_AGENT_TAGS 76 | Value: !Sub "queue=${BuildkiteQueue}" 77 | - Name: AWS_REGION 78 | Value: !Ref AWS::Region 79 | LogConfiguration: 80 | LogDriver: "awslogs" 81 | Options: 82 | "awslogs-group": !Ref ECSLogGroup 83 | "awslogs-region": !Ref AWS::Region 84 | "awslogs-stream-prefix": "agent" 85 | MountPoints: 86 | - SourceVolume: docker-sock 87 | ContainerPath: /var/run/docker.sock 88 | - SourceVolume: buildkite-builds 89 | ContainerPath: /buildkite/builds 90 | - Name: docker 91 | Image: !Ref DockerImage 92 | Essential: false 93 | LogConfiguration: 94 | LogDriver: "awslogs" 95 | Options: 96 | "awslogs-group": !Ref ECSLogGroup 97 | "awslogs-region": !Ref AWS::Region 98 | "awslogs-stream-prefix": "docker" 99 | MountPoints: 100 | - SourceVolume: docker-sock 101 | ContainerPath: /var/run/docker.sock 102 | Volumes: 103 | - Name: docker-sock 104 | Host: 105 | SourcePath: /var/run/docker.sock 106 | - Name: buildkite-builds 107 | Host: 108 | SourcePath: /buildkite/builds 109 | 110 | ECSAgentService: 111 | Type: AWS::ECS::Service 112 | Properties: 113 | Cluster: !Ref ECSCluster 114 | DesiredCount: 0 115 | TaskDefinition: !Ref ECSAgentTaskDefinition 116 | PlacementStrategies: 117 | - Type: binpack 118 | Field: cpu 119 | 120 | ScheduledRule: 121 | Type: "AWS::Events::Rule" 122 | Properties: 123 | Description: "ScheduledRule" 124 | ScheduleExpression: "rate(1 minute)" 125 | State: !Ref LambdaScheduleState 126 | Targets: 127 | - Arn: !GetAtt ECSServiceScalerFunction.Arn 128 | Id: "ECSServiceScalerFunction" 129 | 130 | PermissionForEventsToInvokeLambda: 131 | Type: "AWS::Lambda::Permission" 132 | Properties: 133 | FunctionName: !Ref ECSServiceScalerFunction 134 | Action: "lambda:InvokeFunction" 135 | Principal: "events.amazonaws.com" 136 | SourceArn: !GetAtt ScheduledRule.Arn 137 | 138 | ECSServiceScalerLambdaExecutionRole: 139 | Type: AWS::IAM::Role 140 | Properties: 141 | Path: "/" 142 | AssumeRolePolicyDocument: 143 | Version: '2012-10-17' 144 | Statement: 145 | - Effect: Allow 146 | Principal: 147 | Service: 148 | - lambda.amazonaws.com 149 | Action: 150 | - sts:AssumeRole 151 | ManagedPolicyArns: 152 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 153 | Policies: 154 | - PolicyName: DescribeECSService 155 | PolicyDocument: 156 | Version: '2012-10-17' 157 | Statement: 158 | - Effect: Allow 159 | Action: 160 | - ecs:Describe* 161 | Resource: '*' 162 | - PolicyName: UpdateECSService 163 | PolicyDocument: 164 | Version: '2012-10-17' 165 | Statement: 166 | - Effect: Allow 167 | Action: 168 | - ecs:UpdateService 169 | - ecs:DescribeServices 170 | Resource: '*' 171 | 172 | # This mirrors the group that would be created by the lambda, but enforces 173 | # a retention period and also ensures it's removed when the stack is removed 174 | ECSServiceScalerLogGroup: 175 | Type: "AWS::Logs::LogGroup" 176 | Properties: 177 | LogGroupName: !Join ["/", ["/aws/lambda", !Ref ECSServiceScalerFunction]] 178 | RetentionInDays: 1 179 | 180 | ECSServiceScalerFunction: 181 | Type: AWS::Lambda::Function 182 | Properties: 183 | Code: 184 | S3Bucket: !Ref LambdaBucket 185 | S3Key: "ecs-service-scaler.zip" 186 | S3ObjectVersion: !Ref LambdaObjectVersion 187 | Role: !GetAtt ECSServiceScalerLambdaExecutionRole.Arn 188 | Timeout: 120 189 | Handler: handler 190 | Runtime: go1.x 191 | MemorySize: 128 192 | Environment: 193 | Variables: 194 | BUILDKITE_TOKEN: !Ref BuildkiteAgentToken 195 | BUILDKITE_QUEUE: !Ref BuildkiteQueue 196 | ECS_CLUSTER: !Ref ECSCluster 197 | ECS_SERVICE: !Ref ECSAgentService 198 | LAMBDA_TIMEOUT: 1m 199 | LAMBDA_INTERVAL: 10s 200 | 201 | Outputs: 202 | AgentService: 203 | Description: The ECS Service for the Agent 204 | Value: !Ref ECSAgentService 205 | -------------------------------------------------------------------------------- /templates/compute/spotfleet/README.md: -------------------------------------------------------------------------------- 1 | # Spotfleet Compute Stack 2 | 3 | ## Creating 4 | 5 | ```bash 6 | # Create the stack 7 | aws cloudformation create-stack \ 8 | --output text \ 9 | --stack-name buildkite-spotfleet-default \ 10 | --template-body "file://$PWD/templates/compute/spotfleet/template.yaml" \ 11 | --capabilities CAPABILITY_IAM \ 12 | --parameters \ 13 | "ParameterKey=VPC,ParameterValue=${EC2_VPC_ID?}" \ 14 | "ParameterKey=Subnets,ParameterValue=${EC2_VPC_SUBNETS//,/\\,}" \ 15 | "ParameterKey=ECSCluster,ParameterValue=buildkite-agent-default" 16 | ``` 17 | -------------------------------------------------------------------------------- /templates/compute/spotfleet/template.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | Description: Spot Fleet for Buildkite Elastic Stack 3 | 4 | Mappings: 5 | # These are the latest ECS optimized AMIs as of November 2018: 6 | # 7 | # amzn2-ami-ecs-hvm-2.0.20181112-x86_64-ebs 8 | # ECS agent: 1.22.0 9 | # Docker: 18.06.1-ce 10 | # ecs-init: 1.22.0-1 11 | # 12 | # See https://docs.aws.amazon.com/AmazonECS/latest/developerguide/al2ami.html 13 | 14 | AWSRegionToAMI: 15 | us-east-2: 16 | AMI: ami-037a92bf1efdb11a2 17 | us-east-1: 18 | AMI: ami-0a6b7e0cc0b1f464f 19 | us-west-2: 20 | AMI: ami-0c1f4871ebaae6d86 21 | us-west-1: 22 | AMI: ami-0184f498956de7db5 23 | eu-west-3: 24 | AMI: ami-0caadc4f0db31a303 25 | eu-west-2: 26 | AMI: ami-0b5225210a12d9951 27 | eu-west-1: 28 | AMI: ami-0acc9f8be17a41897 29 | eu-central-1: 30 | AMI: ami-055aa9664ef169e25 31 | ap-northeast-2: 32 | AMI: ami-0bdc871079baf9649 33 | ap-northeast-1: 34 | AMI: ami-0c38293d60d98af86 35 | ap-southeast-2: 36 | AMI: ami-0eed1c915ea891aca 37 | ap-southeast-1: 38 | AMI: ami-0e28ff4e3f1776d86 39 | ca-central-1: 40 | AMI: ami-02c80e9173258d289 41 | ap-south-1: 42 | AMI: ami-0b7c3be99909df6ef 43 | sa-east-1: 44 | AMI: ami-078146697425f25a7 45 | us-gov-est-1: 46 | AMI: ami-31b5d150 47 | 48 | Parameters: 49 | VPC: 50 | Description: The VPC to deploy SpotFleet into 51 | Type: AWS::EC2::VPC::Id 52 | 53 | Subnets: 54 | Description: The subnets of a VPC to deploy SpotFleet into 55 | Type: List 56 | 57 | MaxSize: 58 | Description: The maximum capacity for the Spot fleet 59 | Type: Number 60 | Default: 50 61 | 62 | MinSize: 63 | Description: The minimum capacity for the Spot fleet 64 | Type: Number 65 | Default: 0 66 | 67 | KeyName: 68 | Type: String 69 | Default: "" 70 | 71 | LambdaBucket: 72 | Type: String 73 | Default: "buildkite-aws-stack-ecs-dev" 74 | 75 | LambdaObjectVersion: 76 | Type: String 77 | Default: "zLuyHtrJoI4cmW0ncXb5eEpUTdjvgwXS" 78 | 79 | LambdaScheduleState: 80 | Type: String 81 | Default: "ENABLED" 82 | 83 | Conditions: 84 | HasKeyName: !Not [ !Equals [ !Ref KeyName, "" ] ] 85 | 86 | Resources: 87 | ECSSecurityGroup: 88 | Type: AWS::EC2::SecurityGroup 89 | Properties: 90 | GroupDescription: "Security group for ECS instances" 91 | VpcId: !Ref VPC 92 | 93 | ECSSecurityGroupIngress: 94 | Type: AWS::EC2::SecurityGroupIngress 95 | Properties: 96 | GroupId: !Ref ECSSecurityGroup 97 | SourceSecurityGroupId: !Ref ECSSecurityGroup 98 | IpProtocol: tcp 99 | FromPort: 0 100 | ToPort: 65535 101 | 102 | SpotFleetIAMRole: 103 | Type: AWS::IAM::Role 104 | Properties: 105 | AssumeRolePolicyDocument: | 106 | { 107 | "Statement": [{ 108 | "Action": "sts:AssumeRole", 109 | "Effect": "Allow", 110 | "Principal": { 111 | "Service": "spotfleet.amazonaws.com" 112 | } 113 | }] 114 | } 115 | Path: / 116 | ManagedPolicyArns: 117 | - arn:aws:iam::aws:policy/service-role/AmazonEC2SpotFleetTaggingRole 118 | 119 | ECSRole: 120 | Type: AWS::IAM::Role 121 | Properties: 122 | Path: / 123 | AssumeRolePolicyDocument: | 124 | { 125 | "Statement": [{ 126 | "Action": "sts:AssumeRole", 127 | "Effect": "Allow", 128 | "Principal": { 129 | "Service": "ec2.amazonaws.com" 130 | } 131 | }] 132 | } 133 | Policies: 134 | - PolicyName: ecs-service 135 | PolicyDocument: | 136 | { 137 | "Statement": [{ 138 | "Effect": "Allow", 139 | "Action": [ 140 | "ecs:CreateCluster", 141 | "ecs:DeregisterContainerInstance", 142 | "ecs:DiscoverPollEndpoint", 143 | "ecs:Poll", 144 | "ecs:RegisterContainerInstance", 145 | "ecs:StartTelemetrySession", 146 | "ecs:Submit*", 147 | "logs:CreateLogStream", 148 | "logs:PutLogEvents", 149 | "ecr:BatchCheckLayerAvailability", 150 | "ecr:BatchGetImage", 151 | "ecr:GetDownloadUrlForLayer", 152 | "ecr:GetAuthorizationToken" 153 | ], 154 | "Resource": "*" 155 | }] 156 | } 157 | 158 | ECSInstanceProfile: 159 | Type: AWS::IAM::InstanceProfile 160 | Properties: 161 | Path: / 162 | Roles: 163 | - !Ref ECSRole 164 | 165 | ECSLaunchTemplate: 166 | Type: AWS::EC2::LaunchTemplate 167 | Metadata: 168 | AWS::CloudFormation::Init: 169 | config: 170 | commands: 171 | 01_configure_ecs_agent: 172 | command: !Sub | 173 | echo ECS_CLUSTER=${ECSCluster} > /etc/ecs/ecs.config 174 | echo ECS_DISABLE_PRIVILEGED=false >> /etc/ecs/ecs.config 175 | echo ECS_AWSVPC_BLOCK_IMDS=true >> /etc/ecs/ecs.config 176 | echo ECS_ENABLE_TASK_IAM_ROLE=true >> /etc/ecs/ecs.config 177 | echo ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true >> /etc/ecs/ecs.config 178 | 02_configure_firewall: 179 | command: | 180 | yum install -y iptables-services 181 | iptables --insert FORWARD 1 --in-interface docker+ --destination 169.254.169.254/32 --jump DROP 182 | sysctl -w net.ipv4.conf.all.route_localnet=1 183 | iptables -t nat -A PREROUTING -p tcp -d 169.254.170.2 --dport 80 -j DNAT --to-destination 127.0.0.1:51679 184 | iptables -t nat -A OUTPUT -d 169.254.170.2 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679 185 | iptables-save | tee /etc/sysconfig/iptables 186 | systemctl enable --now iptables 187 | files: 188 | "/etc/cfn/cfn-hup.conf": 189 | mode: 000400 190 | owner: root 191 | group: root 192 | content: !Sub | 193 | [main] 194 | stack=${AWS::StackId} 195 | region=${AWS::Region} 196 | "/etc/cfn/hooks.d/cfn-auto-reloader.conf": 197 | content: !Sub | 198 | [cfn-auto-reloader-hook] 199 | triggers=post.update 200 | path=Resources.ContainerInstances.Metadata.AWS::CloudFormation::Init 201 | action=/opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource ECSLaunchTemplate 202 | services: 203 | sysvinit: 204 | cfn-hup: 205 | enabled: true 206 | ensureRunning: true 207 | files: 208 | - /etc/cfn/cfn-hup.conf 209 | - /etc/cfn/hooks.d/cfn-auto-reloader.conf 210 | Properties: 211 | LaunchTemplateData: 212 | NetworkInterfaces: 213 | - DeviceIndex: 0 214 | Groups: [ !GetAtt ECSSecurityGroup.GroupId ] 215 | KeyName: !If [HasKeyName, !Ref KeyName, !Ref "AWS::NoValue"] 216 | IamInstanceProfile: { Arn: !GetAtt ECSInstanceProfile.Arn } 217 | ImageId: !FindInMap [ AWSRegionToAMI, !Ref "AWS::Region", "AMI" ] 218 | UserData: 219 | "Fn::Base64": !Sub | 220 | #!/bin/bash 221 | yum install -y aws-cfn-bootstrap 222 | /opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource ECSLaunchTemplate 223 | /opt/aws/bin/cfn-signal -e $? --region ${AWS::Region} --stack ${AWS::StackName} --resource SpotFleet 224 | 225 | SpotFleet: 226 | Type: AWS::EC2::SpotFleet 227 | Properties: 228 | SpotFleetRequestConfigData: 229 | AllocationStrategy: lowestPrice 230 | IamFleetRole: !GetAtt SpotFleetIAMRole.Arn 231 | TargetCapacity: !Ref MinSize 232 | ReplaceUnhealthyInstances: true 233 | TerminateInstancesWithExpiration: true 234 | LaunchTemplateConfigs: 235 | - LaunchTemplateSpecification: 236 | LaunchTemplateId: !Ref ECSLaunchTemplate 237 | Version: !GetAtt "ECSLaunchTemplate.LatestVersionNumber" 238 | Overrides: 239 | - WeightedCapacity: 4 240 | InstanceType: m5.large 241 | SubnetId: !Join [ ',', !Ref Subnets ] 242 | - LaunchTemplateSpecification: 243 | LaunchTemplateId: !Ref ECSLaunchTemplate 244 | Version: !GetAtt "ECSLaunchTemplate.LatestVersionNumber" 245 | Overrides: 246 | - WeightedCapacity: 8 247 | InstanceType: m5.xlarge 248 | SubnetId: !Join [ ',', !Ref Subnets ] 249 | - LaunchTemplateSpecification: 250 | LaunchTemplateId: !Ref ECSLaunchTemplate 251 | Version: !GetAtt "ECSLaunchTemplate.LatestVersionNumber" 252 | Overrides: 253 | - WeightedCapacity: 16 254 | InstanceType: m5.2xlarge 255 | SubnetId: !Join [ ',', !Ref Subnets ] 256 | - LaunchTemplateSpecification: 257 | LaunchTemplateId: !Ref ECSLaunchTemplate 258 | Version: !GetAtt "ECSLaunchTemplate.LatestVersionNumber" 259 | Overrides: 260 | - WeightedCapacity: 32 261 | InstanceType: m5.4xlarge 262 | SubnetId: !Join [ ',', !Ref Subnets ] 263 | - LaunchTemplateSpecification: 264 | LaunchTemplateId: !Ref ECSLaunchTemplate 265 | Version: !GetAtt "ECSLaunchTemplate.LatestVersionNumber" 266 | Overrides: 267 | - WeightedCapacity: 4 268 | InstanceType: m4.large 269 | SubnetId: !Join [ ',', !Ref Subnets ] 270 | - LaunchTemplateSpecification: 271 | LaunchTemplateId: !Ref ECSLaunchTemplate 272 | Version: !GetAtt "ECSLaunchTemplate.LatestVersionNumber" 273 | Overrides: 274 | - WeightedCapacity: 8 275 | InstanceType: m4.xlarge 276 | SubnetId: !Join [ ',', !Ref Subnets ] 277 | - LaunchTemplateSpecification: 278 | LaunchTemplateId: !Ref ECSLaunchTemplate 279 | Version: !GetAtt "ECSLaunchTemplate.LatestVersionNumber" 280 | Overrides: 281 | - WeightedCapacity: 16 282 | InstanceType: m4.2xlarge 283 | SubnetId: !Join [ ',', !Ref Subnets ] 284 | - LaunchTemplateSpecification: 285 | LaunchTemplateId: !Ref ECSLaunchTemplate 286 | Version: !GetAtt "ECSLaunchTemplate.LatestVersionNumber" 287 | Overrides: 288 | - WeightedCapacity: 32 289 | InstanceType: m4.4xlarge 290 | SubnetId: !Join [ ',', !Ref Subnets ] 291 | 292 | ECSCluster: 293 | Type: AWS::ECS::Cluster 294 | Properties: 295 | ClusterName: !Ref AWS::StackName 296 | 297 | ScheduledRule: 298 | Type: "AWS::Events::Rule" 299 | Properties: 300 | Description: "ScheduledRule" 301 | ScheduleExpression: "rate(1 minute)" 302 | State: !Ref LambdaScheduleState 303 | Targets: 304 | - Arn: !GetAtt ECSSpotFleetScalerFunction.Arn 305 | Id: "ECSSpotFleetScalerFunction" 306 | 307 | PermissionForEventsToInvokeLambda: 308 | Type: "AWS::Lambda::Permission" 309 | Properties: 310 | FunctionName: !Ref ECSSpotFleetScalerFunction 311 | Action: "lambda:InvokeFunction" 312 | Principal: "events.amazonaws.com" 313 | SourceArn: !GetAtt ScheduledRule.Arn 314 | 315 | ECSSpotFleetScalerLambdaExecutionRole: 316 | Type: AWS::IAM::Role 317 | Properties: 318 | Path: "/" 319 | AssumeRolePolicyDocument: 320 | Version: '2012-10-17' 321 | Statement: 322 | - Effect: Allow 323 | Principal: 324 | Service: 325 | - lambda.amazonaws.com 326 | Action: 327 | - sts:AssumeRole 328 | ManagedPolicyArns: 329 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 330 | Policies: 331 | - PolicyName: DescribeECSResources 332 | PolicyDocument: 333 | Version: '2012-10-17' 334 | Statement: 335 | - Effect: Allow 336 | Action: 337 | - ecs:Describe* 338 | - ecs:List* 339 | Resource: '*' 340 | - PolicyName: ModifySpotFleet 341 | PolicyDocument: 342 | Version: '2012-10-17' 343 | Statement: 344 | - Effect: Allow 345 | Action: 346 | - ec2:DescribeSpotFleetRequests 347 | - ec2:ModifySpotFleetRequest 348 | Resource: '*' 349 | 350 | # This mirrors the group that would be created by the lambda, but enforces 351 | # a retention period and also ensures it's removed when the stack is removed 352 | ECSSpotFleetScalerLogGroup: 353 | Type: "AWS::Logs::LogGroup" 354 | Properties: 355 | LogGroupName: !Join ["/", ["/aws/lambda", !Ref ECSSpotFleetScalerFunction]] 356 | RetentionInDays: 1 357 | 358 | ECSSpotFleetScalerFunction: 359 | Type: AWS::Lambda::Function 360 | Properties: 361 | Code: 362 | S3Bucket: !Ref LambdaBucket 363 | S3Key: "ecs-spotfleet-scaler.zip" 364 | S3ObjectVersion: !Ref LambdaObjectVersion 365 | Role: !GetAtt ECSSpotFleetScalerLambdaExecutionRole.Arn 366 | Timeout: 120 367 | Handler: handler 368 | Runtime: go1.x 369 | MemorySize: 128 370 | Environment: 371 | Variables: 372 | ECS_CLUSTER: !Ref ECSCluster 373 | SPOT_FLEET: !Ref SpotFleet 374 | MIN_SIZE: !Ref MinSize 375 | MAX_SIZE: !Ref MaxSize 376 | LAMBDA_TIMEOUT: 1m 377 | LAMBDA_INTERVAL: 20s 378 | 379 | Outputs: 380 | SpotFleet: 381 | Description: The SpotFleet created 382 | Value: !Ref SpotFleet 383 | 384 | ECSCluster: 385 | Description: The ECS Cluster created 386 | Value: !Ref ECSCluster 387 | -------------------------------------------------------------------------------- /templates/vpc/README.md: -------------------------------------------------------------------------------- 1 | # Elastic Stack VPC Template 2 | 3 | The VPC stack creates a VPC with as many availabilty zones as are available in your region to optimize spot pricing. It makes the following configurations available: 4 | 5 | * 2 public subnets 6 | * 3 public subnets 7 | * 4 public subnets 8 | * 5 public subnets 9 | * 6 public subnets 10 | * 2 private subnets + 2 public subnets with NAT Gateways 11 | * 3 private subnets + 2 public subnets with NAT Gateways 12 | * 4 private subnets + 2 public subnets with NAT Gateways 13 | * 5 private subnets + 2 public subnets with NAT Gateways 14 | * 6 private subnets + 2 public subnets with NAT Gateways 15 | 16 | Note that NAT Gateways are kind of expensive and are charged at $0.045-0.095 an hour. 17 | 18 | ## Creating 19 | 20 | ```bash 21 | ## Figure out what availability zones are available 22 | export EC2_AVAILABILITY_ZONES=$(aws ec2 describe-availability-zones \ 23 | --query 'AvailabilityZones[?State==`available`].ZoneName' \ 24 | --output text | sed -E -e 's/[[:blank:]]+/,/g') 25 | 26 | export EC2_AVAILABILITY_ZONES_COUNT=$(awk -F, '{print NF-1}' <<< "$EC2_AVAILABILITY_ZONES") 27 | 28 | ## Create a VPC stack 29 | aws cloudformation create-stack \ 30 | --output text \ 31 | --stack-name buildkite-vpc \ 32 | --template-body "file://$PWD/templates/vpc/template.yaml" \ 33 | --parameters \ 34 | "ParameterKey=AvailabilityZones,ParameterValue=${EC2_AVAILABILITY_ZONES//,/\\\\\\,}" \ 35 | "ParameterKey=SubnetConfiguration,ParameterValue=${EC2_AVAILABILITY_ZONES_COUNT} public subnets" 36 | 37 | ## Get Private Subnets and Vpc from VPC stack 38 | export EC2_VPC_ID="$(aws cloudformation describe-stacks \ 39 | --stack-name buildkite-vpc \ 40 | --query 'Stacks[0].Outputs[?OutputKey==`Vpc`].OutputValue' \ 41 | --output text)" 42 | 43 | export EC2_VPC_SUBNETS="$(aws cloudformation describe-stacks \ 44 | --stack-name buildkite-vpc \ 45 | --query 'Stacks[0].Outputs[?OutputKey==`PublicSubnets`].OutputValue' \ 46 | --output text)" 47 | ``` 48 | -------------------------------------------------------------------------------- /templates/vpc/template.yaml: -------------------------------------------------------------------------------- 1 | Description: VPC for Buildkite Elastic Stack 2 | 3 | Parameters: 4 | AvailabilityZones: 5 | Type: List 6 | Description: The Availability Zones to use. 7 | MinLength: 10 8 | 9 | SubnetConfiguration: 10 | Type: String 11 | Description: "How to configure your subnets, must match the availability zones you provided. Note: NAT Gateways are charged at $0.045-0.095 an hour." 12 | Default: "2 public subnets" 13 | AllowedValues: 14 | - 2 public subnets 15 | - 3 public subnets 16 | - 4 public subnets 17 | - 5 public subnets 18 | - 6 public subnets 19 | - 2 private subnets + 2 public subnets with NAT Gateways 20 | - 3 private subnets + 2 public subnets with NAT Gateways 21 | - 4 private subnets + 2 public subnets with NAT Gateways 22 | - 5 private subnets + 2 public subnets with NAT Gateways 23 | - 6 private subnets + 2 public subnets with NAT Gateways 24 | 25 | Conditions: 26 | HasPrivateSubnets: !Or [ 27 | !Equals [ !Ref SubnetConfiguration, "2 private subnets + 2 public subnets with NAT Gateways" ], 28 | !Equals [ !Ref SubnetConfiguration, "3 private subnets + 2 public subnets with NAT Gateways" ], 29 | !Equals [ !Ref SubnetConfiguration, "4 private subnets + 2 public subnets with NAT Gateways" ], 30 | !Equals [ !Ref SubnetConfiguration, "5 private subnets + 2 public subnets with NAT Gateways" ], 31 | !Equals [ !Ref SubnetConfiguration, "6 private subnets + 2 public subnets with NAT Gateways" ]] 32 | 33 | HasTwoAZs: !Or [ 34 | !Equals [ !Ref SubnetConfiguration, "2 public subnets" ], 35 | !Equals [ !Ref SubnetConfiguration, "2 private subnets + 2 public subnets with NAT Gateways" ]] 36 | 37 | HasThreeAZs: !Or [ 38 | !Equals [ !Ref SubnetConfiguration, "3 public subnets" ], 39 | !Equals [ !Ref SubnetConfiguration, "3 private subnets + 2 public subnets with NAT Gateways" ]] 40 | 41 | HasFourAZs: !Or [ 42 | !Equals [ !Ref SubnetConfiguration, "4 public subnets" ], 43 | !Equals [ !Ref SubnetConfiguration, "4 private subnets + 2 public subnets with NAT Gateways" ]] 44 | 45 | HasFiveAZs: !Or [ 46 | !Equals [ !Ref SubnetConfiguration, "5 public subnets" ], 47 | !Equals [ !Ref SubnetConfiguration, "5 private subnets + 2 public subnets with NAT Gateways" ]] 48 | 49 | HasSixAZs: !Or [ 50 | !Equals [ !Ref SubnetConfiguration, "6 public subnets" ], 51 | !Equals [ !Ref SubnetConfiguration, "6 private subnets + 2 public subnets with NAT Gateways" ]] 52 | 53 | HasNatGateway1: { Condition: HasPrivateSubnets } 54 | HasNatGateway2: { Condition: HasPrivateSubnets } 55 | 56 | HasPublicSubnet3: !And [ !Not [ { Condition: HasPrivateSubnets } ], !Or [ { Condition: HasThreeAZs }, { Condition: HasFourAZs }, { Condition: HasFiveAZs }, { Condition: HasSixAZs } ] ] 57 | HasPublicSubnet4: !And [ !Not [ { Condition: HasPrivateSubnets } ], !Or [ { Condition: HasFourAZs }, { Condition: HasFiveAZs }, { Condition: HasSixAZs } ] ] 58 | HasPublicSubnet5: !And [ !Not [ { Condition: HasPrivateSubnets } ], !Or [ { Condition: HasFiveAZs }, { Condition: HasSixAZs } ] ] 59 | HasPublicSubnet6: !And [ !Not [ { Condition: HasPrivateSubnets } ], { Condition: HasSixAZs } ] 60 | 61 | HasPrivateSubnet1: { Condition: HasPrivateSubnets } 62 | HasPrivateSubnet2: !And [ { Condition: HasPrivateSubnets }, !Or [ { Condition: HasTwoAZs }, { Condition: HasThreeAZs }, { Condition: HasFourAZs }, { Condition: HasFiveAZs }, { Condition: HasSixAZs } ] ] 63 | HasPrivateSubnet3: !And [ { Condition: HasPrivateSubnets }, !Or [ { Condition: HasThreeAZs }, { Condition: HasFourAZs }, { Condition: HasFiveAZs }, { Condition: HasSixAZs } ] ] 64 | HasPrivateSubnet4: !And [ { Condition: HasPrivateSubnets }, !Or [ { Condition: HasFourAZs }, { Condition: HasFiveAZs }, { Condition: HasSixAZs } ] ] 65 | HasPrivateSubnet5: !And [ { Condition: HasPrivateSubnets }, !Or [ { Condition: HasFiveAZs }, { Condition: HasSixAZs } ] ] 66 | HasPrivateSubnet6: !And [ { Condition: HasPrivateSubnets }, { Condition: HasSixAZs } ] 67 | 68 | Resources: 69 | Vpc: 70 | Type: AWS::EC2::VPC 71 | Properties: 72 | CidrBlock: 10.0.0.0/16 73 | Tags: 74 | - Key: Name 75 | Value: !Ref AWS::StackName 76 | 77 | InternetGateway: 78 | Type: AWS::EC2::InternetGateway 79 | Properties: 80 | Tags: 81 | - Key: Name 82 | Value: !Ref AWS::StackName 83 | 84 | InternetGatewayAttachment: 85 | Type: AWS::EC2::VPCGatewayAttachment 86 | Properties: 87 | InternetGatewayId: !Ref InternetGateway 88 | VpcId: !Ref Vpc 89 | 90 | PublicSubnet1: 91 | Type: AWS::EC2::Subnet 92 | Properties: 93 | VpcId: !Ref Vpc 94 | AvailabilityZone: !Select [ 0, !Ref AvailabilityZones ] 95 | CidrBlock: 10.0.0.0/19 96 | MapPublicIpOnLaunch: true 97 | Tags: 98 | - Key: Name 99 | Value: !Sub ${AWS::StackName} Public Subnet (AZ1) 100 | 101 | PublicSubnet2: 102 | Type: AWS::EC2::Subnet 103 | Properties: 104 | VpcId: !Ref Vpc 105 | AvailabilityZone: !Select [ 1, !Ref AvailabilityZones ] 106 | CidrBlock: 10.0.32.0/19 107 | MapPublicIpOnLaunch: true 108 | Tags: 109 | - Key: Name 110 | Value: !Sub ${AWS::StackName} Public Subnet (AZ2) 111 | 112 | PublicSubnet3: 113 | Type: AWS::EC2::Subnet 114 | Condition: HasPublicSubnet3 115 | Properties: 116 | VpcId: !Ref Vpc 117 | AvailabilityZone: !Select [ 2, !Ref AvailabilityZones ] 118 | CidrBlock: 10.0.128.0/19 119 | MapPublicIpOnLaunch: true 120 | Tags: 121 | - Key: Name 122 | Value: !Sub ${AWS::StackName} Public Subnet (AZ3) 123 | 124 | PublicSubnet4: 125 | Type: AWS::EC2::Subnet 126 | Condition: HasPublicSubnet4 127 | Properties: 128 | VpcId: !Ref Vpc 129 | AvailabilityZone: !Select [ 3, !Ref AvailabilityZones ] 130 | CidrBlock: 10.0.160.0/19 131 | MapPublicIpOnLaunch: true 132 | Tags: 133 | - Key: Name 134 | Value: !Sub ${AWS::StackName} Public Subnet (AZ4) 135 | 136 | PublicSubnet5: 137 | Type: AWS::EC2::Subnet 138 | Condition: HasPublicSubnet5 139 | Properties: 140 | VpcId: !Ref Vpc 141 | AvailabilityZone: !Select [ 4, !Ref AvailabilityZones ] 142 | CidrBlock: 10.0.192.0/19 143 | MapPublicIpOnLaunch: true 144 | Tags: 145 | - Key: Name 146 | Value: !Sub ${AWS::StackName} Public Subnet (AZ5) 147 | 148 | PublicSubnet6: 149 | Type: AWS::EC2::Subnet 150 | Condition: HasPublicSubnet6 151 | Properties: 152 | VpcId: !Ref Vpc 153 | AvailabilityZone: !Select [ 5, !Ref AvailabilityZones ] 154 | CidrBlock: 10.0.224.0/19 155 | MapPublicIpOnLaunch: true 156 | Tags: 157 | - Key: Name 158 | Value: !Sub ${AWS::StackName} Public Subnet (AZ6) 159 | 160 | NatGateway1EIP: 161 | Type: AWS::EC2::EIP 162 | Condition: HasNatGateway1 163 | DependsOn: InternetGatewayAttachment 164 | Properties: 165 | Domain: vpc 166 | 167 | NatGateway2EIP: 168 | Type: AWS::EC2::EIP 169 | Condition: HasNatGateway2 170 | DependsOn: InternetGatewayAttachment 171 | Properties: 172 | Domain: vpc 173 | 174 | NatGateway1: 175 | Type: AWS::EC2::NatGateway 176 | Condition: HasNatGateway1 177 | Properties: 178 | AllocationId: !GetAtt NatGateway1EIP.AllocationId 179 | SubnetId: !Ref PublicSubnet1 180 | 181 | NatGateway2: 182 | Type: AWS::EC2::NatGateway 183 | Condition: HasNatGateway2 184 | Properties: 185 | AllocationId: !GetAtt NatGateway2EIP.AllocationId 186 | SubnetId: !Ref PublicSubnet2 187 | 188 | PublicRouteTable: 189 | Type: AWS::EC2::RouteTable 190 | Properties: 191 | VpcId: !Ref Vpc 192 | Tags: 193 | - Key: Name 194 | Value: !Sub ${AWS::StackName} Public Routes 195 | 196 | DefaultPublicRoute: 197 | Type: AWS::EC2::Route 198 | DependsOn: InternetGatewayAttachment 199 | Properties: 200 | RouteTableId: !Ref PublicRouteTable 201 | DestinationCidrBlock: 0.0.0.0/0 202 | GatewayId: !Ref InternetGateway 203 | 204 | PublicSubnet1RouteTableAssociation: 205 | Type: AWS::EC2::SubnetRouteTableAssociation 206 | Properties: 207 | RouteTableId: !Ref PublicRouteTable 208 | SubnetId: !Ref PublicSubnet1 209 | 210 | PublicSubnet2RouteTableAssociation: 211 | Type: AWS::EC2::SubnetRouteTableAssociation 212 | Properties: 213 | RouteTableId: !Ref PublicRouteTable 214 | SubnetId: !Ref PublicSubnet2 215 | 216 | PublicSubnet3RouteTableAssociation: 217 | Type: AWS::EC2::SubnetRouteTableAssociation 218 | Condition: HasPublicSubnet3 219 | Properties: 220 | RouteTableId: !Ref PublicRouteTable 221 | SubnetId: !Ref PublicSubnet3 222 | 223 | PublicSubnet4RouteTableAssociation: 224 | Type: AWS::EC2::SubnetRouteTableAssociation 225 | Condition: HasPublicSubnet4 226 | Properties: 227 | RouteTableId: !Ref PublicRouteTable 228 | SubnetId: !Ref PublicSubnet4 229 | 230 | PublicSubnet5RouteTableAssociation: 231 | Type: AWS::EC2::SubnetRouteTableAssociation 232 | Condition: HasPublicSubnet5 233 | Properties: 234 | RouteTableId: !Ref PublicRouteTable 235 | SubnetId: !Ref PublicSubnet5 236 | 237 | PublicSubnet6RouteTableAssociation: 238 | Type: AWS::EC2::SubnetRouteTableAssociation 239 | Condition: HasPublicSubnet6 240 | Properties: 241 | RouteTableId: !Ref PublicRouteTable 242 | SubnetId: !Ref PublicSubnet6 243 | 244 | PrivateSubnet1: 245 | Type: AWS::EC2::Subnet 246 | Condition: HasPrivateSubnet1 247 | Properties: 248 | VpcId: !Ref Vpc 249 | AvailabilityZone: !Select [ 0, !Ref AvailabilityZones ] 250 | CidrBlock: 10.0.64.0/19 251 | MapPublicIpOnLaunch: false 252 | 253 | PrivateSubnet2: 254 | Type: AWS::EC2::Subnet 255 | Condition: HasPrivateSubnet2 256 | Properties: 257 | VpcId: !Ref Vpc 258 | AvailabilityZone: !Select [ 1, !Ref AvailabilityZones ] 259 | CidrBlock: 10.0.96.0/19 260 | MapPublicIpOnLaunch: false 261 | Tags: 262 | - Key: Name 263 | Value: !Sub ${AWS::StackName} Private Subnet (AZ2) 264 | 265 | PrivateSubnet3: 266 | Type: AWS::EC2::Subnet 267 | Condition: HasPrivateSubnet3 268 | Properties: 269 | VpcId: !Ref Vpc 270 | AvailabilityZone: !Select [ 2, !Ref AvailabilityZones ] 271 | CidrBlock: 10.0.128.0/19 272 | MapPublicIpOnLaunch: false 273 | Tags: 274 | - Key: Name 275 | Value: !Sub ${AWS::StackName} Private Subnet (AZ3) 276 | 277 | PrivateSubnet4: 278 | Type: AWS::EC2::Subnet 279 | Condition: HasPrivateSubnet4 280 | Properties: 281 | VpcId: !Ref Vpc 282 | AvailabilityZone: !Select [ 3, !Ref AvailabilityZones ] 283 | CidrBlock: 10.0.160.0/19 284 | MapPublicIpOnLaunch: false 285 | Tags: 286 | - Key: Name 287 | Value: !Sub ${AWS::StackName} Private Subnet (AZ4) 288 | 289 | PrivateSubnet5: 290 | Type: AWS::EC2::Subnet 291 | Condition: HasPrivateSubnet5 292 | Properties: 293 | VpcId: !Ref Vpc 294 | AvailabilityZone: !Select [ 4, !Ref AvailabilityZones ] 295 | CidrBlock: 10.0.192.0/19 296 | MapPublicIpOnLaunch: false 297 | Tags: 298 | - Key: Name 299 | Value: !Sub ${AWS::StackName} Private Subnet (AZ5) 300 | 301 | PrivateSubnet6: 302 | Type: AWS::EC2::Subnet 303 | Condition: HasPrivateSubnet6 304 | Properties: 305 | VpcId: !Ref Vpc 306 | AvailabilityZone: !Select [ 5, !Ref AvailabilityZones ] 307 | CidrBlock: 10.0.224.0/19 308 | MapPublicIpOnLaunch: false 309 | Tags: 310 | - Key: Name 311 | Value: !Sub ${AWS::StackName} Private Subnet (AZ6) 312 | 313 | PrivateRouteTable1: 314 | Type: AWS::EC2::RouteTable 315 | Condition: HasPrivateSubnet1 316 | Properties: 317 | VpcId: !Ref Vpc 318 | Tags: 319 | - Key: Name 320 | Value: !Sub ${AWS::StackName} Private Routes (AZ1) 321 | 322 | PrivateRouteTable2: 323 | Type: AWS::EC2::RouteTable 324 | Condition: HasPrivateSubnet2 325 | Properties: 326 | VpcId: !Ref Vpc 327 | Tags: 328 | - Key: Name 329 | Value: !Sub ${AWS::StackName} Private Routes (AZ2) 330 | 331 | PrivateRouteTable3: 332 | Type: AWS::EC2::RouteTable 333 | Condition: HasPrivateSubnet3 334 | Properties: 335 | VpcId: !Ref Vpc 336 | Tags: 337 | - Key: Name 338 | Value: !Sub ${AWS::StackName} Private Routes (AZ3) 339 | 340 | PrivateRouteTable4: 341 | Type: AWS::EC2::RouteTable 342 | Condition: HasPrivateSubnet4 343 | Properties: 344 | VpcId: !Ref Vpc 345 | Tags: 346 | - Key: Name 347 | Value: !Sub ${AWS::StackName} Private Routes (AZ4) 348 | 349 | PrivateRouteTable5: 350 | Type: AWS::EC2::RouteTable 351 | Condition: HasPrivateSubnet5 352 | Properties: 353 | VpcId: !Ref Vpc 354 | Tags: 355 | - Key: Name 356 | Value: !Sub ${AWS::StackName} Private Routes (AZ5) 357 | 358 | PrivateRouteTable6: 359 | Type: AWS::EC2::RouteTable 360 | Condition: HasPrivateSubnet6 361 | Properties: 362 | VpcId: !Ref Vpc 363 | Tags: 364 | - Key: Name 365 | Value: !Sub ${AWS::StackName} Private Routes (AZ6) 366 | 367 | DefaultPrivateRoute1: 368 | Type: AWS::EC2::Route 369 | Condition: HasPrivateSubnet1 370 | Properties: 371 | RouteTableId: !Ref PrivateRouteTable1 372 | DestinationCidrBlock: 0.0.0.0/0 373 | NatGatewayId: !Ref NatGateway1 374 | 375 | DefaultPrivateRoute2: 376 | Type: AWS::EC2::Route 377 | Condition: HasPrivateSubnet2 378 | Properties: 379 | RouteTableId: !Ref PrivateRouteTable2 380 | DestinationCidrBlock: 0.0.0.0/0 381 | NatGatewayId: !Ref NatGateway2 382 | 383 | DefaultPrivateRoute3: 384 | Type: AWS::EC2::Route 385 | Condition: HasPrivateSubnet3 386 | Properties: 387 | RouteTableId: !Ref PrivateRouteTable3 388 | DestinationCidrBlock: 0.0.0.0/0 389 | NatGatewayId: !Ref NatGateway1 390 | 391 | DefaultPrivateRoute4: 392 | Type: AWS::EC2::Route 393 | Condition: HasPrivateSubnet4 394 | Properties: 395 | RouteTableId: !Ref PrivateRouteTable4 396 | DestinationCidrBlock: 0.0.0.0/0 397 | NatGatewayId: !Ref NatGateway2 398 | 399 | DefaultPrivateRoute5: 400 | Type: AWS::EC2::Route 401 | Condition: HasPrivateSubnet5 402 | Properties: 403 | RouteTableId: !Ref PrivateRouteTable5 404 | DestinationCidrBlock: 0.0.0.0/0 405 | NatGatewayId: !Ref NatGateway1 406 | 407 | DefaultPrivateRoute6: 408 | Type: AWS::EC2::Route 409 | Condition: HasPrivateSubnet6 410 | Properties: 411 | RouteTableId: !Ref PrivateRouteTable6 412 | DestinationCidrBlock: 0.0.0.0/0 413 | NatGatewayId: !Ref NatGateway2 414 | 415 | PrivateSubnet1RouteTableAssociation: 416 | Type: AWS::EC2::SubnetRouteTableAssociation 417 | Condition: HasPrivateSubnet1 418 | Properties: 419 | RouteTableId: !Ref PrivateRouteTable1 420 | SubnetId: !Ref PrivateSubnet1 421 | 422 | PrivateSubnet2RouteTableAssociation: 423 | Type: AWS::EC2::SubnetRouteTableAssociation 424 | Condition: HasPrivateSubnet2 425 | Properties: 426 | RouteTableId: !Ref PrivateRouteTable2 427 | SubnetId: !Ref PrivateSubnet2 428 | 429 | PrivateSubnet3RouteTableAssociation: 430 | Type: AWS::EC2::SubnetRouteTableAssociation 431 | Condition: HasPrivateSubnet3 432 | Properties: 433 | RouteTableId: !Ref PrivateRouteTable3 434 | SubnetId: !Ref PrivateSubnet3 435 | 436 | PrivateSubnet4RouteTableAssociation: 437 | Type: AWS::EC2::SubnetRouteTableAssociation 438 | Condition: HasPrivateSubnet4 439 | Properties: 440 | RouteTableId: !Ref PrivateRouteTable4 441 | SubnetId: !Ref PrivateSubnet4 442 | 443 | PrivateSubnet5RouteTableAssociation: 444 | Type: AWS::EC2::SubnetRouteTableAssociation 445 | Condition: HasPrivateSubnet5 446 | Properties: 447 | RouteTableId: !Ref PrivateRouteTable5 448 | SubnetId: !Ref PrivateSubnet5 449 | 450 | PrivateSubnet6RouteTableAssociation: 451 | Type: AWS::EC2::SubnetRouteTableAssociation 452 | Condition: HasPrivateSubnet6 453 | Properties: 454 | RouteTableId: !Ref PrivateRouteTable6 455 | SubnetId: !Ref PrivateSubnet6 456 | 457 | Outputs: 458 | Vpc: 459 | Description: The created VPC 460 | Value: !Ref Vpc 461 | 462 | PublicSubnets: 463 | Description: A list of the public subnets 464 | Value: !Join 465 | - ',' 466 | - - !Ref PublicSubnet1 467 | - !Ref PublicSubnet2 468 | - !If [HasPublicSubnet3, !Ref PublicSubnet3, !Ref 'AWS::NoValue'] 469 | - !If [HasPublicSubnet4, !Ref PublicSubnet4, !Ref 'AWS::NoValue'] 470 | - !If [HasPublicSubnet5, !Ref PublicSubnet5, !Ref 'AWS::NoValue'] 471 | - !If [HasPublicSubnet6, !Ref PublicSubnet6, !Ref 'AWS::NoValue'] 472 | 473 | PrivateSubnets: 474 | Description: A list of the private subnets 475 | Value: !Join 476 | - ',' 477 | - - !If [HasPrivateSubnet1, !Ref PrivateSubnet1, !Ref 'AWS::NoValue'] 478 | - !If [HasPrivateSubnet2, !Ref PrivateSubnet2, !Ref 'AWS::NoValue'] 479 | - !If [HasPrivateSubnet3, !Ref PrivateSubnet3, !Ref 'AWS::NoValue'] 480 | - !If [HasPrivateSubnet4, !Ref PrivateSubnet4, !Ref 'AWS::NoValue'] 481 | - !If [HasPrivateSubnet5, !Ref PrivateSubnet5, !Ref 'AWS::NoValue'] 482 | - !If [HasPrivateSubnet6, !Ref PrivateSubnet6, !Ref 'AWS::NoValue'] 483 | --------------------------------------------------------------------------------