├── .gitignore ├── Dockerfile ├── README.md ├── bin └── .gitignore ├── cmd └── root.go ├── go.mod ├── go.sum ├── internal ├── broker │ └── broker.go ├── config │ └── config.go ├── fly │ ├── api.go │ ├── apps.go │ ├── errors.go │ ├── http.go │ ├── machines.go │ ├── org.go │ ├── requests-app.go │ ├── requests-machine.go │ └── types.go ├── logging │ └── logging.go └── sqs │ ├── client.go │ ├── delete_messages.go │ └── get_events.go ├── main.go └── runtimes ├── fly-builder ├── Dockerfile ├── docker-entrypoint.d │ ├── docker │ └── sysctl ├── entrypoint ├── etc │ └── docker │ │ └── daemon.json └── readme.md ├── go ├── Dockerfile └── README.md ├── js ├── Dockerfile ├── README.md ├── build.sh ├── sample-project │ ├── Dockerfile │ ├── events.json │ └── src │ │ └── index.js └── src │ └── index.js └── php ├── Dockerfile ├── README.md ├── build.sh ├── sample-project ├── Dockerfile ├── events.json └── src │ └── index.php └── src └── index.php /.gitignore: -------------------------------------------------------------------------------- 1 | # If you prefer the allow list template instead of the deny list, see community template: 2 | # https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore 3 | # 4 | # Binaries for programs and plugins 5 | *.exe 6 | *.exe~ 7 | *.dll 8 | *.so 9 | *.dylib 10 | 11 | # Test binary, built with `go test -c` 12 | *.test 13 | 14 | # Output of the go coverage tool, specifically when used with LiteIDE 15 | *.out 16 | 17 | # Dependency directories (remove the comment below to include it) 18 | # vendor/ 19 | 20 | # Go workspace file 21 | go.work 22 | 23 | # Fly things 24 | todo.md 25 | .idea 26 | 27 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM golang:1.21 as builder 2 | 3 | WORKDIR /app 4 | COPY . /app 5 | RUN GOOS=linux GOARCH=amd64 go build -o /app/lambdo 6 | 7 | FROM debian:bookworm-slim 8 | 9 | COPY --from=builder /app/lambdo /usr/local/bin/lambdo 10 | 11 | CMD ["lambdo"] 12 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Lambdo 2 | 3 | Run workloads on Fly Machines based on external events. It's like serverless, but you can run your whole code base (or whatever you want). 4 | 5 | **There are three components:** 6 | 7 | 1. The server (this code base), likelly run as an app within Fly 8 | 2. Your code, which is built into a Docker image (like normal on Fly.io) 9 | 3. An SQS queue you own and populate with your own events 10 | 11 | This will poll the SQS queue. When 1+ messages are received, it will create 1+ Fly Machines and run your code (with the event as context). 12 | 13 | ## The Server 14 | 15 | Compile the program: 16 | 17 | ```bash 18 | # Locally: 19 | go build -o bin/lambdo 20 | 21 | # For Fly VMs: 22 | GOOS=linux GOARCH=amd64 -o bin/lambdo go build 23 | ``` 24 | 25 | Run the program, with some environment variables: 26 | 27 | ```bash 28 | # Any valid AWS credential env vars will do 29 | AWS_REGION=us-east-2 \ 30 | AWS_PROFILE=some-profile \ 31 | LAMBDO_FLY_TOKEN="$(fly auth token)" \ 32 | LAMBDO_FLY_APP=some-app \ 33 | LAMBDO_FLY_REGION=bos \ 34 | LAMBDO_SQS_QUEUE_URL 35 | =https://sqs..amazonaws.com// \ 36 | bin/lambdo 37 | ``` 38 | 39 | ### Run on Fly.io 40 | 41 | The [`Dockerfile`](Dockerfile) in this repository will create a Docker image you can use to run this program. 42 | 43 | You can clone the repository and run `fly launch` to get started. 44 | 45 | Be sure to set the appropriate environment variables in your `fly.toml` file, and set sensitive variables via `fly secrets`. 46 | 47 | If you run the app within Fly.io, you can omit the following environment variables 48 | (they'll be picked up automatically based on where this app is deployed): 49 | 50 | * `LAMBDO_FLY_APP` 51 | * `LAMBDO_FLY_REGION` 52 | 53 | ## Your Code 54 | 55 | You need some code that reads in a JSON string from file `/tmp/events.json`. This is an array of arbitrary events that you create via the SQS queue. 56 | 57 | Example content of `/tmp/events.json`: 58 | 59 | ```json 60 | [ 61 | {"some": "object"}, 62 | {"that": "you", "created": true} 63 | ] 64 | ``` 65 | 66 | You can either program up your own code to handle this file, or if you like the "serverless function" style, you can use a base image provided by this project. 67 | Either way, you'll be running code you produce. 68 | 69 | ### Use Your Code Base 70 | 71 | One way to go about this is to use your existing code base, and add a command that can be run (`php artisan foo`, `rake foo`, `node index.js foo`, whatever) 72 | that reads in the JSON from `/tmp/events.json` and handles each event provided. 73 | 74 | I'm not a JS developer, but here's some JS I developed as an example of what your code might do: 75 | 76 | ```js 77 | const fs = require('fs'); 78 | 79 | // EVENTS_PATH will be set to "/tmp/events.json" 80 | // when created by this code base 81 | const eventsString = fs.readFileSync(process.env.EVENTS_PATH) 82 | const events = JSON.parse(eventsString) 83 | 84 | for (let key in events) { 85 | // Draw the rest of the owl 86 | do_something_with_this_event(events[key]); 87 | } 88 | ``` 89 | 90 | ### Use a Serverless Function 91 | 92 | There's 2 base images here you can use, which are a little more like serverless in that you can provide them a function to run for each event. 93 | 94 | The provided images default to Node 20 or PHP 8.2 (you can, of course, make your own). You just need to add in your own code to `/app/index.js` or `/app/index.php`. 95 | 96 | See the [sample JS project](runtimes/js/sample-project) or the [sample PHP project](runtimes/php/sample-project) to see what that looks like. 97 | 98 | ## The SQS Queue 99 | 100 | The SQS queue is the source of events. Sending messages to this queue will result in Machines being created to process them. 101 | 102 | The message `Body` should be a valid JSON string (your event, its contens are arbitrary). 103 | 104 | The message `Attributes` have up to 3 values to help the project know how to spin up a Machine and process the event. 105 | 106 | It looks like this (forgive the lame need for escaping double quotes): 107 | 108 | ```bash 109 | QUEUE_URL="https://sqs..amazonaws.com//" 110 | JSON_BODY='{"foo": "bar"}' 111 | 112 | aws sqs send-message \ 113 | --queue-url=$QUEUE_URL \ 114 | --message-body=$JSON_BODY \ 115 | --message-attributes='{ 116 | "image":{"DataType":"String","StringValue":"registry.fly.io/app:tag"}, 117 | "size":{"DataType":"String","StringValue":"performance-2x"}, 118 | "command":{"DataType":"String","StringValue":"[\"php\", \"artisan\", \"foo\"]"}, 119 | }' 120 | ``` 121 | 122 | There are 3 attribute values to care about: 123 | 124 | | Attribute | Description | Default | 125 | |-----------|-----------------------------------------------------------------------|--------------------------| 126 | | `image` | **required** - The image to run in the Machine to process that event | | 127 | | `size` | The VM size† | `performance-2x` | 128 | | `command` | The command to run, which is the Docker `CMD` equivalent†† | Your `Dockerfile`'s `CMD` | 129 | 130 | - † Use `fly platform vm-sizes` for valid values. 131 | - †† Use the array syntax, e.g.`["foo", "--bar"]` -------------------------------------------------------------------------------- /bin/.gitignore: -------------------------------------------------------------------------------- 1 | * 2 | !.gitignore -------------------------------------------------------------------------------- /cmd/root.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "context" 5 | "github.com/aws/aws-sdk-go-v2/service/sqs/types" 6 | "github.com/spf13/cobra" 7 | "github.com/superfly/lambdo/internal/broker" 8 | "github.com/superfly/lambdo/internal/logging" 9 | "github.com/superfly/lambdo/internal/sqs" 10 | "go.uber.org/zap" 11 | "os" 12 | "sync" 13 | ) 14 | 15 | var rootCmd = &cobra.Command{ 16 | Use: "lambdo", 17 | Short: "Lambdo runs workloads based on events", 18 | Long: `This is like Lambda, but Flyier 19 | Configuration should be set via environment variables. Possible values: 20 | required: 21 | LAMBDO_SQS_QUEUE_URL: string, full sqs queue url 22 | AWS_*: Any needed AWS credential environment variables (region, key, secret, profile) 23 | LAMBDO_FLY_TOKEN string, a valid Fly API token 24 | LAMBDO_FLY_REGION, FLY_REGION string, one of these must be set. FLY_REGION is already set when running in Fly 25 | LAMBDO_FLY_APP, FLY_APP_NAME string, one of these must be set. FLY_APP_NAME is already set when running in Fly 26 | 27 | optional: 28 | LAMBDO_ENV: string, default: local 29 | LAMBDO_SQS_LONG_POLL_SECONDS: int, default: 10 30 | LAMDBO_EVENTS_PER_MACHINE: int, default 5 31 | `, 32 | Run: RunRootCommand, 33 | } 34 | 35 | var errors chan error 36 | 37 | func Execute(ctx context.Context) error { 38 | errors = make(chan error) 39 | defer close(errors) 40 | 41 | go func() { 42 | for { 43 | select { 44 | case err := <-errors: 45 | logging.GetLogger().Error("broker error", zap.Error(err)) 46 | case <-ctx.Done(): 47 | logging.GetLogger().Info("Shutdown: No longer listening to error channel") 48 | return 49 | } 50 | } 51 | }() 52 | 53 | if err := rootCmd.ExecuteContext(ctx); err != nil { 54 | return err 55 | } 56 | 57 | return nil 58 | } 59 | 60 | func RunRootCommand(cmd *cobra.Command, args []string) { 61 | messages := make(chan []types.Message) 62 | defer close(messages) 63 | 64 | var brokerWorking sync.WaitGroup 65 | 66 | go func(ctx context.Context, m chan []types.Message) { 67 | for { 68 | select { 69 | case msgs := <-m: 70 | logging.GetLogger().Debug("messages received", zap.Any("messages", msgs)) 71 | // todo: Should we send to broker in go routine to prevent 72 | // holding up messages being processed (channel blocked until broker finishes)? 73 | brokerWorking.Add(1) 74 | err := broker.SendToMachine(msgs) 75 | if err != nil { 76 | errors <- err 77 | } 78 | brokerWorking.Done() 79 | case <-ctx.Done(): 80 | logging.GetLogger().Info("Shutdown: no longer creating machines") 81 | return 82 | } 83 | } 84 | }(cmd.Context(), messages) 85 | 86 | // Listen for messages in SQS 87 | if err := sqs.Listen(cmd.Context(), messages); err != nil { 88 | logging.GetLogger().Error("SQS error", zap.Error(err)) 89 | os.Exit(1) 90 | } 91 | 92 | logging.GetLogger().Info("Shutdown: waiting on broker to finish current job") 93 | 94 | brokerWorking.Wait() 95 | } 96 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/superfly/lambdo 2 | 3 | go 1.21.5 4 | 5 | require ( 6 | github.com/aws/aws-sdk-go-v2 v1.24.0 // indirect 7 | github.com/aws/aws-sdk-go-v2/config v1.26.2 // indirect 8 | github.com/aws/aws-sdk-go-v2/credentials v1.16.13 // indirect 9 | github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.14.10 // indirect 10 | github.com/aws/aws-sdk-go-v2/internal/configsources v1.2.9 // indirect 11 | github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.5.9 // indirect 12 | github.com/aws/aws-sdk-go-v2/internal/ini v1.7.2 // indirect 13 | github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.10.4 // indirect 14 | github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.10.9 // indirect 15 | github.com/aws/aws-sdk-go-v2/service/sqs v1.29.6 // indirect 16 | github.com/aws/aws-sdk-go-v2/service/sso v1.18.5 // indirect 17 | github.com/aws/aws-sdk-go-v2/service/ssooidc v1.21.5 // indirect 18 | github.com/aws/aws-sdk-go-v2/service/sts v1.26.6 // indirect 19 | github.com/aws/smithy-go v1.19.0 // indirect 20 | github.com/fsnotify/fsnotify v1.7.0 // indirect 21 | github.com/hashicorp/hcl v1.0.0 // indirect 22 | github.com/inconshreveable/mousetrap v1.1.0 // indirect 23 | github.com/magiconair/properties v1.8.7 // indirect 24 | github.com/mitchellh/mapstructure v1.5.0 // indirect 25 | github.com/pelletier/go-toml/v2 v2.1.1 // indirect 26 | github.com/sagikazarmark/locafero v0.4.0 // indirect 27 | github.com/sagikazarmark/slog-shim v0.1.0 // indirect 28 | github.com/sourcegraph/conc v0.3.0 // indirect 29 | github.com/spf13/afero v1.11.0 // indirect 30 | github.com/spf13/cast v1.6.0 // indirect 31 | github.com/spf13/cobra v1.8.0 // indirect 32 | github.com/spf13/pflag v1.0.5 // indirect 33 | github.com/spf13/viper v1.18.2 // indirect 34 | github.com/subosito/gotenv v1.6.0 // indirect 35 | go.uber.org/atomic v1.11.0 // indirect 36 | go.uber.org/multierr v1.11.0 // indirect 37 | go.uber.org/zap v1.26.0 // indirect 38 | golang.org/x/exp v0.0.0-20231226003508-02704c960a9b // indirect 39 | golang.org/x/sys v0.15.0 // indirect 40 | golang.org/x/text v0.14.0 // indirect 41 | gopkg.in/ini.v1 v1.67.0 // indirect 42 | gopkg.in/yaml.v3 v3.0.1 // indirect 43 | ) 44 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/aws/aws-sdk-go-v2 v1.24.0 h1:890+mqQ+hTpNuw0gGP6/4akolQkSToDJgHfQE7AwGuk= 2 | github.com/aws/aws-sdk-go-v2 v1.24.0/go.mod h1:LNh45Br1YAkEKaAqvmE1m8FUx6a5b/V0oAKV7of29b4= 3 | github.com/aws/aws-sdk-go-v2/config v1.26.2 h1:+RWLEIWQIGgrz2pBPAUoGgNGs1TOyF4Hml7hCnYj2jc= 4 | github.com/aws/aws-sdk-go-v2/config v1.26.2/go.mod h1:l6xqvUxt0Oj7PI/SUXYLNyZ9T/yBPn3YTQcJLLOdtR8= 5 | github.com/aws/aws-sdk-go-v2/credentials v1.16.13 h1:WLABQ4Cp4vXtXfOWOS3MEZKr6AAYUpMczLhgKtAjQ/8= 6 | github.com/aws/aws-sdk-go-v2/credentials v1.16.13/go.mod h1:Qg6x82FXwW0sJHzYruxGiuApNo31UEtJvXVSZAXeWiw= 7 | github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.14.10 h1:w98BT5w+ao1/r5sUuiH6JkVzjowOKeOJRHERyy1vh58= 8 | github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.14.10/go.mod h1:K2WGI7vUvkIv1HoNbfBA1bvIZ+9kL3YVmWxeKuLQsiw= 9 | github.com/aws/aws-sdk-go-v2/internal/configsources v1.2.9 h1:v+HbZaCGmOwnTTVS86Fleq0vPzOd7tnJGbFhP0stNLs= 10 | github.com/aws/aws-sdk-go-v2/internal/configsources v1.2.9/go.mod h1:Xjqy+Nyj7VDLBtCMkQYOw1QYfAEZCVLrfI0ezve8wd4= 11 | github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.5.9 h1:N94sVhRACtXyVcjXxrwK1SKFIJrA9pOJ5yu2eSHnmls= 12 | github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.5.9/go.mod h1:hqamLz7g1/4EJP+GH5NBhcUMLjW+gKLQabgyz6/7WAU= 13 | github.com/aws/aws-sdk-go-v2/internal/ini v1.7.2 h1:GrSw8s0Gs/5zZ0SX+gX4zQjRnRsMJDJ2sLur1gRBhEM= 14 | github.com/aws/aws-sdk-go-v2/internal/ini v1.7.2/go.mod h1:6fQQgfuGmw8Al/3M2IgIllycxV7ZW7WCdVSqfBeUiCY= 15 | github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.10.4 h1:/b31bi3YVNlkzkBrm9LfpaKoaYZUxIAj4sHfOTmLfqw= 16 | github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.10.4/go.mod h1:2aGXHFmbInwgP9ZfpmdIfOELL79zhdNYNmReK8qDfdQ= 17 | github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.10.9 h1:Nf2sHxjMJR8CSImIVCONRi4g0Su3J+TSTbS7G0pUeMU= 18 | github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.10.9/go.mod h1:idky4TER38YIjr2cADF1/ugFMKvZV7p//pVeV5LZbF0= 19 | github.com/aws/aws-sdk-go-v2/service/sqs v1.29.6 h1:UdbDTllc7cmusTTMy1dcTrYKRl4utDEsmKh9ZjvhJCc= 20 | github.com/aws/aws-sdk-go-v2/service/sqs v1.29.6/go.mod h1:mCUv04gd/7g+/HNzDB4X6dzJuygji0ckvB3Lg/TdG5Y= 21 | github.com/aws/aws-sdk-go-v2/service/sso v1.18.5 h1:ldSFWz9tEHAwHNmjx2Cvy1MjP5/L9kNoR0skc6wyOOM= 22 | github.com/aws/aws-sdk-go-v2/service/sso v1.18.5/go.mod h1:CaFfXLYL376jgbP7VKC96uFcU8Rlavak0UlAwk1Dlhc= 23 | github.com/aws/aws-sdk-go-v2/service/ssooidc v1.21.5 h1:2k9KmFawS63euAkY4/ixVNsYYwrwnd5fIvgEKkfZFNM= 24 | github.com/aws/aws-sdk-go-v2/service/ssooidc v1.21.5/go.mod h1:W+nd4wWDVkSUIox9bacmkBP5NMFQeTJ/xqNabpzSR38= 25 | github.com/aws/aws-sdk-go-v2/service/sts v1.26.6 h1:HJeiuZ2fldpd0WqngyMR6KW7ofkXNLyOaHwEIGm39Cs= 26 | github.com/aws/aws-sdk-go-v2/service/sts v1.26.6/go.mod h1:XX5gh4CB7wAs4KhcF46G6C8a2i7eupU19dcAAE+EydU= 27 | github.com/aws/smithy-go v1.19.0 h1:KWFKQV80DpP3vJrrA9sVAHQ5gc2z8i4EzrLhLlWXcBM= 28 | github.com/aws/smithy-go v1.19.0/go.mod h1:NukqUGpCZIILqqiV0NIjeFh24kd/FAa4beRb6nbIUPE= 29 | github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= 30 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 31 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 32 | github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA= 33 | github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM= 34 | github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4= 35 | github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= 36 | github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= 37 | github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= 38 | github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0VQdvPDY= 39 | github.com/magiconair/properties v1.8.7/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0= 40 | github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY= 41 | github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= 42 | github.com/pelletier/go-toml/v2 v2.1.1 h1:LWAJwfNvjQZCFIDKWYQaM62NcYeYViCmWIwmOStowAI= 43 | github.com/pelletier/go-toml/v2 v2.1.1/go.mod h1:tJU2Z3ZkXwnxa4DPO899bsyIoywizdUvyaeZurnPPDc= 44 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 45 | github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= 46 | github.com/sagikazarmark/locafero v0.4.0 h1:HApY1R9zGo4DBgr7dqsTH/JJxLTTsOt7u6keLGt6kNQ= 47 | github.com/sagikazarmark/locafero v0.4.0/go.mod h1:Pe1W6UlPYUk/+wc/6KFhbORCfqzgYEpgQ3O5fPuL3H4= 48 | github.com/sagikazarmark/slog-shim v0.1.0 h1:diDBnUNK9N/354PgrxMywXnAwEr1QZcOr6gto+ugjYE= 49 | github.com/sagikazarmark/slog-shim v0.1.0/go.mod h1:SrcSrq8aKtyuqEI1uvTDTK1arOWRIczQRv+GVI1AkeQ= 50 | github.com/sourcegraph/conc v0.3.0 h1:OQTbbt6P72L20UqAkXXuLOj79LfEanQ+YQFNpLA9ySo= 51 | github.com/sourcegraph/conc v0.3.0/go.mod h1:Sdozi7LEKbFPqYX2/J+iBAM6HpqSLTASQIKqDmF7Mt0= 52 | github.com/spf13/afero v1.11.0 h1:WJQKhtpdm3v2IzqG8VMqrr6Rf3UYpEF239Jy9wNepM8= 53 | github.com/spf13/afero v1.11.0/go.mod h1:GH9Y3pIexgf1MTIWtNGyogA5MwRIDXGUr+hbWNoBjkY= 54 | github.com/spf13/cast v1.6.0 h1:GEiTHELF+vaR5dhz3VqZfFSzZjYbgeKDpBxQVS4GYJ0= 55 | github.com/spf13/cast v1.6.0/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo= 56 | github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0= 57 | github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho= 58 | github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= 59 | github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= 60 | github.com/spf13/viper v1.18.2 h1:LUXCnvUvSM6FXAsj6nnfc8Q2tp1dIgUfY9Kc8GsSOiQ= 61 | github.com/spf13/viper v1.18.2/go.mod h1:EKmWIqdnk5lOcmR72yw6hS+8OPYcwD0jteitLMVB+yk= 62 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 63 | github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= 64 | github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= 65 | github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 66 | github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= 67 | github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= 68 | github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8= 69 | github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU= 70 | go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE= 71 | go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0= 72 | go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= 73 | go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= 74 | go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo= 75 | go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= 76 | golang.org/x/exp v0.0.0-20231226003508-02704c960a9b h1:kLiC65FbiHWFAOu+lxwNPujcsl8VYyTYYEZnsOO1WK4= 77 | golang.org/x/exp v0.0.0-20231226003508-02704c960a9b/go.mod h1:iRJReGqOEeBhDZGkGbynYwcHlctCvnjTYIamk7uXpHI= 78 | golang.org/x/sys v0.15.0 h1:h48lPFYpsTvQJZF4EKyI4aLHaev3CxivZmv7yZig9pc= 79 | golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= 80 | golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ= 81 | golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= 82 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 83 | gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= 84 | gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= 85 | gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 86 | gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= 87 | gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 88 | -------------------------------------------------------------------------------- /internal/broker/broker.go: -------------------------------------------------------------------------------- 1 | package broker 2 | 3 | import ( 4 | "crypto/md5" 5 | "encoding/base64" 6 | "encoding/hex" 7 | "encoding/json" 8 | "fmt" 9 | "github.com/aws/aws-sdk-go-v2/service/sqs/types" 10 | "github.com/superfly/lambdo/internal/config" 11 | "github.com/superfly/lambdo/internal/fly" 12 | "github.com/superfly/lambdo/internal/logging" 13 | events "github.com/superfly/lambdo/internal/sqs" 14 | "go.uber.org/zap" 15 | ) 16 | 17 | type Event struct { 18 | Image string 19 | Size string 20 | Body string 21 | Cmd []string 22 | Meta map[string]string 23 | } 24 | 25 | type EventCollection struct { 26 | Events []*Event 27 | } 28 | 29 | func SendToMachine(messages []types.Message) error { 30 | api := fly.NewApi(config.GetConfig().FlyToken) 31 | appName := config.GetConfig().FlyApp 32 | // Each attempt iteration will try a new region 33 | // TODO: Respect config.GetConfig().FlyRegion setting 34 | regions := []string{"bos", "dfw", "den", "mia"} 35 | 36 | eventsPerMachine := map[string]*EventCollection{} 37 | 38 | // Group messages (events) by which image, size, and command they require 39 | for _, m := range messages { 40 | image, imageErr := findAttribute("image", m) 41 | if imageErr != nil { 42 | logging.GetLogger().Warn("an event had no image", zap.String("error", imageErr.Error())) 43 | continue 44 | } 45 | 46 | size, sizeErr := findAttribute("size", m) 47 | 48 | if sizeErr != nil { 49 | logging.GetLogger().Warn("an event had no size, defaulting to performance-2x", zap.String("error", sizeErr.Error())) 50 | size = "performance-2x" 51 | } 52 | 53 | var cmd []string 54 | cmdString, cmdErr := findAttribute("command", m) 55 | 56 | if cmdErr != nil { 57 | logging.GetLogger().Warn("an event had no command, defaulting to no command", zap.String("error", cmdErr.Error())) 58 | } 59 | 60 | if len(cmdString) > 0 { 61 | if jErr := json.Unmarshal([]byte(cmdString), &cmd); jErr != nil { 62 | logging.GetLogger().Warn("could not parse command, no machine will be created", zap.String("error", jErr.Error())) 63 | continue 64 | } 65 | } 66 | 67 | // md5 of attributes that affect machine creation, so we can group like-events 68 | // into machines that run the same way 69 | eventsPerMachineKeyHash := md5.Sum([]byte(fmt.Sprintf("%s-%s-%s", image, size, cmdString))) 70 | eventsPerMachineKey := hex.EncodeToString(eventsPerMachineKeyHash[:]) 71 | if _, ok := eventsPerMachine[eventsPerMachineKey]; !ok { 72 | eventsPerMachine[eventsPerMachineKey] = &EventCollection{} 73 | } 74 | 75 | eventsPerMachine[eventsPerMachineKey].Events = append(eventsPerMachine[eventsPerMachineKey].Events, &Event{ 76 | Image: image, 77 | Body: *m.Body, 78 | Size: size, 79 | Cmd: cmd, 80 | Meta: map[string]string{"receipt": *m.ReceiptHandle}, 81 | }) 82 | } 83 | 84 | // Build JSON array of events, for each unique image 85 | // This assumes each event body is a 86 | // valid JSON string (lol) 87 | // This is dumb af, but good enough for now 88 | for _, collection := range eventsPerMachine { 89 | eventStrings := "" 90 | receipts := []string{} 91 | image := "" 92 | size := "" 93 | var cmd []string 94 | for k, e := range collection.Events { 95 | if k == 0 { 96 | eventStrings += e.Body 97 | } else { 98 | eventStrings += "," + e.Body 99 | } 100 | receipts = append(receipts, e.Meta["receipt"]) 101 | 102 | // These get reset on every iteration, but in our scenario here, 103 | // they'll always get set to the same values since we segregated 104 | // them above. Just another code smell, no worries. 105 | image = e.Image 106 | size = e.Size 107 | cmd = e.Cmd 108 | } 109 | 110 | eventStringJson := fmt.Sprintf("[%s]", eventStrings) 111 | encodedJson := base64.StdEncoding.EncodeToString([]byte(eventStringJson)) 112 | 113 | logging.GetLogger().Debug("creating Machine", zap.String("app-name", appName), zap.String("image", image)) 114 | 115 | success := false 116 | 117 | for k, region := range regions { 118 | machine := fly.CreateMachineInput{ 119 | AppName: config.GetConfig().FlyApp, 120 | Machine: fly.Machine{ 121 | Region: region, 122 | Config: fly.MachineConfig{ 123 | Image: image, 124 | Env: map[string]string{ 125 | "EVENTS_PATH": "/tmp/events.json", 126 | }, 127 | /* 128 | Guest: fly.MachineSize{ 129 | CpuCount: 2, 130 | RAM: 2048, 131 | Type: "shared", 132 | }, 133 | */ 134 | Size: size, 135 | Files: []fly.MachineFile{ 136 | { 137 | GuestPath: "/tmp/events.json", 138 | RawValue: encodedJson, 139 | }, 140 | }, 141 | AutoDestroy: true, 142 | }, 143 | }, 144 | } 145 | 146 | if len(cmd) > 0 { 147 | machine.Machine.Config.Processes = []fly.MachineProcess{ 148 | { 149 | Cmd: cmd, 150 | }, 151 | } 152 | } 153 | 154 | m, err := api.CreateMachine(&machine) 155 | 156 | if err != nil { 157 | logging.GetLogger().Error("could not create Machine", zap.Error(err), zap.Int("attempt", k), zap.String("region", region)) 158 | continue // try next region 159 | } 160 | 161 | success = true 162 | logging.GetLogger().Debug("created machine", zap.String("machine-id", m.Id)) 163 | break // Break out of region retry loop 164 | } 165 | 166 | if success == false { 167 | logging.GetLogger().Error("could not create a Machine for this workload") 168 | } else { 169 | logging.GetLogger().Debug("machine created, deleting messages", zap.String("image", image)) 170 | 171 | // TODO: Handle if messages could not be deleted (so it does not get re-tried?) - perhaps retry logic? 172 | for _, receipt := range receipts { 173 | delErr := events.DeleteMessage(receipt) 174 | if delErr != nil { 175 | logging.GetLogger().Error("machine crated but could not delete message", zap.Error(delErr)) 176 | } 177 | } 178 | } 179 | } 180 | 181 | // We don't return an error when a machine fails to be created 182 | return nil 183 | } 184 | 185 | func findAttribute(attr string, message types.Message) (string, error) { 186 | for k, v := range message.MessageAttributes { 187 | if k == attr { 188 | return *v.StringValue, nil 189 | } 190 | } 191 | 192 | return "", fmt.Errorf("could not find an event %s", attr) 193 | } 194 | -------------------------------------------------------------------------------- /internal/config/config.go: -------------------------------------------------------------------------------- 1 | package config 2 | 3 | import ( 4 | "fmt" 5 | "github.com/spf13/viper" 6 | "log" 7 | "os" 8 | ) 9 | 10 | type LambdoConfig struct { 11 | Environment string `mapstructure:"env"` 12 | SQSLongPollSeconds int `mapstructure:"sqs_long_poll_seconds"` 13 | SQSQueueUrl string `mapstructure:"sqs_queue_url"` 14 | EventsPerMachine int `mapstructure:"events_per_machine"` 15 | FlyApp string `mapstructure:"fly_app"` 16 | FlyRegion string `mapstructure:"fly_region"` 17 | FlyToken string `mapstructure:"fly_token"` 18 | } 19 | 20 | var lambdoConfig *LambdoConfig 21 | 22 | func Configure() error { 23 | v := viper.New() 24 | v.SetEnvPrefix("lambdo") 25 | v.AutomaticEnv() 26 | 27 | v.BindEnv("env") 28 | v.BindEnv("sqs_long_poll_seconds") 29 | v.BindEnv("sqs_queue_url") 30 | v.BindEnv("events_per_machine") 31 | v.BindEnv("fly_app") 32 | v.BindEnv("fly_region") 33 | v.BindEnv("fly_token") 34 | 35 | v.SetDefault("env", "local") 36 | v.SetDefault("sqs_long_poll_seconds", 10) 37 | v.SetDefault("events_per_machine", 5) 38 | 39 | config := &LambdoConfig{} 40 | err := v.Unmarshal(&config) 41 | 42 | if err != nil { 43 | return err 44 | } 45 | 46 | // Pick these up from Fly runtime environment variables 47 | // if not set explicitly 48 | if len(config.FlyApp) == 0 { 49 | config.FlyApp = os.Getenv("FLY_APP_NAME") 50 | } 51 | 52 | if len(config.FlyRegion) == 0 { 53 | config.FlyRegion = os.Getenv("FLY_REGION") 54 | } 55 | 56 | // Some quick validation 57 | if len(config.FlyToken) == 0 { 58 | return fmt.Errorf("config LAMBDO_FLY_TOKEN must be set") 59 | } 60 | 61 | if len(config.FlyApp) == 0 { 62 | return fmt.Errorf("No values found for LAMDBO_FLY_APP nor FLY_APP_NAME") 63 | } 64 | 65 | if len(config.FlyRegion) == 0 { 66 | return fmt.Errorf("No values found for LAMBDO_FLY_REGION nor FLY_REGION") 67 | } 68 | 69 | if config.EventsPerMachine > 10 { 70 | log.Println("config events_per_machine set higher than 10, using value 10") 71 | config.EventsPerMachine = 10 72 | } 73 | 74 | lambdoConfig = config 75 | 76 | return nil 77 | } 78 | 79 | func GetConfig() *LambdoConfig { 80 | return lambdoConfig 81 | } 82 | -------------------------------------------------------------------------------- /internal/fly/api.go: -------------------------------------------------------------------------------- 1 | package fly 2 | 3 | type Api struct { 4 | Token string 5 | } 6 | 7 | // NewApi returns an instance of the Fly API 8 | func NewApi(token string) *Api { 9 | return &Api{ 10 | Token: token, 11 | } 12 | } 13 | -------------------------------------------------------------------------------- /internal/fly/apps.go: -------------------------------------------------------------------------------- 1 | package fly 2 | 3 | import ( 4 | "encoding/json" 5 | "errors" 6 | "fmt" 7 | "github.com/superfly/lambdo/internal/logging" 8 | "go.uber.org/zap" 9 | "io" 10 | "net/http" 11 | ) 12 | 13 | type CreateAppInput struct { 14 | Name string 15 | Org string 16 | } 17 | 18 | // CreateApp attempts to create an app, specifically 19 | // using the Fly Machines API 20 | func (api *Api) CreateApp(i *CreateAppInput) (*App, error) { 21 | app := i.Name 22 | org := i.Org 23 | 24 | if len(org) == 0 { 25 | org = OrgName() 26 | } 27 | 28 | req := &CreateAppRequest{ 29 | Name: app, 30 | Org: org, 31 | Network: fmt.Sprintf("%s-net", i.Name), 32 | } 33 | 34 | response, err := DoRequest(api.Token, req) 35 | 36 | if err != nil { 37 | return nil, fmt.Errorf("request error: %w", err) 38 | } 39 | defer response.Body.Close() 40 | 41 | if response.StatusCode > 299 { 42 | return nil, fmt.Errorf("could not create app, status: %d", response.StatusCode) 43 | } 44 | 45 | return &App{ 46 | Name: app, 47 | Organization: Organization{ 48 | Slug: org, 49 | }, 50 | }, nil 51 | 52 | return nil, nil 53 | } 54 | 55 | type GetAppInput struct { 56 | Name string 57 | } 58 | 59 | // GetApp attempts to retrieve an application. 60 | // 61 | // An app that is not found returns an AppNotFoundError error 62 | func (api *Api) GetApp(i *GetAppInput) (*App, error) { 63 | req := &GetAppRequest{ 64 | App: App{ 65 | Name: i.Name, 66 | }, 67 | } 68 | 69 | response, err := DoRequest(api.Token, req) 70 | 71 | if err != nil { 72 | return nil, fmt.Errorf("request error: %w", err) 73 | } 74 | defer response.Body.Close() 75 | 76 | if response.StatusCode == http.StatusNotFound { 77 | return nil, AppNotFoundError{ 78 | App: i.Name, 79 | Err: fmt.Errorf("fly returned 404 for app: %s", i.Name), 80 | } 81 | } 82 | 83 | // Other errors 84 | if response.StatusCode > 299 { 85 | return nil, fmt.Errorf("could not get app, status: %d", response.StatusCode) 86 | } 87 | 88 | responseBody, err := io.ReadAll(response.Body) 89 | 90 | if err != nil { 91 | return nil, fmt.Errorf("response body reading error: %w", err) 92 | } 93 | 94 | b := string(responseBody) 95 | if response.StatusCode > 399 { 96 | return nil, fmt.Errorf("invalid HTTP response '%d': %s", b) 97 | } 98 | 99 | logging.GetLogger().Debug("GetApp response", zap.String("body", b)) 100 | 101 | a := &App{} 102 | err = json.Unmarshal(responseBody, a) 103 | 104 | if err != nil { 105 | return nil, fmt.Errorf("could not unmarshall json: %w", err) 106 | } 107 | 108 | return a, nil 109 | } 110 | 111 | type DeleteAppInput struct { 112 | Name string 113 | } 114 | 115 | // DeleteApp deletes an application and 116 | // any created Machines 117 | func (api *Api) DeleteApp(i *DeleteAppInput) error { 118 | req := &DeleteAppRequest{ 119 | App: App{ 120 | Name: i.Name, 121 | }, 122 | } 123 | 124 | response, err := DoRequest(api.Token, req) 125 | 126 | if err != nil { 127 | return fmt.Errorf("request error: %w", err) 128 | } 129 | 130 | defer response.Body.Close() 131 | 132 | // Log "error" http responses, but otherwise ignore errors here 133 | if response.StatusCode > 299 { 134 | logging.GetLogger().Error("could not delete app", zap.Int("status", response.StatusCode)) 135 | } 136 | 137 | return nil 138 | } 139 | 140 | // FindCreateApp will return an app if it already exists 141 | // or attempts to create the app if the given app does 142 | // not yet exist 143 | func (api *Api) FindCreateApp(i *CreateAppInput) (*App, error) { 144 | var app *App 145 | var err error 146 | var appNotFoundError AppNotFoundError 147 | if app, err = api.GetApp(&GetAppInput{i.Name}); err != nil { 148 | if errors.As(err, &appNotFoundError) { 149 | app, err = api.CreateApp(i) 150 | } else { 151 | return nil, err 152 | } 153 | } 154 | 155 | // We have this err check here instead of nesting 156 | // another layer down after call to CreateApp() 157 | if err != nil { 158 | return nil, fmt.Errorf("could not create app: %s", i.Name) 159 | } 160 | 161 | return app, nil 162 | } 163 | -------------------------------------------------------------------------------- /internal/fly/errors.go: -------------------------------------------------------------------------------- 1 | package fly 2 | 3 | import "fmt" 4 | 5 | type AppNotFoundError struct { 6 | App string 7 | Err error 8 | } 9 | 10 | func (e AppNotFoundError) Error() string { 11 | return fmt.Sprintf("app '%s' not found: %v", e.App, e.Err) 12 | } 13 | 14 | type MachineNotFoundError struct { 15 | App string 16 | Machine string 17 | Err error 18 | } 19 | 20 | func (e MachineNotFoundError) Error() string { 21 | return fmt.Sprintf("app '%s' machine '%s' not found: %v", e.App, e.Machine, e.Err) 22 | } 23 | -------------------------------------------------------------------------------- /internal/fly/http.go: -------------------------------------------------------------------------------- 1 | package fly 2 | 3 | import ( 4 | "bytes" 5 | "fmt" 6 | "github.com/superfly/lambdo/internal/logging" 7 | "go.uber.org/zap" 8 | "net/http" 9 | "os" 10 | "time" 11 | ) 12 | 13 | var client = &http.Client{ 14 | Timeout: 10 * time.Second, 15 | } 16 | 17 | // FlyRequest is an interface for API requests made to 18 | // the Fly Machines API 19 | type FlyRequest interface { 20 | ToRequest(token string) (*http.Request, error) 21 | } 22 | 23 | // StandardRequestHeaders adds standard request headers for Fly API 24 | // requests, including the Authorization header with a Bearer token 25 | func StandardRequestHeaders(req *http.Request, token string) { 26 | req.Header.Set("Authorization", "Bearer "+token) 27 | req.Header.Set("Content-Type", "application/json; charset=UTF-8") 28 | req.Header.Set("Accept", "application/json") 29 | } 30 | 31 | // DoRequest runs an HTTP request, retrying any that time out 32 | // due to possible "instability" in the Fly Machines API 33 | func DoRequest(token string, r FlyRequest) (*http.Response, error) { 34 | defer func() { 35 | if r := recover(); r != nil { 36 | logging.GetLogger().Error("DoRequest panic", zap.Any("maybe-error", r)) 37 | } 38 | }() 39 | 40 | req, err := r.ToRequest(token) 41 | if err != nil { 42 | return nil, fmt.Errorf("could not create request: %w", err) 43 | } 44 | 45 | result, err := doRequestWithRetries(req) 46 | if err != nil { 47 | return nil, err 48 | } 49 | 50 | // TODO: Result should never be nil here 51 | if result != nil && result.StatusCode > 299 { 52 | logging.GetLogger().Error( 53 | "API request to Fly returned unsuccessful status", 54 | zap.Int("status", result.StatusCode), 55 | zap.String("method", req.Method), 56 | zap.String("url", req.URL.String()), 57 | ) 58 | } 59 | 60 | return result, nil 61 | } 62 | 63 | func doRequestWithRetries(req *http.Request) (*http.Response, error) { 64 | defer func() { 65 | if r := recover(); r != nil { 66 | logging.GetLogger().Error("doRequestWithRetries panic", zap.Any("maybe-error", r)) 67 | } 68 | }() 69 | 70 | var result *http.Response 71 | var err error 72 | for attempts := 1; attempts <= 5; attempts++ { 73 | logging.GetLogger().Debug( 74 | "making API request to Fly", 75 | zap.Int("attempt", attempts), 76 | zap.String("method", req.Method), 77 | zap.String("url", req.URL.String()), 78 | ) 79 | result, err = client.Do(req) 80 | if err != nil { 81 | if os.IsTimeout(err) { 82 | // Log body if available 83 | body := "nil" 84 | reqBody, reqBodyErr := req.GetBody() 85 | if reqBodyErr == nil { 86 | buf := new(bytes.Buffer) 87 | buf.ReadFrom(reqBody) 88 | body = buf.String() 89 | } 90 | 91 | logging.GetLogger().Debug("client timeout, retrying soon", zap.String("method", req.Method), zap.String("url", req.URL.String()), zap.String("body", body)) 92 | time.Sleep(time.Second * 1) 93 | continue 94 | } 95 | 96 | // If it's not a timeout, break out and return the error 97 | return nil, fmt.Errorf("http client error: %w", err) 98 | } 99 | 100 | // Deleting a machine may result in a 409 response 101 | // (even with the force flag) 102 | // TODO: Result should never be nil here 103 | if result != nil && result.StatusCode == http.StatusConflict { 104 | logging.GetLogger().Debug("conflict response, retrying soon", zap.String("method", req.Method), zap.String("url", req.URL.String())) 105 | time.Sleep(time.Second * 2) 106 | continue 107 | } 108 | 109 | // Deleting a machine may result in a 412 response 110 | // (even with the force flag) 111 | // TODO: Result should never be nil here 112 | if result != nil && result.StatusCode == http.StatusPreconditionFailed { 113 | logging.GetLogger().Debug("precondition failed response, retrying soon", zap.String("method", req.Method), zap.String("url", req.URL.String())) 114 | time.Sleep(time.Second * 2) 115 | continue 116 | } 117 | 118 | // 400 (bad request), preferably we know what exactly is wrong 119 | if result != nil && result.StatusCode == http.StatusBadRequest { 120 | // Log body if available 121 | body := "nil" 122 | buf := new(bytes.Buffer) 123 | _, bodyErr := buf.ReadFrom(result.Body) 124 | if bodyErr == nil { 125 | body = buf.String() 126 | } 127 | 128 | logging.GetLogger().Debug("bad request response", zap.String("method", req.Method), zap.String("url", req.URL.String()), zap.String("body", body)) 129 | } 130 | 131 | logging.GetLogger().Debug( 132 | "made API request to Fly", 133 | zap.Int("attempt", attempts), 134 | zap.String("method", req.Method), 135 | zap.String("url", req.URL.String()), 136 | zap.Int("status", result.StatusCode), 137 | ) 138 | return result, nil 139 | } 140 | 141 | return nil, err 142 | } 143 | -------------------------------------------------------------------------------- /internal/fly/machines.go: -------------------------------------------------------------------------------- 1 | package fly 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "github.com/superfly/lambdo/internal/logging" 7 | "go.uber.org/zap" 8 | "io" 9 | "net/http" 10 | "time" 11 | ) 12 | 13 | type CreateMachineInput struct { 14 | AppName string 15 | Machine Machine 16 | } 17 | 18 | func (api *Api) CreateMachine(i *CreateMachineInput) (*Machine, error) { 19 | req := &CreateMachineRequest{ 20 | App: App{ 21 | Name: i.AppName, 22 | }, 23 | Machine: i.Machine, 24 | } 25 | 26 | response, err := DoRequest(api.Token, req) 27 | 28 | if err != nil { 29 | return nil, fmt.Errorf("request error: %w", err) 30 | } 31 | defer response.Body.Close() 32 | 33 | if response.StatusCode > 299 { 34 | // Special handling for 422 error where Fly does not like our request 35 | if response.StatusCode == http.StatusUnprocessableEntity { 36 | loggedResponseBody := "no value yet" 37 | responseBody, err := io.ReadAll(response.Body) 38 | if err != nil { 39 | loggedResponseBody = fmt.Sprintf("error (response body reading): %s", err) 40 | } else { 41 | loggedResponseBody = string(responseBody) 42 | } 43 | return nil, fmt.Errorf("did not create machine, http status: %d, http body: %s", response.StatusCode, loggedResponseBody) 44 | } else { 45 | return nil, fmt.Errorf("did not create machine, http status: %d", response.StatusCode) 46 | } 47 | } 48 | 49 | responseBody, err := io.ReadAll(response.Body) 50 | 51 | if err != nil { 52 | return nil, fmt.Errorf("response body reading error: %w", err) 53 | } 54 | 55 | m := &Machine{} 56 | err = json.Unmarshal(responseBody, m) 57 | 58 | if err != nil { 59 | return nil, fmt.Errorf("could not unmarshall json: %w", err) 60 | } 61 | 62 | return m, nil 63 | } 64 | 65 | type GetMachineInput struct { 66 | AppName string 67 | MachineId string 68 | } 69 | 70 | func (api *Api) GetMachine(i *GetMachineInput) (*Machine, error) { 71 | req := &GetMachineRequest{ 72 | App: App{ 73 | Name: i.AppName, 74 | }, 75 | Machine: Machine{ 76 | Id: i.MachineId, 77 | }, 78 | } 79 | 80 | response, err := DoRequest(api.Token, req) 81 | 82 | if err != nil { 83 | return nil, fmt.Errorf("request error: %w", err) 84 | } 85 | defer response.Body.Close() 86 | 87 | // Incorrect Machine ID's might return a 400 instead of a 404 88 | if response.StatusCode == http.StatusNotFound || response.StatusCode == http.StatusBadRequest { 89 | return nil, MachineNotFoundError{ 90 | App: i.AppName, 91 | Machine: i.MachineId, 92 | Err: err, 93 | } 94 | } 95 | 96 | // Other errors 97 | if response.StatusCode > 299 { 98 | return nil, fmt.Errorf("could not get machine, http status: %d", response.StatusCode) 99 | } 100 | 101 | responseBody, err := io.ReadAll(response.Body) 102 | 103 | b := string(responseBody) 104 | if response.StatusCode > 399 { 105 | return nil, fmt.Errorf("invalid HTTP response '%d': %s", b) 106 | } 107 | 108 | if err != nil { 109 | return nil, fmt.Errorf("response body reading error: %w", err) 110 | } 111 | 112 | logging.GetLogger().Debug("GetMachine response", zap.String("body", b)) 113 | 114 | m := &Machine{} 115 | err = json.Unmarshal(responseBody, m) 116 | 117 | if err != nil { 118 | return nil, fmt.Errorf("could not unmarshall json: %w", err) 119 | } 120 | 121 | return m, nil 122 | } 123 | 124 | type ListMachinesInput struct { 125 | AppName string 126 | } 127 | 128 | func (api *Api) ListMachines(i *ListMachinesInput) (*ListMachinesResponse, error) { 129 | req := &ListMachinesRequest{ 130 | App: App{ 131 | Name: i.AppName, 132 | }, 133 | } 134 | 135 | response, err := DoRequest(api.Token, req) 136 | 137 | if err != nil { 138 | return nil, fmt.Errorf("request error: %w", err) 139 | } 140 | defer response.Body.Close() 141 | 142 | if response.StatusCode > 299 { 143 | return nil, fmt.Errorf("could not list machines, http status: %d", response.StatusCode) 144 | } 145 | 146 | responseBody, err := io.ReadAll(response.Body) 147 | 148 | if err != nil { 149 | return nil, fmt.Errorf("response body reading error: %w", err) 150 | } 151 | 152 | m := &ListMachinesResponse{} 153 | err = json.Unmarshal(responseBody, &m.Machines) 154 | 155 | if err != nil { 156 | return nil, fmt.Errorf("could not unmarshall json: %w", err) 157 | } 158 | 159 | return m, nil 160 | } 161 | 162 | /* TODO: Implement this. It's similar (equivalent to?) 163 | the CreateMachine API call 164 | type UpdateMachineInput struct { 165 | 166 | } 167 | 168 | func (api *Api) UpdateMachine() error { 169 | 170 | } 171 | */ 172 | 173 | type DeleteMachineInput struct { 174 | AppName string 175 | MachineId string 176 | Force bool 177 | } 178 | 179 | func (api *Api) DeleteMachine(i *DeleteMachineInput) error { 180 | req := &DeleteMachineRequest{ 181 | App: App{ 182 | Name: i.AppName, 183 | }, 184 | Machine: Machine{ 185 | Id: i.MachineId, 186 | }, 187 | Force: i.Force, 188 | } 189 | 190 | response, err := DoRequest(api.Token, req) 191 | 192 | if err != nil { 193 | return fmt.Errorf("request error: %w", err) 194 | } 195 | 196 | if response != nil { 197 | // Log "error" http responses, but otherwise ignore errors here 198 | if response.StatusCode > 299 { 199 | logging.GetLogger().Error("could not delete machine", zap.Int("status", response.StatusCode)) 200 | } 201 | } else { 202 | logging.GetLogger().Error("Delete Machine: We should have a response object here, but we do not") 203 | } 204 | 205 | return nil 206 | } 207 | 208 | type StartMachineInput struct { 209 | AppName string 210 | MachineId string 211 | } 212 | 213 | func (api *Api) StartMachine(i *StartMachineInput) error { 214 | req := &StartMachineRequest{ 215 | App: App{ 216 | Name: i.AppName, 217 | }, 218 | Machine: Machine{ 219 | Id: i.MachineId, 220 | }, 221 | } 222 | 223 | _, err := DoRequest(api.Token, req) 224 | 225 | if err != nil { 226 | return fmt.Errorf("request error: %w", err) 227 | } 228 | 229 | return nil 230 | } 231 | 232 | type StopMachineInput struct { 233 | AppName string 234 | MachineId string 235 | } 236 | 237 | func (api *Api) StopMachine(i *StopMachineInput) error { 238 | req := &StopMachineRequest{ 239 | App: App{ 240 | Name: i.AppName, 241 | }, 242 | Machine: Machine{ 243 | Id: i.MachineId, 244 | }, 245 | } 246 | 247 | _, err := DoRequest(api.Token, req) 248 | 249 | if err != nil { 250 | return fmt.Errorf("request error: %w", err) 251 | } 252 | 253 | return nil 254 | } 255 | 256 | // WaitForMachine waits for a newly created Machine 257 | // to become available 258 | func (api *Api) WaitForMachine(i *GetMachineInput) error { 259 | // Total wait time ~5 minutes (should only need a minute or 2) 260 | ticker := time.NewTicker(2 * time.Second) 261 | totalAttempts := 0 262 | attemptsAllowed := 150 263 | 264 | for { 265 | select { 266 | case <-ticker.C: 267 | e, err := api.GetMachine(i) 268 | if err != nil { 269 | ticker.Stop() 270 | return fmt.Errorf("could not get machine: %w", err) 271 | } 272 | 273 | if e.IsInitialized() { 274 | ticker.Stop() 275 | return nil 276 | } 277 | 278 | totalAttempts++ 279 | if totalAttempts >= attemptsAllowed { 280 | ticker.Stop() 281 | return fmt.Errorf("too many GetMachine attempts") 282 | } 283 | } 284 | } 285 | } 286 | -------------------------------------------------------------------------------- /internal/fly/org.go: -------------------------------------------------------------------------------- 1 | package fly 2 | 3 | import ( 4 | "github.com/superfly/lambdo/internal/config" 5 | ) 6 | 7 | func OrgName() string { 8 | if config.GetConfig().Environment == "production" || config.GetConfig().Environment == "prod-debug" { 9 | return "personal" 10 | } 11 | 12 | return "personal" 13 | } 14 | -------------------------------------------------------------------------------- /internal/fly/requests-app.go: -------------------------------------------------------------------------------- 1 | package fly 2 | 3 | import ( 4 | "bytes" 5 | "encoding/json" 6 | "fmt" 7 | "net/http" 8 | ) 9 | 10 | /*********************************** 11 | * Application Requests 12 | ***********************************/ 13 | 14 | /************************* 15 | * Create App Request 16 | *************************/ 17 | 18 | type CreateAppRequest struct { 19 | Name string `json:"app_name"` 20 | Org string `json:"org_slug"` 21 | Network string `json:"network"` 22 | } 23 | 24 | func (r *CreateAppRequest) ToRequest(token string) (*http.Request, error) { 25 | j, err := json.Marshal(r) 26 | 27 | if err != nil { 28 | return nil, fmt.Errorf("could not encode App to JSON: %w", err) 29 | } 30 | 31 | url := fmt.Sprintf("%s/v1/apps", "https://api.machines.dev") 32 | req, err := http.NewRequest(http.MethodPost, url, bytes.NewBuffer(j)) 33 | 34 | if err != nil { 35 | return nil, fmt.Errorf("could not create http request object: %w", err) 36 | } 37 | 38 | StandardRequestHeaders(req, token) 39 | 40 | return req, nil 41 | } 42 | 43 | /************************* 44 | * Get App Request 45 | *************************/ 46 | 47 | type GetAppRequest struct { 48 | App App 49 | } 50 | 51 | func (r *GetAppRequest) ToRequest(token string) (*http.Request, error) { 52 | uri := fmt.Sprintf("%s/v1/apps/%s", "https://api.machines.dev", r.App.Name) 53 | 54 | req, err := http.NewRequest(http.MethodGet, uri, nil) 55 | 56 | if err != nil { 57 | return nil, fmt.Errorf("could not create http request object: %w", err) 58 | } 59 | 60 | StandardRequestHeaders(req, token) 61 | 62 | return req, nil 63 | } 64 | 65 | /************************* 66 | * Delete App Request 67 | *************************/ 68 | 69 | type DeleteAppRequest struct { 70 | App App 71 | } 72 | 73 | func (r *DeleteAppRequest) ToRequest(token string) (*http.Request, error) { 74 | uri := fmt.Sprintf("%s/v1/apps/%s", "https://api.machines.dev", r.App.Name) 75 | 76 | req, err := http.NewRequest(http.MethodDelete, uri, nil) 77 | 78 | if err != nil { 79 | return nil, fmt.Errorf("could not create http request object: %w", err) 80 | } 81 | 82 | StandardRequestHeaders(req, token) 83 | 84 | return req, nil 85 | } 86 | -------------------------------------------------------------------------------- /internal/fly/requests-machine.go: -------------------------------------------------------------------------------- 1 | package fly 2 | 3 | import ( 4 | "bytes" 5 | "encoding/json" 6 | "fmt" 7 | "github.com/superfly/lambdo/internal/logging" 8 | "go.uber.org/zap" 9 | "net/http" 10 | ) 11 | 12 | /*********************************** 13 | * Machine Requests 14 | ***********************************/ 15 | 16 | /************************* 17 | * Create Machine Request 18 | *************************/ 19 | 20 | type CreateMachineRequest struct { 21 | App App 22 | Machine Machine 23 | } 24 | 25 | func (r *CreateMachineRequest) ToRequest(token string) (*http.Request, error) { 26 | j, err := json.Marshal(r.Machine) 27 | 28 | if err != nil { 29 | return nil, fmt.Errorf("could not encode Machine to JSON: %w", err) 30 | } 31 | 32 | logging.GetLogger().Debug("create machine request", zap.ByteString("body", j)) 33 | 34 | uri := fmt.Sprintf("%s/v1/apps/%s/machines", "https://api.machines.dev", r.App.Name) 35 | req, err := http.NewRequest(http.MethodPost, uri, bytes.NewBuffer(j)) 36 | 37 | if err != nil { 38 | return nil, fmt.Errorf("could not create http request object: %w", err) 39 | } 40 | 41 | StandardRequestHeaders(req, token) 42 | 43 | return req, nil 44 | } 45 | 46 | /************************* 47 | * Get Machine Request 48 | *************************/ 49 | 50 | type GetMachineRequest struct { 51 | App App 52 | Machine Machine 53 | } 54 | 55 | func (r *GetMachineRequest) ToRequest(token string) (*http.Request, error) { 56 | uri := fmt.Sprintf("%s/v1/apps/%s/machines/%s", "https://api.machines.dev", r.App.Name, r.Machine.Id) 57 | 58 | req, err := http.NewRequest(http.MethodGet, uri, nil) 59 | 60 | if err != nil { 61 | return nil, fmt.Errorf("could not create http request object: %w", err) 62 | } 63 | 64 | StandardRequestHeaders(req, token) 65 | 66 | return req, nil 67 | } 68 | 69 | /************************* 70 | * List Machines Request 71 | *************************/ 72 | 73 | type ListMachinesRequest struct { 74 | App App 75 | } 76 | 77 | type ListMachinesResponse struct { 78 | Machines []Machine 79 | } 80 | 81 | func (r *ListMachinesRequest) ToRequest(token string) (*http.Request, error) { 82 | uri := fmt.Sprintf("%s/v1/apps/%s/machines", "https://api.machines.dev", r.App.Name) 83 | req, err := http.NewRequest(http.MethodGet, uri, nil) 84 | 85 | if err != nil { 86 | return nil, fmt.Errorf("could not create http request object: %w", err) 87 | } 88 | 89 | StandardRequestHeaders(req, token) 90 | 91 | return req, nil 92 | } 93 | 94 | /************************* 95 | * Update Machine Request 96 | *************************/ 97 | 98 | type UpdateMachineRequest struct { 99 | App App 100 | Machine Machine 101 | } 102 | 103 | func (r *UpdateMachineRequest) ToRequest(token string) (*http.Request, error) { 104 | j, err := json.Marshal(r.Machine) 105 | 106 | if err != nil { 107 | return nil, fmt.Errorf("could not encode Machine to JSON: %w", err) 108 | } 109 | 110 | uri := fmt.Sprintf("%s/v1/apps/%s/machines/%s", "https://api.machines.dev", r.App.Name, r.Machine.Id) 111 | req, err := http.NewRequest(http.MethodPost, uri, bytes.NewBuffer(j)) 112 | 113 | if err != nil { 114 | return nil, fmt.Errorf("could not create http request object: %w", err) 115 | } 116 | 117 | StandardRequestHeaders(req, token) 118 | 119 | return req, nil 120 | } 121 | 122 | /************************* 123 | * Delete Machine Request 124 | *************************/ 125 | 126 | type DeleteMachineRequest struct { 127 | App App 128 | Machine Machine 129 | Force bool 130 | } 131 | 132 | func (r *DeleteMachineRequest) ToRequest(token string) (*http.Request, error) { 133 | var force string 134 | if r.Force { 135 | force = "?kill=true" 136 | } else { 137 | force = "" 138 | } 139 | 140 | uri := fmt.Sprintf("%s/v1/apps/%s/machines/%s%s", "https://api.machines.dev", r.App.Name, r.Machine.Id, force) 141 | 142 | req, err := http.NewRequest(http.MethodDelete, uri, nil) 143 | 144 | if err != nil { 145 | return nil, fmt.Errorf("could not create http request object: %w", err) 146 | } 147 | 148 | StandardRequestHeaders(req, token) 149 | 150 | return req, nil 151 | } 152 | 153 | /************************* 154 | * Start Machine Request 155 | *************************/ 156 | 157 | type StartMachineRequest struct { 158 | App App 159 | Machine Machine 160 | } 161 | 162 | func (r *StartMachineRequest) ToRequest(token string) (*http.Request, error) { 163 | uri := fmt.Sprintf("%s/v1/apps/%s/machines/%s/start", "https://api.machines.dev", r.App.Name, r.Machine.Id) 164 | req, err := http.NewRequest(http.MethodPost, uri, nil) 165 | 166 | if err != nil { 167 | return nil, fmt.Errorf("could not create http request object: %w", err) 168 | } 169 | 170 | StandardRequestHeaders(req, token) 171 | 172 | return req, nil 173 | } 174 | 175 | /************************* 176 | * Stop Machine Request 177 | *************************/ 178 | 179 | type StopMachineRequest struct { 180 | App App 181 | Machine Machine 182 | } 183 | 184 | func (r *StopMachineRequest) ToRequest(token string) (*http.Request, error) { 185 | uri := fmt.Sprintf("%s/v1/apps/%s/machines/%s/stop", "https://api.machines.dev", r.App.Name, r.Machine.Id) 186 | req, err := http.NewRequest(http.MethodPost, uri, nil) 187 | 188 | if err != nil { 189 | return nil, fmt.Errorf("could not create http request object: %w", err) 190 | } 191 | 192 | StandardRequestHeaders(req, token) 193 | 194 | return req, nil 195 | } 196 | -------------------------------------------------------------------------------- /internal/fly/types.go: -------------------------------------------------------------------------------- 1 | package fly 2 | 3 | import ( 4 | "golang.org/x/exp/slices" 5 | ) 6 | 7 | const TypeShared = "shared" 8 | const TypePerf = "performance" 9 | 10 | type Organization struct { 11 | Name string `json:"name"` 12 | Slug string `json:"slug"` 13 | } 14 | 15 | type App struct { 16 | Name string `json:"name"` 17 | Organization Organization `json:"organization"` 18 | } 19 | 20 | type Machine struct { 21 | Id string `json:"id,omitempty"` 22 | Name string `json:"name,omitempty"` 23 | State string `json:"state,omitempty"` 24 | Region string `json:"region"` 25 | PrivateIp string `json:"private_ip,omitempty"` 26 | Config MachineConfig `json:"config"` 27 | } 28 | 29 | // IsInitialized checks the state of the machine to see if it 30 | // is finished being created. 31 | // See https://fly.io/docs/reference/machines/#machine-states 32 | func (m *Machine) IsInitialized() bool { 33 | initValues := []string{"started", "stopped", "stopping"} 34 | 35 | return slices.Contains(initValues, m.State) 36 | } 37 | 38 | type MachineConfig struct { 39 | Image string `json:"image"` 40 | Guest MachineSize `json:"guest,omitempty"` 41 | Size string `json:"size,omitempty"` 42 | Env map[string]string `json:"env,omitempty"` 43 | Services []MachineService `json:"services,omitempty"` 44 | Processes []MachineProcess `json:"processes,omitempty"` 45 | MetaData map[string]string `json:"metadata,omitempty"` 46 | Files []MachineFile `json:"files,omitempty"` 47 | AutoDestroy bool `json:"auto_destroy,omitempty"` 48 | } 49 | 50 | type MachineSize struct { 51 | CpuCount int `json:"cpus"` 52 | RAM int `json:"memory_mb"` 53 | Type string `json:"cpu_kind"` 54 | } 55 | 56 | type MachineService struct { 57 | InternalPort int `json:"internal_port"` // 8000 58 | Protocol string `json:"protocol"` // "tcp" 59 | Ports []Port `json:"ports"` 60 | } 61 | 62 | type MachineProcess struct { 63 | Name string `json:"name,omitempty"` 64 | Entrypoint []string `json:"entrypoint,omitempty"` 65 | Cmd []string `json:"cmd,omitempty"` 66 | Env map[string]string `json:"env,omitempty"` 67 | User string `json:"user,omitempty"` 68 | } 69 | 70 | type MachineFile struct { 71 | GuestPath string `json:"guest_path"` 72 | RawValue string `json:"raw_value,omitempty"` 73 | SecretName string `json:"secret_name,omitempty"` 74 | } 75 | 76 | type Port struct { 77 | Port int `json:"port"` // 80, 443 78 | Handlers []string `json:"handlers"` // ["http"], ["tls", "http"] 79 | } 80 | -------------------------------------------------------------------------------- /internal/logging/logging.go: -------------------------------------------------------------------------------- 1 | package logging 2 | 3 | import ( 4 | "fmt" 5 | "go.uber.org/zap" 6 | ) 7 | 8 | var logger *zap.Logger 9 | 10 | // GetLogger is a global helper to get the logger object 11 | func GetLogger() *zap.Logger { 12 | return logger 13 | } 14 | 15 | // SetupLogging creates the logger object relevant 16 | // to the current application environment 17 | func SetupLogging(production bool) error { 18 | var l *zap.Logger 19 | var err error 20 | if production { 21 | if l, err = zap.NewProduction(); err != nil { 22 | return fmt.Errorf("could not setup production logger: %w", err) 23 | } 24 | } else { 25 | if l, err = zap.NewDevelopment(); err != nil { 26 | return fmt.Errorf("could not setup development logger: %w", err) 27 | } 28 | } 29 | 30 | logger = l 31 | 32 | return nil 33 | } 34 | -------------------------------------------------------------------------------- /internal/sqs/client.go: -------------------------------------------------------------------------------- 1 | package sqs 2 | 3 | import ( 4 | "context" 5 | "github.com/aws/aws-sdk-go-v2/aws" 6 | "github.com/aws/aws-sdk-go-v2/config" 7 | "github.com/aws/aws-sdk-go-v2/service/sqs" 8 | "github.com/superfly/lambdo/internal/logging" 9 | "go.uber.org/zap" 10 | ) 11 | 12 | var awsConfig aws.Config 13 | var client *sqs.Client 14 | 15 | func init() { 16 | config, err := config.LoadDefaultConfig(context.TODO()) 17 | if err != nil { 18 | logging.GetLogger().Error("could not make aws config", zap.Error(err)) 19 | } else { 20 | awsConfig = config 21 | client = sqs.NewFromConfig(awsConfig) 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /internal/sqs/delete_messages.go: -------------------------------------------------------------------------------- 1 | package sqs 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "github.com/aws/aws-sdk-go-v2/aws" 7 | "github.com/aws/aws-sdk-go-v2/service/sqs" 8 | "github.com/superfly/lambdo/internal/config" 9 | ) 10 | 11 | func DeleteMessage(receiptHandler string) error { 12 | _, err := client.DeleteMessage(context.TODO(), &sqs.DeleteMessageInput{ 13 | QueueUrl: aws.String(config.GetConfig().SQSQueueUrl), 14 | ReceiptHandle: aws.String(receiptHandler), 15 | }) 16 | 17 | if err != nil { 18 | return fmt.Errorf("could not delete message: %w", err) 19 | } 20 | 21 | return nil 22 | } 23 | -------------------------------------------------------------------------------- /internal/sqs/get_events.go: -------------------------------------------------------------------------------- 1 | package sqs 2 | 3 | import ( 4 | "context" 5 | "errors" 6 | "fmt" 7 | "github.com/aws/aws-sdk-go-v2/aws" 8 | "github.com/aws/aws-sdk-go-v2/service/sqs" 9 | "github.com/aws/aws-sdk-go-v2/service/sqs/types" 10 | cfg "github.com/superfly/lambdo/internal/config" 11 | "github.com/superfly/lambdo/internal/logging" 12 | "time" 13 | ) 14 | 15 | type MessageHandler func([]types.Message) 16 | 17 | func Listen(ctx context.Context, messages chan []types.Message) error { 18 | logging.GetLogger().Info("listening on SQS queue") 19 | 20 | GETMSGS: 21 | for { 22 | select { 23 | case <-ctx.Done(): 24 | logging.GetLogger().Info("Shutdown: No longer listening for messages") 25 | break GETMSGS 26 | default: 27 | logging.GetLogger().Debug("about to call sqs.ReceiveMessage") 28 | response, sqsErr := client.ReceiveMessage(ctx, &sqs.ReceiveMessageInput{ 29 | QueueUrl: aws.String(cfg.GetConfig().SQSQueueUrl), 30 | MaxNumberOfMessages: int32(cfg.GetConfig().EventsPerMachine), // max of 10 31 | WaitTimeSeconds: int32(cfg.GetConfig().SQSLongPollSeconds), // long polling 32 | VisibilityTimeout: 30, // POC queue defaults to 30, we mirror that here 33 | MessageAttributeNames: []string{"image", "size", "command"}, 34 | }) 35 | 36 | if sqsErr != nil { 37 | if !errors.Is(sqsErr, context.Canceled) { 38 | return fmt.Errorf("could not receive SQS messages: %w", sqsErr) 39 | } else { 40 | logging.GetLogger().Debug("ReceiveMessage stopped: context cancelled") 41 | return nil 42 | } 43 | 44 | } 45 | 46 | if len(response.Messages) > 0 { 47 | // Fire and forget from this function's point of view 48 | messages <- response.Messages 49 | } 50 | 51 | // Add time between calls if we don't long poll 52 | if cfg.GetConfig().SQSLongPollSeconds < 1 { 53 | time.Sleep(time.Millisecond * 250) 54 | } 55 | } 56 | } 57 | 58 | return nil 59 | } 60 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "context" 5 | "github.com/superfly/lambdo/cmd" 6 | "github.com/superfly/lambdo/internal/config" 7 | "github.com/superfly/lambdo/internal/logging" 8 | "go.uber.org/zap" 9 | "log" 10 | "os" 11 | "os/signal" 12 | "syscall" 13 | ) 14 | 15 | func main() { 16 | err := config.Configure() 17 | 18 | if err != nil { 19 | log.Printf("configuration error: %v", err) 20 | os.Exit(1) 21 | } 22 | 23 | err = logging.SetupLogging(config.GetConfig().Environment == "production") 24 | 25 | if err != nil { 26 | log.Printf("logging error: %v", err) 27 | os.Exit(1) 28 | } 29 | 30 | ctx, cancel := context.WithCancel(context.Background()) 31 | defer cancel() 32 | 33 | sigs := make(chan os.Signal, 1) 34 | defer close(sigs) 35 | signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM) 36 | 37 | go func() { 38 | sig := <-sigs 39 | logging.GetLogger().Debug("signal received", zap.String("signal", sig.String())) 40 | cancel() 41 | return 42 | }() 43 | 44 | logging.GetLogger().Info("starting lambdo") 45 | 46 | err = cmd.Execute(ctx) 47 | 48 | if err != nil { 49 | panic(err) 50 | } 51 | 52 | logging.GetLogger().Info("👋 exiting lambdo") 53 | } 54 | -------------------------------------------------------------------------------- /runtimes/fly-builder/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM docker/buildx-bin:v0.12 as buildx 2 | 3 | FROM docker:24-dind 4 | 5 | RUN apk add bash iptables-legacy pigz sysstat procps lsof \ 6 | && mv /sbin/iptables /sbin/iptables.original \ 7 | && mv /sbin/ip6tables /sbin/ip6tables.original \ 8 | && ln -s /sbin/iptables-legacy /sbin/iptables \ 9 | && ln -s /sbin/ip6tables-legacy /sbin/ip6tables 10 | 11 | COPY etc/docker/daemon.json /etc/docker/daemon.json 12 | 13 | COPY --from=buildx /buildx /root/.docker/cli-plugins/docker-buildx 14 | 15 | COPY ./entrypoint ./entrypoint 16 | COPY ./docker-entrypoint.d/* ./docker-entrypoint.d/ 17 | 18 | ENV DOCKER_TMPDIR=/data/docker/tmp 19 | 20 | ENTRYPOINT ["./entrypoint"] 21 | 22 | CMD ["dockerd", "-p", "/var/run/docker.pid"] 23 | -------------------------------------------------------------------------------- /runtimes/fly-builder/docker-entrypoint.d/docker: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e 4 | 5 | echo "Setting up Docker data directory" 6 | mkdir -p /data/docker 7 | 8 | echo "Configuring ipv6 for docker" 9 | ip6tables-legacy -t nat -A POSTROUTING -s 2001:db8:1::/64 ! -o docker0 -j MASQUERADE 10 | 11 | echo "Done setting up docker!" 12 | -------------------------------------------------------------------------------- /runtimes/fly-builder/docker-entrypoint.d/sysctl: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e 4 | 5 | echo "Allowing ipv6 forwarding via sysctl" 6 | sysctl net.ipv6.conf.default.forwarding=1 7 | sysctl net.ipv6.conf.all.forwarding=1 8 | 9 | echo "General sysctl tweaks" 10 | sysctl vm.swappiness=0 11 | sysctl vm.dirty_ratio=6 12 | sysctl vm.dirty_background_ratio=3 13 | 14 | # Default Socket Receive Buffer 15 | sysctl net.core.rmem_default=31457280 16 | 17 | # Maximum Socket Receive Buffer 18 | sysctl net.core.rmem_max=33554432 19 | 20 | # Default Socket Send Buffer 21 | sysctl net.core.wmem_default=31457280 22 | 23 | # Maximum Socket Send Buffer 24 | sysctl net.core.wmem_max=33554432 25 | 26 | # Increase number of incoming connections 27 | sysctl net.core.somaxconn=65535 28 | 29 | # Increase number of incoming connections backlog 30 | sysctl net.core.netdev_max_backlog=65536 31 | 32 | # Increase the maximum amount of option memory buffers 33 | sysctl net.core.optmem_max=25165824 34 | 35 | # Increase the maximum total buffer-space allocatable 36 | # This is measured in units of pages (4096 bytes) 37 | sysctl "net.ipv4.tcp_mem=786432 1048576 26777216" 38 | sysctl "net.ipv4.udp_mem=65536 131072 262144" 39 | 40 | # Increase the read-buffer space allocatable 41 | sysctl "net.ipv4.tcp_rmem=8192 87380 33554432" 42 | sysctl net.ipv4.udp_rmem_min=16384 43 | 44 | # Increase the write-buffer-space allocatable 45 | sysctl "net.ipv4.tcp_wmem=8192 65536 33554432" 46 | sysctl net.ipv4.udp_wmem_min=16384 -------------------------------------------------------------------------------- /runtimes/fly-builder/entrypoint: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e 4 | 5 | if [[ -d "docker-entrypoint.d" ]] 6 | then 7 | echo "Running docker-entrypoint.d files" 8 | /bin/run-parts docker-entrypoint.d 9 | fi 10 | 11 | echo "Running $@" 12 | 13 | exec "$@" -------------------------------------------------------------------------------- /runtimes/fly-builder/etc/docker/daemon.json: -------------------------------------------------------------------------------- 1 | { 2 | "data-root": "/data/docker", 3 | "experimental": true, 4 | "ipv6": true, 5 | "ip6tables": true, 6 | "fixed-cidr-v6": "2001:db8:1::/64", 7 | "default-address-pools": [ 8 | { 9 | "base": "10.100.0.1/16", 10 | "size": 24 11 | } 12 | ], 13 | "debug": true, 14 | "log-level": "debug", 15 | "features": { 16 | "buildkit": true 17 | }, 18 | "hosts": [ 19 | "unix:///var/run/docker.sock", 20 | "tcp://[::]:2375" 21 | ], 22 | "mtu": 1400, 23 | "max-concurrent-downloads": 10, 24 | "max-concurrent-uploads": 5, 25 | "metrics-addr": "0.0.0.0:9323", 26 | "tls": false 27 | } -------------------------------------------------------------------------------- /runtimes/fly-builder/readme.md: -------------------------------------------------------------------------------- 1 | # Fly Builder 2 | 3 | Run Docker in a Fly.io VM. 4 | 5 | This helps run `docker build...` on AMD64 instances, especially useful when you're hacking on this project from an ARM-based infrastructure or a machine that runs on a battery. 6 | 7 | A lot of this is based on [this repository](https://github.com/fly-apps/docker-daemon). 8 | 9 | ## Setting Up the Remote Builder 10 | 11 | Creating a builder for use like this can be done using [Fly Machines](https://fly.io/docs/machines/). 12 | 13 | Here are the steps: 14 | 15 | ```bash 16 | # Create an app to house the VM that will run Docker 17 | fly apps create --name lambdo-docker-builder 18 | 19 | # Create a volume 20 | fly volumes create -r dfw -a lambdo-docker-builder -s 50 lambdo_docker 21 | 22 | # From this directory, which contains a Dockerfile 23 | fly m run . -r dfw -v lambdo_docker:/data -a lambdo-docker-builder 24 | ``` 25 | 26 | ## Using the Remote Builder 27 | 28 | Once you [setup Wireguard](https://fly.io/docs/reference/private-networking/#private-network-vpn) with Fly, you can activate the VPN and run Docker builds remotely by setting `DOCKER_HOST`: 29 | 30 | ```bash 31 | # Get the IP Address of the VM we created 32 | fly m list -a lambdo-docker-builder 33 | 34 | # Use that as your DOCKER_HOST, in one of these 2 formats: 35 | DOCKER_HOST=.vm..internal docker ps 36 | DOCKER_HOST=tcp://[ipv6-here]:2375 docker ps 37 | ``` 38 | -------------------------------------------------------------------------------- /runtimes/go/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM go:1.21 2 | 3 | -------------------------------------------------------------------------------- /runtimes/go/README.md: -------------------------------------------------------------------------------- 1 | # TODO 2 | 3 | A Go runtime requires a bit more work, we can steal ideas from Lambda [[1](https://docs.aws.amazon.com/lambda/latest/dg/golang-handler.html)], [[2](https://github.com/aws/aws-lambda-go/blob/main/lambda/entry.go)]. 4 | 5 | 1. A running web server in the container that will return the JSON events 6 | 2. A package that the user-code incorporates, which takes the handler function, calls the localhost HTTP server, gets the events, and calls the handler function -------------------------------------------------------------------------------- /runtimes/js/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM node:20-slim 2 | 3 | COPY src/index.js /opt/index.js 4 | 5 | CMD ["node", "/opt/index.js"] -------------------------------------------------------------------------------- /runtimes/js/README.md: -------------------------------------------------------------------------------- 1 | # Lambdo JS Runtime 2 | -------------------------------------------------------------------------------- /runtimes/js/build.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Requires building on x86-64 architecture 4 | # for Fly VM's 5 | docker build \ 6 | -t fideloper/lambdo-js:20 \ 7 | -f Dockerfile \ 8 | . 9 | 10 | # aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/i2r3m5g4 11 | # docker push public.ecr.aws/i2r3m5g4/runtime-js:latest 12 | -------------------------------------------------------------------------------- /runtimes/js/sample-project/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM fideloper/lambdo-js:20 2 | 3 | # Hack to run locally, this would otherwise be 4 | # placed automatically in a Fly VM when created 5 | # COPY events.json /tmp/events.json 6 | 7 | COPY src/ /app/ 8 | -------------------------------------------------------------------------------- /runtimes/js/sample-project/events.json: -------------------------------------------------------------------------------- 1 | [ 2 | {"foo": "bar"}, 3 | {"baz": "cux"} 4 | ] 5 | -------------------------------------------------------------------------------- /runtimes/js/sample-project/src/index.js: -------------------------------------------------------------------------------- 1 | exports.handler = async function(event) { 2 | console.log("Let's process an event! The event:", event) 3 | } 4 | -------------------------------------------------------------------------------- /runtimes/js/src/index.js: -------------------------------------------------------------------------------- 1 | const fs = require('fs'); 2 | 3 | try { 4 | const eventsString = fs.readFileSync(process.env.EVENTS_PATH) 5 | const events = JSON.parse(eventsString) 6 | 7 | const handler_module = require('/app/index.js'); 8 | 9 | for (let key in events) { 10 | try { 11 | handler_module.handler(events[key]); 12 | } catch(e) { 13 | console.error("handler execution error", e) 14 | } 15 | } 16 | } catch (e) { 17 | console.error("error retrieving or running events", e) 18 | process.exit(1) 19 | } 20 | 21 | 22 | -------------------------------------------------------------------------------- /runtimes/php/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM php:8.2-cli 2 | 3 | COPY src/index.php /opt/index.php 4 | 5 | CMD ["php", "/opt/index.php"] 6 | -------------------------------------------------------------------------------- /runtimes/php/README.md: -------------------------------------------------------------------------------- 1 | # Lambdo PHP Runtime 2 | -------------------------------------------------------------------------------- /runtimes/php/build.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Requires building on x86-64 architecture 4 | # for Fly VM's 5 | docker build \ 6 | -t fideloper/lambdo-php:8.2 \ 7 | -f Dockerfile \ 8 | . 9 | 10 | # aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/i2r3m5g4 11 | # docker push public.ecr.aws/i2r3m5g4/runtime-js:latest 12 | -------------------------------------------------------------------------------- /runtimes/php/sample-project/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM fideloper/lambdo-php:8.2 2 | 3 | # Hack to run locally, this would otherwise be 4 | # placed automatically in a Fly VM when created 5 | # COPY events.json /tmp/events.json 6 | 7 | COPY src/ /app/ 8 | -------------------------------------------------------------------------------- /runtimes/php/sample-project/events.json: -------------------------------------------------------------------------------- 1 | [ 2 | {"foo": "bar"}, 3 | {"baz": "cux"} 4 | ] 5 | -------------------------------------------------------------------------------- /runtimes/php/sample-project/src/index.php: -------------------------------------------------------------------------------- 1 | getMessage(); 21 | } 22 | } 23 | 24 | } catch(\Exception $e) { 25 | echo "error: " . $e->getMessage(); 26 | exit(1); 27 | } --------------------------------------------------------------------------------