├── .gitignore ├── LICENSE ├── Makefile ├── README.md ├── bin └── recco ├── example_spec.yml ├── go.mod ├── main.go ├── scan.go └── scanners └── rails.yml /.gitignore: -------------------------------------------------------------------------------- 1 | # Binaries for programs and plugins 2 | *.exe 3 | *.exe~ 4 | *.dll 5 | *.so 6 | *.dylib 7 | 8 | # Test binary, built with `go test -c` 9 | *.test 10 | 11 | # Output of the go coverage tool, specifically when used with LiteIDE 12 | *.out 13 | 14 | # Dependency directories (remove the comment below to include it) 15 | # vendor/ 16 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2022, fly.io 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | all: build 2 | 3 | build: 4 | go build -o bin/recco -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Recco extracts information from application source trees to inform and simplify deployment on supported platforms. This work is initially designed to inform Nix-based deployments on [Fly.io](https://fly.io). 2 | 3 | ## Concepts 4 | 5 | Recco has one simple goal: given an application source tree, it will generate a *YAML deployment specification* to be used by build systems and deployment platforms. Recco should have few dependencies and be runnable at deploy time to catch changes made in the source tree. 6 | 7 | Knowledge about runtimes, frameworks and common application services is encoded in YAML files called *scanners*. Here's a sample [scanner for Rails apps](scanners/rails.yml). 8 | 9 | Scanners can be of different types with different rules: runtime (ruby), framework (rails) or service (sidekiq). We know things about each of these, such as whether YJIT is available (Ruby 3.1.1 and above), whether to set `RAILS_MASTER_KEY` (if credentials are present) or whether to run a second process for the app (a Sidekiq worker). 10 | 11 | ## Implementation dieas 12 | 13 | Roughly, here's how it could work: 14 | 15 | 1. Run a source three through all scanners in `scanners/*.yml` 16 | 2. For matched runtimes, extract the version to inform package installers (like the Ruby version from `.ruby-version` or `Gemfile`) 17 | 3. For matched frameworks, extract versions to inform setting secrets (like `RAILS_MASTER_KEY` from `config/master.key`) 18 | 4. For matched services, set env vars and secrets like puma WEB_CONCURRENCY and 'puma -c config/puma.rb' 19 | 5. Export a YAML spec intended to be versioned in source trees and picked up by build systems 20 | 21 | One could imagine this being extended to run commands for deploy preparation, for example to create a [Docker-based release for Phoenix](https://hexdocs.pm/phoenix/Mix.Tasks.Phx.Gen.Release.html). 22 | 23 | ## Why do we need this? 24 | 25 | Today, information about configuring deployments is locked up in Dockerfiles, buildpacks, platform-specific tooling and developers heads. Developers are no longer comfortable saying "Just deploy on Heroku". We need a way to break out of that mindset with confidence and without a degree in Dockerfiles. You've probably run into a lot of problems unrelated to your app code that you've quietly erased from your memory. The goal is to reduce friction at deploy time while also educating developers about what's possible. 26 | 27 | ## What about Buildpacks? 28 | 29 | Buildpacks, at first glance, seem to offer a simple middleware-like API as a way out. But the reality is that buildpacks are fragmented across platforms and make no consistent guarantees about how they'll behave. It's a whole different experience running a standard deployment on Heroku and deploying with Heroku's buildpack-compatible Docker builder. The same runtimes and frameworks have multiple implementations of the same logic in bash, Golang, and so on. 30 | 31 | ## What, are some no-code zealot? Isn't this just reinventing the wheel? 32 | 33 | Nah - let's look at the details of the problems developers have at deploy time. I think they come down to: 34 | 35 | 1. Lack of knowledge on what knobs to turn for deployment, and how 36 | 2. Lack of clarity around the actual build process: do we have caching, persistent storage, etc? 37 | 3. Lack of visibilty into the logic used to make decisions about the deployment environment (disparate dockerfiles, buildpacks) 38 | 4. Lack of confidence making changes to deployment configuration (packages going missing) 39 | 5. Lack of flexibility when trying to compose software (mismatched versions of Ruby, Node, etc) 40 | 41 | A lot of this lack is related to details of the build system. But some is related to the possibilities in each domain being obscured. 42 | 43 | ## Isn't this just declarative versus imperative config? 44 | 45 | I anticipate pushback here from those who shun declarative configuration over code. Here we're not going this far. We're simply extracting what is actually *data* to a digestible and extensible format. The more data we can extract, the simpler the code that consumes this information will need to be. 46 | 47 | ## How does this information get into a build system? 48 | 49 | I think this information may only be useful to systems that offer fine-grained control over deployment configurations. Things like running multiple processes in-VMs, or installing specific versions of runtimes, can be hard to compose in inflexible systems like Dockerfiles or fragmented systems like buildpacks. At Fly, we're looking into Nix to help us here. More on this topic soon. 50 | -------------------------------------------------------------------------------- /bin/recco: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/superfly/recco/6933e24fca4d26592cb7ceedfefad5810db88c12/bin/recco -------------------------------------------------------------------------------- /example_spec.yml: -------------------------------------------------------------------------------- 1 | runtime: 2 | name: ruby 3 | version: 2.7.5 4 | yjit: true 5 | jemalloc: true 6 | framework: 7 | name: rails 8 | version: 7.0.1.2 9 | secrets: 10 | - key: RAILS_MASTER_KEY 11 | from_file: config/master.key 12 | required: true 13 | - key: DATABASE_URL 14 | required: true 15 | services: 16 | - name: puma 17 | command: puma -c config/puma.rb 18 | env: 19 | # Possibly detect CPU and configure this at runtime? 20 | WEB_CONCURRENCY: 2 21 | RAILS_MAX_THREADS: 5 22 | RAILS_SERVE_STATIC_FILES: true 23 | - name: sidekiq 24 | command: sidekiq 25 | env: 26 | # Possibly detect this at runtime? 27 | SIDEKIQ_CONCURRENCY: 2 28 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/superfly/recco 2 | 3 | go 1.17 4 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "os" 7 | "os/signal" 8 | "runtime" 9 | "syscall" 10 | ) 11 | 12 | func main() { 13 | 14 | ctx, cancel := newContext() 15 | defer cancel() 16 | run(ctx) 17 | } 18 | 19 | func newContext() (context.Context, context.CancelFunc) { 20 | // NOTE: when signal.Notify is called for os.Interrupt it traps both 21 | // ^C (Control-C) and ^BREAK (Control-Break) on Windows. 22 | 23 | signals := []os.Signal{os.Interrupt} 24 | if runtime.GOOS != "windows" { 25 | signals = append(signals, syscall.SIGTERM) 26 | } 27 | 28 | return signal.NotifyContext(context.Background(), signals...) 29 | } 30 | 31 | type DetectedSource struct { 32 | Runtime string 33 | Framework string 34 | RuntimeVersion string 35 | FrameworkVersion string 36 | } 37 | 38 | func run(ctx context.Context) { 39 | fmt.Println("implement me!") 40 | // See README for implementation ideas 41 | } 42 | -------------------------------------------------------------------------------- /scan.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bufio" 5 | "os" 6 | "path/filepath" 7 | "regexp" 8 | ) 9 | 10 | func fileExists(filenames ...string) scanFn { 11 | return func(dir string) bool { 12 | for _, filename := range filenames { 13 | info, err := os.Stat(filepath.Join(dir, filename)) 14 | if err != nil { 15 | continue 16 | } 17 | if !info.IsDir() { 18 | return true 19 | } 20 | } 21 | return false 22 | } 23 | } 24 | 25 | func fileContains(path string, pattern string) bool { 26 | file, err := os.Open(path) 27 | 28 | if err != nil { 29 | return false 30 | } 31 | 32 | defer file.Close() 33 | 34 | scanner := bufio.NewScanner(file) 35 | 36 | for scanner.Scan() { 37 | re := regexp.MustCompile(pattern) 38 | if re.MatchString(scanner.Text()) { 39 | return true 40 | } 41 | } 42 | 43 | return false 44 | } 45 | 46 | func dirContains(glob string, patterns ...string) scanFn { 47 | return func(dir string) bool { 48 | for _, pattern := range patterns { 49 | filenames, _ := filepath.Glob(filepath.Join(dir, glob)) 50 | for _, filename := range filenames { 51 | if fileContains(filename, pattern) { 52 | return true 53 | } 54 | } 55 | } 56 | return false 57 | } 58 | } 59 | 60 | type scanFn func(dir string) bool 61 | 62 | func sourceTriggers(sourceDir string, scanners ...scanFn) bool { 63 | for _, check := range scanners { 64 | if check(sourceDir) { 65 | return true 66 | } 67 | } 68 | return false 69 | } 70 | -------------------------------------------------------------------------------- /scanners/rails.yml: -------------------------------------------------------------------------------- 1 | matcher: 2 | - file: Gemfile 3 | # These regex should at least name 'name' and 'version' 4 | regex: \n\s{4}(?.+)\s\((?.+)+\) 5 | package: rails 6 | secrets: 7 | - key: RAILS_MASTER_KEY 8 | error_message: We couldn't find a suitable source for setting RAILS_MASTER_KEY, so it's been set to a placeholder value 9 | source_files: 10 | # Standard location for apps without per-environment credentials 11 | - config/master.key 12 | # Common location for apps with per-environment credentials 13 | - config/credentials/production.key --------------------------------------------------------------------------------