├── README.md ├── alias.go ├── build.sh ├── common.go ├── crosscompile.bash ├── dante.go ├── docker.go ├── docs ├── dante1.jpg ├── layers_base.png ├── layers_base.svg ├── layers_test.png └── layers_test.svg ├── fs.go ├── inventory.go ├── push.go ├── test.go └── test ├── debian └── jessie │ ├── iojs │ └── 1.4.2 │ │ └── Dockerfile │ └── node │ └── 0.12.0 │ └── Dockerfile ├── inventory.yml └── tests ├── iojs └── Dockerfile ├── node └── Dockerfile └── node12 └── Dockerfile /README.md: -------------------------------------------------------------------------------- 1 | # *This project is no longer actively maintained* 😢 2 | 3 | Dante 4 | ===== 5 | 6 | _Build tests against Docker images by harnessing the power of layers_ 7 | 8 | ![dante](/docs/dante1.jpg) 9 | 10 | > When I had journeyed half of our life's way, 11 | > I found myself within a shadowed forest, 12 | > for I had lost the path that does not stray. 13 | 14 | Dante is a tool for building and running validation tests against Dockerfiles. With Dante you can ensure your Dockerfiles produce a safe and stable environment for your applications. 15 | 16 | Dante is the perfect tool for CI servers and local development. We do not recommend using this tool in a production environment, its purpose is to verify images are production ready _before_ they reach production. 17 | 18 | # Usage 19 | 20 | ## Setup 21 | 22 | Getting ready to use Dante is a 3 step process. 23 | 24 | 1. Naturally, you need to have one or more environments defined as `Dockerfile`s 25 | 2. Define tests that run in your environment using `Dockerfile`s 26 | 3. Define an `inventory.yml` file, which describes the structure of your project directory 27 | 28 | ## Commands 29 | 30 | ### test 31 | 32 | Example: `dante test` 33 | 34 | Builds all the images and subsequently runs tests on top of them. 35 | 36 | ### push 37 | 38 | Example: `dante push` 39 | 40 | Pushes any images that exist on the host machine containing the tags defined in `inventoy.yml` to the Docker registry (not including tests). 41 | 42 | ## Flags 43 | 44 | All commands support this set of flags: 45 | 46 | * `-j COUNT` runs COUNT jobs in parallel. 47 | * `-r COUNT` retry failed jobs COUNT times. 48 | 49 | ### `inventory.yml` File 50 | 51 | The tool is driven by a single yaml file in the base of your project directory named `inventory.yml`. 52 | 53 | An `inventory.yml` may look like this: 54 | 55 | ```yaml 56 | images: 57 | - name: "wblankenship/dockeri.co:server" 58 | path: "./dockerico/server" 59 | test: ["./dockerico/tests/http","./dockerico/tests/badges"] 60 | alias: ["wblankenship/dockeri.co:latest"] 61 | - name: "wblankenship/dockeri.co:database" 62 | path: "./dockerico/database" 63 | test: "./dockerico/tests/db" 64 | ``` 65 | 66 | Where the corresponding project directory would look like this: 67 | 68 | ```text 69 | . 70 | ├── dockerico 71 | │   ├── db 72 | │   │   └── Dockerfile 73 | │   ├── server 74 | │   │   └── Dockerfile 75 | │   └── tests 76 | │   ├── badges 77 | │   │   └── Dockerfile 78 | │   ├── db 79 | │   │   ├── dependency.tar 80 | │   │   └── Dockerfile 81 | │   └── http 82 | │   └── Dockerfile 83 | └── inventory.yml 84 | ``` 85 | 86 | ### Tests 87 | 88 | Tests are defined in the `inventory.yml` file using the `test` key, which can accept either a single string or an array of strings as a value. 89 | 90 | A test is simply `Dockerfile` and looks like this: 91 | 92 | ```Dockerfile 93 | WORKDIR /usr/src/app 94 | ADD dependency.tar / 95 | RUN tar -xvf dependency.tar 96 | RUN this_will_fail 97 | RUN echo "SUCCESS!" 98 | ``` 99 | 100 | When Dante runs, it will build each layer defined in the test `Dockerfile` on top of the image produced by the `Dockerfile` it is testing. If any command is unsuccesful, Dante will mark the image as having failed the test. In this example case the line `RUN this_will_fail` will result in the entire test failing. 101 | 102 | It is safe to include dependencies in the directory with the `Dockerfile` as demonstrated with the line `ADD dependency.tar /`. Dante will upload the entire working directory as context to the docker daemon when building the image. 103 | 104 | You may have noticed the missing `FROM` command in the `Dockerfile`. This is intentional as Dante will build this `Dockerfile` from the image it is a test for. If you are interested in how this works or why we do it this way, refer to our [Philosophy](#philosophy) section. 105 | 106 | ### Aliases 107 | 108 | Aliases are used to label a single image with mutliple tags. As opposed to rebuilding an image, which risks creating non-identical hashes for images that should be aliased, the `alias` key will use the `docker tag` command to create a proper alias for each value in the key's array. 109 | 110 | ### Output 111 | 112 | Dante generates two different outputs 113 | 114 | 1. Markdown 115 | 2. Docker Images 116 | 117 | When running, the tool outputs its status to stdout in the form of markdown for easy integration with GitHub and the Docker Registry. 118 | 119 | It also generates docker images tagged with the `name` value from the `inventory.yml` file, and successful test images are built with the same tag but with `-test#` append to the end, where `#` is the number of the current test 120 | 121 | For example, if you have an `inventory.yml` file: 122 | 123 | ```yaml 124 | images: 125 | - name: "wblankenship/dockeri.co:server" 126 | path: "./dockerico/server" 127 | test: ["./dockerico/tests/http","./dockerico/tests/badges"] 128 | ``` 129 | 130 | You will end up with the following Docker images (assuming the image builds and the tests run succesfully) 131 | 132 | * `dockeri.co:server`: the base image 133 | * `dockeri.co:server-test1`: the image built from the http directory 134 | * `dockeri.co:server-test2`: the image built from the badges directory 135 | 136 | 137 | # Philosophy 138 | 139 | We strongly believe that tooling should fit naturally into the existing ecosystem. This belief has driven every aspect of developing Dante. We have taken full advantage of existing tools and formats that exist within the docker ecosystem to produce an unobtrusive approach to testing Dockerfiles and docker images. 140 | 141 | ## Testing Concept 142 | 143 | Our approach to testing docker images is entirely driven by image layers. Now for a quick crash course into what we mean by that. 144 | 145 | ![docker layers](/docs/layers_base.png) 146 | 147 | So lets say you build an image from a Dockerfile, it produces individual layers like in the diagram above. Each command in a Dockerfile produces a layer. The `FROM` command is special, it will build your Dockerfile layers ontop of the layers from another image. 148 | 149 | ![docker test](/docs/layers_test.png) 150 | 151 | What this allows us to do is build your image from a Dockerfile, then build the tests as layers on top of your image. Assuming all of the commands in the tests can succesfully generate layers on top of your image, you have a guarentee that the environment inside of your image is stable enough to run the tasks represented in your tests. We can then throw away the test layers and ship the base image now that we know it is in a stable state! 152 | 153 | ## Technologies 154 | 155 | There were a few design decisions we took under careful consideration when putting together this tool. Primarily: 156 | 157 | * Tests as Dockerfiles 158 | * Inventory file as yaml 159 | * Output as Markdown 160 | 161 | ### Tests as Dockerfiles 162 | 163 | First and foremost, we wanted all tests to be built as layers ontop of the image we are testing. This ensures that we capture not only the environment we are testing, but the tests that we run inside of that environment. Assuming you are archiving the images generated by Dante, when a bug is found in production that the tests _should_ have caught, you can reproduce the testing environment at any layer to inspect why exactly the test passed. 164 | 165 | Tests as Dockerfiles also means that users do not need to learn new tools in order to test their images. They simply create a Dockerfile that makes assertions about the environment produced by the Dockerfile they are testing. For users with already established testing frameworks, these frameworks can easily be built into the Dockerfile and run as a layer ontop of the image itself. 166 | 167 | ### Inventory file as yaml 168 | 169 | We modeled our inventory file after the `docker-compose.yml` specification. This format is already established in the community, and reduces the congnitive overhead of producing the file. 170 | 171 | ### Output as Markdown 172 | 173 | The motivation for writting Markdown to stdout is to allow easy consumption of the results on both the Docker Registry and GitHub. Moving forward, we may include flags that change this behaviour. 174 | 175 | # Changlog 176 | 177 | ## v2.1.0 178 | 179 | * `alias` key now supported in `inventory.yml` 180 | * `test` now tags aliases after successful build 181 | * `push` now pushes both images and their aliases 182 | 183 | ## v2.0.0 184 | 185 | * Subcommands Added (`test` and `push`) 186 | * Dante can now push to repositories from an `inventory.yml` file 187 | * Implemented a `-r` flag for retrying failed tests, builds, and pushes. 188 | 189 | ## v1.1.0 190 | 191 | * Added `j` flag for parallel builds 192 | -------------------------------------------------------------------------------- /alias.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | ) 6 | 7 | func runAlias(inventory Inventory) (errs int) { 8 | i := 0 9 | fmt.Printf("# Tagging Aliases\n\n") 10 | for _, image := range inventory["images"] { 11 | name := image["name"].(string) 12 | aliases := getAliasArray(image) 13 | for _, alias := range aliases { 14 | fmt.Printf("%v. %v -> %v\n", i, name, alias) 15 | i++ 16 | output, err := dockerAlias(name, alias) 17 | if err != nil { 18 | fmt.Printf("Error creating tag:\n\n```\n%v\n```\n\n%v\n", output, err) 19 | errs++ 20 | } 21 | } 22 | } 23 | fmt.Printf("\n") 24 | return 25 | } 26 | 27 | func getAliasArray(image map[string]interface{}) (aliases []string) { 28 | switch image["alias"].(type) { 29 | case string: 30 | // If the value is a single string, append it to the array and be done 31 | aliases = append(aliases, image["alias"].(string)) 32 | break 33 | case []interface{}: 34 | // If the value is an array, iterate through and add all of the strings 35 | // to the array one by one. 36 | for _, str := range image["alias"].([]interface{}) { 37 | aliases = append(aliases, str.(string)) 38 | } 39 | break 40 | } 41 | return 42 | } 43 | -------------------------------------------------------------------------------- /build.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Get Working Directory 4 | 5 | SOURCE="${BASH_SOURCE[0]}" 6 | while [ -h "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symlink 7 | DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )" 8 | SOURCE="$(readlink "$SOURCE")" 9 | [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" 10 | done 11 | DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )" 12 | 13 | # Change to working directory 14 | 15 | cd $DIR 16 | 17 | # Import compile functions 18 | 19 | source crosscompile.bash 20 | 21 | # Compile binaries 22 | 23 | go-build-all 24 | -------------------------------------------------------------------------------- /common.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | ) 6 | 7 | type ImageDefinition map[string]interface{} 8 | 9 | type Job struct { 10 | Image ImageDefinition 11 | Retries int 12 | Output string 13 | Success bool 14 | Id int 15 | } 16 | 17 | func reporter(output chan Job, done chan bool) { 18 | for { 19 | tmp := <-output 20 | fmt.Printf("%v", tmp.Output) 21 | done <- tmp.Success 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /crosscompile.bash: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright 2012 The Go Authors. All rights reserved. 3 | # Use of this source code is governed by a BSD-style 4 | # license that can be found in the LICENSE file. 5 | 6 | # support functions for go cross compilation 7 | 8 | type setopt >/dev/null 2>&1 && setopt shwordsplit 9 | PLATFORMS="darwin/386 darwin/amd64 freebsd/386 freebsd/amd64 freebsd/arm linux/386 linux/amd64 linux/arm windows/386 windows/amd64 openbsd/386 openbsd/amd64" 10 | 11 | function go-alias { 12 | local GOOS=${1%/*} 13 | local GOARCH=${1#*/} 14 | eval "function go-${GOOS}-${GOARCH} { ( GOOS=${GOOS} GOARCH=${GOARCH} go \"\$@\" ) }" 15 | } 16 | 17 | function go-crosscompile-build { 18 | local GOOS=${1%/*} 19 | local GOARCH=${1#*/} 20 | cd $(go env GOROOT)/src ; GOOS=${GOOS} GOARCH=${GOARCH} ./make.bash --no-clean 2>&1 21 | } 22 | 23 | function go-crosscompile-build-all { 24 | local FAILURES="" 25 | for PLATFORM in $PLATFORMS; do 26 | local CMD="go-crosscompile-build ${PLATFORM}" 27 | echo "$CMD" 28 | $CMD || FAILURES="$FAILURES $PLATFORM" 29 | done 30 | if [ "$FAILURES" != "" ]; then 31 | echo "*** go-crosscompile-build-all FAILED on $FAILURES ***" 32 | return 1 33 | fi 34 | } 35 | 36 | function go-all { 37 | local FAILURES="" 38 | for PLATFORM in $PLATFORMS; do 39 | local GOOS=${PLATFORM%/*} 40 | local GOARCH=${PLATFORM#*/} 41 | local CMD="go-${GOOS}-${GOARCH} $@" 42 | echo "$CMD" 43 | $CMD || FAILURES="$FAILURES $PLATFORM" 44 | done 45 | if [ "$FAILURES" != "" ]; then 46 | echo "*** go-all FAILED on $FAILURES ***" 47 | return 1 48 | fi 49 | } 50 | 51 | function go-build-all { 52 | local FAILURES="" 53 | for PLATFORM in $PLATFORMS; do 54 | local GOOS=${PLATFORM%/*} 55 | local GOARCH=${PLATFORM#*/} 56 | local SRCFILENAME=`echo $@ | sed 's/\.go//'` 57 | local CURDIRNAME=${PWD##*/} 58 | local OUTPUT=${SRCFILENAME:-$CURDIRNAME} # if no src file given, use current dir name 59 | local CMD="go-${GOOS}-${GOARCH} build -o $OUTPUT-${GOOS}-${GOARCH} $@" 60 | echo "$CMD" 61 | $CMD || FAILURES="$FAILURES $PLATFORM" 62 | done 63 | if [ "$FAILURES" != "" ]; then 64 | echo "*** go-build-all FAILED on $FAILURES ***" 65 | return 1 66 | fi 67 | } 68 | 69 | for PLATFORM in $PLATFORMS; do 70 | go-alias $PLATFORM 71 | done 72 | 73 | unset -f go-alias 74 | -------------------------------------------------------------------------------- /dante.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "github.com/retrohacker/cli" 6 | "os" 7 | ) 8 | 9 | const version string = "1.1.0" 10 | 11 | // Inventory is static and global 12 | // We will initialize it once and then use it throughout the app 13 | var inventory Inventory 14 | 15 | func main() { 16 | 17 | /* Define cli commands and flags */ 18 | app := cli.NewApp() 19 | app.Commands = []cli.Command{ 20 | { 21 | Name: "test", 22 | Usage: "Build images and run tests defined in inventory.yml", 23 | Action: test, 24 | Flags: []cli.Flag{ 25 | cli.IntFlag{ 26 | Name: "retries,r", 27 | Usage: "Retry on failure", 28 | Value: 0, 29 | }, 30 | cli.IntFlag{ 31 | Name: "parallel,j", 32 | Usage: "Run parallel jobs", 33 | Value: 1, 34 | }, 35 | }, 36 | }, 37 | { 38 | Name: "push", 39 | Usage: "Push local images to remote registry", 40 | Action: push, 41 | Flags: []cli.Flag{ 42 | cli.IntFlag{ 43 | Name: "retries,r", 44 | Usage: "Retry on failure", 45 | Value: 0, 46 | }, 47 | cli.IntFlag{ 48 | Name: "parallel,j", 49 | Usage: "Run parallel jobs", 50 | Value: 1, 51 | }, 52 | }, 53 | }, 54 | } 55 | 56 | app.Version = version 57 | 58 | app.Run(os.Args) 59 | 60 | } 61 | 62 | /* 63 | populateInventory initializes the global state of the application as directed 64 | by the user's inventory.yaml file 65 | */ 66 | func populateInventory() { 67 | // Load the yml definition of images and tests 68 | var err error 69 | inventory, err = GetInventory() 70 | 71 | if err != nil { 72 | // If we can't find the inventory file, there is nothing left for us to do. 73 | fmt.Printf("%v\n", err) 74 | os.Exit(1) 75 | } 76 | } 77 | 78 | func test(c *cli.Context) { 79 | populateInventory() 80 | 81 | opts := scrub_input(TestOpts{ 82 | Threads: c.Int("parallel"), 83 | Retries: c.Int("retries"), 84 | }) 85 | 86 | // Build the images and run the tests defined in the inventory file 87 | errs := runTests(inventory, opts) 88 | 89 | // Determine if the tests passed or failed 90 | if errs > 0 { 91 | // Not all tests passed, this makes docker-test a sad panda 92 | fmt.Printf("# Conclusion\n\n%v tests failed.\n\n", errs) 93 | os.Exit(1) 94 | } 95 | // All tests and builds completed succesfully! 96 | fmt.Printf("# Conclusion\n\nall tests passed.\n\n") 97 | 98 | // Tag images with aliases 99 | errs = runAlias(inventory) 100 | if errs > 0 { 101 | fmt.Printf("# Conclusion\n\n%v aliases failed.\n\n", errs) 102 | os.Exit(1) 103 | } 104 | 105 | fmt.Printf("# Conclusion\n\nall aliases succeeded.\n\n", errs) 106 | os.Exit(0) 107 | 108 | } 109 | 110 | func push(c *cli.Context) { 111 | populateInventory() 112 | 113 | opts := scrub_input(TestOpts{ 114 | Threads: c.Int("parallel"), 115 | Retries: c.Int("retries"), 116 | }) 117 | 118 | errs := runPushes(inventory, opts) 119 | 120 | // Determine if the tests passed or failed 121 | if errs > 0 { 122 | // Not all tests passed, this makes docker-test a sad panda 123 | fmt.Printf("# Conclusion\n\n%v pushes failed.\n\n", errs) 124 | os.Exit(1) 125 | } else { 126 | // All tests and builds completed succesfully! 127 | fmt.Printf("# Conclusion\n\nall pushes succeeded.\n\n") 128 | os.Exit(0) 129 | } 130 | 131 | } 132 | 133 | func scrub_input(opts TestOpts) TestOpts { 134 | if opts.Threads < 1 { 135 | opts.Threads = 1 136 | } 137 | 138 | if opts.Retries < 0 { 139 | opts.Retries = 0 140 | } 141 | 142 | return opts 143 | } 144 | -------------------------------------------------------------------------------- /docker.go: -------------------------------------------------------------------------------- 1 | /* 2 | docker.go contains all of the logic specific to executing docker processes 3 | from inside the application 4 | */ 5 | package main 6 | 7 | import ( 8 | "os/exec" 9 | "path/filepath" 10 | ) 11 | 12 | /* 13 | execDocker is a pretty wrapper around exec.Command("docker",...) 14 | */ 15 | func execDocker(path string, command string, args ...string) (output string, err error) { 16 | // Hold the output from our command 17 | var outputBytes []byte 18 | 19 | // First, we create an array with command and args to pass to exec 20 | tmp := []string{command} 21 | for _, arg := range args { 22 | tmp = append(tmp, arg) 23 | } 24 | 25 | // Next, we build and execute the command 26 | cmd := exec.Command("docker", tmp...) 27 | 28 | cmd.Dir, err = filepath.Abs(path) 29 | if err != nil { 30 | return 31 | } 32 | 33 | outputBytes, err = cmd.CombinedOutput() 34 | output = string(outputBytes) 35 | 36 | return 37 | } 38 | 39 | type DockerOpts struct { 40 | Cache bool 41 | } 42 | 43 | /* 44 | buildImage will take a path to a docker image, and execute docker build as a 45 | child process. It will tag the docker built image as name, this allows us to 46 | later build other images using this one as a base. It captures stdout and 47 | stderr returning them both in output. 48 | */ 49 | func buildImage(name string, path string, opts DockerOpts) (output string, err error) { 50 | args := []string{"-t", name} 51 | 52 | if !opts.Cache { 53 | args = append(args, "--no-cache") 54 | } 55 | 56 | // local directory 57 | args = append(args, ".") 58 | 59 | return execDocker(path, "build", args...) 60 | } 61 | 62 | /* 63 | pushImage will take a docker image and push it to a remote registry. It captures 64 | stdout and stderr returning them both in output 65 | */ 66 | func pushImage(name string) (output string, err error) { 67 | return execDocker("/", "push", name) 68 | } 69 | 70 | func dockerAlias(name string, alias string) (output string, err error) { 71 | return execDocker("/", "tag", "-f", name, alias) 72 | } 73 | -------------------------------------------------------------------------------- /docs/dante1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retrohacker/dante/0a263d839da6373baf50f0c8be15ad1f7c531ce7/docs/dante1.jpg -------------------------------------------------------------------------------- /docs/layers_base.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retrohacker/dante/0a263d839da6373baf50f0c8be15ad1f7c531ce7/docs/layers_base.png -------------------------------------------------------------------------------- /docs/layers_base.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 18 | 20 | 42 | 44 | 45 | 47 | image/svg+xml 48 | 50 | 51 | 52 | 53 | 54 | 59 | 67 | 78 | 86 | 95899b254ad6 97 | 105 | 50e0eff96b21 116 | 124 | fd0d0ffb727d 135 | 143 | fdf6bfe25e0e 154 | 155 | 156 | -------------------------------------------------------------------------------- /docs/layers_test.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retrohacker/dante/0a263d839da6373baf50f0c8be15ad1f7c531ce7/docs/layers_test.png -------------------------------------------------------------------------------- /docs/layers_test.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 18 | 20 | 42 | 44 | 45 | 47 | image/svg+xml 48 | 50 | 51 | 52 | 53 | 54 | 59 | 67 | 78 | 86 | 95899b254ad6 97 | 105 | 50e0eff96b21 116 | 124 | fd0d0ffb727d 135 | 143 | fdf6bfe25e0e 154 | 162 | 86d33ec817a9 173 | 181 | 8cce809aa132 192 | 200 | 31eef4f70ddd 211 | 216 | 217 | 218 | -------------------------------------------------------------------------------- /fs.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "io" 6 | "io/ioutil" 7 | "os" 8 | "path/filepath" 9 | ) 10 | 11 | /* 12 | copyFile moves a file from one place in the filesystem to another 13 | 14 | Source from https://www.socketloop.com/tutorials/golang-copy-directory-including-sub-directories-files 15 | */ 16 | func copyFile(source string, dest string) (err error) { 17 | sourcefile, err := os.Open(source) 18 | if err != nil { 19 | return err 20 | } 21 | 22 | defer sourcefile.Close() 23 | 24 | destfile, err := os.Create(dest) 25 | if err != nil { 26 | return err 27 | } 28 | 29 | defer destfile.Close() 30 | 31 | _, err = io.Copy(destfile, sourcefile) 32 | if err == nil { 33 | sourceinfo, err := os.Stat(source) 34 | if err == nil { 35 | err = os.Chmod(dest, sourceinfo.Mode()) 36 | } 37 | } 38 | 39 | return 40 | } 41 | 42 | /* 43 | copyDir recursively moves the contents of a folder from one place in the filesystem to another 44 | 45 | Source from https://www.socketloop.com/tutorials/golang-copy-directory-including-sub-directories-files 46 | */ 47 | func copyDir(source string, dest string) (err error) { 48 | 49 | // get properties of source dir 50 | sourceinfo, err := os.Stat(source) 51 | if err != nil { 52 | return err 53 | } 54 | 55 | // create dest dir 56 | 57 | err = os.MkdirAll(dest, sourceinfo.Mode()) 58 | if err != nil { 59 | return err 60 | } 61 | 62 | directory, _ := os.Open(source) 63 | 64 | objects, err := directory.Readdir(-1) 65 | 66 | for _, obj := range objects { 67 | 68 | sourcefilepointer := source + "/" + obj.Name() 69 | 70 | destinationfilepointer := dest + "/" + obj.Name() 71 | 72 | if obj.IsDir() { 73 | // create sub-directories - recursively 74 | err = copyDir(sourcefilepointer, destinationfilepointer) 75 | if err != nil { 76 | fmt.Println(err) 77 | } 78 | } else { 79 | // perform copy 80 | err = copyFile(sourcefilepointer, destinationfilepointer) 81 | if err != nil { 82 | fmt.Println(err) 83 | } 84 | } 85 | 86 | } 87 | return 88 | } 89 | 90 | /* 91 | prependToFile takes the string text and places it at the beginning of the file 92 | named filename. This function expects filename to be a vaild path to a file, 93 | and uses filepath.Abs() to get the absolute path of filename. 94 | */ 95 | func prependToFile(filename, text string) (err error) { 96 | // Begin declaring local variables 97 | var contents []byte 98 | // End declaring local variables 99 | 100 | // Get the absolute path to filename (also ensures its a valid path) 101 | filename, err = filepath.Abs(filename) 102 | if err != nil { 103 | return err 104 | } 105 | 106 | // Attempt to read in the entire file's contents. 107 | // NOTE: we are making the assumption that Dockerfiles are a reaasonable 108 | // size, so it is safe to load the entire file into memory when performing 109 | // the prepend operation 110 | contents, err = ioutil.ReadFile(filename) 111 | if err != nil { 112 | return err 113 | } 114 | 115 | // We then prepend the text to the beginning of the file in memory, again 116 | // assuming the dockerfile is a reasonable size 117 | contents = append([]byte(text), contents...) 118 | 119 | // Finally, we write the new string back to the filesystem. 120 | // Note: we assume this function is being used in the context of a temporary 121 | // directory for the purpose of this application. Since the file is a temp 122 | // file and only exists for the lifetime of this application running, it is 123 | // safe to disregard the original permission string on the file and instead 124 | // set it to something we know the docker daemon can use. If this assumption 125 | // proves to be incorrect, we will need to modify this line. 126 | return ioutil.WriteFile(filename, contents, 0666) 127 | } 128 | -------------------------------------------------------------------------------- /inventory.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "gopkg.in/yaml.v2" 5 | "io/ioutil" 6 | "os" 7 | "path/filepath" 8 | ) 9 | 10 | /* 11 | getInventory() simply reads and returns the contents of the inventory.yml file 12 | located in the process's current working directory. 13 | */ 14 | func getInventory() (file []byte, err error) { 15 | // Begin declaring local variables 16 | var cwd, filename string 17 | // End declaring local variables 18 | 19 | // The next few lines attempts to get the current working directory of the 20 | // process and create an absolute path to the inventory.yml file located 21 | // there 22 | cwd, err = os.Getwd() 23 | if err != nil { 24 | return 25 | } 26 | filename = filepath.Join(cwd, "inventory.yml") 27 | 28 | // Attempt to read the file into memory 29 | file, err = ioutil.ReadFile(filename) 30 | if err != nil { 31 | return nil, err 32 | } 33 | return 34 | } 35 | 36 | /* 37 | Inventory is used to unmarshal the inventory.yml file 38 | 39 | NOTE: This places assumptions on the structure of the inventory.yml file. If 40 | we want more intuitive error messages moving forward, we may want to add a 41 | intermediary map[interface{}]interface{} object which the yaml library can 42 | unmarshal into without errors, then have verifyInventory() handle converting 43 | the unstructure map to the Inventory structure while verifying the content and 44 | structure of the file. 45 | */ 46 | type Inventory map[string][]map[string]interface{} 47 | 48 | /* 49 | parseInventory takes in a raw byte array representing an inventory.yml file 50 | unmarshals it into a go map. 51 | */ 52 | func parseInventory(file []byte) (obj Inventory, err error) { 53 | // Create a new inventory structure that will hold our object 54 | obj = Inventory{} 55 | 56 | // Unmarshal and return our new Inventory object 57 | err = yaml.Unmarshal(file, &obj) 58 | if err != nil { 59 | return nil, err 60 | } 61 | return 62 | } 63 | 64 | /* 65 | verifyInventory ensures the structure and contents of an inventory.yml file are 66 | correct before the application attempts to process it. 67 | */ 68 | func verifyInventory(inventory Inventory) (err error) { 69 | // Currently we do absolutely no verification, we simply allow the application 70 | // to fail. This function will be useful moving forward when the need for 71 | // more intuitive error messages arises. 72 | return nil 73 | } 74 | 75 | /* 76 | containsDockerfile ensures that a dockerfile exists in a directory. This is 77 | currently not used, but will serve a purpose in the verifyInventory function 78 | moving forward. 79 | */ 80 | func containsDockerfile(dockerdir string) (err error) { 81 | var dockerDir, dockerfile string 82 | var file *os.File 83 | dockerDir, err = filepath.Abs(dockerdir) 84 | if err != nil { 85 | return 86 | } 87 | dockerfile = filepath.Join(dockerDir, "Dockerfile") 88 | file, err = os.Open(dockerfile) 89 | if err != nil { 90 | return 91 | } 92 | err = file.Close() 93 | if err != nil { 94 | return 95 | } 96 | return 97 | } 98 | 99 | /* 100 | GetInventory is the method you should be calling when interacting with the 101 | contents of this file. It loads in an inventory.yml file, converts it to a go 102 | object, verifies its structure, and returns the object. 103 | */ 104 | func GetInventory() (inventory Inventory, err error) { 105 | // Begin declaring local variables 106 | var file []byte 107 | // End declaring local variables 108 | 109 | // Load the inventory file from disk 110 | file, err = getInventory() 111 | if err != nil { 112 | return nil, err 113 | } 114 | 115 | // Convert the inventory file to a go object 116 | inventory, err = parseInventory(file) 117 | if err != nil { 118 | return nil, err 119 | } 120 | 121 | // Verify the structure of the inventory object 122 | err = verifyInventory(inventory) 123 | if err != nil { 124 | return nil, err 125 | } 126 | return 127 | } 128 | -------------------------------------------------------------------------------- /push.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | ) 6 | 7 | func runPushes(inventory Inventory, opts TestOpts) (errs int) { 8 | 9 | input := make(chan Job) 10 | output := make(chan Job) 11 | 12 | // Get job count 13 | jobs := 0 14 | for _, image := range inventory["images"] { 15 | jobs++ 16 | aliases := getAliasArray(image) 17 | for range aliases { 18 | jobs++ 19 | } 20 | } 21 | 22 | done := make(chan bool, jobs) 23 | 24 | for i := 0; i < opts.Threads; i++ { 25 | go pushWorker(input, output) 26 | } 27 | 28 | go reporter(output, done) 29 | 30 | for i, image := range inventory["images"] { 31 | input <- Job{ 32 | Image: image, 33 | Retries: opts.Retries, 34 | Id: i, 35 | } 36 | aliases := getAliasArray(image) 37 | for _, alias := range aliases { 38 | aliasImage := make(ImageDefinition) 39 | aliasImage["name"] = alias 40 | input <- Job{ 41 | Image: aliasImage, 42 | Retries: opts.Retries, 43 | Id: i, 44 | } 45 | } 46 | } 47 | 48 | errs = 0 49 | for i := 0; i < jobs; i++ { 50 | if <-done == false { 51 | errs++ 52 | } 53 | } 54 | 55 | return 56 | 57 | return 58 | } 59 | 60 | func pushWorker(input chan Job, output chan Job) { 61 | for { 62 | job := <-input 63 | var resultString string 64 | 65 | // Initialize Output For Image 66 | stdout := fmt.Sprintf("# Pushed image `%v`\n\n## Push Log\n\n", job.Image["name"].(string)) 67 | 68 | resultString, job = HandleSinglePushJob(job) 69 | 70 | stdout = stdout + resultString 71 | 72 | job.Output = stdout 73 | output <- job 74 | 75 | } 76 | } 77 | 78 | func HandleSinglePushJob(job Job) (string, Job) { 79 | 80 | var stdout string 81 | 82 | // Attempt to build the image until we run out of retries 83 | for retries := job.Retries; retries >= 0; retries-- { 84 | // Try to build the image 85 | result, err := pushImage(job.Image["name"].(string)) 86 | stdout = stdout + fmt.Sprintf("```\n%v\n```\n\n", string(result)) 87 | 88 | // If we fail, determine proper notice to log, else break out of retry loop 89 | if err != nil { 90 | stdout = stdout + fmt.Sprintf("**Failed** with error: `%v`\nRetries Remaining: %v", err, retries) 91 | if retries == 0 { 92 | stdout = stdout + fmt.Sprintf("... Moving on") 93 | job.Success = false 94 | } 95 | stdout = stdout + fmt.Sprintf("\n\n") 96 | } else { 97 | job.Success = true 98 | break 99 | } 100 | } 101 | 102 | return stdout, job 103 | } 104 | -------------------------------------------------------------------------------- /test.go: -------------------------------------------------------------------------------- 1 | /* 2 | test.go contains all of the logic specific to the test command 3 | */ 4 | 5 | package main 6 | 7 | import ( 8 | "fmt" 9 | "io/ioutil" 10 | "os" 11 | "path/filepath" 12 | "strconv" 13 | ) 14 | 15 | /* 16 | tempPath is the location that all test files will be placed in when running 17 | this application. 18 | 19 | NOTE: Assumption that this is a safe path in the current working directory. 20 | This application will delete this directory when run, which is a high risk for 21 | deleting user data. If this assumption proves not to be safe, we will have to 22 | rethink this constant. 23 | */ 24 | const tempPath = ".~tmp.test" 25 | 26 | var errs []error 27 | 28 | /* 29 | getTestArray takes a single image from the inventory.yml file and converts 30 | its test key (of type interface{}) to an array of strings. This allows us 31 | to accept either a single string or an array of strings as a value for 32 | test. 33 | */ 34 | func getTestArray(image map[string]interface{}) (tests []string) { 35 | switch image["test"].(type) { 36 | case string: 37 | // If the value is a single string, append it to the array and be done 38 | tests = append(tests, image["test"].(string)) 39 | break 40 | case []interface{}: 41 | // If the value is an array, iterate through and add all of the strings 42 | // to the array one by one. 43 | for _, str := range image["test"].([]interface{}) { 44 | tests = append(tests, str.(string)) 45 | } 46 | break 47 | } 48 | return 49 | } 50 | 51 | type TestOpts struct { 52 | Threads int 53 | Retries int 54 | } 55 | 56 | /* 57 | runTests iterates through an Inventory object and builds every image, followed 58 | by running each of the tests listed against the newly built image. We attempt 59 | to build every image defined in inventory, and return an array of errors if any 60 | are encountered. 61 | */ 62 | func runTests(inventory Inventory, opts TestOpts) (errs int) { 63 | 64 | input := make(chan Job) 65 | output := make(chan Job) 66 | done := make(chan bool, len(inventory["images"])) 67 | 68 | for i := 0; i < opts.Threads; i++ { 69 | go testWorker(input, output) 70 | } 71 | 72 | go reporter(output, done) 73 | 74 | for i, image := range inventory["images"] { 75 | input <- Job{ 76 | Image: image, 77 | Retries: opts.Retries, 78 | Id: i, 79 | } 80 | } 81 | 82 | errs = 0 83 | for i := 0; i < len(inventory["images"]); i++ { 84 | if <-done == false { 85 | errs++ 86 | } 87 | } 88 | 89 | return 90 | } 91 | 92 | func testWorker(input chan Job, output chan Job) { 93 | for { 94 | tmp := <-input 95 | var resultString string 96 | 97 | // Initialize Output For Image 98 | stdout := fmt.Sprintf("# Tested image `%v`\n\n## Build Log\n\n", tmp.Image["name"].(string)) 99 | 100 | resultString, tmp = testBuildImage(tmp) 101 | stdout = stdout + resultString 102 | 103 | // If we did not successfully build, there is nothing left to do 104 | if !tmp.Success { 105 | tmp.Output = stdout 106 | output <- tmp 107 | continue 108 | } 109 | 110 | resultString, tmp = testBuildTests(tmp) 111 | stdout = stdout + resultString 112 | 113 | tmp.Output = stdout 114 | output <- tmp 115 | } 116 | } 117 | 118 | func testBuildImage(tmp Job) (string, Job) { 119 | 120 | var stdout string 121 | 122 | // Attempt to build the image until we run out of retries 123 | for retries := tmp.Retries; retries >= 0; retries-- { 124 | // Try to build the image 125 | result, err := buildImage(tmp.Image["name"].(string), tmp.Image["path"].(string), DockerOpts{}) 126 | stdout = stdout + fmt.Sprintf("```\n%v\n```\n\n", string(result)) 127 | 128 | // If we fail, determine proper notice to log, else break out of retry loop 129 | if err != nil { 130 | stdout = stdout + fmt.Sprintf("**Failed** with error: `%v`\nRetries Remaining: %v", err, retries) 131 | if retries == 0 { 132 | stdout = stdout + fmt.Sprintf("... Moving on") 133 | tmp.Success = false 134 | } 135 | stdout = stdout + fmt.Sprintf("\n\n") 136 | } else { 137 | tmp.Success = true 138 | break 139 | } 140 | } 141 | 142 | return stdout, tmp 143 | } 144 | 145 | func testBuildTests(tmp Job) (string, Job) { 146 | 147 | // Get an array of tests we want to run against our newly built image 148 | tests := getTestArray(tmp.Image) 149 | stdout := fmt.Sprintf("Array of tests: `%v`\n\n", tests) 150 | 151 | for testNum, test := range tests { 152 | output, err := testBuildTest(tmp.Image, tmp.Id, tmp.Retries, testNum, test) 153 | stdout = stdout + output 154 | if err != nil { 155 | tmp.Success = false 156 | } 157 | } 158 | 159 | return stdout, tmp 160 | } 161 | 162 | func testBuildTest(image ImageDefinition, id int, retries int, testNum int, test string) (output string, err error) { 163 | 164 | var tempDir string 165 | 166 | // Grab an absolute path to the directory we will store our tests in. 167 | // We need to use a temporary directory since we will be modifying the 168 | // contents of the directory to build the tests against the base image. 169 | localTempPath := tempPath + strconv.Itoa(id) 170 | tempDir, err = filepath.Abs(localTempPath) 171 | 172 | var testpath string 173 | var contents []byte 174 | output = output + fmt.Sprintf("## Running test #%v\n\n", testNum) 175 | 176 | // Generate a unique name for the test image that we will build 177 | testname := image["name"].(string) + "-test" + strconv.Itoa(testNum+1) 178 | 179 | // Get the absolute path to the test Dockerfile and context location 180 | testpath, err = filepath.Abs(test) 181 | if err != nil { 182 | output = output + fmt.Sprintf("**Failed** Could not get path to file `%v`: `%v`\n\n", test, err) 183 | // If we can't get the path, we can't build the image. Moving on. 184 | return 185 | } 186 | 187 | // Delete the tempDir directory if it exists to ensure our test has a 188 | // clean context and isn't polluted by a corrupted previous run of 189 | // docker-test 190 | os.RemoveAll(tempDir) 191 | 192 | // We need to copy the test context to a temp directory. In order to use 193 | // our new docker image as a base, we prepend a FROM statement to the 194 | // test's Dockerfile. This tool should be repeatable, generating the same 195 | // results if run in the same environment multiple times (assuming the 196 | // Dockerfiles it builds are deterministic). This means we can not modify 197 | // the original Dockerfile in place, so we create a temporary directory 198 | // where we copy the test's context and are then able to safely mutate 199 | // the context's state to suite our tool. We delete the temp directory 200 | // when finished. 201 | output = output + fmt.Sprintf("Copying `%v` to `%v`\n\n", testpath, tempDir) 202 | copyDir(testpath, tempDir) 203 | 204 | // tmpDir should already be an absolute path. So we are now getting 205 | // an absolute path to the Dockerfile we just copied into our tempDir 206 | dockerfile := filepath.Join(tempDir, "Dockerfile") 207 | 208 | // We then prepend a FROM statement to our Dockerfile so that when the 209 | // docker daemon builds it, it would build the layers on top of the 210 | // the image we are attempting to test. 211 | prependToFile(dockerfile, "FROM "+image["name"].(string)+"\n") 212 | contents, err = ioutil.ReadFile(dockerfile) 213 | if err != nil { 214 | output = output + fmt.Sprintf("**Failed** Could not get contents of Dockerfile `%v`: `%v`\n\n", test, err) 215 | // If we can't get the Dockerfile, we can't build the image. Moving on. 216 | return 217 | } 218 | output = output + fmt.Sprintf("Contents of dockerfile `%v`:\n\n```\n%v\n```\n\n", dockerfile, string(contents)) 219 | output = output + fmt.Sprintf("Building `%v` from `%v`\n\n", testname, tempDir) 220 | 221 | // Build our test image against our base image until we succeed or run out of retries 222 | for ; retries >= 0; retries-- { 223 | var resultStr string 224 | resultStr, err = buildImage(testname, tempDir, DockerOpts{}) 225 | output = output + fmt.Sprintf("```\n%v\n```\n\n", string(resultStr)) 226 | if err != nil { 227 | output = output + fmt.Sprintf("**Failed** with error: `%v`\nRetries Remaining: %v", err, retries) 228 | if retries == 0 { 229 | output = output + fmt.Sprintf("... Moving on") 230 | } 231 | output = output + fmt.Sprintf("\n\n") 232 | } else { 233 | break 234 | } 235 | } 236 | return 237 | } 238 | -------------------------------------------------------------------------------- /test/debian/jessie/iojs/1.4.2/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM debian:jessie 2 | MAINTAINER William Blankenship 3 | 4 | RUN echo "Hello World. I'm a Node.js image!" 5 | -------------------------------------------------------------------------------- /test/debian/jessie/node/0.12.0/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM debian:jessie 2 | MAINTAINER William Blankenship 3 | 4 | RUN echo "Hello World! I am a Node.js image!" 5 | -------------------------------------------------------------------------------- /test/inventory.yml: -------------------------------------------------------------------------------- 1 | images: 2 | - name: "wblankenship/test:1" 3 | path: "./debian/jessie/node/0.12.0" 4 | #test: ["./tests/node","./tests/node12"] 5 | alias: ["wblankenship/test:3","wblankenship/test:4"] 6 | - name: "wblankenship/test:2" 7 | path: "./debian/jessie/iojs/1.4.2" 8 | test: "./tests/iojs" 9 | alias: "wblankenship/test:5" 10 | -------------------------------------------------------------------------------- /test/tests/iojs/Dockerfile: -------------------------------------------------------------------------------- 1 | RUN echo "Hello World!" 2 | -------------------------------------------------------------------------------- /test/tests/node/Dockerfile: -------------------------------------------------------------------------------- 1 | RUN echo "Hello World!" 2 | -------------------------------------------------------------------------------- /test/tests/node12/Dockerfile: -------------------------------------------------------------------------------- 1 | RUN echo "Hello World!" 2 | RUN this_will_fail 3 | --------------------------------------------------------------------------------