├── .gitignore ├── README.md ├── broadcast.go ├── copy.go ├── errors.go ├── fs.go ├── iptables.go ├── machinectl.go ├── main.go ├── mount.go ├── network.go ├── nspawn.go ├── packages.go ├── query.go ├── storage.go ├── storage_overlayfs.go └── storage_zfs.go /.gitignore: -------------------------------------------------------------------------------- 1 | /hastur 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![hastur](https://cloud.githubusercontent.com/assets/674812/10144792/fadcbd28-660e-11e5-8569-894e874cff14.png) 2 | 3 | hastur is a tool for launching systemd-nspawn containers without the need for 4 | manual configuration. 5 | 6 | It will setup networking, a base root FS and even an overlay FS for containers 7 | automatically. 8 | 9 | The primary usecase for hastur is supporting testcases for distributed systems 10 | and running a local set of trusted containers. 11 | 12 | ![gif](https://cloud.githubusercontent.com/assets/674812/10140037/37f12ea2-65f5-11e5-90c7-eb18e6a9b37b.gif) 13 | 14 | # Motivation 15 | 16 | systemd-nspawn is useful tool, which can create and run lightweight containers 17 | without any additional software, because it's available out-of-the-box systemd. 18 | 19 | However, it requires some configuration to run a working container, such as 20 | managing the network configuration, and downloading and extracting packages. 21 | 22 | hastur offers all this configuration automatically, which makes it possible to 23 | run fully-configured systemd-nspawn containers in seconds. 24 | 25 | # Installation 26 | 27 | ## Arch Linux 28 | 29 | hastur is available for Arch Linux (for now, only) through the AUR: 30 | 31 | https://aur4.archlinux.org/packages/hastur/ 32 | 33 | ## go get 34 | 35 | hastur is also go-gettable: 36 | 37 | ``` 38 | go get github.com/seletskiy/hastur 39 | ``` 40 | 41 | # Usage 42 | 43 | ## Testing water 44 | 45 | The most simple usage is testing hastur out-of-the-box: you can tell it to 46 | create an ephemeral container with a basic set of packages: 47 | 48 | ``` 49 | sudo hastur -S 50 | ``` 51 | 52 | After invoking that command, you will end up ~~in Cthulhu's void~~ in the bash 53 | shell. 54 | 55 | This test container will be deleted after you exit its shell. 56 | 57 | ## Creating non-ephemeral containers 58 | 59 | Passing the `-k` flag will tell hastur to keep the container after exit: 60 | 61 | ``` 62 | sudo hastur -kS 63 | ``` 64 | 65 | If you don't like the fantastical autogenerated names, you can pass the flag 66 | `-n`: 67 | 68 | ``` 69 | sudo hastur -Sn my-cool-name 70 | ``` 71 | 72 | Note that ephemeral containers are only the ones that both have autogenerated 73 | names and were not started with the `-k` flag. 74 | 75 | ## Networking 76 | 77 | hastur will take care of setting up the networking by creating a bridge and 78 | setting up a shared network. By default, hastur will automatically generate IP 79 | addresses, and you can see a container's address either in its starting message 80 | or by running the query command: 81 | 82 | ``` 83 | sudo hastur -Q 84 | ``` 85 | 86 | You can specify your own IP address by passing the `-a` flag, like this: 87 | 88 | ``` 89 | sudo hastur -S -a 10.0.0.2/8 90 | ``` 91 | 92 | ## But what about software? 93 | 94 | hastur uses package-based container configurations and will happily populate 95 | your container with the packages that you want. You can use the `-p` flag for 96 | this: 97 | 98 | ``` 99 | sudo hastur -S -p nginx 100 | ``` 101 | 102 | That will create a container with the `nginx` package pre-installed. In 103 | actuality, hastur uses overlays to keep base dirs separate from container data. 104 | The base dirs, or, if you like, images, are just prepared root filesystems, 105 | which have pre-installed packages. You can query the cached base dirs by 106 | running hastur with the `-Qi` flag: 107 | 108 | ``` 109 | sudo hastur -Qi 110 | ``` 111 | 112 | In fact, from hastur's standpoint, a container is just a data dir, which gets 113 | overlayed on top of a root filesystem and then given network capability, so 114 | it will not remember what IP address a container has or what set of packages it 115 | has installed if you forget to specify the correct options. 116 | 117 | For example: 118 | 119 | ``` 120 | sudo hastur -Sn test -p git -- /bin/git --version 121 | ``` 122 | 123 | Will output: 124 | 125 | ``` 126 | git version 2.5.3 127 | ``` 128 | 129 | But running this container the next time without `-p git` will tell you that 130 | git is not installed: 131 | 132 | ``` 133 | sudo hastur -Sn test -- /bin/git --version 134 | ``` 135 | 136 | This outputs: 137 | 138 | ``` 139 | No such file or directory 140 | ``` 141 | 142 | However, this `test` container will have a separate FS and all data files will 143 | persist across the two runs. 144 | 145 | # Additional information 146 | 147 | hastur can operate over several root directories and keep container instances 148 | separately from each other. The `-r` flag is used for this: 149 | 150 | ``` 151 | sudo hastur -r solar-system -Sn earth 152 | sudo hastur -r solar-system -Sn moon 153 | sudo hastur -r alpha-centauri -Sn a 154 | sudo hastur -r alpha-centauri -Sn b 155 | sudo hastur -r alpha-centauri -Sn c 156 | ``` 157 | 158 | The output will be different for these two commands: 159 | 160 | ``` 161 | sudo hastur -r solar-system -Q 162 | sudo hastur -r alpha-centauri -Q 163 | ``` 164 | 165 | The first one will list only the `earth` and `moon` containers, and the second 166 | will only list the `a`, `b` and `c` containers. 167 | 168 | # License 169 | 170 | MIT. 171 | -------------------------------------------------------------------------------- /broadcast.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import "net" 4 | 5 | func allFF(b []byte) bool { 6 | for _, c := range b { 7 | if c != 0xff { 8 | return false 9 | } 10 | } 11 | return true 12 | } 13 | 14 | func bytesEqual(x, y []byte) bool { 15 | if len(x) != len(y) { 16 | return false 17 | } 18 | for i, b := range x { 19 | if y[i] != b { 20 | return false 21 | } 22 | } 23 | return true 24 | } 25 | 26 | var v4InV6Prefix = []byte{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0xff, 0xff} 27 | 28 | func broadcast(ip net.IP, mask net.IPMask) net.IP { 29 | if len(mask) == net.IPv6len && len(ip) == net.IPv4len && allFF(mask[:12]) { 30 | mask = mask[12:] 31 | } 32 | if len(mask) == net.IPv4len && len(ip) == net.IPv6len && bytesEqual(ip[:12], v4InV6Prefix) { 33 | ip = ip[12:] 34 | } 35 | n := len(ip) 36 | if n != len(mask) { 37 | return nil 38 | } 39 | out := make(net.IP, n) 40 | for i := 0; i < n; i++ { 41 | out[i] = ip[i] | ^mask[i] 42 | } 43 | return out 44 | } 45 | -------------------------------------------------------------------------------- /copy.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "io" 5 | "io/ioutil" 6 | "os" 7 | "path/filepath" 8 | 9 | "github.com/reconquest/ser-go" 10 | ) 11 | 12 | func copyFile(src string, dest string) error { 13 | srcFile, err := os.Open(src) 14 | if err != nil { 15 | return err 16 | } 17 | defer srcFile.Close() 18 | 19 | destFile, err := os.Create(dest) 20 | if err != nil { 21 | return err 22 | } 23 | defer destFile.Close() 24 | 25 | _, err = io.Copy(destFile, srcFile) 26 | if err != nil { 27 | return err 28 | } 29 | 30 | stat, err := os.Stat(src) 31 | if err != nil { 32 | return err 33 | } 34 | 35 | err = os.Chmod(dest, stat.Mode()) 36 | if err != nil { 37 | return ser.Errorf( 38 | err, "can't change file mode: %s", dest, 39 | ) 40 | } 41 | 42 | return nil 43 | } 44 | 45 | func copyDir(src string, dest string) (err error) { 46 | srcStat, err := os.Stat(src) 47 | if err != nil { 48 | return err 49 | } 50 | 51 | if !srcStat.IsDir() { 52 | return ser.Errorf( 53 | err, "%s is not directory", src, 54 | ) 55 | } 56 | 57 | err = os.MkdirAll(dest, srcStat.Mode()) 58 | if err != nil { 59 | return ser.Errorf( 60 | err, "can't mkdir %s", dest, 61 | ) 62 | } 63 | 64 | entries, err := ioutil.ReadDir(src) 65 | 66 | for _, entry := range entries { 67 | srcEntry := filepath.Join(src, entry.Name()) 68 | destEntry := filepath.Join(dest, entry.Name()) 69 | 70 | if entry.IsDir() { 71 | err = copyDir(srcEntry, destEntry) 72 | } else { 73 | err = copyFile(srcEntry, destEntry) 74 | } 75 | 76 | if err != nil { 77 | return ser.Errorf( 78 | err, 79 | "can't copy %s -> %s", srcEntry, destEntry, 80 | ) 81 | } 82 | } 83 | 84 | return nil 85 | } 86 | -------------------------------------------------------------------------------- /errors.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import "github.com/reconquest/ser-go" 4 | 5 | func formatAbsPathError(path string, err error) error { 6 | return ser.Errorf( 7 | err, "can't get abs path for '%s'", path, err, 8 | ) 9 | } 10 | -------------------------------------------------------------------------------- /fs.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "crypto/sha256" 5 | "fmt" 6 | "os" 7 | "os/exec" 8 | "path/filepath" 9 | "strings" 10 | 11 | "github.com/reconquest/executil-go" 12 | "github.com/reconquest/ser-go" 13 | ) 14 | 15 | func getFSType(root string) (string, error) { 16 | command := exec.Command("findmnt", "-o", "fstype", "-nfT", root) 17 | output, _, err := executil.Run(command) 18 | if err != nil { 19 | return "", err 20 | } 21 | 22 | return strings.TrimSpace(string(output)), nil 23 | } 24 | 25 | func createBaseDirForPackages( 26 | rootDir string, 27 | packages []string, 28 | storageEngine storage, 29 | ) (exists bool, dirName string, err error) { 30 | imageName := fmt.Sprintf( 31 | "%x", 32 | sha256.Sum224([]byte(strings.Join(packages, ","))), 33 | ) 34 | 35 | imageDir := getImageDir(rootDir, imageName) 36 | if isExists(imageDir) && !isExists(imageDir, ".hastur") { 37 | err = storageEngine.DeInitImage(imageName) 38 | if err != nil { 39 | return false, "", ser.Errorf( 40 | err, "can't deinitialize image %s", imageName, 41 | ) 42 | } 43 | } 44 | 45 | if !isExists(imageDir) { 46 | err = storageEngine.InitImage(imageName) 47 | if err != nil { 48 | return false, "", ser.Errorf( 49 | err, "can't initialize image %s", imageName, 50 | ) 51 | } 52 | 53 | return false, imageName, nil 54 | } else { 55 | return true, imageName, nil 56 | } 57 | } 58 | 59 | func installBootstrapExecutable(root string, target string) error { 60 | path, err := os.Readlink("/proc/self/exe") 61 | if err != nil { 62 | return ser.Errorf( 63 | err, "can't read link to /proc/self/exe", 64 | ) 65 | } 66 | 67 | command := exec.Command("cp", path, filepath.Join(root, target)) 68 | _, _, err = executil.Run(command) 69 | if err != nil { 70 | return err 71 | } 72 | 73 | return nil 74 | } 75 | 76 | func listContainers(rootDir string) ([]string, error) { 77 | containers := []string{} 78 | 79 | filepath.Walk( 80 | rootDir, 81 | func(path string, info os.FileInfo, err error) error { 82 | if path == rootDir { 83 | return nil 84 | } 85 | 86 | if info.IsDir() { 87 | containers = append(containers, filepath.Base(path)) 88 | return filepath.SkipDir 89 | } 90 | 91 | return nil 92 | }, 93 | ) 94 | 95 | return containers, nil 96 | } 97 | 98 | func getContainerDir(rootDir string, containerName string) string { 99 | return filepath.Join(rootDir, "containers", containerName) 100 | } 101 | 102 | func getImageDir(rootDir string, imageName string) string { 103 | return filepath.Join(rootDir, "images", imageName) 104 | } 105 | 106 | func getBaseDirs(rootDir string) ([]string, error) { 107 | return filepath.Glob(filepath.Join(rootDir, "base.#*")) 108 | } 109 | 110 | func removeContainerDir(containerDir string) error { 111 | command := exec.Command("rm", "-rf", containerDir) 112 | _, _, err := executil.Run(command) 113 | if err != nil { 114 | return err 115 | } 116 | 117 | return nil 118 | } 119 | 120 | func isExists(path ...string) bool { 121 | _, err := os.Stat(filepath.Join(path...)) 122 | return !os.IsNotExist(err) 123 | } 124 | -------------------------------------------------------------------------------- /iptables.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "os/exec" 5 | 6 | "github.com/reconquest/executil-go" 7 | ) 8 | 9 | func addPostroutingMasquarading(dev string) error { 10 | args := []string{"-t", "nat", "-A", "POSTROUTING", "-o", dev, 11 | "-j", "MASQUERADE"} 12 | 13 | command := exec.Command("iptables", args...) 14 | _, _, err := executil.Run(command) 15 | if err != nil { 16 | return err 17 | } 18 | 19 | return nil 20 | } 21 | 22 | func removePostroutingMasquarading(dev string) error { 23 | args := []string{"-t", "nat", "-D", "POSTROUTING", "-o", dev, 24 | "-j", "MASQUERADE"} 25 | 26 | command := exec.Command("iptables", args...) 27 | _, _, err := executil.Run(command) 28 | if err != nil { 29 | return err 30 | } 31 | 32 | return nil 33 | } 34 | -------------------------------------------------------------------------------- /machinectl.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "os/exec" 6 | "strconv" 7 | "strings" 8 | 9 | "github.com/reconquest/executil-go" 10 | "github.com/reconquest/ser-go" 11 | ) 12 | 13 | func listActiveContainers( 14 | containerSuffix string, 15 | ) (map[string]struct{}, error) { 16 | command := exec.Command("machinectl", "--no-legend") 17 | output, _, err := executil.Run(command) 18 | if err != nil { 19 | return nil, err 20 | } 21 | 22 | containers := map[string]struct{}{} 23 | rawContainers := strings.Split(string(output), "\n") 24 | 25 | for _, rawContainer := range rawContainers { 26 | if rawContainer == "" { 27 | continue 28 | } 29 | 30 | fields := strings.Fields(rawContainer) 31 | if len(fields) < 3 { 32 | return nil, fmt.Errorf( 33 | "invalid output from machinectl: %s", rawContainer, 34 | ) 35 | } 36 | 37 | if strings.HasSuffix(fields[0], containerSuffix) { 38 | nameWithoutSuffix := strings.TrimSuffix(fields[0], containerSuffix) 39 | containers[nameWithoutSuffix] = struct{}{} 40 | } 41 | } 42 | 43 | return containers, nil 44 | } 45 | 46 | func getContainerLeaderPID(name string) (int, error) { 47 | command := exec.Command("machinectl", "show", name+containerSuffix) 48 | output, _, err := executil.Run(command) 49 | if err != nil { 50 | return 0, err 51 | } 52 | 53 | config := strings.Split(string(output), "\n") 54 | for _, line := range config { 55 | if strings.HasPrefix(line, "Leader=") { 56 | pid, err := strconv.Atoi(strings.Split(line, "=")[1]) 57 | if err != nil { 58 | return 0, ser.Errorf( 59 | err, 60 | "can't convert Leader value from '%s' to PID", 61 | line, 62 | ) 63 | } 64 | 65 | return pid, nil 66 | } 67 | } 68 | 69 | return 0, fmt.Errorf( 70 | "PID info is not found in machinectl show '%s'", name, 71 | ) 72 | } 73 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "io/ioutil" 6 | "log" 7 | "math/rand" 8 | "net" 9 | "os" 10 | "path/filepath" 11 | "strings" 12 | "syscall" 13 | "time" 14 | 15 | "github.com/docopt/docopt-go" 16 | "github.com/reconquest/executil-go" 17 | "github.com/reconquest/ser-go" 18 | ) 19 | 20 | const ( 21 | containerSuffix = `.hastur` 22 | defaultPackages = `bash,coreutils,iproute2,iputils,libidn,nettle` 23 | version = `3.5` 24 | usage = `hastur the unspeakable - zero-conf systemd container manager. 25 | 26 | hastur is a simple wrapper around systemd-nspawn, that will start container 27 | with overlayfs, pre-installed packages and bridged network available out 28 | of the box. 29 | 30 | hastur operates over specified root directory, which will hold base FS 31 | for containers and numerous amount of containers derived from base FS. 32 | 33 | Primary use of hastur is testing purposes, running testcases for distributed 34 | services and running local network of trusted containers. 35 | 36 | Usage: 37 | hastur -h | --help 38 | hastur [options] [-b=] [-s=] [-a=] [-p ...] [-n=] -S [--] [...] 39 | hastur [options] [-s=] -Q [-j] [...] 40 | hastur [options] [-s=] -D [-f] 41 | hastur [options] [-s=] --free 42 | 43 | Options: 44 | -h --help Show this help. 45 | -r Root directory which will hold containers. 46 | [default: /var/lib/hastur/] 47 | -q Be quiet. Do not report status messages from nspawn. 48 | -f Force operation. 49 | -s Use specified storageSpec backend for container base 50 | images and containers themselves. By default, overlayfs 51 | will be used to provide COW for base system. If overlayfs 52 | is not possible on current FS and no other storageSpec 53 | engine is possible, tmpfs will be mounted in specified 54 | root dir to provide groundwork for overlayfs. 55 | [default: autodetect] 56 | Possible values are: 57 | * autodetect - use one of available storage engines 58 | depending on current FS. 59 | * overlayfs:N - use current FS and overlayfs on top; 60 | if overlayfs is unsupported on current FS, mount tmpfs of 61 | size N first. 62 | * zfs:POOL - use ZFS and use located on POOL. 63 | 64 | Create options: 65 | -S Create and start container. 66 | Execute specified command in created 67 | container. 68 | -b Bridge interface name and, optionally, an address, 69 | separated by colon. 70 | If bridge does not exists, it will be automatically 71 | created. 72 | [default: br0:10.0.0.1/8] 73 | -t Use host network and gain access to external network. 74 | Interface will pair given interface with bridge. 75 | -p Packages to install, separated by comma. 76 | [default: ` + defaultPackages + `] 77 | -n Use specified container name. If not specified, randomly 78 | generated name will be used and container will be 79 | considered ephemeral, e.g. will be destroyed on 80 | exit. 81 | -a
Use specified IP address/netmask. If not specified, 82 | automatically generated adress from 10.0.0.0/8 will 83 | be used. 84 | -k Keep container after exit if it name was autogenerated. 85 | -x Copy entries of specified directory into created 86 | container root directory. 87 | -e Keep container after exit if executed failed. 88 | 89 | Query options: 90 | -Q Show information about containers in the dir. 91 | Query container's options. 92 | -j Output information using JSON format. 93 | Destroy options: 94 | -D Destroy specified container. 95 | --free Completely remove all data in directory with 96 | containers and base images. 97 | ` 98 | ) 99 | 100 | func fatal(err error) { 101 | fmt.Fprintln(os.Stderr, err.Error()) 102 | os.Exit(1) 103 | } 104 | 105 | func main() { 106 | rand.Seed(time.Now().UnixNano()) 107 | 108 | if os.Args[0] == "/.hastur.exec" && len(os.Args) >= 2 { 109 | err := execBootstrap() 110 | if err != nil { 111 | fatal(err) 112 | } 113 | } 114 | 115 | args, err := docopt.Parse(usage, nil, true, version, false) 116 | if err != nil { 117 | panic(err) 118 | } 119 | 120 | var ( 121 | rootDir = args["-r"].(string) 122 | storageSpec = args["-s"].(string) 123 | ) 124 | 125 | storageEngine, err := createStorageFromSpec(rootDir, storageSpec) 126 | if err != nil { 127 | fatal(ser.Errorf(err, "can't initialize storage")) 128 | } 129 | 130 | switch { 131 | case args["-S"].(bool): 132 | err = createAndStart(args, storageEngine) 133 | case args["-Q"].(bool): 134 | err = queryContainers(args, storageEngine) 135 | case args["-D"].(bool): 136 | err = destroyContainer(args, storageEngine) 137 | case args["--free"].(bool): 138 | err = destroyRoot(args, storageEngine) 139 | } 140 | 141 | if err != nil { 142 | fatal(err) 143 | } 144 | } 145 | 146 | func execBootstrap() error { 147 | command := []string{} 148 | 149 | if len(os.Args) == 2 { 150 | command = []string{"/bin/bash"} 151 | } else { 152 | if len(os.Args) == 3 && strings.Contains(os.Args[2], " ") { 153 | command = []string{"/bin/bash", "-c", os.Args[2]} 154 | } else { 155 | command = os.Args[2:] 156 | } 157 | } 158 | 159 | ioutil.WriteFile(os.Args[1], []byte{}, 0) 160 | ioutil.ReadFile(os.Args[1]) 161 | 162 | err := os.Remove(os.Args[1]) 163 | if err != nil { 164 | return ser.Errorf( 165 | err, 166 | "can't remove control file '%s'", os.Args[1], 167 | ) 168 | } 169 | 170 | err = syscall.Exec(command[0], command[0:], os.Environ()) 171 | if err != nil { 172 | return ser.Errorf( 173 | err, 174 | "can't execute command %q", os.Args[2:], 175 | ) 176 | } 177 | 178 | return nil 179 | } 180 | 181 | func destroyContainer( 182 | args map[string]interface{}, 183 | storageEngine storage, 184 | ) error { 185 | var ( 186 | containerName = args[""].([]string)[0] 187 | ) 188 | 189 | err := storageEngine.DestroyContainer(containerName) 190 | 191 | _ = umountNetorkNamespace(containerName) 192 | 193 | err = cleanupNetworkInterface(containerName) 194 | if err != nil { 195 | log.Println(err) 196 | } 197 | 198 | return err 199 | } 200 | 201 | func showBaseDirsInfo( 202 | args map[string]interface{}, 203 | storageEngine storage, 204 | ) error { 205 | var ( 206 | rootDir = args["-r"].(string) 207 | ) 208 | 209 | baseDirs, err := getBaseDirs(rootDir) 210 | if err != nil { 211 | return ser.Errorf( 212 | err, 213 | "can't get base dirs from '%s'", rootDir, 214 | ) 215 | } 216 | 217 | for _, baseDir := range baseDirs { 218 | packages, err := listExplicitlyInstalled(baseDir) 219 | if err != nil { 220 | return ser.Errorf( 221 | err, 222 | "can't list explicitly installed packages in '%s'", 223 | baseDir, 224 | ) 225 | } 226 | 227 | fmt.Println(baseDir) 228 | for _, packageName := range packages { 229 | fmt.Printf("\t%s\n", packageName) 230 | } 231 | } 232 | 233 | return nil 234 | } 235 | 236 | func createAndStart( 237 | args map[string]interface{}, 238 | storageEngine storage, 239 | ) error { 240 | var ( 241 | bridgeInfo = args["-b"].(string) 242 | rootDir = args["-r"].(string) 243 | packagesList = args["-p"].([]string) 244 | containerName, _ = args["-n"].(string) 245 | commandLine = args[""].([]string) 246 | networkAddress, _ = args["-a"].(string) 247 | force = args["-f"].(bool) 248 | keep = args["-k"].(bool) 249 | keepFailed = args["-e"].(bool) 250 | copyingDir, _ = args["-x"].(string) 251 | hostInterface, _ = args["-t"].(string) 252 | quiet = args["-q"].(bool) 253 | ) 254 | 255 | err := ensureIPv4Forwarding() 256 | if err != nil { 257 | return ser.Errorf( 258 | err, 259 | "can't enable ipv4 forwarding", 260 | ) 261 | } 262 | 263 | bridgeDevice, bridgeAddress := parseBridgeInfo(bridgeInfo) 264 | err = ensureBridge(bridgeDevice) 265 | if err != nil { 266 | return ser.Errorf( 267 | err, 268 | "can't create bridge interface '%s'", bridgeDevice, 269 | ) 270 | } 271 | 272 | err = ensureBridgeInterfaceUp(bridgeDevice) 273 | if err != nil { 274 | return ser.Errorf( 275 | err, 276 | "can't set bridge '%s' up", 277 | bridgeDevice, 278 | ) 279 | } 280 | 281 | if bridgeAddress != "" { 282 | err = setupBridge(bridgeDevice, bridgeAddress) 283 | if err != nil { 284 | return ser.Errorf( 285 | err, 286 | "can't assign address '%s' on bridge '%s'", 287 | bridgeAddress, 288 | bridgeDevice, 289 | ) 290 | } 291 | } 292 | 293 | if hostInterface != "" { 294 | err := addInterfaceToBridge(hostInterface, bridgeDevice) 295 | if err != nil { 296 | return ser.Errorf( 297 | err, 298 | "can't bind host's ethernet '%s' to '%s'", 299 | hostInterface, 300 | bridgeDevice, 301 | ) 302 | } 303 | 304 | err = copyInterfaceAddressToBridge(hostInterface, bridgeDevice) 305 | if err != nil { 306 | return ser.Errorf( 307 | err, 308 | "can't copy address from host's '%s' to '%s'", 309 | hostInterface, 310 | bridgeDevice, 311 | ) 312 | } 313 | 314 | err = copyInterfaceRoutesToBridge(hostInterface, bridgeDevice) 315 | if err != nil { 316 | return ser.Errorf( 317 | err, 318 | "can't copy routes from host's '%s' to '%s'", 319 | hostInterface, 320 | bridgeDevice, 321 | ) 322 | } 323 | } 324 | 325 | ephemeral := false 326 | if containerName == "" { 327 | generatedName := generateContainerName() 328 | if !keep { 329 | ephemeral = true 330 | 331 | if !keepFailed && !quiet { 332 | fmt.Println( 333 | "Container is ephemeral and will be deleted after exit.", 334 | ) 335 | } 336 | } 337 | 338 | containerName = generatedName 339 | 340 | fmt.Printf("Container name: %s\n", containerName) 341 | } 342 | 343 | allPackages := []string{} 344 | for _, packagesGroup := range packagesList { 345 | packages := strings.Split(packagesGroup, ",") 346 | allPackages = append(allPackages, packages...) 347 | } 348 | 349 | cacheExists, baseDir, err := createBaseDirForPackages( 350 | rootDir, 351 | allPackages, 352 | storageEngine, 353 | ) 354 | if err != nil { 355 | return ser.Errorf( 356 | err, 357 | "can't create base dir '%s'", baseDir, 358 | ) 359 | } 360 | 361 | if !cacheExists || force { 362 | fmt.Println("Installing packages") 363 | err = installPackages(getImageDir(rootDir, baseDir), allPackages) 364 | if err != nil { 365 | return ser.Errorf( 366 | err, 367 | "can't install packages into '%s'", rootDir, 368 | ) 369 | } 370 | 371 | err = ioutil.WriteFile( 372 | filepath.Join(getImageDir(rootDir, baseDir), ".hastur"), 373 | nil, 0644, 374 | ) 375 | if err != nil { 376 | return ser.Errorf( 377 | err, "can't create .hastur file in image directory", 378 | ) 379 | } 380 | } 381 | 382 | err = storageEngine.InitContainer(baseDir, containerName) 383 | if err != nil { 384 | return ser.Errorf( 385 | err, 386 | "can't create directory layout under '%s'", rootDir, 387 | ) 388 | } 389 | 390 | if networkAddress == "" { 391 | _, baseIPNet, _ := net.ParseCIDR("10.0.0.0/8") 392 | networkAddress = generateRandomNetwork(baseIPNet) 393 | 394 | if !quiet { 395 | fmt.Printf("Container will use IP: %s\n", networkAddress) 396 | } 397 | } 398 | 399 | if copyingDir != "" { 400 | err = copyDir(copyingDir, getImageDir(rootDir, baseDir)) 401 | if err != nil { 402 | return ser.Errorf( 403 | err, 404 | "can't copy %s to container root", copyingDir, 405 | ) 406 | } 407 | } 408 | 409 | err = nspawn( 410 | storageEngine, 411 | containerName, 412 | bridgeDevice, networkAddress, bridgeAddress, 413 | ephemeral, keepFailed, quiet, 414 | commandLine, 415 | ) 416 | 417 | if err != nil { 418 | if executil.IsExitError(err) { 419 | os.Exit(executil.GetExitStatus(err)) 420 | } 421 | 422 | return ser.Errorf(err, "command execution failed") 423 | } 424 | 425 | return nil 426 | } 427 | 428 | func destroyRoot( 429 | args map[string]interface{}, 430 | storageEngine storage, 431 | ) error { 432 | err := storageEngine.Destroy() 433 | if err != nil { 434 | return ser.Errorf( 435 | err, "can't destroy storage", 436 | ) 437 | } 438 | 439 | return nil 440 | } 441 | 442 | func generateContainerName() string { 443 | tuples := []string{"ir", "oh", "at", "op", "un", "ed"} 444 | triples := []string{"gep", "vin", "kut", "lop", "man", "zod"} 445 | all := append(append([]string{}, tuples...), triples...) 446 | 447 | getTuple := func() string { 448 | return tuples[rand.Intn(len(tuples))] 449 | } 450 | 451 | getTriple := func() string { 452 | return triples[rand.Intn(len(triples))] 453 | } 454 | 455 | getAny := func() string { 456 | return all[rand.Intn(len(all))] 457 | } 458 | 459 | id := []string{ 460 | getTuple(), 461 | getTriple(), 462 | "-", 463 | getTuple(), 464 | getTriple(), 465 | getTuple(), 466 | "-", 467 | getAny(), 468 | } 469 | 470 | return strings.Join(id, "") 471 | } 472 | 473 | func parseBridgeInfo(bridgeInfo string) (dev, address string) { 474 | parts := strings.Split(bridgeInfo, ":") 475 | if len(parts) == 1 { 476 | return parts[0], "" 477 | } else { 478 | return parts[0], parts[1] 479 | } 480 | } 481 | 482 | func createStorageFromSpec(rootDir, storageSpec string) (storage, error) { 483 | var storageEngine storage 484 | var err error 485 | 486 | switch { 487 | case storageSpec == "autodetect": 488 | storageSpec = "overlayfs" 489 | fallthrough 490 | 491 | case strings.HasPrefix(storageSpec, "overlayfs"): 492 | storageEngine, err = NewOverlayFSStorage(rootDir, storageSpec) 493 | 494 | case strings.HasPrefix(storageSpec, "zfs"): 495 | storageEngine, err = NewZFSStorage(rootDir, storageSpec) 496 | } 497 | 498 | if err != nil { 499 | return nil, ser.Errorf( 500 | err, "can't create storage '%s'", storageSpec, 501 | ) 502 | } 503 | 504 | err = storageEngine.Init() 505 | if err != nil { 506 | return nil, ser.Errorf( 507 | err, "can't init storage '%s'", storageSpec, 508 | ) 509 | } 510 | 511 | return storageEngine, nil 512 | } 513 | -------------------------------------------------------------------------------- /mount.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "io/ioutil" 6 | "os" 7 | "os/exec" 8 | "path/filepath" 9 | "strings" 10 | 11 | "github.com/reconquest/executil-go" 12 | "github.com/reconquest/ser-go" 13 | ) 14 | 15 | func isMounted(device, mountpoint string) (bool, error) { 16 | absPath, err := filepath.Abs(mountpoint) 17 | if err != nil { 18 | return false, err 19 | } 20 | 21 | command := exec.Command("findmnt", device, absPath) 22 | _, _, err = executil.Run(command) 23 | if err != nil { 24 | if executil.IsExitError(err) { 25 | return false, nil 26 | } 27 | 28 | return false, err 29 | } 30 | 31 | return true, nil 32 | } 33 | 34 | func mountTmpfs(target string, size string) error { 35 | command := exec.Command( 36 | "mount", "-t", "tmpfs", "-o", "size="+size, "tmpfs", target, 37 | ) 38 | 39 | _, _, err := executil.Run(command) 40 | if err != nil { 41 | return err 42 | } 43 | 44 | return nil 45 | } 46 | 47 | func mountOverlay(lower, upper, work, target string) error { 48 | lowerAbsPath, err := filepath.Abs(lower) 49 | if err != nil { 50 | return formatAbsPathError(lower, err) 51 | } 52 | 53 | upperAbsPath, err := filepath.Abs(upper) 54 | if err != nil { 55 | return formatAbsPathError(upper, err) 56 | } 57 | 58 | workAbsPath, err := filepath.Abs(work) 59 | if err != nil { 60 | return formatAbsPathError(work, err) 61 | } 62 | 63 | command := exec.Command( 64 | "mount", "-t", "overlay", "-o", 65 | strings.Join([]string{ 66 | "lowerdir=" + lowerAbsPath, 67 | "upperdir=" + upperAbsPath, 68 | "workdir=" + workAbsPath, 69 | }, ","), 70 | "overlay", target, 71 | ) 72 | 73 | _, _, err = executil.Run(command) 74 | if err != nil { 75 | return err 76 | } 77 | 78 | return nil 79 | } 80 | 81 | func mountNetworkNamespace(PID int, target string) error { 82 | netnsDir := "/var/run/netns" 83 | if _, err := os.Stat(netnsDir); os.IsNotExist(err) { 84 | err := os.Mkdir(netnsDir, 0755) 85 | if err != nil { 86 | return ser.Errorf( 87 | err, 88 | "can't create dir '%s'", netnsDir, 89 | ) 90 | } 91 | } 92 | 93 | bindTarget := filepath.Join(netnsDir, target) 94 | 95 | err := ioutil.WriteFile(bindTarget, []byte{}, 0644) 96 | if err != nil { 97 | return ser.Errorf( 98 | err, "can't touch file '%s'", bindTarget, 99 | ) 100 | } 101 | 102 | return mountBind( 103 | filepath.Join("/proc", fmt.Sprint(PID), "ns/net"), bindTarget, 104 | ) 105 | } 106 | 107 | func mountBind(source, target string) error { 108 | command := exec.Command("mount", "--bind", source, target) 109 | 110 | _, _, err := executil.Run(command) 111 | if err != nil { 112 | return err 113 | } 114 | 115 | return nil 116 | } 117 | 118 | func umountNetorkNamespace(name string) error { 119 | bindTarget := filepath.Join("/var/run/netns", name) 120 | 121 | err := umount(bindTarget) 122 | if err != nil { 123 | return ser.Errorf( 124 | err, "can't umount %s", bindTarget, 125 | ) 126 | } 127 | 128 | err = os.Remove(bindTarget) 129 | if err != nil { 130 | return ser.Errorf( 131 | err, "can't remove %s", bindTarget, 132 | ) 133 | } 134 | 135 | return nil 136 | } 137 | 138 | func umountRecursively(target string) error { 139 | command := exec.Command("umount", "-R", target) 140 | _, _, err := executil.Run(command) 141 | if err != nil { 142 | return err 143 | } 144 | 145 | return nil 146 | } 147 | 148 | func umount(target string) error { 149 | command := exec.Command("umount", target) 150 | _, _, err := executil.Run(command) 151 | if err != nil { 152 | return err 153 | } 154 | 155 | return nil 156 | } 157 | -------------------------------------------------------------------------------- /network.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bytes" 5 | "errors" 6 | "fmt" 7 | "io/ioutil" 8 | "math" 9 | "net" 10 | "os" 11 | "os/exec" 12 | "strings" 13 | "time" 14 | 15 | "github.com/reconquest/executil-go" 16 | "github.com/reconquest/ser-go" 17 | ) 18 | 19 | func ensureBridge(bridge string) error { 20 | command := exec.Command("brctl", "addbr", bridge) 21 | _, stderr, err := executil.Run(command) 22 | if err != nil { 23 | prefix := fmt.Sprintf("device %s already exists;", bridge) 24 | if strings.HasPrefix(string(stderr), prefix) { 25 | return nil 26 | } 27 | 28 | return err 29 | } 30 | 31 | return nil 32 | } 33 | 34 | func ensureBridgeInterfaceUp(bridge string) error { 35 | command := exec.Command("ip", "link", "set", "dev", bridge, "up") 36 | _, _, err := executil.Run(command) 37 | if err != nil { 38 | return err 39 | } 40 | 41 | return nil 42 | } 43 | 44 | func ensureIPv4Forwarding() error { 45 | fileIpForward := "/proc/sys/net/ipv4/ip_forward" 46 | valueIpForward, err := ioutil.ReadFile(fileIpForward) 47 | if err != nil { 48 | return ser.Errorf( 49 | err, "can't read file %s", fileIpForward, 50 | ) 51 | } 52 | 53 | if strings.Contains(string(valueIpForward), "0") { 54 | err = ioutil.WriteFile( 55 | fileIpForward, []byte("1\n"), 56 | os.FileMode(0644), 57 | ) 58 | if err != nil { 59 | return ser.Errorf( 60 | err, "can't write '1' to file %s", fileIpForward, 61 | ) 62 | } 63 | } 64 | 65 | return nil 66 | } 67 | 68 | func copyInterfaceRoutesToBridge(iface, bridge string) error { 69 | command := exec.Command("ip", "route", "show", "dev", iface) 70 | output, _, err := executil.Run(command) 71 | if err != nil { 72 | return err 73 | } 74 | 75 | rawIPOutput := strings.Split(string(output), "\n") 76 | for _, line := range rawIPOutput { 77 | if line == "" { 78 | continue 79 | } 80 | 81 | trimmedLine := strings.TrimSpace(line) 82 | trimmedLine = strings.Replace(trimmedLine, " ", " ", -1) 83 | 84 | err = execIpRoute( 85 | "delete", iface, 86 | strings.Split(trimmedLine, " ")..., 87 | ) 88 | if err != nil { 89 | return err 90 | } 91 | 92 | err = execIpRoute( 93 | "add", bridge, 94 | strings.Split(trimmedLine, " ")..., 95 | ) 96 | if err != nil { 97 | return err 98 | } 99 | } 100 | 101 | return nil 102 | } 103 | 104 | func execIpRoute(action string, iface string, args ...string) error { 105 | command := exec.Command( 106 | "ip", append( 107 | []string{"route", action, "dev", iface}, 108 | args..., 109 | )..., 110 | ) 111 | 112 | _, stderr, err := executil.Run(command) 113 | if err != nil { 114 | if bytes.HasPrefix( 115 | stderr, 116 | []byte("RTNETLINK answers: File exists"), 117 | ) { 118 | return nil 119 | } 120 | 121 | return err 122 | } 123 | 124 | return nil 125 | } 126 | 127 | func copyInterfaceAddressToBridge(iface string, bridge string) error { 128 | addrs, err := getHostIPs(iface) 129 | if err != nil { 130 | return ser.Errorf( 131 | err, "can't get host ip addresses for interface %s", iface, 132 | ) 133 | } 134 | 135 | for _, addr := range addrs { 136 | ip, _, err := net.ParseCIDR(addr.String()) 137 | if err != nil { 138 | return ser.Errorf( 139 | err, "can't parse net address '%s'", addr.String(), 140 | ) 141 | } 142 | if ip.To4() == nil { 143 | continue 144 | } 145 | 146 | broadcast := broadcast(ip, ip.DefaultMask()) 147 | 148 | command := exec.Command( 149 | "ip", "addr", "add", 150 | "dev", bridge, addr.String(), 151 | "broadcast", broadcast.String(), 152 | ) 153 | _, stderr, err := executil.Run(command) 154 | if err != nil { 155 | if bytes.HasPrefix( 156 | stderr, 157 | []byte("RTNETLINK answers: File exists"), 158 | ) { 159 | return nil 160 | } 161 | 162 | return err 163 | } 164 | } 165 | 166 | return nil 167 | } 168 | 169 | func addInterfaceToBridge(iface, bridge string) error { 170 | command := exec.Command("brctl", "addif", bridge, iface) 171 | _, stderr, err := executil.Run(command) 172 | if err != nil { 173 | prefix := fmt.Sprintf("device %s is already a member", iface) 174 | if strings.HasPrefix(string(stderr), prefix) { 175 | return nil 176 | } 177 | 178 | return err 179 | } 180 | 181 | return nil 182 | } 183 | 184 | func getContainerIP(containerName string) (string, error) { 185 | command := exec.Command("ip", "-n", containerName, "addr", "show", "host0") 186 | output, _, err := executil.Run(command) 187 | if err != nil { 188 | return "", err 189 | } 190 | 191 | rawIPOutput := strings.Split(string(output), "\n") 192 | for _, line := range rawIPOutput { 193 | trimmedLine := strings.TrimSpace(line) 194 | if strings.HasPrefix(trimmedLine, "inet ") { 195 | inet := strings.Fields(trimmedLine) 196 | if len(inet) < 2 { 197 | return "", fmt.Errorf( 198 | "invalid output from ip: %q", line, 199 | ) 200 | } 201 | 202 | return inet[1], nil 203 | } 204 | } 205 | 206 | return "", nil 207 | } 208 | 209 | func setupNetwork(namespace string, address string, gateway string) error { 210 | err := ensureAddress(namespace, address, "host0") 211 | if err != nil { 212 | return err 213 | } 214 | 215 | err = upInterface(namespace, "host0") 216 | if err != nil { 217 | return err 218 | } 219 | 220 | gatewayIP, _, err := net.ParseCIDR(gateway) 221 | if err != nil { 222 | return err 223 | } 224 | 225 | err = addDefaultRoute(namespace, "host0", gatewayIP.String()) 226 | if err != nil { 227 | return err 228 | } 229 | 230 | return nil 231 | } 232 | 233 | func addDefaultRoute(namespace string, dev string, gateway string) error { 234 | args := []string{"route", "add", "default", "via", gateway} 235 | if namespace != "" { 236 | args = append([]string{"-n", namespace}, args...) 237 | } 238 | 239 | command := exec.Command("ip", args...) 240 | 241 | _, stderr, err := executil.Run(command) 242 | if err != nil { 243 | if bytes.HasPrefix(stderr, []byte("RTNETLINK answers: File exists")) { 244 | return nil 245 | } 246 | 247 | return err 248 | } 249 | 250 | return nil 251 | } 252 | 253 | func ensureAddress(namespace string, address string, dev string) error { 254 | args := []string{"addr", "add", address, "dev", dev} 255 | if namespace != "" { 256 | args = append([]string{"-n", namespace}, args...) 257 | } 258 | 259 | command := exec.Command("ip", args...) 260 | 261 | _, stderr, err := executil.Run(command) 262 | if err != nil { 263 | if bytes.HasPrefix(stderr, []byte("RTNETLINK answers: File exists")) { 264 | return nil 265 | } 266 | 267 | return err 268 | } 269 | 270 | return nil 271 | } 272 | 273 | func cleanupNetworkInterface(name string) error { 274 | interfaceName := "vb-" + name 275 | if len(interfaceName) > 14 { 276 | interfaceName = interfaceName[:14] // seems like it get cutted by 14 chars 277 | } 278 | 279 | args := []string{"link", "delete", interfaceName} 280 | 281 | command := exec.Command("ip", args...) 282 | 283 | _, stderr, err := executil.Run(command) 284 | if err != nil { 285 | if bytes.Contains(stderr, []byte("Cannot find device")) { 286 | return nil 287 | } 288 | 289 | return err 290 | } 291 | 292 | return nil 293 | } 294 | 295 | func setupBridge(dev string, address string) error { 296 | return ensureAddress("", address, dev) 297 | } 298 | 299 | func upInterface(namespace string, dev string) error { 300 | command := exec.Command( 301 | "ip", "-n", namespace, "link", "set", "up", dev, 302 | ) 303 | 304 | _, _, err := executil.Run(command) 305 | if err != nil { 306 | return err 307 | } 308 | 309 | return nil 310 | } 311 | 312 | func generateRandomNetwork(address *net.IPNet) string { 313 | tick := float64(time.Now().UnixNano() / 1000000) 314 | 315 | ones, bits := address.Mask.Size() 316 | zeros := bits - ones 317 | uniqIPsAmount := math.Pow(2.0, float64(zeros)) 318 | 319 | rawIP := math.Mod(tick, uniqIPsAmount) 320 | 321 | remainder := rawIP 322 | 323 | remainder, octet4 := math.Modf(remainder / 255.0) 324 | remainder, octet3 := math.Modf(remainder / 255.0) 325 | remainder, octet2 := math.Modf(remainder / 255.0) 326 | 327 | base := address.IP 328 | 329 | address.IP = net.IPv4( 330 | byte(remainder)|base[0], 331 | byte(octet2*255)|base[1], 332 | byte(octet3*255)|base[2], 333 | byte(octet4*255)|base[3], 334 | ) 335 | 336 | address.IP.Mask(address.Mask) 337 | 338 | return address.String() 339 | } 340 | 341 | func getHostIPs(interfaceName string) ([]net.Addr, error) { 342 | hostInterfaces, err := net.Interfaces() 343 | if err != nil { 344 | return nil, err 345 | } 346 | 347 | var iface net.Interface 348 | for _, hostInterface := range hostInterfaces { 349 | if hostInterface.Name == interfaceName { 350 | iface = hostInterface 351 | break 352 | } 353 | } 354 | 355 | addrs, err := iface.Addrs() 356 | if err != nil { 357 | return nil, err 358 | } 359 | 360 | if len(addrs) == 0 { 361 | return nil, errors.New("no ip addresses assigned to interface") 362 | } 363 | 364 | return addrs, nil 365 | } 366 | -------------------------------------------------------------------------------- /nspawn.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "io/ioutil" 6 | "os" 7 | "os/exec" 8 | "path/filepath" 9 | "syscall" 10 | 11 | "github.com/reconquest/executil-go" 12 | "github.com/reconquest/ser-go" 13 | ) 14 | 15 | func nspawn( 16 | storageEngine storage, 17 | containerName string, 18 | bridge string, 19 | networkAddress string, bridgeAddress string, 20 | ephemeral bool, keepFailed bool, quiet bool, 21 | commandLine []string, 22 | ) (err error) { 23 | defer storageEngine.DeInitContainer(containerName) 24 | 25 | if err != nil { 26 | return ser.Errorf( 27 | err, 28 | "storage can't create rootfs for nspawn", 29 | ) 30 | } 31 | 32 | if err != nil { 33 | return ser.Errorf( 34 | err, 35 | "can't setup overlayfs for '%s'", containerName, 36 | ) 37 | } 38 | 39 | if ephemeral { 40 | defer func() { 41 | if err != nil && keepFailed { 42 | return 43 | } 44 | 45 | removeErr := storageEngine.DestroyContainer(containerName) 46 | if removeErr != nil { 47 | err = removeErr 48 | 49 | fmt.Fprintln( 50 | os.Stderr, 51 | ser.Errorf( 52 | err, "can't remove container '%s'", containerName, 53 | ).HierarchicalError(), 54 | ) 55 | } 56 | }() 57 | } 58 | 59 | bootstrapper := "/.hastur.exec" 60 | err = installBootstrapExecutable( 61 | storageEngine.GetContainerRoot(containerName), 62 | bootstrapper, 63 | ) 64 | if err != nil { 65 | return err 66 | } 67 | 68 | controlPipeName := bootstrapper + ".control" 69 | controlPipePath := filepath.Join( 70 | storageEngine.GetContainerRoot(containerName), 71 | controlPipeName, 72 | ) 73 | 74 | err = syscall.Mknod(controlPipePath, syscall.S_IFIFO|0644, 0) 75 | if err != nil { 76 | return ser.Errorf( 77 | err, 78 | "can't create control pipe for bootstrapper", 79 | ) 80 | } 81 | 82 | defer os.Remove(controlPipePath) 83 | 84 | // we ignore error there because interface may not exist 85 | _ = umountNetorkNamespace(containerName) 86 | _ = cleanupNetworkInterface(containerName) 87 | 88 | defer cleanupNetworkInterface(containerName) 89 | 90 | err = addPostroutingMasquarading(bridge) 91 | if err != nil { 92 | return ser.Errorf( 93 | err, 94 | "can't add masquarading rules on the '%s'", 95 | bridge, 96 | ) 97 | } 98 | 99 | defer removePostroutingMasquarading(bridge) 100 | 101 | command := exec.Command( 102 | "systemd-machine-id-setup", 103 | "--root", storageEngine.GetContainerRoot(containerName), 104 | ) 105 | _, _, err = executil.Run(command) 106 | if err != nil { 107 | return err 108 | } 109 | 110 | args := []string{ 111 | "--pipe", 112 | "-M", containerName + containerSuffix, 113 | "-D", storageEngine.GetContainerRoot(containerName), 114 | } 115 | 116 | args = append(args, "-n", "--network-bridge", bridge) 117 | 118 | if quiet { 119 | args = append(args, "-q") 120 | } 121 | 122 | args = append(args, bootstrapper, controlPipeName) 123 | 124 | command = exec.Command( 125 | "systemd-nspawn", 126 | append(args, commandLine...)..., 127 | ) 128 | 129 | command.Env = []string{} 130 | 131 | command.Stdin = os.Stdin 132 | command.Stdout = os.Stdout 133 | command.Stderr = os.Stderr 134 | 135 | err = command.Start() 136 | if err != nil { 137 | return err 138 | } 139 | 140 | defer command.Process.Kill() 141 | 142 | _, err = ioutil.ReadFile(controlPipePath) 143 | if err != nil { 144 | return err 145 | } 146 | 147 | pid, err := getContainerLeaderPID(containerName) 148 | if err != nil { 149 | return err 150 | } 151 | 152 | err = mountNetworkNamespace(pid, containerName) 153 | if err != nil { 154 | return err 155 | } 156 | 157 | defer umountNetorkNamespace(containerName) 158 | 159 | err = setupNetwork(containerName, networkAddress, bridgeAddress) 160 | if err != nil { 161 | return err 162 | } 163 | 164 | err = ioutil.WriteFile(controlPipePath, []byte{}, 0) 165 | if err != nil { 166 | return ser.Errorf( 167 | err, "can't write to control pipe") 168 | } 169 | 170 | err = command.Wait() 171 | return err 172 | } 173 | -------------------------------------------------------------------------------- /packages.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "io/ioutil" 5 | "os" 6 | "os/exec" 7 | "path/filepath" 8 | "strings" 9 | 10 | "github.com/reconquest/executil-go" 11 | ) 12 | 13 | func installPackages(target string, packages []string) error { 14 | args := []string{"-c", "-d", target} 15 | command := exec.Command("pacstrap", append(args, packages...)...) 16 | 17 | command.Stdout = os.Stderr 18 | command.Stderr = os.Stderr 19 | 20 | _, _, err := executil.Run( 21 | command, 22 | executil.IgnoreStderr, 23 | executil.IgnoreStdout, 24 | ) 25 | if err != nil { 26 | return err 27 | } 28 | 29 | err = ioutil.WriteFile(filepath.Join(target, ".packages"), []byte( 30 | strings.Join(packages, "\n"), 31 | ), 0644) 32 | 33 | return err 34 | } 35 | 36 | func listExplicitlyInstalled(baseDir string) ([]string, error) { 37 | rawPackages, err := ioutil.ReadFile(filepath.Join(baseDir, ".packages")) 38 | if err != nil { 39 | return nil, err 40 | } 41 | 42 | packages := strings.Split(strings.TrimSpace(string(rawPackages)), "\n") 43 | 44 | return packages, nil 45 | } 46 | -------------------------------------------------------------------------------- /query.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "os" 7 | "path/filepath" 8 | "text/tabwriter" 9 | 10 | "github.com/reconquest/karma-go" 11 | ) 12 | 13 | type container struct { 14 | Name string `json:"name"` 15 | Status string `json:"status"` 16 | Root string `json:"root"` 17 | Address string `json:"address"` 18 | } 19 | 20 | func queryContainers( 21 | args map[string]interface{}, storageEngine storage, 22 | ) error { 23 | var ( 24 | rootDir = args["-r"].(string) 25 | useJSON = args["-j"].(bool) 26 | filter = args[""].([]string) 27 | ) 28 | 29 | all, err := listContainers(filepath.Join(rootDir, "containers")) 30 | if err != nil { 31 | return err 32 | } 33 | 34 | active, err := listActiveContainers(containerSuffix) 35 | if err != nil { 36 | return err 37 | } 38 | 39 | containers := []container{} 40 | for _, name := range all { 41 | if len(filter) > 0 { 42 | found := false 43 | for _, target := range filter { 44 | if target == name { 45 | found = true 46 | break 47 | } 48 | } 49 | 50 | if !found { 51 | continue 52 | } 53 | } 54 | 55 | container := container{ 56 | Name: name, 57 | Status: "inactive", 58 | Root: storageEngine.GetContainerRoot(name), 59 | Address: "", 60 | } 61 | 62 | _, ok := active[name] 63 | if ok { 64 | container.Status = "active" 65 | container.Address, err = getContainerIP(name) 66 | if err != nil { 67 | fmt.Fprintln(os.Stderr, karma.Format(err, 68 | "WARNING: can't obtain container '%s' address", 69 | name, 70 | )) 71 | } 72 | } 73 | 74 | containers = append(containers, container) 75 | } 76 | 77 | if !useJSON { 78 | writer := tabwriter.NewWriter(os.Stdout, 2, 4, 2, ' ', 0) 79 | for _, container := range containers { 80 | fmt.Fprintf( 81 | writer, 82 | "%s\t%s\t%s\t%s\n", 83 | container.Name, container.Status, 84 | container.Address, container.Root, 85 | ) 86 | } 87 | 88 | err = writer.Flush() 89 | if err != nil { 90 | return err 91 | } 92 | 93 | return nil 94 | } 95 | 96 | output, err := json.MarshalIndent(containers, "", " ") 97 | if err != nil { 98 | return err 99 | } 100 | 101 | fmt.Println(string(output)) 102 | 103 | return nil 104 | } 105 | -------------------------------------------------------------------------------- /storage.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | type storage interface { 4 | Init() error 5 | DeInit() error 6 | InitContainer(baseDir, container string) error 7 | DeInitContainer(container string) error 8 | InitImage(image string) error 9 | DeInitImage(image string) error 10 | DestroyContainer(container string) error 11 | GetContainerRoot(container string) string 12 | Destroy() error 13 | } 14 | -------------------------------------------------------------------------------- /storage_overlayfs.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "os" 6 | "path/filepath" 7 | "strings" 8 | 9 | "github.com/reconquest/ser-go" 10 | ) 11 | 12 | const defaultOverlayFSSize = "1G" 13 | 14 | type overlayFSStorage struct { 15 | tmpfsSize string 16 | rootDir string 17 | } 18 | 19 | func NewOverlayFSStorage(rootDir, spec string) (storage, error) { 20 | args := strings.Split(spec, ":") 21 | size := defaultOverlayFSSize 22 | 23 | // TODO: validate size parameter 24 | if len(args) == 2 { 25 | size = args[1] 26 | } 27 | 28 | return &overlayFSStorage{ 29 | rootDir: rootDir, 30 | tmpfsSize: size, 31 | }, nil 32 | } 33 | 34 | func (storage *overlayFSStorage) Init() error { 35 | FSType, err := getFSType(storage.rootDir) 36 | if err != nil { 37 | return ser.Errorf( 38 | err, 39 | "can't get FS type for '%s'", storage.rootDir, 40 | ) 41 | } 42 | 43 | switch FSType { 44 | case "tmpfs", "ext", "ext2", "ext3", "ext4", "btrfs": 45 | return nil 46 | 47 | default: 48 | fmt.Printf("WARNING! %s is not currently supported.\n", FSType) 49 | fmt.Println(" overlayfs over tmpfs will be used and") 50 | fmt.Println(" containers will not persist across reboots.") 51 | fmt.Println() 52 | 53 | err := storage.fixUnsupportedFS() 54 | if err != nil { 55 | return err 56 | } 57 | } 58 | 59 | return nil 60 | } 61 | 62 | func (storage *overlayFSStorage) InitImage(image string) error { 63 | return os.MkdirAll(getImageDir(storage.rootDir, image), 0755) 64 | } 65 | 66 | func (storage *overlayFSStorage) DeInitImage(image string) error { 67 | return os.RemoveAll(getImageDir(storage.rootDir, image)) 68 | } 69 | 70 | func (storage *overlayFSStorage) DeInit() error { 71 | return nil 72 | } 73 | 74 | func (storage *overlayFSStorage) InitContainer( 75 | baseDir string, 76 | containerName string, 77 | ) error { 78 | containerDir := getContainerDir(storage.rootDir, containerName) 79 | 80 | containerRoot := storage.GetContainerRoot(containerName) 81 | 82 | for _, dir := range []string{"root", ".nspawn.root", ".overlay.workdir"} { 83 | err := os.MkdirAll( 84 | filepath.Join(containerDir, dir), 85 | 0755, 86 | ) 87 | 88 | if err != nil { 89 | return err 90 | } 91 | } 92 | 93 | err := mountOverlay( 94 | getImageDir(storage.rootDir, baseDir), 95 | filepath.Join(containerDir, "root"), 96 | filepath.Join(containerDir, ".overlay.workdir"), 97 | containerRoot, 98 | ) 99 | if err != nil { 100 | return ser.Errorf( 101 | err, 102 | "can't mount overlay fs [%s] for '%s'", 103 | baseDir, containerName, 104 | ) 105 | } 106 | 107 | return nil 108 | } 109 | 110 | func (storage *overlayFSStorage) GetContainerRoot(containerName string) string { 111 | containerDir := getContainerDir(storage.rootDir, containerName) 112 | 113 | return filepath.Join(containerDir, ".nspawn.root") 114 | } 115 | 116 | func (storage *overlayFSStorage) DeInitContainer(containerName string) error { 117 | return umount(storage.GetContainerRoot(containerName)) 118 | } 119 | 120 | func (storage *overlayFSStorage) Destroy() error { 121 | return umountRecursively(storage.rootDir) 122 | } 123 | 124 | func (storage *overlayFSStorage) DestroyContainer(containerName string) error { 125 | _ = storage.DeInitContainer(containerName) 126 | 127 | return removeContainerDir(getContainerDir(storage.rootDir, containerName)) 128 | } 129 | 130 | func (storage *overlayFSStorage) fixUnsupportedFS() error { 131 | tmpfsMounted, err := isMounted("tmpfs", storage.rootDir) 132 | if err != nil { 133 | return ser.Errorf( 134 | err, 135 | "can't check is tmpfs mounted on '%s'", storage.rootDir, 136 | ) 137 | } 138 | 139 | if !tmpfsMounted { 140 | err := os.MkdirAll(storage.rootDir, 0644) 141 | if err != nil { 142 | return ser.Errorf( 143 | err, 144 | "can't create directory for tmpfs mountpoint", 145 | ) 146 | } 147 | 148 | err = mountTmpfs(storage.rootDir, storage.tmpfsSize) 149 | if err != nil { 150 | return ser.Errorf( 151 | err, 152 | "can't mount tmpfs of size %s on '%s'", 153 | storage.tmpfsSize, storage.rootDir, 154 | ) 155 | } 156 | } 157 | 158 | return nil 159 | } 160 | -------------------------------------------------------------------------------- /storage_zfs.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "errors" 5 | "os/exec" 6 | "path/filepath" 7 | "strings" 8 | 9 | "github.com/reconquest/executil-go" 10 | ) 11 | 12 | type zfsStorage struct { 13 | pool string 14 | rootDir string 15 | } 16 | 17 | func doZFSCommand(parameters ...string) error { 18 | command := exec.Command("zfs", parameters...) 19 | _, _, err := executil.Run(command) 20 | if err != nil { 21 | return err 22 | } 23 | 24 | return nil 25 | } 26 | 27 | func NewZFSStorage(rootDir, spec string) (storage, error) { 28 | args := strings.Split(spec, ":") 29 | pool := "" 30 | 31 | // TODO: validate pool parameter 32 | if len(args) == 2 { 33 | pool = args[1] 34 | } 35 | 36 | if pool == "" { 37 | return nil, errors.New( 38 | "pool name should be specified", 39 | ) 40 | } 41 | 42 | return &zfsStorage{ 43 | pool: pool, 44 | rootDir: rootDir, 45 | }, nil 46 | } 47 | 48 | func (storage *zfsStorage) Init() error { 49 | err := doZFSCommand( 50 | "create", 51 | "-p", 52 | filepath.Join(storage.pool, getContainerDir(storage.rootDir, "")), 53 | ) 54 | if err != nil { 55 | return err 56 | } 57 | 58 | err = doZFSCommand( 59 | "create", 60 | "-p", 61 | filepath.Join(storage.pool, getImageDir(storage.rootDir, "")), 62 | ) 63 | if err != nil { 64 | return err 65 | } 66 | 67 | return nil 68 | } 69 | 70 | func (storage *zfsStorage) InitImage(image string) error { 71 | err := doZFSCommand( 72 | "create", 73 | "-p", 74 | filepath.Join(storage.pool, getImageDir(storage.rootDir, image)), 75 | ) 76 | if err != nil { 77 | return err 78 | } 79 | 80 | return nil 81 | } 82 | 83 | func (storage *zfsStorage) DeInitImage(image string) error { 84 | err := doZFSCommand( 85 | "destroy", 86 | filepath.Join(storage.pool, getImageDir(storage.rootDir, image)), 87 | ) 88 | if err != nil { 89 | return err 90 | } 91 | 92 | return nil 93 | } 94 | 95 | func (storage *zfsStorage) DeInit() error { 96 | return nil 97 | } 98 | 99 | func (storage *zfsStorage) InitContainer( 100 | baseDir string, 101 | containerName string, 102 | ) error { 103 | err := doZFSCommand( 104 | "list", 105 | filepath.Join( 106 | storage.pool, 107 | getImageDir(storage.rootDir, baseDir), 108 | )+"@"+containerName, 109 | ) 110 | 111 | if err != nil { 112 | err := doZFSCommand( 113 | "snapshot", 114 | filepath.Join( 115 | storage.pool, 116 | getImageDir(storage.rootDir, baseDir), 117 | )+"@"+containerName, 118 | ) 119 | 120 | if err != nil { 121 | return err 122 | } 123 | } 124 | 125 | err = doZFSCommand( 126 | "list", 127 | filepath.Join( 128 | storage.pool, 129 | getContainerDir(storage.rootDir, containerName), 130 | ), 131 | ) 132 | 133 | if err != nil { 134 | err = doZFSCommand( 135 | "clone", 136 | filepath.Join( 137 | storage.pool, 138 | getImageDir(storage.rootDir, baseDir), 139 | )+"@"+containerName, 140 | filepath.Join( 141 | storage.pool, 142 | getContainerDir(storage.rootDir, containerName), 143 | ), 144 | ) 145 | 146 | if err != nil { 147 | return err 148 | } 149 | } 150 | 151 | return nil 152 | } 153 | 154 | func (storage *zfsStorage) GetContainerRoot(containerName string) string { 155 | containerDir := getContainerDir(storage.rootDir, containerName) 156 | 157 | return containerDir 158 | } 159 | 160 | func (storage *zfsStorage) DeInitContainer(containerName string) error { 161 | return nil 162 | } 163 | 164 | func (storage *zfsStorage) Destroy() error { 165 | return doZFSCommand( 166 | "destroy", 167 | "-r", 168 | filepath.Join(storage.pool, storage.rootDir), 169 | ) 170 | } 171 | 172 | func (storage *zfsStorage) DestroyContainer(containerName string) error { 173 | return doZFSCommand( 174 | "destroy", 175 | filepath.Join(storage.pool, getContainerDir( 176 | storage.rootDir, 177 | containerName, 178 | )), 179 | ) 180 | } 181 | --------------------------------------------------------------------------------