├── .gitignore ├── Makefile ├── README.md ├── Vagrantfile ├── commit.go ├── git.go ├── git_test.go ├── integration_test.go ├── main.go ├── pull.go ├── pullV2.go ├── pullingV2_job.go ├── pulling_job.go ├── push.go ├── pushV2.go ├── queue.go ├── registry_session.go ├── sample_configs ├── container.json └── lxc-config ├── setup.sh ├── utils.go └── vendor.sh /.gitignore: -------------------------------------------------------------------------------- 1 | rootfs 2 | .vagrant 3 | vendor 4 | release 5 | krgo 6 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | GOPATH:=`pwd`/vendor:$(GOPATH) #inject vendored package 2 | GOPATH:=`pwd`/vendor/src/github.com/docker/docker/vendor:$(GOPATH) #inject docker vendored package 3 | 4 | VERSION:=1.5.0 5 | HARDWARE=$(shell uname -m) 6 | 7 | build: vendor 8 | GOPATH=$(GOPATH) go build 9 | 10 | test: vendor build 11 | GOPATH=$(GOPATH) PATH=$(PATH):`pwd` go test 12 | 13 | release: 14 | mkdir -p release 15 | GOPATH=$(GOPATH) GOOS=linux go build -o release/krgo 16 | cd release && tar -zcf krgo-v$(VERSION)_$(HARDWARE).tgz krgo 17 | rm release/krgo 18 | 19 | clean: 20 | rm -rf ./krgo ./release ./vendor/pkg/* 21 | 22 | vendor: 23 | sh vendor.sh 24 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | > krgo was formerly [dlrootfs](https://github.com/robinmonjo/dlrootfs) and cargo but has been renamed because of this [issue]() 2 | 3 | # krgo 4 | 5 | docker hub without docker. `krgo` is a command line tool to pull and push docker images from/to the docker hub. 6 | `krgo` brings the docker hub content and delivery capabilities to any container engine. 7 | 8 | [Read the launch article and how to](https://gist.github.com/robinmonjo/f6ca0f85a204c8103e10) 9 | 10 | ## Why krgo ? 11 | 12 | docker is really popular and a lot of people and organisations are building docker images. These images are stored 13 | and shared on the docker hub. However they are only available to docker users. Metadata apart, a docker 14 | image is a linux root file system that can be used with any container engine 15 | ([LXC](https://linuxcontainers.org/lxc/introduction/), 16 | [libcontainer nsinit](https://github.com/docker/libcontainer#nsinit), 17 | [systemd-nspawn](http://www.freedesktop.org/software/systemd/man/systemd-nspawn.html), 18 | [rocket](https://github.com/coreos/rocket) 19 | ...). 20 | Using `krgo`, non docker users would be able to pull and share linux images using the [docker hub](https://hub.docker.com/). 21 | 22 | ## Installation 23 | 24 | ````bash 25 | curl -sL https://github.com/robinmonjo/krgo/releases/download/v1.5.0/krgo-v1.5.0_x86_64.tgz | tar -C /usr/local/bin -zxf - 26 | ```` 27 | 28 | Provided binary is linux only but `krgo` may be used on OSX and (probably) Windows too. 29 | 30 | ## Usage 31 | 32 | ```` 33 | NAME: 34 | krgo - docker hub without docker 35 | 36 | USAGE: 37 | krgo [global options] command [command options] [arguments...] 38 | 39 | VERSION: 40 | krgo 1.5.0 (docker 1.5.0) 41 | 42 | COMMANDS: 43 | pull pull an image 44 | push push an image 45 | commit commit changes to an image pulled with -g 46 | help, h Shows a list of commands or help for one command 47 | 48 | GLOBAL OPTIONS: 49 | --help, -h show help 50 | --version, -v print the version 51 | ```` 52 | 53 | ### krgo pull 54 | 55 | `krgo pull image [-r rootfs] [-u user] [-g] [-v2]` 56 | 57 | Pull `image` into `rootfs` directory: 58 | - `-u` flag allows you to specify your docker hub credentials: `username:password` 59 | - `-g` flag download the image into a git repository. Each branch contains a layer 60 | of the image. This is the resulting rootfs of `krgo pull busybox -g`: 61 | 62 | ![Alt text](https://dl.dropboxusercontent.com/u/6543817/cargo-readme/cargo_br.png) 63 | 64 | Branches are named `layer__`. layer_n is a `checkout -b` from layer_n-1, so 65 | the layer_3 branch contains the full image. You can then use it as is. 66 | 67 | The `-g` flag brings the power of git to container images (versionning, inspecting diffs ...). But more importantly, it will allow to 68 | push image modifications to the docker hub (see `krgo push`) 69 | 70 | - `-v2` flag makes `krgo` download the image using docker [v2 registry](https://github.com/docker/docker-registry/issues/612). Because everything is not yet production ready, images pulled with the `-v2` flag won't be pushable to the docker hub 71 | 72 | **Examples**: 73 | - `krgo pull debian -v2 #library/debian:latest using v2 registry` 74 | - `krgo pull progrium/busybox -r busybox -g` 75 | - `krgo pull robinmonjo/debian:latest -r debian -u $DHUB_CREDS` 76 | 77 | ### krgo push 78 | 79 | Push an image downloaded with the `-g` option to the docker hub 80 | (a [docker hub account](https://hub.docker.com/account/signup/) is needed). Images downloaded with the `-v2` flag can't be pushed at this time as registry v2 is not yet fully operational. 81 | 82 | In order to push your modification you **must commit** them beforehand: 83 | 84 | `krgo commit [-r rootfs] -m "commit message"` 85 | 86 | This will take every changes on the current branch, and commit them onto a new branch. 87 | The new branch will be properly named and some additional metadata will be written, so 88 | this new layer can be pushed: 89 | 90 | ````bash 91 | $> krgo commit -m "adding new user" 92 | Changes commited in layer_4_804c37249306321b90bbfa07d7cfe02d5f3d056971eb069d7bc37647de484a35 93 | Image ID: 804c37249306321b90bbfa07d7cfe02d5f3d056971eb069d7bc37647de484a35 94 | Parent: 4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125 95 | Layer size: 1536 96 | Done 97 | ```` 98 | 99 | If you plan to use `krgo push`, branches should not be created manually and commit must be done via `krgo`. 100 | Also, branches other than the last one should never be modified. 101 | 102 | `krgo push image [-r rootfs] -u username:password` 103 | 104 | Push the image in the `rootfs` directory onto the docker hub. 105 | 106 | **Examples:** 107 | - `krgo push username/debian:krgo -u $DHUB_CREDS` 108 | - `krgo push username/busybox -r busybox -u $DHUB_CREDS` 109 | 110 | ## Dependency 111 | 112 | If you plan to use `krgo` to push images, you will need git >= 1.8 113 | 114 | ## Notes on docker v2 registry 115 | 116 | docker 1.5.0 pulls official images (library/*) from the v2 registry. Push are still made using the v1 registry. v2 registry brings a lot of [changes](https://github.com/docker/docker-registry/issues/612), the most noticeable ones for `krgo` are: 117 | - images are now addressed by content (IDs are tarsum calculation) 118 | - images are described in a manifest 119 | - images metadata are no more stored in a json file at the root of the file system 120 | 121 | A lot of layers in v1 where created only because the json metadata file changed. Since this file is no more distributed, some (all ?) images have "dulpicated empty layers". `krgo` clean the manifest to download only what's needed. 122 | 123 | 124 | ## Hacking on krgo 125 | 126 | `krgo` directly uses some of docker source code. Docker is moving fast, and `krgo` must keep up. 127 | I will maintain it but if you want to contribute every pull requests / bug reports are welcome. 128 | 129 | You don't need linux, `krgo` can run on OSX (Windows ?). Fork the repository and clone it into your 130 | go workspace. Then `make vendor`, `make build` and you are ready to go. Tests can be run 131 | with `make test`. Note that most `krgo` command must be run as sudo. 132 | 133 | ## Resources 134 | 135 | - [docker image specification](https://github.com/docker/docker/blob/master/image/spec/v1.md) 136 | - [docker image layering](https://docs.docker.com/terms/layer/) 137 | - [docker repository](https://github.com/docker/docker) 138 | - [docker hub](https://hub.docker.com/) 139 | 140 | ## License 141 | 142 | MIT 143 | -------------------------------------------------------------------------------- /Vagrantfile: -------------------------------------------------------------------------------- 1 | # -*- mode: ruby -*- 2 | # vi: set ft=ruby : 3 | 4 | # Vagrantfile API/syntax version. Don't touch unless you know what you're doing! 5 | VAGRANTFILE_API_VERSION = "2" 6 | 7 | Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| 8 | config.vm.box = "trusty64" 9 | config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box" 10 | 11 | config.vm.synced_folder ".", "/vagrant", disabled: true 12 | config.vm.synced_folder ".", "/vagrant/krgo" 13 | 14 | config.vm.provision "shell", path: "setup.sh" 15 | 16 | config.vm.network :public_network 17 | 18 | config.vm.provider "virtualbox" do |v| 19 | v.memory = 2048 20 | end 21 | 22 | end 23 | -------------------------------------------------------------------------------- /commit.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "io/ioutil" 7 | "os" 8 | "path" 9 | "time" 10 | 11 | "github.com/docker/docker/image" 12 | "github.com/docker/docker/pkg/archive" 13 | "github.com/docker/docker/utils" 14 | ) 15 | 16 | //krgo commit -r rootfs 17 | //commit current changes in a new properly formated branch ready for pushing 18 | func commitChanges(rootfs, message string) error { 19 | if !isGitRepo(rootfs) { 20 | return fmt.Errorf("%v not a git repository", rootfs) 21 | } 22 | gitRepo, _ := newGitRepo(rootfs) 23 | 24 | layerData, err := gitRepo.exportUncommitedChangeSet() 25 | if err != nil { 26 | return err 27 | } 28 | defer layerData.Close() 29 | 30 | //Load image data 31 | image, err := image.LoadImage(gitRepo.Path) //reading json file in rootfs 32 | if err != nil { 33 | return err 34 | } 35 | 36 | //fill new infos 37 | image.Parent = image.ID 38 | image.ID = utils.GenerateRandomID() 39 | image.Created = time.Now() 40 | image.Comment = message 41 | 42 | layer, err := archive.NewTempArchive(layerData, "") 43 | if err != nil { 44 | return err 45 | } 46 | image.Size = layer.Size 47 | os.RemoveAll(layer.Name()) 48 | 49 | if err := image.SaveSize(rootfs); err != nil { 50 | return err 51 | } 52 | 53 | jsonRaw, err := json.Marshal(image) 54 | if err != nil { 55 | return err 56 | } 57 | 58 | err = ioutil.WriteFile(path.Join(rootfs, "json"), jsonRaw, 0600) 59 | if err != nil { 60 | return err 61 | } 62 | 63 | //commit the changes in a new branch 64 | n, _ := gitRepo.countBranch() 65 | br := newBranch(n, image.ID) 66 | if _, err = gitRepo.checkoutB(br); err != nil { 67 | return err 68 | } 69 | if _, err := gitRepo.addAllAndCommit(message); err != nil { 70 | return err 71 | } 72 | 73 | fmt.Printf("Changes commited in %v\n", br) 74 | fmt.Printf("Image ID: %v\nParent: %v\nChecksum: %v\nLayer size: %v\n", image.ID, image.Parent, image.Size) 75 | 76 | return nil 77 | } 78 | -------------------------------------------------------------------------------- /git.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bufio" 5 | "bytes" 6 | "fmt" 7 | "os/exec" 8 | "path" 9 | "strconv" 10 | "strings" 11 | 12 | "github.com/docker/docker/pkg/archive" 13 | ) 14 | 15 | const ( 16 | DIFF_ADDED = "A" 17 | DIFF_MODIFIED = "M" 18 | DIFF_DELETED = "D" 19 | ) 20 | 21 | var ErrNoChange = fmt.Errorf("no changes to extract") 22 | 23 | type gitRepo struct { 24 | Path string 25 | } 26 | 27 | func isGitRepo(repoPath string) bool { 28 | return fileExists(path.Join(repoPath, ".git")) 29 | } 30 | 31 | func newGitRepo(path string) (*gitRepo, error) { 32 | r := &gitRepo{Path: path} 33 | 34 | if isGitRepo(r.Path) { 35 | return r, nil 36 | } 37 | 38 | _, err := r.exec("init", path) 39 | if err != nil { 40 | return nil, err 41 | } 42 | 43 | email, _ := r.userConfig("email") 44 | if len(email) == 0 { 45 | if _, err := r.execInWorkTree("config", "user.email", "fake@krgo.com"); err != nil { 46 | return nil, err 47 | } 48 | } 49 | 50 | name, _ := r.userConfig("name") 51 | if len(name) == 0 { 52 | if _, err := r.execInWorkTree("config", "user.name", "krgo"); err != nil { 53 | return nil, err 54 | } 55 | } 56 | 57 | return r, nil 58 | } 59 | 60 | func (r *gitRepo) userConfig(key string) ([]byte, error) { 61 | return r.exec("config", "user."+key) 62 | } 63 | 64 | func (r *gitRepo) checkout(br branch) ([]byte, error) { 65 | return r.execInWorkTree("checkout", br.string()) 66 | } 67 | 68 | func (r *gitRepo) checkoutB(br branch) ([]byte, error) { 69 | return r.execInWorkTree("checkout", "-b", br.string()) 70 | } 71 | 72 | func (r *gitRepo) addAllAndCommit(message string) ([]byte, error) { 73 | badd, err := r.add(".") 74 | if err != nil { 75 | return badd, err 76 | } 77 | bCi, err := r.commit(message) 78 | return append(badd, bCi...), err 79 | } 80 | 81 | func (r *gitRepo) add(file string) ([]byte, error) { 82 | return r.execInWorkTree("add", file, "--all") 83 | } 84 | 85 | func (r *gitRepo) commit(message string) ([]byte, error) { 86 | out, err := r.execInWorkTree("status", "--porcelain") 87 | if err != nil { 88 | return out, err 89 | } 90 | if len(out) == 0 { 91 | return nil, nil //nothing to commit 92 | } 93 | return r.execInWorkTree("commit", "-m", message) 94 | } 95 | 96 | func (r *gitRepo) branch() ([]branch, error) { 97 | b, err := r.execInWorkTree("branch") 98 | if err != nil { 99 | return nil, err 100 | } 101 | rawBrs := strings.Split(string(b), "\n") 102 | brs := make([]branch, len(rawBrs)) 103 | for i, br := range rawBrs { 104 | brs[i] = branch(strings.TrimLeft(br, " *")) 105 | } 106 | return brs[:len(brs)-1], nil //remove the last empty line 107 | } 108 | 109 | func (r *gitRepo) currentBranch() (branch, error) { 110 | b, err := r.execInWorkTree("symbolic-ref", "--short", "HEAD") 111 | return branch(strings.TrimSuffix(string(b), "\n")), err 112 | } 113 | 114 | func (r *gitRepo) describeBranch(br branch, descr string) error { 115 | _, err := r.execInWorkTree("config", "branch."+br.string()+".description", descr) 116 | return err 117 | } 118 | 119 | func (r *gitRepo) branchDescription(br branch) ([]byte, error) { 120 | return r.execInWorkTree("config", "branch."+br.string()+".description") 121 | } 122 | 123 | func (r *gitRepo) countBranch() (int, error) { 124 | branches, err := r.branch() 125 | if err != nil { 126 | return -1, err 127 | } 128 | return len(branches), nil 129 | } 130 | 131 | func (r *gitRepo) diffCached() ([]byte, error) { 132 | return r.execInWorkTree("diff", "--cached", "--name-status") 133 | } 134 | 135 | func (r *gitRepo) diff(br1, br2 branch) ([]byte, error) { 136 | return r.execInWorkTree("diff", br1.string()+".."+br2.string(), "--name-status") 137 | } 138 | 139 | //export every uncommited changes in the current branch 140 | func (r *gitRepo) exportUncommitedChangeSet() (archive.Archive, error) { 141 | r.add(".") 142 | 143 | diff, err := r.diffCached() 144 | if err != nil { 145 | return nil, err 146 | } 147 | return exportChanges(r.Path, diff) 148 | } 149 | 150 | func (r *gitRepo) exportChangeSet(br branch) (archive.Archive, error) { 151 | currentBr, err := r.currentBranch() 152 | if err != nil { 153 | return nil, err 154 | } 155 | 156 | _, err = r.checkout(br) 157 | if err != nil { 158 | return nil, err 159 | } 160 | 161 | defer func() { 162 | r.checkout(currentBr) 163 | }() 164 | 165 | branches, err := r.branch() 166 | if err != nil { 167 | return nil, err 168 | } 169 | 170 | switch br.number() { 171 | case 0: 172 | changes, err := archive.ChangesDirs(r.Path, "") 173 | if err != nil { 174 | return nil, err 175 | } 176 | var curatedChanges []archive.Change 177 | for _, ch := range changes { 178 | if !strings.HasPrefix(ch.Path, "/.git") { 179 | curatedChanges = append(curatedChanges, ch) 180 | } 181 | } 182 | return archive.ExportChanges(r.Path, curatedChanges) 183 | default: 184 | parentBr := branches[br.number()-1] 185 | diff, _ := r.diff(parentBr, br) 186 | return exportChanges(r.Path, diff) 187 | } 188 | } 189 | 190 | func exportChanges(rootfs string, diff []byte) (archive.Archive, error) { 191 | var changes []archive.Change 192 | 193 | scanner := bufio.NewScanner(bytes.NewReader(diff)) 194 | for scanner.Scan() { 195 | line := scanner.Text() 196 | dType := strings.SplitN(line, "\t", 2)[0] 197 | path := "/" + strings.SplitN(line, "\t", 2)[1] // important to consider the / for ExportChanges 198 | 199 | change := archive.Change{Path: path} 200 | 201 | switch dType { 202 | case DIFF_MODIFIED: 203 | change.Kind = archive.ChangeModify 204 | case DIFF_ADDED: 205 | change.Kind = archive.ChangeAdd 206 | case DIFF_DELETED: 207 | change.Kind = archive.ChangeDelete 208 | } 209 | 210 | changes = append(changes, change) 211 | 212 | if err := scanner.Err(); err != nil { 213 | return nil, err 214 | } 215 | } 216 | if len(changes) == 0 { 217 | return nil, ErrNoChange 218 | } 219 | return archive.ExportChanges(rootfs, changes) 220 | } 221 | 222 | func (r *gitRepo) execInWorkTree(args ...string) ([]byte, error) { 223 | args = append([]string{"--git-dir=" + path.Join(r.Path, "/.git"), "--work-tree=" + r.Path}, args...) 224 | return r.exec(args...) 225 | } 226 | 227 | func (r *gitRepo) exec(args ...string) ([]byte, error) { 228 | gitPath, err := exec.LookPath("git") 229 | if err != nil { 230 | return nil, err 231 | } 232 | cmd := exec.Command(gitPath, args...) 233 | out, err := cmd.CombinedOutput() 234 | if err != nil { 235 | return out, fmt.Errorf("%v (%v)", string(out), err) 236 | } 237 | return out, nil 238 | } 239 | 240 | //branch specific type for krgo 241 | type branch string 242 | 243 | func newBranch(n int, ID string) branch { 244 | return branch("layer_" + strconv.Itoa(n) + "_" + ID) 245 | } 246 | 247 | func (br branch) number() int { 248 | n, _ := strconv.ParseInt(strings.Split(string(br), "_")[1], 10, 64) 249 | return int(n) 250 | } 251 | 252 | func (br branch) imageID() string { 253 | return strings.Split(string(br), "_")[2] 254 | } 255 | 256 | func (br branch) string() string { 257 | return string(br) 258 | } 259 | -------------------------------------------------------------------------------- /git_test.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "io/ioutil" 6 | "os" 7 | "path" 8 | "strconv" 9 | "testing" 10 | 11 | "github.com/docker/docker/pkg/archive" 12 | ) 13 | 14 | const REPO_PATH = "/tmp/git_repo" 15 | 16 | var ( 17 | branches = []branch{ 18 | newBranch(0, "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125"), 19 | newBranch(1, "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8126"), 20 | newBranch(2, "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8127"), 21 | newBranch(3, "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8128"), 22 | } 23 | ) 24 | 25 | func asserErrNil(err error, t *testing.T) { 26 | if err != nil { 27 | t.Fatal(err) 28 | } 29 | } 30 | 31 | func TestGitFlow(t *testing.T) { 32 | fmt.Printf("Testing git ... ") 33 | r, err := newGitRepo(REPO_PATH) 34 | asserErrNil(err, t) 35 | 36 | defer os.RemoveAll(REPO_PATH) 37 | 38 | //Create 3 branches 39 | for i := 0; i < 3; i++ { 40 | br := branches[i] 41 | _, err = r.checkoutB(br) 42 | asserErrNil(err, t) 43 | 44 | curBr, err := r.currentBranch() 45 | asserErrNil(err, t) 46 | 47 | if br != curBr { 48 | t.Fatalf("current branch: %v expected %v", curBr, br) 49 | } 50 | 51 | f, err := os.Create(path.Join(r.Path, "br"+strconv.Itoa(i)+".txt")) 52 | asserErrNil(err, t) 53 | f.Close() 54 | 55 | _, err = r.addAllAndCommit("commit message") 56 | asserErrNil(err, t) 57 | } 58 | 59 | exportChangeSet(r, branches[0], []string{"br0.txt"}, []string{"br1.txt", "br2.txt", ".git"}, t) 60 | exportChangeSet(r, branches[1], []string{"br1.txt"}, []string{"br0.txt", "br2.txt"}, t) 61 | exportChangeSet(r, branches[2], []string{"br2.txt"}, []string{"br0.txt", "br1.txt"}, t) 62 | 63 | //Modify files 64 | err = ioutil.WriteFile(path.Join(r.Path, "br0.txt"), []byte("hello world !!"), 0777) 65 | asserErrNil(err, t) 66 | _, err = r.addAllAndCommit("commit message") 67 | asserErrNil(err, t) 68 | exportChangeSet(r, branches[2], []string{"br2.txt", "br0.txt"}, []string{"br1.txt"}, t) 69 | 70 | //Delete file 71 | err = os.Remove(path.Join(r.Path, "br1.txt")) 72 | asserErrNil(err, t) 73 | _, err = r.addAllAndCommit("commit message") 74 | exportChangeSet(r, branches[2], []string{"br2.txt", ".wh.br1.txt", "br0.txt"}, []string{"br1.txt"}, t) 75 | 76 | //Uncommited changes 77 | _, err = r.checkoutB(branches[3]) 78 | asserErrNil(err, t) 79 | 80 | f, err := os.Create(path.Join(r.Path, "br3.txt")) 81 | asserErrNil(err, t) 82 | f.Close() 83 | exportUncommitedChangeSet(r, []string{"br3.txt"}, []string{"br1.txt", ".wh.br1.txt", "br0.txt", "br2.txt"}, t) 84 | fmt.Printf("OK\n") 85 | } 86 | 87 | func exportUncommitedChangeSet(r *gitRepo, expectedFiles, unexpectedFiles []string, t *testing.T) { 88 | tar, err := r.exportUncommitedChangeSet() 89 | asserErrNil(err, t) 90 | defer tar.Close() 91 | checkTarCorrect(tar, expectedFiles, unexpectedFiles, t) 92 | } 93 | 94 | func exportChangeSet(r *gitRepo, br branch, expectedFiles, unexpectedFiles []string, t *testing.T) { 95 | tar, err := r.exportChangeSet(br) 96 | asserErrNil(err, t) 97 | defer tar.Close() 98 | checkTarCorrect(tar, expectedFiles, unexpectedFiles, t) 99 | } 100 | 101 | func checkTarCorrect(tar archive.Archive, expectedFiles, unexpectedFiles []string, t *testing.T) { 102 | err := archive.Untar(tar, "/tmp/tar", nil) 103 | asserErrNil(err, t) 104 | defer os.RemoveAll("/tmp/tar") 105 | filesShouldExist(true, expectedFiles, "/tmp/tar", t) 106 | filesShouldExist(false, unexpectedFiles, "/tmp/tar", t) 107 | } 108 | 109 | func filesShouldExist(shouldExist bool, files []string, basePath string, t *testing.T) { 110 | for _, f := range files { 111 | exist := fileExists(path.Join(basePath, f)) 112 | if exist != shouldExist { 113 | t.Fatalf("file %v should exist ? %v", f, shouldExist) 114 | } 115 | } 116 | } 117 | -------------------------------------------------------------------------------- /integration_test.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "os" 6 | "os/exec" 7 | "path" 8 | "strconv" 9 | "testing" 10 | "time" 11 | ) 12 | 13 | const CREDS_ENV string = "DHUB_CREDS" 14 | 15 | var ( 16 | krgo = &krgoBin{"krgo"} 17 | testImages = []string{"busybox", "progrium/busybox"} 18 | testV2RegImages = []string{"busybox"} 19 | privateImage = "robinmonjo/busybox" 20 | gitImage = "busybox:latest" 21 | 22 | minimalLinuxRootfs = []string{"bin", "dev", "etc", "home", "lib", "mnt", "opt", "proc", "root", "run", "sbin", "sys", "tmp", "usr", "var"} 23 | ) 24 | 25 | type krgoBin struct { 26 | binary string 27 | } 28 | 29 | func (krgo *krgoBin) exec(args ...string) error { 30 | cmd := exec.Command(krgo.binary, args...) 31 | out, err := cmd.CombinedOutput() 32 | if err != nil { 33 | return fmt.Errorf("Error: %v, Out: %s", err, string(out)) 34 | } 35 | return nil 36 | } 37 | 38 | func TestPullImages(t *testing.T) { 39 | for _, imageName := range testImages { 40 | rootfs := uniqueStr("rootfs") 41 | fmt.Printf("Testing %s image ... ", imageName) 42 | err := krgo.exec("pull", imageName, "-r", rootfs) 43 | defer os.RemoveAll(rootfs) 44 | if err != nil { 45 | t.Fatal(err) 46 | } 47 | if err := checkFS(rootfs); err != nil { 48 | t.Fatal(err) 49 | } 50 | fmt.Printf("Ok\n") 51 | } 52 | } 53 | 54 | func TestPullImagesV2(t *testing.T) { 55 | for _, imageName := range testV2RegImages { 56 | rootfs := uniqueStr("rootfs") 57 | fmt.Printf("Testing registry v2 with %s image ... ", imageName) 58 | err := krgo.exec("pull", imageName, "-r", rootfs, "-v2") 59 | defer os.RemoveAll(rootfs) 60 | if err != nil { 61 | t.Fatal(err) 62 | } 63 | if err := checkFS(rootfs); err != nil { 64 | t.Fatal(err) 65 | } 66 | fmt.Printf("Ok\n") 67 | } 68 | } 69 | 70 | func TestPullPrivateImage(t *testing.T) { 71 | creds := os.Getenv(CREDS_ENV) 72 | if creds == "" { 73 | fmt.Printf("Skipping private image test (%s not set)\n", CREDS_ENV) 74 | return 75 | } 76 | fmt.Printf("Testing private %s image ... ", privateImage) 77 | rootfs := uniqueStr("rootfs") 78 | err := krgo.exec("pull", privateImage, "-r", rootfs, "-u", creds) 79 | defer os.RemoveAll(rootfs) 80 | if err != nil { 81 | t.Fatal(err) 82 | } 83 | if err := checkFS(rootfs); err != nil { 84 | t.Fatal(err) 85 | } 86 | fmt.Printf("Ok\n") 87 | } 88 | 89 | func TestPullAndPushImage(t *testing.T) { 90 | //1: download the image 91 | fmt.Printf("Testing pulling and pushing on image %s\n\tPulling ... ", gitImage) 92 | rootfs := uniqueStr("rootfs") 93 | err := krgo.exec("pull", gitImage, "-r", rootfs, "-g") 94 | defer os.RemoveAll(rootfs) 95 | if err != nil { 96 | t.Fatal(err) 97 | } 98 | if err := checkFS(rootfs); err != nil { 99 | t.Fatal(err) 100 | } 101 | fmt.Printf("Ok\n") 102 | 103 | gitRepo, _ := newGitRepo(rootfs) 104 | branches, _ := gitRepo.branch() 105 | 106 | fmt.Printf("\tChecking git layering ... ") 107 | 108 | expectedbranches := []string{ 109 | "layer_0_511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158", 110 | "layer_1_df7546f9f060a2268024c8a230d8639878585defcc1bc6f79d2728a13957871b", 111 | "layer_2_ea13149945cb6b1e746bf28032f02e9b5a793523481a0a18645fc77ad53c4ea2", 112 | "layer_3_4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125", 113 | } 114 | 115 | for i, br := range branches { 116 | if br.string() != expectedbranches[i] { 117 | t.Fatal("Expected branch", expectedbranches[i], "got", br) 118 | } 119 | } 120 | fmt.Printf("Ok\n") 121 | 122 | //2: make some modification to it 123 | creds := os.Getenv(CREDS_ENV) 124 | if creds == "" { 125 | fmt.Printf("Skipping push image test (%s not set)\n", CREDS_ENV) 126 | return 127 | } 128 | 129 | fmt.Printf("\tModifying and commiting the image ... ") 130 | newImageName := uniqueStr("robinmonjo/krgo_bb_") 131 | 132 | //make some modifications on the image 133 | createdFile := "modification.txt" 134 | deletedFile := "sbin/ifconfig" 135 | 136 | f, err := os.Create(path.Join(rootfs, createdFile)) 137 | if err != nil { 138 | t.Fatal(err) 139 | } 140 | f.Close() 141 | 142 | err = os.RemoveAll(path.Join(rootfs, deletedFile)) 143 | if err != nil { 144 | t.Fatal(err) 145 | } 146 | 147 | //3: commit the image 148 | err = krgo.exec("commit", "-r", rootfs, "-m", "commit message") 149 | if err != nil { 150 | t.Fatal(err) 151 | } 152 | fmt.Printf("Ok\n") 153 | 154 | //4: push the image 155 | fmt.Printf("\tPushing %s into %s image ... ", gitImage, newImageName) 156 | err = krgo.exec("push", newImageName, "-r", rootfs, "-u", creds) 157 | if err != nil { 158 | t.Fatal(err) 159 | } 160 | fmt.Printf("Ok\n") 161 | 162 | //5: donload the image and make sure modifications where applied 163 | fmt.Printf("\tPulling %s image to make sure modifications were properly applied ... ", newImageName) 164 | rootfsNew := uniqueStr("rootfs") 165 | err = krgo.exec("pull", newImageName, "-r", rootfsNew) 166 | defer os.RemoveAll(rootfsNew) 167 | if err != nil { 168 | t.Fatal(err) 169 | } 170 | 171 | if !fileExists(path.Join(rootfsNew, createdFile)) { 172 | t.Fatal("expected file %s doesn't exists", createdFile) 173 | } 174 | 175 | if fileExists(path.Join(rootfsNew, deletedFile)) { 176 | t.Fatal("file %s should have been deleted", deletedFile) 177 | } 178 | fmt.Printf("Ok\n") 179 | } 180 | 181 | func checkFS(rootfs string) error { 182 | for _, file := range minimalLinuxRootfs { 183 | if !fileExists(path.Join(rootfs, file)) { 184 | return fmt.Errorf("expected file %s doesn't exists", file) 185 | } 186 | } 187 | return nil 188 | } 189 | 190 | func uniqueStr(prefix string) string { 191 | timestamp := time.Now().Unix() 192 | timestampStr := strconv.FormatInt(timestamp, 10) 193 | return prefix + timestampStr 194 | } 195 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "log" 6 | "os" 7 | 8 | "github.com/codegangsta/cli" 9 | "github.com/docker/docker/dockerversion" 10 | ) 11 | 12 | const ( 13 | VERSION = "1.5.0" 14 | DOCKER_VERSION = "1.5.0" 15 | ) 16 | 17 | var ( 18 | //shared flags 19 | userFlag = cli.StringFlag{Name: "u, user", Usage: "dockerhub credentials (format: username:password)"} 20 | rootfsFlag = cli.StringFlag{Name: "r, rootfs", Usage: "path of the root FS (default: rootfs)", Value: "rootfs"} 21 | 22 | //commands 23 | pullCmd = cli.Command{ 24 | Name: "pull", 25 | Usage: "pull an image", 26 | Description: "pull image [-r rootfs] [-u user] [-g] [-v2]", 27 | Action: pull, 28 | Flags: []cli.Flag{ 29 | cli.BoolFlag{Name: "g, git-layering", Usage: "use git layering (needed to push afteward)"}, 30 | userFlag, 31 | rootfsFlag, 32 | cli.BoolFlag{Name: "v2", Usage: "use docker V2 registry (push not available yet for images pulled with this flag)"}, 33 | }, 34 | } 35 | 36 | pushCmd = cli.Command{ 37 | Name: "push", 38 | Usage: "push an image", 39 | Description: "push image [-r rootfs] -u user", 40 | Action: push, 41 | Flags: []cli.Flag{ 42 | userFlag, 43 | rootfsFlag, 44 | }, 45 | } 46 | 47 | commitCmd = cli.Command{ 48 | Name: "commit", 49 | Usage: "commit changes to an image pulled with -g", 50 | Description: "commit [-r rootfs] -m message", 51 | Action: commit, 52 | Flags: []cli.Flag{ 53 | cli.StringFlag{Name: "m, message", Usage: "commit message"}, 54 | rootfsFlag, 55 | }, 56 | } 57 | ) 58 | 59 | func init() { 60 | dockerversion.VERSION = DOCKER_VERSION //needed otherwise error 500 on push 61 | } 62 | 63 | func main() { 64 | app := cli.NewApp() 65 | app.Name = "krgo" 66 | app.Version = "krgo " + VERSION + " (docker " + DOCKER_VERSION + ")" 67 | app.Usage = "docker hub without docker" 68 | app.Author = "Robin Monjo" 69 | app.Email = "robinmonjo@gmail.com" 70 | app.Commands = []cli.Command{pullCmd, pushCmd, commitCmd} 71 | 72 | app.Run(os.Args) 73 | } 74 | 75 | func pull(c *cli.Context) { 76 | imageName, imageTag := parseImageNameTag(c.Args().First()) 77 | userName, password := parseCredentials(c.String("user")) 78 | 79 | fmt.Printf("Pulling image %v:%v ...\n", imageName, imageTag) 80 | session, err := newRegistrySession(userName, password) 81 | if err != nil { 82 | log.Fatal(err) 83 | } 84 | 85 | if c.Bool("git-layering") { 86 | if c.Bool("v2") { 87 | err = session.pullRepositoryV2(imageName, imageTag, c.String("rootfs")) 88 | } else { 89 | err = session.pullRepository(imageName, imageTag, c.String("rootfs")) 90 | } 91 | } else { 92 | if c.Bool("v2") { 93 | err = session.pullImageV2(imageName, imageTag, c.String("rootfs")) 94 | } else { 95 | err = session.pullImage(imageName, imageTag, c.String("rootfs")) 96 | } 97 | } 98 | if err != nil { 99 | log.Fatal(err) 100 | } 101 | 102 | fmt.Printf("Done. Rootfs of %v:%v in %v\n", imageName, imageTag, c.String("rootfs")) 103 | } 104 | 105 | func commit(c *cli.Context) { 106 | err := commitChanges(c.String("rootfs"), c.String("message")) 107 | if err != nil { 108 | log.Fatalf("Something went wrong: %v\nGit repo may have been altered. Please make sure it's fine before commiting again\n", err) 109 | } 110 | fmt.Printf("Done\n") 111 | } 112 | 113 | func push(c *cli.Context) { 114 | imageName, imageTag := parseImageNameTag(c.Args().First()) 115 | userName, password := parseCredentials(c.String("user")) 116 | 117 | fmt.Printf("Pushing image %v:%v ...\n", imageName, imageTag) 118 | session, err := newRegistrySession(userName, password) 119 | if err != nil { 120 | log.Fatal(err) 121 | } 122 | 123 | err = session.pushRepository(imageName, imageTag, c.String("rootfs")) 124 | if err != nil { 125 | log.Fatal(err) 126 | } 127 | fmt.Printf("Done: https://registry.hub.docker.com/%s/%s\n", userName, imageName) 128 | } 129 | -------------------------------------------------------------------------------- /pull.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "io/ioutil" 6 | "os" 7 | "path" 8 | "strconv" 9 | 10 | "github.com/docker/docker/pkg/archive" 11 | ) 12 | 13 | const ( 14 | MAX_DL_CONCURRENCY = 7 15 | ONE_MB = 1000000 16 | ) 17 | 18 | //krgo pull image -r rootfs 19 | //download a flattened docker image from the V1 registry 20 | func (s *registrySession) pullImage(imageName, imageTag, rootfsDest string) error { 21 | return s.downloadImage(imageName, imageTag, rootfsDest, false) 22 | } 23 | 24 | //krgo pull image -r rootfs -g 25 | //download a docker image from the V1 registry putting each layer in a git branch "on top of each other" 26 | func (s *registrySession) pullRepository(imageName, imageTag, rootfsDest string) error { 27 | return s.downloadImage(imageName, imageTag, rootfsDest, true) 28 | } 29 | 30 | //pulling using V1 registry 31 | func (s *registrySession) downloadImage(imageName, imageTag, rootfsDest string, gitLayering bool) error { 32 | repoData, err := s.GetRepositoryData(imageName) 33 | if err != nil { 34 | return err 35 | } 36 | fmt.Printf("Registry endpoint: %v\n", repoData.Endpoints) 37 | 38 | tagsList, err := s.GetRemoteTags(repoData.Endpoints, imageName, repoData.Tokens) 39 | if err != nil { 40 | return err 41 | } 42 | 43 | imageId := tagsList[imageTag] 44 | fmt.Printf("Image ID: %v\n", imageId) 45 | 46 | //Download image history 47 | var imageHistory []string 48 | for _, ep := range repoData.Endpoints { 49 | imageHistory, err = s.GetRemoteHistory(imageId, ep, repoData.Tokens) 50 | if err == nil { 51 | break 52 | } 53 | } 54 | if err != nil { 55 | return err 56 | } 57 | 58 | err = os.MkdirAll(rootfsDest, 0700) 59 | if err != nil { 60 | return err 61 | } 62 | 63 | var gitRepo *gitRepo 64 | if gitLayering { 65 | if gitRepo, err = newGitRepo(rootfsDest); err != nil { 66 | return err 67 | } 68 | } 69 | 70 | queue := NewQueue(MAX_DL_CONCURRENCY) 71 | fmt.Printf("Pulling %d layers:\n", len(imageHistory)) 72 | 73 | for i := len(imageHistory) - 1; i >= 0; i-- { 74 | layerId := imageHistory[i] 75 | job := NewPullingJob(s, repoData, layerId) 76 | queue.Enqueue(job) 77 | } 78 | <-queue.DoneChan 79 | 80 | fmt.Printf("Downloading layers:\n") 81 | 82 | cpt := 0 83 | 84 | for i := len(imageHistory) - 1; i >= 0; i-- { 85 | 86 | //for each layers 87 | layerID := imageHistory[i] 88 | 89 | if gitLayering { 90 | //create a git branch 91 | if _, err = gitRepo.checkoutB(newBranch(cpt, layerID)); err != nil { 92 | return err 93 | } 94 | } 95 | 96 | //download and untar the layer 97 | job := queue.CompletedJobWithID(layerID).(*PullingJob) 98 | fmt.Printf("\t%s (%.2f MB) ... ", layerID, float64(job.LayerSize)/ONE_MB) 99 | _, err = archive.ApplyLayer(rootfsDest, job.LayerData) 100 | job.LayerData.Close() 101 | if err != nil { 102 | return err 103 | } 104 | 105 | ioutil.WriteFile(path.Join(rootfsDest, "json"), job.LayerInfo, 0644) 106 | if gitLayering { 107 | ioutil.WriteFile(path.Join(rootfsDest, "layersize"), []byte(strconv.Itoa(job.LayerSize)), 0644) 108 | } 109 | 110 | if gitLayering { 111 | if _, err = gitRepo.addAllAndCommit("adding layer " + layerID); err != nil { 112 | return err 113 | } 114 | } 115 | 116 | cpt++ 117 | 118 | fmt.Printf("done\n") 119 | } 120 | return nil 121 | } 122 | -------------------------------------------------------------------------------- /pullV2.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "io/ioutil" 7 | "os" 8 | "strings" 9 | 10 | "github.com/docker/docker/pkg/archive" 11 | "github.com/docker/docker/registry" 12 | ) 13 | 14 | //krgo pull image -r rootfs -v2 15 | //download a flattened docker image from the V2 registry 16 | func (s *registrySession) pullImageV2(imageName, imageTag, rootfsDest string) error { 17 | return s.downloadImageV2(imageName, imageTag, rootfsDest, false) 18 | } 19 | 20 | //krgo pull image -r rootfs -g -v2 21 | //download a docker image from the V1 registry putting each layer in a git branch "on top of each other" 22 | func (s *registrySession) pullRepositoryV2(imageName, imageTag, rootfsDest string) error { 23 | return s.downloadImageV2(imageName, imageTag, rootfsDest, true) 24 | } 25 | 26 | //pulling using V2 registry (much nicer !) 27 | func (s *registrySession) downloadImageV2(imageName, imageTag, rootfsDest string, gitLayering bool) error { 28 | endpoint, err := s.V2RegistryEndpoint(s.indexInfo) 29 | if err != nil { 30 | return err 31 | } 32 | auth, err := s.GetV2Authorization(endpoint, imageName, true) 33 | if err != nil { 34 | return err 35 | } 36 | fmt.Printf("Registry endpoint: %v\n", endpoint) 37 | 38 | rawManifest, err := s.GetV2ImageManifest(endpoint, imageName, imageTag, auth) 39 | if err != nil { 40 | return err 41 | } 42 | 43 | var manifest registry.ManifestData 44 | if err := json.Unmarshal(rawManifest, &manifest); err != nil { 45 | return err 46 | } 47 | 48 | err = os.MkdirAll(rootfsDest, 0700) 49 | if err != nil { 50 | return err 51 | } 52 | 53 | var gitRepo *gitRepo 54 | if gitLayering { 55 | if gitRepo, err = newGitRepo(rootfsDest); err != nil { 56 | return err 57 | } 58 | } 59 | 60 | queue := NewQueue(MAX_DL_CONCURRENCY) 61 | fmt.Printf("Manifest contains %d layers, try to cleanup ...\n", len(manifest.FSLayers)) 62 | cleanupManifest(&manifest) 63 | fmt.Printf("Pulling %d layers:\n", len(manifest.FSLayers)) 64 | 65 | for i := len(manifest.FSLayers) - 1; i >= 0; i-- { 66 | sumStr := manifest.FSLayers[i].BlobSum 67 | job := NewPullingV2Job(s, endpoint, auth, imageName, sumStr) 68 | queue.Enqueue(job) 69 | } 70 | <-queue.DoneChan 71 | 72 | fmt.Printf("Downloading layers:\n") 73 | cpt := 0 74 | for i := len(manifest.FSLayers) - 1; i >= 0; i-- { 75 | sumStr := manifest.FSLayers[i].BlobSum 76 | sumType := strings.Split(sumStr, ":")[0] 77 | checksum := strings.Split(sumStr, ":")[1] 78 | 79 | if gitLayering { 80 | //create a git branch 81 | br := newBranch(cpt, checksum) 82 | if _, err = gitRepo.checkoutB(br); err != nil { 83 | return err 84 | } 85 | //set tarsum info into branch description 86 | if err := gitRepo.describeBranch(br, sumType); err != nil { 87 | return err 88 | } 89 | } 90 | 91 | job := queue.CompletedJobWithID(sumStr).(*PullingV2Job) 92 | fmt.Printf("\t%s (%.2f MB) ... ", checksum, float64(job.LayerSize)/ONE_MB) 93 | _, err = archive.ApplyLayer(rootfsDest, ioutil.NopCloser(job.LayerTarSumReader)) 94 | if err != nil { 95 | return err 96 | } 97 | finalChecksum := job.LayerTarSumReader.Sum(nil) 98 | job.LayerDataReader.Close() 99 | 100 | if gitLayering { 101 | if _, err = gitRepo.addAllAndCommit("adding layer " + checksum); err != nil { 102 | return err 103 | } 104 | } 105 | 106 | verified := strings.EqualFold(finalChecksum, sumStr) 107 | fmt.Printf("done (tarsum verified: %v)\n", verified) 108 | 109 | cpt++ 110 | } 111 | return nil 112 | } 113 | 114 | //Layers are now addressed by content, i.e identified by their tarsum (https://github.com/docker/docker-registry/issues/612) 115 | //v1 registry required to push the layer json, that made a lot of "duplicated layer" 116 | //So images manifests contain duplicated layers (layers with same content and then same tarsum), we can clean them up 117 | func cleanupManifest(manifest *registry.ManifestData) { 118 | found := make(map[string]bool) 119 | cleanFSLayers := []*registry.FSLayer{} 120 | for _, layer := range manifest.FSLayers { 121 | if !found[layer.BlobSum] { 122 | found[layer.BlobSum] = true 123 | cleanFSLayers = append(cleanFSLayers, ®istry.FSLayer{BlobSum: layer.BlobSum}) 124 | } 125 | } 126 | manifest.FSLayers = cleanFSLayers 127 | } 128 | -------------------------------------------------------------------------------- /pullingV2_job.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "io" 6 | "strings" 7 | 8 | "github.com/docker/docker/pkg/tarsum" 9 | "github.com/docker/docker/registry" 10 | ) 11 | 12 | type PullingV2Job struct { 13 | Session *registrySession 14 | Endpoint *registry.Endpoint 15 | Auth *registry.RequestAuthorization 16 | ImageName string 17 | SumStr string 18 | 19 | LayerId string 20 | 21 | LayerDataReader io.ReadCloser 22 | LayerTarSumReader tarsum.TarSum 23 | LayerSize int64 24 | 25 | Err error 26 | } 27 | 28 | func NewPullingV2Job(session *registrySession, endpoint *registry.Endpoint, auth *registry.RequestAuthorization, imageName, sumStr string) *PullingV2Job { 29 | return &PullingV2Job{Session: session, Endpoint: endpoint, Auth: auth, ImageName: imageName, SumStr: sumStr} 30 | } 31 | 32 | func (job *PullingV2Job) Start() { 33 | chunks := strings.SplitN(job.SumStr, ":", 2) 34 | if len(chunks) < 2 { 35 | job.Err = fmt.Errorf("expected 2 parts in the sumStr, got %#v", chunks) 36 | return 37 | } 38 | sumType, checksum := chunks[0], chunks[1] 39 | fmt.Printf("\t%s ...\n", checksum) 40 | job.LayerDataReader, job.LayerSize, job.Err = job.Session.GetV2ImageBlobReader(job.Endpoint, job.ImageName, sumType, checksum, job.Auth) 41 | if job.Err != nil { 42 | return 43 | } 44 | 45 | job.LayerTarSumReader, job.Err = tarsum.NewTarSumForLabel(job.LayerDataReader, true, sumType) 46 | if job.Err != nil { 47 | return 48 | } 49 | 50 | fmt.Printf("\tDone %s\n", checksum) 51 | } 52 | 53 | func (job *PullingV2Job) Error() error { 54 | return job.Err 55 | } 56 | 57 | func (job *PullingV2Job) ID() string { 58 | return job.SumStr 59 | } 60 | -------------------------------------------------------------------------------- /pulling_job.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "io" 6 | 7 | "github.com/docker/docker/registry" 8 | ) 9 | 10 | type PullingJob struct { 11 | Session *registrySession 12 | RepoData *registry.RepositoryData 13 | 14 | LayerId string 15 | 16 | LayerData io.ReadCloser 17 | LayerInfo []byte 18 | LayerSize int 19 | 20 | Err error 21 | } 22 | 23 | func NewPullingJob(session *registrySession, repoData *registry.RepositoryData, layerId string) *PullingJob { 24 | return &PullingJob{Session: session, RepoData: repoData, LayerId: layerId} 25 | } 26 | 27 | func (job *PullingJob) Start() { 28 | fmt.Printf("\t%v\n", job.LayerId) 29 | endpoints := job.RepoData.Endpoints 30 | tokens := job.RepoData.Tokens 31 | 32 | for _, ep := range endpoints { 33 | job.LayerInfo, job.LayerSize, job.Err = job.Session.GetRemoteImageJSON(job.LayerId, ep, tokens) 34 | if job.Err != nil { 35 | continue 36 | } 37 | job.LayerData, job.Err = job.Session.GetRemoteImageLayer(job.LayerId, ep, tokens, int64(job.LayerSize)) 38 | } 39 | 40 | fmt.Printf("\tDone %v\n", job.LayerId) 41 | } 42 | 43 | func (job *PullingJob) Error() error { 44 | return job.Err 45 | } 46 | 47 | func (job *PullingJob) ID() string { 48 | return job.LayerId 49 | } 50 | -------------------------------------------------------------------------------- /push.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "io/ioutil" 6 | "path" 7 | 8 | "github.com/docker/docker/registry" 9 | ) 10 | 11 | func (s *registrySession) pushRepository(imageName, imageTag, rootfs string) error { 12 | if !isGitRepo(rootfs) { 13 | return fmt.Errorf("%v not a git repository", rootfs) 14 | } 15 | gitRepo, _ := newGitRepo(rootfs) 16 | 17 | branches, err := gitRepo.branch() 18 | if err != nil { 19 | return err 20 | } 21 | var imageIds []string = make([]string, len(branches)) 22 | for _, br := range branches { 23 | imageIds[br.number()] = br.imageID() 24 | } 25 | 26 | fmt.Printf("Pushing %d layers:\n", len(imageIds)) 27 | 28 | //Push image index 29 | var imageIndex []*registry.ImgData 30 | for _, id := range imageIds { 31 | imageIndex = append(imageIndex, ®istry.ImgData{ID: id, Tag: imageTag}) 32 | } 33 | repoData, err := s.PushImageJSONIndex(imageName, imageIndex, false, nil) 34 | if err != nil { 35 | return err 36 | } 37 | 38 | ep := repoData.Endpoints[0] 39 | //make sure existing branches are pushed 40 | for i, imageId := range imageIds { 41 | fmt.Printf("\t%v ... ", imageId) 42 | if err := s.LookupRemoteImage(imageId, ep, repoData.Tokens); err == nil { 43 | fmt.Printf("done (already pushed)\n") 44 | } else { 45 | err = s.pushImageLayer(gitRepo, branches[i], imageId, ep, repoData.Tokens) 46 | if err != nil { 47 | if err == registry.ErrAlreadyExists { 48 | fmt.Printf("done (already pushed)\n") 49 | } else { 50 | return err 51 | } 52 | } else { 53 | fmt.Printf("done\n") 54 | } 55 | } 56 | 57 | //push tag 58 | if err := s.PushRegistryTag(imageName, imageId, imageTag, ep, repoData.Tokens); err != nil { 59 | return err 60 | } 61 | } 62 | 63 | //Finalize push 64 | if _, err = s.PushImageJSONIndex(imageName, imageIndex, true, repoData.Endpoints); err != nil { 65 | return err 66 | } 67 | return nil 68 | } 69 | 70 | func (s *registrySession) pushImageLayer(gitRepo *gitRepo, br branch, imgID, ep string, token []string) error { 71 | if _, err := gitRepo.checkout(br); err != nil { 72 | return err 73 | } 74 | 75 | jsonRaw, err := ioutil.ReadFile(path.Join(gitRepo.Path, "json")) 76 | if err != nil { 77 | //if json is not found, this probably means that user pull the image using V2 registry 78 | fmt.Printf("Hint: you can't push images pulled using the -v2 flag yet") 79 | return err 80 | } 81 | 82 | imgData := ®istry.ImgData{ 83 | ID: imgID, 84 | } 85 | 86 | // Send the json 87 | if err := s.PushImageJSONRegistry(imgData, jsonRaw, ep, token); err != nil { 88 | return err 89 | } 90 | 91 | layerData, err := gitRepo.exportChangeSet(br) 92 | if err != nil { 93 | return err 94 | } 95 | defer layerData.Close() 96 | 97 | checksum, checksumPayload, err := s.PushImageLayerRegistry(imgID, layerData, ep, token, jsonRaw) 98 | if err != nil { 99 | return err 100 | } 101 | imgData.Checksum = checksum 102 | imgData.ChecksumPayload = checksumPayload 103 | 104 | return s.PushImageChecksumRegistry(imgData, ep, token) 105 | } 106 | -------------------------------------------------------------------------------- /pushV2.go: -------------------------------------------------------------------------------- 1 | // +build ignore 2 | 3 | package main 4 | 5 | import ( 6 | "bytes" 7 | "encoding/json" 8 | "fmt" 9 | "io" 10 | "io/ioutil" 11 | "strings" 12 | 13 | "github.com/docker/docker/pkg/tarsum" 14 | "github.com/docker/docker/registry" 15 | ) 16 | 17 | /* 18 | Unused code, prepare the ground for full registry v2 when it's production ready 19 | */ 20 | 21 | func generateManifest(gitRepo *gitRepo, imageName, imageTag string) (*registry.ManifestData, error) { 22 | branches, err := gitRepo.branch() 23 | if err != nil { 24 | return nil, err 25 | } 26 | var imageChecksums []string = make([]string, len(branches)) 27 | for _, br := range branches { 28 | checksum := br.imageID() 29 | sumTypeBytes, err := gitRepo.branchDescription(br) 30 | if err != nil { 31 | return nil, err 32 | } 33 | imageChecksums[br.number()] = string(sumTypeBytes) + ":" + checksum 34 | } 35 | 36 | manifest := ®istry.ManifestData{ 37 | Name: imageName, 38 | Architecture: "amd64", //unclean but so far looks ok ... 39 | Tag: imageTag, 40 | SchemaVersion: 1, 41 | FSLayers: make([]*registry.FSLayer, 0, 4), 42 | } 43 | 44 | for i, checksum := range imageChecksums { 45 | if tarsum.VersionLabelForChecksum(checksum) != tarsum.Version1.String() { 46 | //need to calculate the tarsum V1 for each layer ... 47 | layerData, err := gitRepo.exportChangeSet(branches[i]) 48 | if err == ErrNoChange { 49 | continue 50 | } 51 | if err != nil { 52 | return nil, err 53 | } 54 | defer layerData.Close() 55 | 56 | tarSum, err := tarsum.NewTarSum(layerData, true, tarsum.Version1) 57 | if err != nil { 58 | return nil, err 59 | } 60 | if _, err := io.Copy(ioutil.Discard, tarSum); err != nil { 61 | return nil, err 62 | } 63 | 64 | checksum = tarSum.Sum(nil) 65 | } 66 | manifest.FSLayers = append(manifest.FSLayers, ®istry.FSLayer{BlobSum: checksum}) 67 | } 68 | return manifest, nil 69 | } 70 | 71 | func (s *registrySession) pushRepositoryV2(imageName, imageTag, rootfs string) error { 72 | if !isGitRepo(rootfs) { 73 | return fmt.Errorf("%v not a git repository", rootfs) 74 | } 75 | gitRepo, _ := newGitRepo(rootfs) 76 | 77 | endpoint, err := s.V2RegistryEndpoint(s.indexInfo) 78 | if err != nil { 79 | return err 80 | } 81 | auth, err := s.GetV2Authorization(endpoint, imageName, true) 82 | if err != nil { 83 | return err 84 | } 85 | fmt.Printf("Registry endpoint: %v\n", endpoint) 86 | 87 | manifest, err := generateManifest(gitRepo, imageName, imageTag) 88 | if err != nil { 89 | return err 90 | } 91 | 92 | manifestBytes, err := json.MarshalIndent(manifest, "", " ") 93 | if err != nil { 94 | return err 95 | } 96 | 97 | branches, err := gitRepo.branch() 98 | if err != nil { 99 | return err 100 | } 101 | orderedBranches := make([]branch, len(branches)) 102 | for _, br := range branches { 103 | orderedBranches[br.number()] = br 104 | } 105 | 106 | for i := len(manifest.FSLayers) - 1; i >= 0; i-- { 107 | sumStr := manifest.FSLayers[i].BlobSum 108 | sumParts := strings.SplitN(sumStr, ":", 2) 109 | if len(sumParts) < 2 { 110 | return fmt.Errorf("Invalid checksum: %s", sumStr) 111 | } 112 | manifestSum := sumParts[1] 113 | 114 | // Call mount blob 115 | exists, err := s.HeadV2ImageBlob(endpoint, imageName, sumParts[0], manifestSum, auth) 116 | if err != nil { 117 | return err 118 | } 119 | 120 | if !exists { 121 | fmt.Println("Image doesn't exist") 122 | layerData, err := gitRepo.exportChangeSet(orderedBranches[i]) 123 | if err != nil { 124 | return err 125 | } 126 | defer layerData.Close() 127 | fmt.Println(endpoint) 128 | err = s.PutV2ImageBlob(endpoint, imageName, sumParts[0], sumParts[1], layerData, auth) 129 | if err != nil { 130 | return err 131 | } 132 | 133 | //todo push manifest 134 | } else { 135 | fmt.Println("Image already exists") 136 | } 137 | 138 | // push the manifest 139 | if err := s.PutV2ImageManifest(endpoint, imageName, imageTag, bytes.NewReader([]byte(manifestBytes)), auth); err != nil { 140 | return err 141 | } 142 | } 143 | 144 | return nil 145 | } 146 | -------------------------------------------------------------------------------- /queue.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "log" 5 | "sync" 6 | ) 7 | 8 | type Job interface { 9 | Start() 10 | Error() error 11 | ID() string 12 | } 13 | 14 | type Queue struct { 15 | Concurrency int 16 | NbRunningJob int 17 | WaitingJobs []Job 18 | Lock *sync.Mutex 19 | DoneChan chan bool 20 | PerJobChan chan string 21 | CompletedJobs map[string]Job 22 | } 23 | 24 | func NewQueue(concurrency int) *Queue { 25 | doneChan := make(chan bool) 26 | perJobChan := make(chan string, 10000) 27 | return &Queue{Concurrency: concurrency, Lock: &sync.Mutex{}, DoneChan: doneChan, PerJobChan: perJobChan, CompletedJobs: make(map[string]Job)} 28 | } 29 | 30 | func (queue *Queue) Enqueue(job Job) { 31 | queue.Lock.Lock() 32 | defer queue.Lock.Unlock() 33 | 34 | if !queue.canLaunchJob() { 35 | //concurrency limit reached, make the job wait 36 | queue.WaitingJobs = append(queue.WaitingJobs, job) 37 | return 38 | } 39 | queue.startJob(job) 40 | } 41 | 42 | func (queue *Queue) startJob(job Job) { 43 | queue.NbRunningJob++ 44 | go func() { 45 | //start the job 46 | job.Start() 47 | queue.dequeue(job) 48 | }() 49 | } 50 | 51 | func (queue *Queue) dequeue(job Job) { 52 | queue.Lock.Lock() 53 | defer queue.Lock.Unlock() 54 | 55 | if job.Error() != nil { 56 | log.Fatal(job.Error()) 57 | } 58 | queue.CompletedJobs[job.ID()] = job 59 | queue.PerJobChan <- job.ID() 60 | 61 | queue.NbRunningJob-- 62 | if queue.canLaunchJob() && len(queue.WaitingJobs) > 0 { 63 | queue.startJob(queue.WaitingJobs[0]) 64 | queue.WaitingJobs = append(queue.WaitingJobs[:0], queue.WaitingJobs[1:]...) 65 | } 66 | if len(queue.WaitingJobs) == 0 && queue.NbRunningJob == 0 { 67 | queue.DoneChan <- true 68 | } 69 | } 70 | 71 | func (queue *Queue) canLaunchJob() bool { 72 | return queue.NbRunningJob < queue.Concurrency 73 | } 74 | 75 | func (queue *Queue) CompletedJobWithID(jobId string) Job { 76 | return queue.CompletedJobs[jobId] 77 | } 78 | -------------------------------------------------------------------------------- /registry_session.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | 6 | "github.com/docker/docker/registry" 7 | ) 8 | 9 | type registrySession struct { 10 | registry.Session 11 | indexInfo *registry.IndexInfo 12 | } 13 | 14 | //return a registrySession associated with the repository contained in imageName 15 | func newRegistrySession(userName, password string) (*registrySession, error) { 16 | //IndexInfo 17 | indexInfo := ®istry.IndexInfo{ 18 | Name: registry.INDEXNAME, 19 | Mirrors: []string{}, 20 | Secure: true, 21 | Official: true, 22 | } 23 | 24 | endpoint, err := registry.NewEndpoint(indexInfo) 25 | if err != nil { 26 | return nil, err 27 | } 28 | fmt.Printf("Index endpoint: %s\n", endpoint) 29 | 30 | authConfig := ®istry.AuthConfig{Username: userName, Password: password} 31 | 32 | var metaHeaders map[string][]string 33 | 34 | session, err := registry.NewSession(authConfig, registry.HTTPRequestFactory(metaHeaders), endpoint, true) 35 | if err != nil { 36 | return nil, fmt.Errorf("failed to create registry session: %v", err) 37 | } 38 | 39 | return ®istrySession{*session, indexInfo}, nil 40 | } 41 | -------------------------------------------------------------------------------- /sample_configs/container.json: -------------------------------------------------------------------------------- 1 | { 2 | "capabilities": [ 3 | "CHOWN", 4 | "DAC_OVERRIDE", 5 | "FOWNER", 6 | "MKNOD", 7 | "NET_RAW", 8 | "SETGID", 9 | "SETUID", 10 | "SETFCAP", 11 | "SETPCAP", 12 | "NET_BIND_SERVICE", 13 | "SYS_CHROOT", 14 | "KILL" 15 | ], 16 | "cgroups": { 17 | "allowed_devices": [ 18 | { 19 | "cgroup_permissions": "m", 20 | "major_number": -1, 21 | "minor_number": -1, 22 | "type": 99 23 | }, 24 | { 25 | "cgroup_permissions": "m", 26 | "major_number": -1, 27 | "minor_number": -1, 28 | "type": 98 29 | }, 30 | { 31 | "cgroup_permissions": "rwm", 32 | "major_number": 5, 33 | "minor_number": 1, 34 | "path": "/dev/console", 35 | "type": 99 36 | }, 37 | { 38 | "cgroup_permissions": "rwm", 39 | "major_number": 4, 40 | "path": "/dev/tty0", 41 | "type": 99 42 | }, 43 | { 44 | "cgroup_permissions": "rwm", 45 | "major_number": 4, 46 | "minor_number": 1, 47 | "path": "/dev/tty1", 48 | "type": 99 49 | }, 50 | { 51 | "cgroup_permissions": "rwm", 52 | "major_number": 136, 53 | "minor_number": -1, 54 | "type": 99 55 | }, 56 | { 57 | "cgroup_permissions": "rwm", 58 | "major_number": 5, 59 | "minor_number": 2, 60 | "type": 99 61 | }, 62 | { 63 | "cgroup_permissions": "rwm", 64 | "major_number": 10, 65 | "minor_number": 200, 66 | "type": 99 67 | }, 68 | { 69 | "cgroup_permissions": "rwm", 70 | "file_mode": 438, 71 | "major_number": 1, 72 | "minor_number": 3, 73 | "path": "/dev/null", 74 | "type": 99 75 | }, 76 | { 77 | "cgroup_permissions": "rwm", 78 | "file_mode": 438, 79 | "major_number": 1, 80 | "minor_number": 5, 81 | "path": "/dev/zero", 82 | "type": 99 83 | }, 84 | { 85 | "cgroup_permissions": "rwm", 86 | "file_mode": 438, 87 | "major_number": 1, 88 | "minor_number": 7, 89 | "path": "/dev/full", 90 | "type": 99 91 | }, 92 | { 93 | "cgroup_permissions": "rwm", 94 | "file_mode": 438, 95 | "major_number": 5, 96 | "path": "/dev/tty", 97 | "type": 99 98 | }, 99 | { 100 | "cgroup_permissions": "rwm", 101 | "file_mode": 438, 102 | "major_number": 1, 103 | "minor_number": 9, 104 | "path": "/dev/urandom", 105 | "type": 99 106 | }, 107 | { 108 | "cgroup_permissions": "rwm", 109 | "file_mode": 438, 110 | "major_number": 1, 111 | "minor_number": 8, 112 | "path": "/dev/random", 113 | "type": 99 114 | } 115 | ], 116 | "name": "docker-koye", 117 | "parent": "docker" 118 | }, 119 | "restrict_sys": true, 120 | "mount_config": { 121 | "device_nodes": [ 122 | { 123 | "cgroup_permissions": "rwm", 124 | "file_mode": 438, 125 | "major_number": 1, 126 | "minor_number": 3, 127 | "path": "/dev/null", 128 | "type": 99 129 | }, 130 | { 131 | "cgroup_permissions": "rwm", 132 | "file_mode": 438, 133 | "major_number": 1, 134 | "minor_number": 5, 135 | "path": "/dev/zero", 136 | "type": 99 137 | }, 138 | { 139 | "cgroup_permissions": "rwm", 140 | "file_mode": 438, 141 | "major_number": 1, 142 | "minor_number": 7, 143 | "path": "/dev/full", 144 | "type": 99 145 | }, 146 | { 147 | "cgroup_permissions": "rwm", 148 | "file_mode": 438, 149 | "major_number": 5, 150 | "path": "/dev/tty", 151 | "type": 99 152 | }, 153 | { 154 | "cgroup_permissions": "rwm", 155 | "file_mode": 438, 156 | "major_number": 1, 157 | "minor_number": 9, 158 | "path": "/dev/urandom", 159 | "type": 99 160 | }, 161 | { 162 | "cgroup_permissions": "rwm", 163 | "file_mode": 438, 164 | "major_number": 1, 165 | "minor_number": 8, 166 | "path": "/dev/random", 167 | "type": 99 168 | } 169 | ], 170 | "mounts": [ 171 | { 172 | "type": "tmpfs", 173 | "destination": "/tmp" 174 | } 175 | ] 176 | }, 177 | "environment": [ 178 | "HOME=/", 179 | "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", 180 | "HOSTNAME=koye", 181 | "TERM=xterm" 182 | ], 183 | "hostname": "krgo-demo", 184 | "namespaces": [ 185 | {"type": "NEWIPC"}, 186 | {"type": "NEWNS"}, 187 | {"type": "NEWPID"}, 188 | {"type": "NEWUTS"} 189 | ], 190 | "tty": true, 191 | "user": "root" 192 | } 193 | -------------------------------------------------------------------------------- /sample_configs/lxc-config: -------------------------------------------------------------------------------- 1 | 2 | # Container specific configuration 3 | #CHANGE ME 4 | lxc.rootfs = /root/ubuntu 5 | lxc.utsname = krgo-demo 6 | lxc.arch = amd64 7 | 8 | # Network configuration 9 | lxc.network.type = veth 10 | 11 | # Default pivot location 12 | lxc.pivotdir = lxc_putold 13 | 14 | # Default mount entries 15 | lxc.mount.entry = proc proc proc nodev,noexec,nosuid 0 0 16 | lxc.mount.entry = sysfs sys sysfs defaults 0 0 17 | lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none bind,optional 0 0 18 | lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0 19 | lxc.mount.entry = /sys/kernel/security sys/kernel/security none bind,optional 0 0 20 | lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0 21 | 22 | # Default console settings 23 | lxc.devttydir = lxc 24 | lxc.tty = 4 25 | lxc.pts = 1024 26 | 27 | # Default capabilities 28 | lxc.cap.drop = sys_module mac_admin mac_override sys_time 29 | 30 | # When using LXC with apparmor, the container will be confined by default. 31 | # If you wish for it to instead run unconfined, copy the following line 32 | # (uncommented) to the container's configuration file. 33 | #lxc.aa_profile = unconfined 34 | 35 | # To support container nesting on an Ubuntu host while retaining most of 36 | # apparmor's added security, use the following two lines instead. 37 | #lxc.aa_profile = lxc-container-default-with-nesting 38 | #lxc.hook.mount = /usr/share/lxc/hooks/mountcgroups 39 | 40 | # Uncomment the following line to autodetect squid-deb-proxy configuration on the 41 | # host and forward it to the guest at start time. 42 | #lxc.hook.pre-start = /usr/share/lxc/hooks/squid-deb-proxy-client 43 | 44 | # If you wish to allow mounting block filesystems, then use the following 45 | # line instead, and make sure to grant access to the block device and/or loop 46 | # devices below in lxc.cgroup.devices.allow. 47 | #lxc.aa_profile = lxc-container-default-with-mounting 48 | 49 | # Default cgroup limits 50 | lxc.cgroup.devices.deny = a 51 | ## Allow any mknod (but not using the node) 52 | lxc.cgroup.devices.allow = c *:* m 53 | lxc.cgroup.devices.allow = b *:* m 54 | ## /dev/null and zero 55 | lxc.cgroup.devices.allow = c 1:3 rwm 56 | lxc.cgroup.devices.allow = c 1:5 rwm 57 | ## consoles 58 | lxc.cgroup.devices.allow = c 5:0 rwm 59 | lxc.cgroup.devices.allow = c 5:1 rwm 60 | ## /dev/{,u}random 61 | lxc.cgroup.devices.allow = c 1:8 rwm 62 | lxc.cgroup.devices.allow = c 1:9 rwm 63 | ## /dev/pts/* 64 | lxc.cgroup.devices.allow = c 5:2 rwm 65 | lxc.cgroup.devices.allow = c 136:* rwm 66 | ## rtc 67 | lxc.cgroup.devices.allow = c 254:0 rm 68 | ## fuse 69 | lxc.cgroup.devices.allow = c 10:229 rwm 70 | ## tun 71 | lxc.cgroup.devices.allow = c 10:200 rwm 72 | ## full 73 | lxc.cgroup.devices.allow = c 1:7 rwm 74 | ## hpet 75 | lxc.cgroup.devices.allow = c 10:228 rwm 76 | ## kvm 77 | lxc.cgroup.devices.allow = c 10:232 rwm 78 | ## To use loop devices, copy the following line to the container's 79 | ## configuration file (uncommented). 80 | #lxc.cgroup.devices.allow = b 7:* rwm 81 | 82 | # Blacklist some syscalls which are not safe in privileged 83 | # containers 84 | lxc.seccomp = /usr/share/lxc/config/common.seccomp 85 | -------------------------------------------------------------------------------- /setup.sh: -------------------------------------------------------------------------------- 1 | ##!/bin/ash 2 | 3 | set -e 4 | 5 | sudo apt-get update -qq 6 | 7 | echo "Installing base stack" 8 | 9 | packagelist=( 10 | cgroup-lite #this is important !! 11 | curl 12 | build-essential 13 | bison 14 | openssl 15 | libreadline6 16 | libreadline-dev 17 | git-core 18 | zlib1g 19 | zlib1g-dev 20 | libssl-dev 21 | libyaml-dev 22 | libxml2-dev 23 | libxslt-dev 24 | autoconf 25 | ssl-cert 26 | libcurl4-openssl-dev 27 | lxc 28 | zsh 29 | git 30 | python-software-properties 31 | golang 32 | ) 33 | 34 | sudo DEBIAN_FRONTEND=noninteractive apt-get install -y ${packagelist[@]} 35 | 36 | curl -L http://install.ohmyz.sh | sh 37 | 38 | echo "GOPATH=~/code/go" >> ~/.bashrc 39 | -------------------------------------------------------------------------------- /utils.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "os" 5 | "strings" 6 | ) 7 | 8 | //credentials format: : 9 | func parseCredentials(credentials string) (string, string) { 10 | comps := strings.SplitN(credentials, ":", 2) 11 | if len(comps) != 2 { 12 | return "", "" 13 | } 14 | return comps[0], comps[1] 15 | } 16 | 17 | //image format: /:. tag defaults to latest, repository defaults to library 18 | func parseImageNameTag(imageNameTag string) (imageName, imageTag string) { 19 | if strings.Contains(imageNameTag, ":") { 20 | imageName = strings.SplitN(imageNameTag, ":", 2)[0] 21 | imageTag = strings.SplitN(imageNameTag, ":", 2)[1] 22 | } else { 23 | imageName = imageNameTag 24 | imageTag = "latest" 25 | } 26 | 27 | if !strings.Contains(imageName, "/") { 28 | imageName = "library/" + imageName 29 | } 30 | return 31 | } 32 | 33 | //return whether imageName is an official image or not 34 | func isOfficialImage(imageName string) bool { 35 | return strings.HasPrefix(imageName, "library/") 36 | } 37 | 38 | //fileExists reports whether the named file or directory exists 39 | func fileExists(path string) bool { 40 | if _, err := os.Stat(path); err != nil { 41 | if os.IsNotExist(err) { 42 | return false 43 | } 44 | } 45 | return true 46 | } 47 | -------------------------------------------------------------------------------- /vendor.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -e 3 | 4 | cd "$(dirname "$BASH_SOURCE")" 5 | 6 | # Downloads dependencies into vendor/ directory 7 | mkdir -p vendor 8 | cd vendor 9 | 10 | clone() { 11 | vcs=$1 12 | pkg=$2 13 | rev=$3 14 | 15 | pkg_url=https://$pkg 16 | target_dir=src/$pkg 17 | 18 | echo "$pkg @ $rev: " 19 | 20 | if [ -d $target_dir ]; then 21 | echo "rm old, $pkg" 22 | rm -fr $target_dir 23 | fi 24 | 25 | echo "clone, $pkg" 26 | case $vcs in 27 | git) 28 | git clone --quiet --no-checkout $pkg_url $target_dir 29 | ( cd $target_dir && git reset --quiet --hard $rev ) 30 | ;; 31 | hg) 32 | hg clone --quiet --updaterev $rev $pkg_url $target_dir 33 | ;; 34 | esac 35 | 36 | echo "rm VCS, $vcs" 37 | ( cd $target_dir && rm -rf .{git,hg} ) 38 | 39 | echo "done" 40 | } 41 | 42 | #docker dependencies 43 | clone git github.com/docker/docker v1.5.0 44 | 45 | clone git github.com/codegangsta/cli v1.2.0 46 | 47 | echo "if not using make, don't forget to add vendor folder to your GOPATH (export GOPATH=\$GOPATH:\`pwd\`/vendor)" 48 | --------------------------------------------------------------------------------