├── LICENSE ├── README.md └── main.go /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2018, Péter Szilágyi 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | * Redistributions of source code must retain the above copyright 7 | notice, this list of conditions and the following disclaimer. 8 | * Redistributions in binary form must reproduce the above copyright 9 | notice, this list of conditions and the following disclaimer in the 10 | documentation and/or other materials provided with the distribution. 11 | * Neither the name of the nor the 12 | names of its contributors may be used to endorse or promote products 13 | derived from this software without specific prior written permission. 14 | 15 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 16 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 17 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 18 | DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY 19 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 20 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 21 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 22 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 23 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 24 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 25 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | The [`gx`](https://github.com/whyrusleeping/gx) project is a powerful package manager built on top of [IPFS](https://ipfs.io/). It is highly performant, fully decentralized and incorruptible. The goal of `ungx` is to get rid of it 😇 2 | 3 | ## Why?! 4 | 5 | As powerful as `gx` is, it doesn't yet play nice with non-gx workflows: 6 | 7 | * You can't `go get` a `gx` based project, as the dependencies are hosted via IPFS, so special tooling support is needed to interpret the import paths and also to retrieve their content. This also means no other Go project can depend on your `gx` based package. 8 | * The import paths in `gx` based packages lose all canonical URL context and get replaced with random strings of letters and numbers (e.g. `gx/ipfs/QmPXvegq26x982cQjSfbTvSzZXn7GiaMwhhVPHkeTEhrPT/sys`). [These will make your head spin](https://github.com/ipfs/go-ipfs/blob/master/core/core.go#L42). It will also make `goimports`' head spin when it realises there are 15 versions of `sys`under different hashes. 9 | * Finally, if you vendor in a `gx` based package with all its juicy hash-based import paths and use these in your own code too, you'll be very unhappy at the first occasion when you need to update dependencies and realize you have to update **all** the import statements in your code with new hashes, that you won't even know yourself. 10 | 11 | TL;DR `gx` is an amazing project, but until the Go ecosystem builds a tool that can bridge the two worlds, I need a way to use `gx` projects without forcing me to switch over to `gx` myself. 12 | 13 | ## How? 14 | 15 | With the *why* out of the way, lets see *how* `ungx` helps make our lives easier. The goal of `ungx` is to take a `gx` based package/repository, resolve all the dependencies in it via `gx` and then rewrite/vendor all the dependencies into legacy Go style. 16 | 17 | It's operation is fairly simplistic: 18 | 19 | * Run `gx install --local` to fetch all `gx` dependencies and vendor them in with hashes. 20 | * Find all `gx` dependencies that do not have multiple versions (we can't rewrite clashes). 21 | * Vendor all non-clashing plain Go dependencies under `vendor` with their canonical path. 22 | * Embed all non-clashing `gx` dependencies under `gxdeps` with their canonical path. 23 | * Rewrite all import statements for all non-clashing dependencies to the new paths. 24 | * Optionally rewrite the root import path to a custom one specified via `--fork`. 25 | 26 | **Note, it will overwrite your original checked out repo!** 27 | 28 | ## Example 29 | 30 | If we'd want to make a nice `go-ipfs` fork that doesn't contain strange import paths and plays nice with existing Go toolings, we could use `ungx` for it: 31 | 32 | First up we need the original `go-ipfs` repo. 33 | 34 | ``` 35 | $ go get -u -d github.com/ipfs/go-ipfs 36 | ``` 37 | 38 | Then we need `gx` for dependency retrieval and `ungx` for rewrites: 39 | 40 | ``` 41 | $ go get -u github.com/whyrusleeping/gx 42 | $ go get -u github.com/karalabe/ungx 43 | ``` 44 | 45 | Finally we can let `ungx` do its magic: 46 | 47 | ``` 48 | $ cd $GOPATH/github.com/ipfs/go-ipfs 49 | $ ungx 50 | 51 | 2018/04/13 17:27:02 Vendoring in gx dependencies 52 | [done] [fetch] go-libp2p-secio QmT8TkDNBDyBsnZ4JJ2ecHU7qN184jkw1tY8y4chFfeWsy 835ms 53 | [done] [fetch] go-log QmRb5jh8z2E8hMGN2tkvs1yHynUanqnZ3UeKwgN1i9P1F8 834ms 54 | [done] [fetch] goleveldb QmbBhyDKsY4mbY6xsKt3qu9Y7FPvMJ6qbD8AMjYYvPRw1g 507ms 55 | [...] 56 | [done] [install] opentracing-go QmWLWmRVSiagqP15jczsGME1qpob6HDbtbHAY2he9W5iUo 0s 57 | [done] [install] go-fs-lock QmPdqSMmiwtQCBC515gFtMW2mP14HsfgnyQ2k5xPQVxMge 8ms 58 | [done] [install] go-bitfield QmTbBs3Y3u5F69XNJzdnnc6SP5GKgcXxCDzx6w8m6piVRT 4ms 59 | 60 | 2018/04/13 17:28:36 Rewriting gx/ipfs/QmNeSwALyTCrgtCTsPiF7tcDN6uLtdi8qCMtFm7nct1nm1/httprouter to github.com/julienschmidt/httprouter 61 | 2018/04/13 17:28:37 Rewriting gx/ipfs/QmQFhPsJCp82az4SXbziP9QcVSqggEELnV9wGZqMR1EfMB/go-smux-spdystream to github.com/whyrusleeping/go-smux-spdystream 62 | 2018/04/13 17:28:37 Rewriting gx/ipfs/QmT8TkDNBDyBsnZ4JJ2ecHU7qN184jkw1tY8y4chFfeWsy/go-libp2p-secio to github.com/libp2p/go-libp2p-secio 63 | [...] 64 | 2018/04/13 17:28:59 Rewriting gx/ipfs/QmTEmsyNnckEq8rEfALfdhLHjrEHGoSGFDrAYReuetn7MC/go-net to golang.org/x/go-net 65 | 2018/04/13 17:28:59 Rewriting gx/ipfs/QmVYxfoJQiZijTgPNHCHgHELvQpbsJNTg6Crmc3dQkj3yy/golang-lru to github.com/hashicorp/golang-lru 66 | 2018/04/13 17:28:59 Rewriting gx/ipfs/QmZyZDi491cCNTLfAhwcaDii2Kg4pwKRkhqQzURGDvY6ua/go-multihash to github.com/multiformats/go-multihash 67 | ``` 68 | 69 | And voila, we have a fork of `go-ipfs` that does not contain cryptic hash import paths and is a joy to work with. If you want to update your fork to a new version, repeat the above procedure in a pristine GOPATH and overwrite your old fork with the newly generated one. 70 | 71 | *Note, if you want to publish your dependency publicly, you'll need to rewrite all the package's internal imports to your fork paths (e.g. `ungx --fork=github.com/myipfs/go-ipfs`). and manually move the repository contents to `$GOPATH/github.com/myipfs/go-ipfs`.* 72 | 73 | ## Disclaimer 74 | 75 | This tool is a toy. I built it for my personal hobby projects. You're welcome to use it, but don't expect support, stability or even responses from me 😋 76 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | // Copyright 2018 Péter Szilágyi. All rights reserved. 2 | // Use of this source code is governed by a BSD-style 3 | // license that can be found in the LICENSE file. 4 | 5 | package main 6 | 7 | import ( 8 | "bytes" 9 | "encoding/json" 10 | "flag" 11 | "fmt" 12 | "io/ioutil" 13 | "log" 14 | "net/http" 15 | "os" 16 | "os/exec" 17 | "path/filepath" 18 | "regexp" 19 | "strings" 20 | ) 21 | 22 | // fork defines an optional import path to rewrite the main package to. It's main 23 | // use is when a gx package is forked into a different repo and avoids having to 24 | // do an extra rewrite after copying the code. 25 | var fork = flag.String("fork", "", "Optional root import path to rewrite to") 26 | 27 | // embed defines an optional list of import paths which should be embedded into 28 | // the sources directly instead of vendoring. This can be used to pin an external 29 | // dependency who's API is broken. 30 | var embed = flag.String("embed", "", "Comma-separated packages to force embedding") 31 | 32 | func main() { 33 | flag.Parse() 34 | 35 | embeds := make(map[string]bool) 36 | for _, embed := range strings.Split(*embed, ",") { 37 | embeds[embed] = true 38 | } 39 | // Create a temporary Go workspace to download canonical packages into 40 | workspace, err := ioutil.TempDir("", "") 41 | if err != nil { 42 | log.Fatalf("Failed to create temporary workspace: %v", err) 43 | } 44 | defer os.RemoveAll(workspace) 45 | 46 | // Resolve the current package's import path 47 | root, err := exec.Command("go", "list").CombinedOutput() 48 | if err != nil { 49 | log.Fatalf("Failed to resolve package import path: %v", err) 50 | } 51 | root = bytes.TrimSpace(root) 52 | 53 | // Retrieve all the gx dependencies into the local vendor folder 54 | deps := exec.Command("gx", "install", "--local") 55 | deps.Stdout = os.Stdout 56 | deps.Stderr = os.Stderr 57 | 58 | log.Printf("Vendoring in gx dependencies") 59 | if err := deps.Run(); err != nil { 60 | log.Fatalf("Failed to vendor dependencies: %v", err) 61 | } 62 | // Find all the unique import paths (duplicates remain unmodified) 63 | gxpkgs := filepath.Join("vendor", "gx", "ipfs") 64 | 65 | hashes, err := ioutil.ReadDir(gxpkgs) 66 | if err != nil { 67 | log.Fatalf("Failed to list vendored packages: %v", err) 68 | } 69 | versions := make(map[string]int) 70 | mappings := make(map[string]string) 71 | 72 | for _, hash := range hashes { 73 | // Retrieve the package spec from the dependency 74 | dirs, err := ioutil.ReadDir(filepath.Join(gxpkgs, hash.Name())) 75 | if err != nil { 76 | log.Fatalf("Failed to list package contents: %v", err) 77 | } 78 | blob, err := ioutil.ReadFile(filepath.Join(gxpkgs, hash.Name(), dirs[0].Name(), "package.json")) 79 | if err != nil { 80 | log.Fatalf("Failed to read package definition: %v", err) 81 | } 82 | // Extract the canonical package import path 83 | var pkg struct { 84 | Gx struct { 85 | Path string `json:"dvcsimport"` 86 | } `json:"gx"` 87 | } 88 | if err := json.Unmarshal(blob, &pkg); err != nil { 89 | log.Fatalf("Failed to parse package definition: %v", err) 90 | } 91 | // Save the hash to path mapping and clash count 92 | mappings[hash.Name()] = pkg.Gx.Path 93 | versions[pkg.Gx.Path]++ 94 | } 95 | // Move the package from hash to canonical path 96 | rewrite := make(map[string]string) 97 | 98 | log.Printf("Converting gx dependencies to canonical paths") 99 | for hash, path := range mappings { 100 | // Clashing dependencies cannot be rewritten, so they need to be embedded 101 | if versions[path] > 1 { 102 | if err := os.MkdirAll(filepath.Join("gxlibs", "ipfs"), 0700); err != nil { 103 | log.Fatalf("Failed to create canonical embed path: %v", err) 104 | } 105 | log.Printf("Embedding gx/ipfs/%s to gxlibs/ipfs/%s", hash, hash) 106 | if err := os.Rename(filepath.Join(gxpkgs, hash), filepath.Join("gxlibs", "ipfs", hash)); err != nil { 107 | log.Fatalf("Failed to move embedded package: %v", err) 108 | } 109 | rewrite["gx/ipfs/"+hash] = string(root) + "/gxlibs/ipfs/" + hash 110 | 111 | continue 112 | } 113 | // Any gx-based dependency should be embedded directly to allow library reuse 114 | if embeds[path] || shouldEmbed(workspace, path) { 115 | if err := os.MkdirAll(filepath.Join("gxlibs", filepath.Dir(path)), 0700); err != nil { 116 | log.Fatalf("Failed to create canonical embed path: %v", err) 117 | } 118 | dirs, err := ioutil.ReadDir(filepath.Join(gxpkgs, hash)) 119 | if err != nil { 120 | log.Fatalf("Failed to list package contents: %v", err) 121 | } 122 | for _, dir := range dirs { 123 | log.Printf("Embedding gx/ipfs/%s/%s to gxlibs/%s", hash, dir.Name(), path) 124 | if err := os.Rename(filepath.Join(gxpkgs, hash, dir.Name()), filepath.Join("gxlibs", path)); err != nil { 125 | log.Fatalf("Failed to move embedded package: %v", err) 126 | } 127 | rewrite["gx/ipfs/"+hash+"/"+dir.Name()] = string(root) + "/gxlibs/" + path 128 | rewrite[path] = string(root) + "/gxlibs/" + path 129 | } 130 | } else { 131 | // Non-clashing plain Go dependencies can be vendored in 132 | if err := os.MkdirAll(filepath.Join("vendor", filepath.Dir(path)), 0700); err != nil { 133 | log.Fatalf("Failed to create canonical vendor path: %v", err) 134 | } 135 | dirs, err := ioutil.ReadDir(filepath.Join(gxpkgs, hash)) 136 | if err != nil { 137 | log.Fatalf("Failed to list package contents: %v", err) 138 | } 139 | for _, dir := range dirs { 140 | log.Printf("Vendoring gx/ipfs/%s/%s to vendor/%s", hash, dir.Name(), path) 141 | if err := os.Rename(filepath.Join(gxpkgs, hash, dir.Name()), filepath.Join("vendor", path)); err != nil { 142 | log.Fatalf("Failed to move vendored package: %v", err) 143 | } 144 | rewrite["gx/ipfs/"+hash+"/"+dir.Name()] = path 145 | } 146 | } 147 | // Delete the empty hash dependency path 148 | if err := os.Remove(filepath.Join(gxpkgs, hash)); err != nil { 149 | log.Fatalf("Failed to remove gx leftover: %v", err) 150 | } 151 | } 152 | // Rewrite packages to their canonical paths 153 | log.Printf("Rewriting import statements to canonical paths") 154 | restrict := regexp.MustCompile(`// import ".*"`) 155 | 156 | if err := filepath.Walk(".", func(fp string, fi os.FileInfo, err error) error { 157 | // Abort if any error occurred, descend into directories 158 | if err != nil { 159 | return err 160 | } 161 | if fi.IsDir() { 162 | return nil 163 | } 164 | // Replace the relevant import path in all Go files 165 | if strings.HasSuffix(fi.Name(), ".go") { 166 | oldblob, err := ioutil.ReadFile(fp) 167 | if err != nil { 168 | return err 169 | } 170 | newblob := oldblob 171 | for gxpath, gopath := range rewrite { 172 | newblob = bytes.Replace(newblob, []byte("\""+gxpath+"/"), []byte("\""+gopath+"/"), -1) 173 | newblob = bytes.Replace(newblob, []byte("\""+gxpath+"\""), []byte("\""+gopath+"\""), -1) 174 | } 175 | if *fork != "" { 176 | newblob = bytes.Replace(newblob, []byte("\""+string(root)+"/"), []byte("\""+*fork+"/"), -1) 177 | newblob = bytes.Replace(newblob, []byte("\""+string(root)+"\""), []byte("\""+*fork+"\""), -1) 178 | } 179 | newblob = restrict.ReplaceAll(newblob, []byte{}) 180 | if !bytes.Equal(oldblob, newblob) { 181 | if err = ioutil.WriteFile(fp, newblob, 0); err != nil { 182 | return err 183 | } 184 | } 185 | } 186 | return nil 187 | }); err != nil { 188 | log.Fatalf("Failed to rewrite import paths: %v", err) 189 | } 190 | } 191 | 192 | // shouldEmbed returns whether a package identified by its import path should be 193 | // embedded directly into a ungx-ed package or whether vendoring is enough. The 194 | // deciding factor is whether the package's canonical version is gx based or not, 195 | // since we can't vendor gx packages. 196 | func shouldEmbed(gopath string, path string) bool { 197 | log.Printf("Deciding whether to vendor or embed %s", path) 198 | 199 | // If the import path points to GitHub, we can cheat and directly decide 200 | if strings.HasPrefix(path, "github.com/") { 201 | // Try to retrieve the gx package spec, embed on hard failure 202 | res, err := http.Get(fmt.Sprintf("https://%s/master/package.json", strings.Replace(path, "github.com", "raw.githubusercontent.com", 1))) 203 | if err != nil { 204 | return true 205 | } 206 | defer res.Body.Close() 207 | 208 | // If the file exists, assume its a gx based project, otherwise vendor 209 | return res.StatusCode == http.StatusOK 210 | } 211 | // Non-github package or something failed, we need to download the canonical code 212 | get := exec.Command("go", "get", "-d", path+"/...") 213 | get.Stdout = os.Stdout 214 | get.Stderr = os.Stderr 215 | get.Env = append(os.Environ(), "GOPATH="+gopath) 216 | 217 | if err := get.Run(); err == nil { 218 | if _, err := os.Stat(filepath.Join(gopath, "src", path, "package.json")); err != nil { 219 | return false 220 | } 221 | } 222 | return true 223 | } 224 | --------------------------------------------------------------------------------