├── .gitignore ├── LICENSE ├── README.md ├── main.go └── reporoot.go /.gitignore: -------------------------------------------------------------------------------- 1 | vendetta 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2016 David Wragg. All rights reserved. 2 | 3 | Redistribution and use in source and binary forms, with or without 4 | modification, are permitted provided that the following conditions are 5 | met: 6 | 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above 10 | copyright notice, this list of conditions and the following disclaimer 11 | in the documentation and/or other materials provided with the 12 | distribution. 13 | 14 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 15 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 16 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 17 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 18 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 19 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 20 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 21 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 22 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 23 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 24 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 25 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Vendetta 2 | 3 | The go dependency management tool for people who don't like go 4 | dependency management tools. 5 | 6 | ## Introduction 7 | 8 | Vendetta is a minimal tool for managing the dependencies in the 9 | `vendor` directory of a go project. Go supports such directories 10 | [from version 1.5](https://golang.org/s/go15vendor), but doesn't 11 | provide a way to manage their contents. Vendetta is less obtrusive 12 | than other go dependency management tools because it relies on git 13 | submodules. You don't need vendetta to build a project, or for most 14 | other development tasks. Vendetta is just used to populate the 15 | `vendor` directory with submodules, or to update them when a project's 16 | dependencies change. Because it uses git submodules, your project 17 | repository remains small, and it is easy to relate the contents of the 18 | `vendor` directory back to their origin repositories. 19 | 20 | ## Installation 21 | 22 | If you have your GOPATH set up: 23 | 24 | ```sh 25 | go get github.com/dpw/vendetta 26 | ``` 27 | 28 | This will install `vendetta` in `$GOPATH/bin` 29 | 30 | If you don't: 31 | 32 | ```sh 33 | git clone https://github.com/dpw/vendetta.git && (cd vendetta ; go build) 34 | ``` 35 | 36 | The `vendetta` binary will be in the cloned `vendetta` directory. 37 | 38 | ## Use 39 | 40 | Usage: `vendetta `_`[options] [directory]`_ 41 | 42 | The directory specified should be the top-level directory of the git 43 | repo that holds your Go project. If it is omitted, the current 44 | directory is used. 45 | 46 | Like `go get`, vendetta identifies any missing packages needed to 47 | build your top-level project (including packages needed by other 48 | dependencies). It then finds the projects containing those missing 49 | packages, and runs the git commands to add submodules for them. 50 | 51 | When you clone a project with submodules, as produced by vendetta, the 52 | submodule directories will initially be empty. Do `git submodule 53 | update --init --recursive` in order to retrieve the submodule 54 | contents. 55 | 56 | Vendetta follows all the relevant Go conventions, such as ignoring 57 | `testdata` directories. 58 | 59 | ### Options 60 | 61 | * `-p`: _Prune_ unneeded submodules under `vendor/`. 62 | 63 | * `-u`: _Update_ dependencies of your project. This pulls from the 64 | remote repositories for required submodules under `vendor/`. 65 | 66 | ## Background 67 | 68 | Go 1.5 introduced the [Go Vendor](https://golang.org/s/go15vendor) 69 | feature. This provides support in the standard go tool set for 70 | `vendor` directories which contain the source code for dependencies of 71 | a project. Note that in Go 1.5, you must set the 72 | GO15VENDOREXPERIMENT environment variable to enable this feature; but 73 | Go 1.6 enables it by default. 74 | 75 | The Go Vendor feature is a significant step forward for dependency 76 | management in go. But out of the box, it does not provide a way to 77 | populate the `vendor` directory for a project with its dependencies, 78 | or manage those dependencies as the project evolves. Trying to do 79 | this by hand is cumbersome and error prone. 80 | 81 | [Other go vendoring tools are 82 | available](https://github.com/golang/go/wiki/PackageManagementTools). 83 | But they support two approaches: Either they copy the source code of 84 | dependencies into `vendor/`, which bloats the repository of your 85 | project. Or, they write a dependency metadata file under `vendor/` 86 | which says how to get the dependencies. But then anyone who wants to 87 | build the project needs to use a specific tool to retrieve the 88 | dependencies. (And there is no dominant standard for the dependency 89 | metadata files – there are even two different formats for a file 90 | called `vendor.json`.) 91 | 92 | Instead, vendetta relies on the 93 | [submodule](https://git-scm.com/docs/git-submodule) feature of git, 94 | which provides a way for one git repository to point to another git 95 | repository (and a specific commit within it). And submodules are a 96 | standard feature of git, so git will retrieve them for you. You may 97 | already have experience with submodules. And tools built on top of 98 | git understand submodules (e.g. github knows about submodules, and 99 | will display a submodule pointing to another project on github as a 100 | link). 101 | 102 | When you clone a repository containing submodules, you need to do `git 103 | submodule update --init --recursive` in order to retrieve the 104 | submodule contents. This step is sometimes surprising to those new to 105 | git submodules. But it can be hidden by incorporating it into build 106 | scripts or makefiles. And `go get` will do `git submodule update` 107 | after cloning a repo, so it is not necessary to run it explicitly when 108 | fetching go packages in that way. 109 | 110 | A downside of git submodules is that, being a git-specific feature, 111 | they only allow dependencies that live in (or are mirrored to) git 112 | repositories. But given the prevalence of git within the go 113 | community, and the ease of mirroring other VCSes to git, this is not 114 | much of a limitation. 115 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bufio" 5 | "flag" 6 | "fmt" 7 | "go/build" 8 | "io" 9 | "io/ioutil" 10 | "os" 11 | "os/exec" 12 | "path/filepath" 13 | "regexp" 14 | "sort" 15 | "strings" 16 | ) 17 | 18 | // TODO: 19 | // 20 | // check the directory is a git repo, and error if not 21 | // 22 | // option to run in any directory of git repo. Needs to figure out 23 | // path from repo root. 24 | // 25 | // verbose option to print git commands being run 26 | // 27 | // popen should include command in errors 28 | // 29 | // Deal with git being fussy when a submodule is removed then re-added 30 | // 31 | // warn when it looks like a package ought to be present at the 32 | // particular path, but it's not. E.g. when resolve an import of 33 | // github.com/foo/bar/baz, we find that only github.com/foo exists. 34 | // 35 | // check that declared package names match dirs 36 | // 37 | // Support relative (aka local) imports 38 | // 39 | // Warn on diamond problem 40 | 41 | type config struct { 42 | rootDir string 43 | projectName string 44 | update bool 45 | prune bool 46 | } 47 | 48 | func main() { 49 | flag.Usage = func() { 50 | fmt.Fprintf(os.Stderr, "Usage: %s [ ]\n", 51 | os.Args[0]) 52 | flag.PrintDefaults() 53 | } 54 | 55 | var cf config 56 | 57 | flag.StringVar(&cf.projectName, "n", "", 58 | "base package name for the project, e.g. github.com/user/proj") 59 | flag.BoolVar(&cf.update, "u", false, 60 | "update dependency submodules from their remote repos") 61 | flag.BoolVar(&cf.prune, "p", false, 62 | "prune unused dependency submodules") 63 | 64 | flag.Parse() 65 | 66 | switch { 67 | case flag.NArg() == 1: 68 | cf.rootDir = flag.Arg(0) 69 | case flag.NArg() > 1: 70 | flag.Usage() 71 | os.Exit(2) 72 | } 73 | 74 | if err := run(&cf); err != nil { 75 | fmt.Fprintln(os.Stderr, err) 76 | os.Exit(1) 77 | } 78 | } 79 | 80 | type vendetta struct { 81 | *config 82 | goPath 83 | goPaths map[string]*goPath 84 | dirPackages map[string]*build.Package 85 | submodules []submodule 86 | } 87 | 88 | // A goPath says where to search for packages (analogous to 89 | // GOPATH). Different directories have different gopaths because there 90 | // can be vendor directories anywhere, not just at the top level. 91 | // These gopaths are produced by getGoPath and memoized in the goPaths 92 | // map. 93 | type goPath struct { 94 | dir string 95 | next *goPath 96 | 97 | // prefixes is nil for goPaths corresponding to vendor 98 | // directories. But there is also a goPath corresponding to 99 | // the top-level project directory. When searching for a 100 | // package in that top-level directory, we need to remove any 101 | // prefix of the package name corresponding to the root name 102 | // of the project (e.g. github.com/user/proj). prefixes is 103 | // the set of such prefixes. 104 | prefixes map[string]struct{} 105 | } 106 | 107 | type submodule struct { 108 | dir string 109 | used bool 110 | } 111 | 112 | func run(cf *config) error { 113 | v := vendetta{ 114 | config: cf, 115 | goPaths: make(map[string]*goPath), 116 | dirPackages: make(map[string]*build.Package), 117 | } 118 | 119 | v.goPaths[""] = &goPath{dir: "vendor", next: &v.goPath} 120 | v.prefixes = make(map[string]struct{}) 121 | 122 | rootPkgs, err := v.scanRootProject() 123 | if err != nil { 124 | return err 125 | } 126 | 127 | if cf.projectName != "" { 128 | v.prefixes[cf.projectName] = struct{}{} 129 | } else { 130 | if err := v.inferProjectNameFromGoPath(); err != nil { 131 | return err 132 | } 133 | 134 | if err := v.inferProjectNameFromGit(); err != nil { 135 | return err 136 | } 137 | 138 | v.inferProjectNameFromImportComments(rootPkgs) 139 | 140 | if !mainOnly(rootPkgs) && len(v.prefixes) == 0 { 141 | return fmt.Errorf("Unable to infer project name; specify it explicitly with the '-n' option.") 142 | } 143 | } 144 | 145 | if err := v.checkSubmodules(); err != nil { 146 | return err 147 | } 148 | 149 | if err := v.populateSubmodules(); err != nil { 150 | return err 151 | } 152 | 153 | if err := v.resolveRootProjectDeps(rootPkgs); err != nil { 154 | return err 155 | } 156 | 157 | return v.pruneSubmodules() 158 | } 159 | 160 | // Attempt to infer the project name from GOPATH, by seeing if the 161 | // project dir resides under any element of the GOPATH. 162 | func (v *vendetta) inferProjectNameFromGoPath() error { 163 | gp := os.Getenv("GOPATH") 164 | if gp == "" { 165 | return nil 166 | } 167 | 168 | gpparts := filepath.SplitList(gp) 169 | gpfis := make([]os.FileInfo, len(gpparts)) 170 | for i, p := range gpparts { 171 | var err error 172 | gpfis[i], err = os.Stat(filepath.Join(p, "src")) 173 | if err != nil { 174 | if os.IsNotExist(err) { 175 | continue 176 | } 177 | 178 | return err 179 | } 180 | } 181 | 182 | // Get the absolute path to the project dir 183 | dir := v.rootDir 184 | if dir == "" { 185 | dir = "." 186 | } 187 | 188 | dir, err := filepath.Abs(dir) 189 | if err != nil { 190 | return err 191 | } 192 | 193 | // walk from the project dir upwards towards the filesystem root 194 | var proj, projsep string 195 | for { 196 | var subdir string 197 | dir, subdir = filepath.Split(dir) 198 | 199 | proj = subdir + projsep + proj 200 | projsep = "/" 201 | 202 | fi, err := os.Stat(dir) 203 | if err != nil { 204 | return err 205 | } 206 | 207 | for i, gpfi := range gpfis { 208 | if gpfi != nil && os.SameFile(fi, gpfi) { 209 | v.inferredProjectName(proj, "GOPATH element", 210 | gpparts[i]) 211 | return nil 212 | } 213 | } 214 | 215 | if dir != "" && dir[len(dir)-1] == os.PathSeparator { 216 | dir = dir[:len(dir)-1] 217 | } 218 | 219 | if dir == "" { 220 | return nil 221 | } 222 | } 223 | } 224 | 225 | var remoteUrlRE = regexp.MustCompile(`^(?:https://github\.com/|git@github\.com:)(.*\.?)$`) 226 | 227 | func (v *vendetta) inferProjectNameFromGit() error { 228 | remotes, err := v.popen("git", "remote", "-v") 229 | if err != nil { 230 | return err 231 | } 232 | 233 | defer remotes.close() 234 | 235 | for remotes.Scan() { 236 | fields := splitWS(remotes.Text()) 237 | if len(fields) < 2 { 238 | return fmt.Errorf("could not parse 'git remote' output") 239 | } 240 | 241 | m := remoteUrlRE.FindStringSubmatch(fields[1]) 242 | if m != nil { 243 | name := m[1] 244 | if strings.HasSuffix(name, ".git") { 245 | name = name[:len(name)-4] 246 | } 247 | 248 | v.inferredProjectName("github.com/"+name, 249 | "git remote", fields[0]) 250 | } 251 | } 252 | 253 | if err := remotes.close(); err != nil { 254 | return err 255 | } 256 | 257 | return nil 258 | } 259 | 260 | func (v *vendetta) inferProjectNameFromImportComments(rootPkgs []rootPackage) { 261 | for _, pkg := range rootPkgs { 262 | ic := pkg.ImportComment 263 | if ic == "" { 264 | continue 265 | } 266 | 267 | // For an import comment to suggest a project name, it 268 | // should have the path of the package within the 269 | // project as a suffix. 270 | if pkg.dir != "" { 271 | suffix := pathToPackage(pkg.dir) 272 | if len(ic) <= len(suffix) || 273 | ic[len(ic)-len(suffix):] != suffix && 274 | ic[len(ic)-len(suffix)-1] != '/' { 275 | continue 276 | } 277 | 278 | ic = ic[:len(ic)-len(suffix)-1] 279 | } 280 | 281 | v.inferredProjectName(ic, "import comment in", 282 | v.realDir(pkg.dir)) 283 | } 284 | } 285 | 286 | func (v *vendetta) inferredProjectName(proj string, source ...interface{}) { 287 | if _, found := v.prefixes[proj]; !found { 288 | fmt.Println(append([]interface{}{ 289 | "Inferred root package name", proj, "from", 290 | }, source...)...) 291 | v.prefixes[proj] = struct{}{} 292 | } 293 | } 294 | 295 | // Check for submodules that seem to be missing in the working tree. 296 | func (v *vendetta) checkSubmodules() error { 297 | var err2 error 298 | if err := v.querySubmodules(func(path string) bool { 299 | err2 = v.checkSubmodule(path) 300 | return err2 == nil 301 | }, "--recursive"); err != nil { 302 | return err 303 | } 304 | 305 | return err2 306 | } 307 | 308 | func (v *vendetta) checkSubmodule(dir string) error { 309 | foundSomething := false 310 | if err := readDir(v.realDir(dir), func(fi os.FileInfo) bool { 311 | foundSomething = true 312 | return false 313 | }); err != nil && !os.IsNotExist(err) { 314 | return err 315 | } 316 | 317 | if !foundSomething { 318 | return fmt.Errorf("The submodule '%s' doesn't seem to the present in the working tree. Maybe you forgot to update with 'git submodule update --init --recursive'?", dir) 319 | } 320 | 321 | return nil 322 | } 323 | 324 | func (v *vendetta) querySubmodules(f func(string) bool, args ...string) error { 325 | status, err := v.popen("git", 326 | append([]string{"submodule", "status"}, args...)...) 327 | if err != nil { 328 | return err 329 | } 330 | 331 | defer status.close() 332 | 333 | for status.Scan() { 334 | fields := splitWS(strings.TrimSpace(status.Text())) 335 | if len(fields) < 2 { 336 | return fmt.Errorf("could not parse 'git submodule status' output") 337 | } 338 | 339 | path := fields[1] 340 | 341 | if !f(path) { 342 | return nil 343 | } 344 | } 345 | 346 | return status.close() 347 | } 348 | 349 | func (v *vendetta) populateSubmodules() error { 350 | var submodules []string 351 | if err := v.querySubmodules(func(path string) bool { 352 | submodules = append(submodules, path) 353 | return true 354 | }); err != nil { 355 | return err 356 | } 357 | 358 | sort.Strings(submodules) 359 | 360 | v.submodules = make([]submodule, 0, len(submodules)) 361 | for _, p := range submodules { 362 | v.submodules = append(v.submodules, submodule{dir: p}) 363 | } 364 | 365 | return nil 366 | } 367 | 368 | func (v *vendetta) pathInSubmodule(path string) *submodule { 369 | i := sort.Search(len(v.submodules), func(i int) bool { 370 | return v.submodules[i].dir >= path 371 | }) 372 | if i < len(v.submodules) && v.submodules[i].dir == path { 373 | return &v.submodules[i] 374 | } 375 | if i > 0 && isSubpath(path, v.submodules[i-1].dir) { 376 | return &v.submodules[i-1] 377 | } 378 | return nil 379 | } 380 | 381 | func (v *vendetta) addSubmodule(dir string) { 382 | i := sort.Search(len(v.submodules), func(i int) bool { 383 | return v.submodules[i].dir >= dir 384 | }) 385 | 386 | submodules := make([]submodule, len(v.submodules)+1) 387 | copy(submodules, v.submodules[:i]) 388 | submodules[i] = submodule{dir: dir, used: true} 389 | copy(submodules[i+1:], v.submodules[i:]) 390 | v.submodules = submodules 391 | } 392 | 393 | func isSubpath(path, dir string) bool { 394 | return path == dir || 395 | (strings.HasPrefix(path, dir) && path[len(dir)] == os.PathSeparator) 396 | } 397 | 398 | func (v *vendetta) updateSubmodule(sm *submodule) error { 399 | fmt.Fprintf(os.Stderr, "Updating submodule %s from remote\n", sm.dir) 400 | if err := v.git("submodule", "update", "--remote", "--recursive", sm.dir); err != nil { 401 | return err 402 | } 403 | 404 | // If we don't put the updated submodule into the index, a 405 | // subsequent "git submodule update" will revert it, which can 406 | // lead to surprises. 407 | return v.git("add", sm.dir) 408 | } 409 | 410 | func (v *vendetta) pruneSubmodules() error { 411 | for _, sm := range v.submodules { 412 | if sm.used || !isSubpath(sm.dir, "vendor") { 413 | continue 414 | } 415 | 416 | if v.prune { 417 | fmt.Fprintf(os.Stderr, "Removing unused submodule %s\n", 418 | sm.dir) 419 | if err := v.git("rm", "-f", sm.dir); err != nil { 420 | return err 421 | } 422 | 423 | if err := v.removeEmptyDirsAbove(sm.dir); err != nil { 424 | return err 425 | } 426 | } else { 427 | fmt.Fprintf(os.Stderr, "Unused submodule %s (use -p option to prune)\n", sm.dir) 428 | } 429 | } 430 | 431 | return nil 432 | } 433 | 434 | func (v *vendetta) removeEmptyDirsAbove(dir string) error { 435 | for { 436 | dir = parentDir(dir) 437 | if dir == "" { 438 | return nil 439 | } 440 | 441 | empty := true 442 | if err := readDir(v.realDir(dir), func(_ os.FileInfo) bool { 443 | empty = false 444 | return false 445 | }); err != nil { 446 | return err 447 | } 448 | 449 | if !empty { 450 | return nil 451 | } 452 | 453 | if err := os.Remove(v.realDir(dir)); err != nil { 454 | return err 455 | } 456 | } 457 | } 458 | 459 | // Get the directory name from a path. path.Dir doesn't 460 | // do what we want in the case where there is no path 461 | // separator: 462 | func parentDir(path string) string { 463 | slash := strings.LastIndexByte(path, os.PathSeparator) 464 | dir := "" 465 | if slash >= 0 { 466 | dir = path[:slash] 467 | } 468 | return dir 469 | } 470 | 471 | var wsRE = regexp.MustCompile(`[ \t]+`) 472 | 473 | func splitWS(s string) []string { 474 | return wsRE.Split(s, -1) 475 | } 476 | 477 | func (v *vendetta) gitSubmoduleAdd(url, dir string) error { 478 | fmt.Fprintf(os.Stderr, "Adding %s at %s\n", url, dir) 479 | err := v.git("submodule", "add", url, dir) 480 | if err != nil { 481 | return err 482 | } 483 | 484 | v.addSubmodule(dir) 485 | return nil 486 | } 487 | 488 | func (v *vendetta) git(args ...string) error { 489 | return v.system("git", args...) 490 | } 491 | 492 | func (v *vendetta) system(name string, args ...string) error { 493 | cmd := exec.Command(name, args...) 494 | cmd.Dir = v.rootDir 495 | cmd.Stdout = os.Stdout 496 | cmd.Stderr = os.Stderr 497 | 498 | err := cmd.Start() 499 | if err == nil { 500 | err = cmd.Wait() 501 | if err == nil { 502 | return nil 503 | } 504 | } 505 | 506 | return fmt.Errorf("Command failed: %s %s (%s)", 507 | name, strings.Join(args, " "), err) 508 | } 509 | 510 | type popenLines struct { 511 | cmd *exec.Cmd 512 | stdout io.ReadCloser 513 | *bufio.Scanner 514 | } 515 | 516 | func (v *vendetta) popen(name string, args ...string) (popenLines, error) { 517 | cmd := exec.Command(name, args...) 518 | cmd.Dir = v.rootDir 519 | stdout, err := cmd.StdoutPipe() 520 | if err != nil { 521 | return popenLines{}, err 522 | } 523 | 524 | cmd.Stderr = os.Stderr 525 | p := popenLines{cmd: cmd, stdout: stdout} 526 | 527 | if err := cmd.Start(); err != nil { 528 | return popenLines{}, err 529 | } 530 | 531 | p.Scanner = bufio.NewScanner(stdout) 532 | return p, nil 533 | } 534 | 535 | func (p popenLines) close() error { 536 | res := p.Scanner.Err() 537 | setRes := func(err error) { 538 | if res == nil { 539 | res = err 540 | } 541 | } 542 | 543 | if p.stdout != nil { 544 | _, err := io.Copy(ioutil.Discard, p.stdout) 545 | p.stdout = nil 546 | if err != nil { 547 | setRes(err) 548 | p.cmd.Process.Kill() 549 | } 550 | } 551 | 552 | if p.cmd != nil { 553 | setRes(p.cmd.Wait()) 554 | p.cmd = nil 555 | } 556 | 557 | return res 558 | } 559 | 560 | func (v *vendetta) realDir(dir string) string { 561 | res := filepath.Join(v.rootDir, dir) 562 | if res == "" { 563 | res = "." 564 | } 565 | 566 | return res 567 | } 568 | 569 | type rootPackage struct { 570 | dir string 571 | *build.Package 572 | } 573 | 574 | func (v *vendetta) scanRootProject() ([]rootPackage, error) { 575 | // Load each package in the root project without resolving 576 | // dependencies, because we process packages in the root 577 | // project slightly differently to dependency packages. 578 | var pkgs []rootPackage 579 | var err error 580 | 581 | var traverseDir func(dir string, root bool) 582 | traverseDir = func(dir string, root bool) { 583 | var pkg *build.Package 584 | pkg, err = v.loadPackage(dir, true) 585 | if err != nil { 586 | return 587 | } 588 | 589 | if pkg != nil { 590 | pkgs = append(pkgs, rootPackage{dir, pkg}) 591 | } 592 | 593 | err = readDir(v.realDir(dir), func(fi os.FileInfo) bool { 594 | if !fi.IsDir() { 595 | return true 596 | } 597 | 598 | switch fi.Name() { 599 | case "vendor": 600 | if root { 601 | return true 602 | } 603 | case "testdata": 604 | return true 605 | } 606 | 607 | traverseDir(filepath.Join(dir, fi.Name()), false) 608 | return err == nil 609 | }) 610 | } 611 | 612 | traverseDir("", true) 613 | if err != nil { 614 | return nil, err 615 | } 616 | 617 | return pkgs, nil 618 | } 619 | 620 | func (v *vendetta) resolveRootProjectDeps(pkgs []rootPackage) error { 621 | for _, pkg := range pkgs { 622 | if err := v.resolveDependencies(pkg.dir, pkg.Imports); err != nil { 623 | return err 624 | } 625 | if err := v.resolveDependencies(pkg.dir, pkg.TestImports); err != nil { 626 | return err 627 | } 628 | if err := v.resolveDependencies(pkg.dir, pkg.XTestImports); err != nil { 629 | return err 630 | } 631 | } 632 | 633 | return nil 634 | } 635 | 636 | func mainOnly(pkgs []rootPackage) bool { 637 | for _, pkg := range pkgs { 638 | if pkg.Name != "main" { 639 | return false 640 | } 641 | } 642 | 643 | return true 644 | } 645 | 646 | func (v *vendetta) scanPackage(dir string) (*build.Package, error) { 647 | if pkg := v.dirPackages[dir]; pkg != nil { 648 | return pkg, nil 649 | } 650 | 651 | pkg, err := v.loadPackage(dir, false) 652 | if err != nil { 653 | return nil, err 654 | } 655 | 656 | if err = v.resolveDependencies(dir, pkg.Imports); err != nil { 657 | return nil, err 658 | } 659 | 660 | return pkg, nil 661 | } 662 | 663 | func (v *vendetta) loadPackage(dir string, noGoOk bool) (*build.Package, error) { 664 | pkg, err := build.Default.ImportDir(v.realDir(dir), build.ImportComment) 665 | if err != nil { 666 | if _, ok := err.(*build.NoGoError); ok && noGoOk { 667 | return nil, nil 668 | } 669 | 670 | return nil, fmt.Errorf("gathering imports in %s: %s", 671 | v.realDir(dir), err) 672 | } 673 | 674 | v.dirPackages[dir] = pkg 675 | return pkg, nil 676 | } 677 | 678 | func (v *vendetta) resolveDependencies(dir string, deps []string) error { 679 | for _, dep := range deps { 680 | if err := v.resolveDependency(dir, dep); err != nil { 681 | return err 682 | } 683 | } 684 | 685 | return nil 686 | } 687 | 688 | func (v *vendetta) resolveDependency(dir string, pkg string) error { 689 | found, pkgdir, err := v.searchGoPath(dir, pkg) 690 | switch { 691 | case err != nil: 692 | return err 693 | case found: 694 | // Does the package fall within an existing submodule 695 | // under vendor/ ? 696 | if sm := v.pathInSubmodule(pkgdir); sm != nil && !sm.used { 697 | sm.used = true 698 | if v.update { 699 | if err := v.updateSubmodule(sm); err != nil { 700 | return err 701 | } 702 | } 703 | } 704 | 705 | default: 706 | pkgdir, err = v.obtainPackage(pkg) 707 | if err != nil || pkgdir == "" { 708 | return err 709 | } 710 | } 711 | 712 | pi, err := v.scanPackage(pkgdir) 713 | if err != nil { 714 | return err 715 | } 716 | 717 | if pi.ImportComment != "" && pkg != pi.ImportComment { 718 | fmt.Printf("Warning: Package with import comment %s referred to as %s (from directory %s)\n", 719 | pi.ImportComment, pkg, v.realDir(dir)) 720 | } 721 | 722 | return nil 723 | } 724 | 725 | func (v *vendetta) obtainPackage(pkg string) (string, error) { 726 | bits := strings.Split(pkg, "/") 727 | 728 | // Exclude golang standard packages 729 | if !strings.Contains(bits[0], ".") { 730 | return "", nil 731 | } 732 | 733 | // Figure out how to obtain the package. Packages on 734 | // github.com are treated as a special case, because that is 735 | // most of them. Otherwise, we use the queryRepoRoot code 736 | // borrowed from vcs.go to figure out how to obtain the 737 | // package. 738 | var basePkg, url string 739 | if bits[0] == "github.com" { 740 | if len(bits) < 3 { 741 | return "", fmt.Errorf("github.com package name %s seems to be truncated", pkg) 742 | } 743 | 744 | basePkg = strings.Join(bits[:3], "/") 745 | url = "https://" + basePkg 746 | } else if bits[0] == "bitbucket.org" { 747 | if len(bits) < 3 { 748 | return "", fmt.Errorf("bitbucket.org package name %s seems to be truncated", pkg) 749 | } 750 | 751 | basePkg = strings.Join(bits[:3], "/") 752 | url = "https://" + basePkg 753 | 754 | // Probe to see if it is a git repo 755 | if exec.Command("git", "ls-remote", url).Run() != nil { 756 | return "", fmt.Errorf("Package %s does not seem to be git repo at %s; maybe it's an hg repo?", pkg, url) 757 | } 758 | } else if rr, err := queryRepoRoot(pkg, secure); err == nil { 759 | if rr.vcs != "git" { 760 | return "", fmt.Errorf("Package %s does not live in a git repo", pkg) 761 | } 762 | 763 | basePkg = rr.root 764 | url = rr.repo 765 | } else if strings.HasSuffix(err.Error(), "no go-import meta tags") && len(bits) >= 3 { 766 | // When no go-import meta tag is found, guess the base 767 | // package and repo URL, so that e.g. package names on 768 | // gitlab work. The test above is gross, but it 769 | // avoids changes to the borrowed reporoot code. 770 | basePkg = strings.Join(bits[:3], "/") 771 | url = fmt.Sprintf("https://%s.git", basePkg) 772 | fmt.Printf("Warning: no go-import meta tags found for package '%s'. Guessing git repo URL '%s'\n", pkg, url) 773 | } else { 774 | return "", err 775 | } 776 | 777 | projDir := filepath.Join("vendor", packageToPath(basePkg)) 778 | if err := v.gitSubmoduleAdd(url, projDir); err != nil { 779 | return "", err 780 | } 781 | 782 | return filepath.Join("vendor", packageToPath(pkg)), nil 783 | } 784 | 785 | // Search the gopath for the given dir to find an existing package 786 | func (v *vendetta) searchGoPath(dir, pkg string) (bool, string, error) { 787 | gp, err := v.getGoPath(dir) 788 | if err != nil { 789 | return false, "", err 790 | } 791 | 792 | for gp != nil { 793 | found, pkgdir, err := gp.provides(pkg, v) 794 | if err != nil { 795 | return false, "", err 796 | } 797 | 798 | if found { 799 | return found, pkgdir, nil 800 | } 801 | 802 | gp = gp.next 803 | } 804 | 805 | return false, "", nil 806 | } 807 | 808 | func (v *vendetta) getGoPath(dir string) (*goPath, error) { 809 | gp := v.goPaths[dir] 810 | if gp != nil { 811 | return gp, nil 812 | } 813 | 814 | gp, err := v.getGoPath(parentDir(dir)) 815 | if err != nil { 816 | return nil, err 817 | } 818 | 819 | // If there's a vendor/ dir here, we need to put it on the 820 | // front of the gopath 821 | vendorDir := filepath.Join(dir, "vendor") 822 | fi, err := os.Stat(v.realDir(vendorDir)) 823 | if err != nil { 824 | if !os.IsNotExist(err) { 825 | return nil, err 826 | } 827 | } else if fi.IsDir() { 828 | gp = &goPath{dir: vendorDir, next: gp} 829 | } 830 | 831 | v.goPaths[dir] = gp 832 | return gp, nil 833 | } 834 | 835 | func (gp *goPath) provides(pkg string, v *vendetta) (bool, string, error) { 836 | matched, pkg := gp.removePrefix(pkg) 837 | if !matched { 838 | return false, "", nil 839 | } 840 | 841 | foundGoSrc := false 842 | pkgdir := filepath.Join(gp.dir, packageToPath(pkg)) 843 | if err := readDir(v.realDir(pkgdir), func(fi os.FileInfo) bool { 844 | // Should check for symlinks here? 845 | if fi.Mode().IsRegular() && strings.HasSuffix(fi.Name(), ".go") { 846 | foundGoSrc = true 847 | return false 848 | } 849 | return true 850 | }); err != nil { 851 | if os.IsNotExist(err) { 852 | err = nil 853 | } 854 | return false, "", err 855 | } 856 | 857 | return foundGoSrc, pkgdir, nil 858 | } 859 | 860 | func (gp *goPath) removePrefix(pkg string) (bool, string) { 861 | if gp.prefixes == nil { 862 | return true, pkg 863 | } 864 | 865 | for prefix := range gp.prefixes { 866 | if pkg == prefix { 867 | return true, "" 868 | } else if isSubpath(pkg, prefix) { 869 | return true, pkg[len(prefix)+1:] 870 | } 871 | } 872 | 873 | return false, "" 874 | } 875 | 876 | // Convert a package name to a filesystem path 877 | func packageToPath(name string) string { 878 | return filepath.FromSlash(name) 879 | } 880 | 881 | // Convert a filesystem path to a package name 882 | func pathToPackage(path string) string { 883 | return filepath.ToSlash(path) 884 | } 885 | 886 | func readDir(dir string, f func(os.FileInfo) bool) error { 887 | dh, err := os.Open(dir) 888 | if err != nil { 889 | return err 890 | } 891 | 892 | defer dh.Close() 893 | 894 | for { 895 | fis, err := dh.Readdir(100) 896 | if err != nil { 897 | if err == io.EOF { 898 | return nil 899 | } 900 | 901 | return err 902 | } 903 | 904 | for _, fi := range fis { 905 | if !f(fi) { 906 | return nil 907 | } 908 | } 909 | } 910 | } 911 | -------------------------------------------------------------------------------- /reporoot.go: -------------------------------------------------------------------------------- 1 | // This file contains code taken from the "go" source tree, governed 2 | // by the go license: 3 | 4 | // Copyright (c) 2012 The Go Authors. All rights reserved. 5 | // 6 | // Redistribution and use in source and binary forms, with or without 7 | // modification, are permitted provided that the following conditions are 8 | // met: 9 | // 10 | // * Redistributions of source code must retain the above copyright 11 | // notice, this list of conditions and the following disclaimer. 12 | // * Redistributions in binary form must reproduce the above 13 | // copyright notice, this list of conditions and the following disclaimer 14 | // in the documentation and/or other materials provided with the 15 | // distribution. 16 | // * Neither the name of Google Inc. nor the names of its 17 | // contributors may be used to endorse or promote products derived from 18 | // this software without specific prior written permission. 19 | // 20 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 21 | // "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 22 | // LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 23 | // A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 24 | // OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 25 | // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 26 | // LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 27 | // DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 28 | // THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 29 | // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 30 | // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 31 | 32 | package main 33 | 34 | import ( 35 | "crypto/tls" 36 | "encoding/xml" 37 | "errors" 38 | "fmt" 39 | "io" 40 | "io/ioutil" 41 | "log" 42 | "net/http" 43 | "net/url" 44 | "strings" 45 | "sync" 46 | "time" 47 | ) 48 | 49 | const buildV = false 50 | 51 | // From go/src/cmd/go/vcs.go 52 | 53 | // securityMode specifies whether a function should make network 54 | // calls using insecure transports (eg, plain text HTTP). 55 | // The zero value is "secure". 56 | type securityMode int 57 | 58 | const ( 59 | secure securityMode = iota 60 | insecure 61 | ) 62 | 63 | // repoRoot represents a version control system, a repo, and a root of 64 | // where to put it on disk. 65 | type repoRoot struct { 66 | vcs string 67 | 68 | // repo is the repository URL, including scheme 69 | repo string 70 | 71 | // root is the import path corresponding to the root of the 72 | // repository 73 | root string 74 | } 75 | 76 | // repoRootForImportDynamic finds a *repoRoot for a custom domain that's not 77 | // statically known by repoRootForImportPathStatic. 78 | // 79 | // This handles custom import paths like "name.tld/pkg/foo" or just "name.tld". 80 | func queryRepoRoot(importPath string, security securityMode) (*repoRoot, error) { 81 | slash := strings.Index(importPath, "/") 82 | if slash < 0 { 83 | slash = len(importPath) 84 | } 85 | host := importPath[:slash] 86 | if !strings.Contains(host, ".") { 87 | return nil, errors.New("import path does not begin with hostname") 88 | } 89 | urlStr, body, err := httpsOrHTTP(importPath, security) 90 | if err != nil { 91 | msg := "https fetch: %v" 92 | if security == insecure { 93 | msg = "http/" + msg 94 | } 95 | return nil, fmt.Errorf(msg, err) 96 | } 97 | defer body.Close() 98 | imports, err := parseMetaGoImports(body) 99 | if err != nil { 100 | return nil, fmt.Errorf("parsing %s: %v", importPath, err) 101 | } 102 | // Find the matched meta import. 103 | mmi, err := matchGoImport(imports, importPath) 104 | if err != nil { 105 | if err != errNoMatch { 106 | return nil, fmt.Errorf("parse %s: %v", urlStr, err) 107 | } 108 | return nil, fmt.Errorf("parse %s: no go-import meta tags", urlStr) 109 | } 110 | if buildV { 111 | log.Printf("get %q: found meta tag %#v at %s", importPath, mmi, urlStr) 112 | } 113 | // If the import was "uni.edu/bob/project", which said the 114 | // prefix was "uni.edu" and the RepoRoot was "evilroot.com", 115 | // make sure we don't trust Bob and check out evilroot.com to 116 | // "uni.edu" yet (possibly overwriting/preempting another 117 | // non-evil student). Instead, first verify the root and see 118 | // if it matches Bob's claim. 119 | if mmi.Prefix != importPath { 120 | if buildV { 121 | log.Printf("get %q: verifying non-authoritative meta tag", importPath) 122 | } 123 | urlStr0 := urlStr 124 | var imports []metaImport 125 | urlStr, imports, err = metaImportsForPrefix(mmi.Prefix, security) 126 | if err != nil { 127 | return nil, err 128 | } 129 | metaImport2, err := matchGoImport(imports, importPath) 130 | if err != nil || mmi != metaImport2 { 131 | return nil, fmt.Errorf("%s and %s disagree about go-import for %s", urlStr0, urlStr, mmi.Prefix) 132 | } 133 | } 134 | 135 | if !strings.Contains(mmi.RepoRoot, "://") { 136 | return nil, fmt.Errorf("%s: invalid repo root %q; no scheme", urlStr, mmi.RepoRoot) 137 | } 138 | return &repoRoot{ 139 | vcs: mmi.VCS, 140 | repo: mmi.RepoRoot, 141 | root: mmi.Prefix, 142 | }, nil 143 | } 144 | 145 | var ( 146 | fetchCacheMu sync.Mutex 147 | fetchCache = map[string]fetchResult{} // key is metaImportsForPrefix's importPrefix 148 | ) 149 | 150 | // metaImportsForPrefix takes a package's root import path as declared in a tag 151 | // and returns its HTML discovery URL and the parsed metaImport lines 152 | // found on the page. 153 | // 154 | // The importPath is of the form "golang.org/x/tools". 155 | // It is an error if no imports are found. 156 | // urlStr will still be valid if err != nil. 157 | // The returned urlStr will be of the form "https://golang.org/x/tools?go-get=1" 158 | func metaImportsForPrefix(importPrefix string, security securityMode) (urlStr string, imports []metaImport, err error) { 159 | setCache := func(res fetchResult) fetchResult { 160 | fetchCacheMu.Lock() 161 | defer fetchCacheMu.Unlock() 162 | fetchCache[importPrefix] = res 163 | return res 164 | } 165 | 166 | fetch := func() fetchResult { 167 | fetchCacheMu.Lock() 168 | if res, ok := fetchCache[importPrefix]; ok { 169 | fetchCacheMu.Unlock() 170 | return res 171 | } 172 | fetchCacheMu.Unlock() 173 | 174 | urlStr, body, err := httpsOrHTTP(importPrefix, security) 175 | if err != nil { 176 | return setCache(fetchResult{urlStr: urlStr, err: fmt.Errorf("fetch %s: %v", urlStr, err)}) 177 | } 178 | imports, err := parseMetaGoImports(body) 179 | if err != nil { 180 | return setCache(fetchResult{urlStr: urlStr, err: fmt.Errorf("parsing %s: %v", urlStr, err)}) 181 | } 182 | if len(imports) == 0 { 183 | err = fmt.Errorf("fetch %s: no go-import meta tag", urlStr) 184 | } 185 | return setCache(fetchResult{urlStr: urlStr, imports: imports, err: err}) 186 | } 187 | 188 | res := fetch() 189 | return res.urlStr, res.imports, res.err 190 | } 191 | 192 | type fetchResult struct { 193 | urlStr string // e.g. "https://foo.com/x/bar?go-get=1" 194 | imports []metaImport 195 | err error 196 | } 197 | 198 | // metaImport represents the parsed tags from HTML files. 200 | type metaImport struct { 201 | Prefix, VCS, RepoRoot string 202 | } 203 | 204 | // errNoMatch is returned from matchGoImport when there's no applicable match. 205 | var errNoMatch = errors.New("no import match") 206 | 207 | // matchGoImport returns the metaImport from imports matching importPath. 208 | // An error is returned if there are multiple matches. 209 | // errNoMatch is returned if none match. 210 | func matchGoImport(imports []metaImport, importPath string) (_ metaImport, err error) { 211 | match := -1 212 | for i, im := range imports { 213 | if !strings.HasPrefix(importPath, im.Prefix) { 214 | continue 215 | } 216 | if match != -1 { 217 | err = fmt.Errorf("multiple meta tags match import path %q", importPath) 218 | return 219 | } 220 | match = i 221 | } 222 | if match == -1 { 223 | err = errNoMatch 224 | return 225 | } 226 | return imports[match], nil 227 | } 228 | 229 | // From go/src/cmd/go/http.go 230 | 231 | // httpClient is the default HTTP client, but a variable so it can be 232 | // changed by tests, without modifying http.DefaultClient. 233 | var httpClient = http.DefaultClient 234 | 235 | // impatientInsecureHTTPClient is used in -insecure mode, 236 | // when we're connecting to https servers that might not be there 237 | // or might be using self-signed certificates. 238 | var impatientInsecureHTTPClient = &http.Client{ 239 | Timeout: time.Duration(5 * time.Second), 240 | Transport: &http.Transport{ 241 | TLSClientConfig: &tls.Config{ 242 | InsecureSkipVerify: true, 243 | }, 244 | }, 245 | } 246 | 247 | type httpError struct { 248 | status string 249 | statusCode int 250 | url string 251 | } 252 | 253 | func (e *httpError) Error() string { 254 | return fmt.Sprintf("%s: %s", e.url, e.status) 255 | } 256 | 257 | // httpGET returns the data from an HTTP GET request for the given URL. 258 | func httpGET(url string) ([]byte, error) { 259 | resp, err := httpClient.Get(url) 260 | if err != nil { 261 | return nil, err 262 | } 263 | defer resp.Body.Close() 264 | if resp.StatusCode != 200 { 265 | err := &httpError{status: resp.Status, statusCode: resp.StatusCode, url: url} 266 | 267 | return nil, err 268 | } 269 | b, err := ioutil.ReadAll(resp.Body) 270 | if err != nil { 271 | return nil, fmt.Errorf("%s: %v", url, err) 272 | } 273 | return b, nil 274 | } 275 | 276 | // httpsOrHTTP returns the body of either the importPath's 277 | // https resource or, if unavailable, the http resource. 278 | func httpsOrHTTP(importPath string, security securityMode) (urlStr string, body io.ReadCloser, err error) { 279 | fetch := func(scheme string) (urlStr string, res *http.Response, err error) { 280 | u, err := url.Parse(scheme + "://" + importPath) 281 | if err != nil { 282 | return "", nil, err 283 | } 284 | u.RawQuery = "go-get=1" 285 | urlStr = u.String() 286 | if buildV { 287 | log.Printf("Fetching %s", urlStr) 288 | } 289 | if security == insecure && scheme == "https" { // fail earlier 290 | res, err = impatientInsecureHTTPClient.Get(urlStr) 291 | } else { 292 | res, err = httpClient.Get(urlStr) 293 | } 294 | return 295 | } 296 | closeBody := func(res *http.Response) { 297 | if res != nil { 298 | res.Body.Close() 299 | } 300 | } 301 | urlStr, res, err := fetch("https") 302 | if err != nil { 303 | if buildV { 304 | log.Printf("https fetch failed: %v", err) 305 | } 306 | if security == insecure { 307 | closeBody(res) 308 | urlStr, res, err = fetch("http") 309 | } 310 | } 311 | if err != nil { 312 | closeBody(res) 313 | return "", nil, err 314 | } 315 | // Note: accepting a non-200 OK here, so people can serve a 316 | // meta import in their http 404 page. 317 | if buildV { 318 | log.Printf("Parsing meta tags from %s (status code %d)", urlStr, res.StatusCode) 319 | } 320 | return urlStr, res.Body, nil 321 | } 322 | 323 | // From go/src/cmd/go/discovery.go 324 | 325 | // charsetReader returns a reader for the given charset. Currently 326 | // it only supports UTF-8 and ASCII. Otherwise, it returns a meaningful 327 | // error which is printed by go get, so the user can find why the package 328 | // wasn't downloaded if the encoding is not supported. Note that, in 329 | // order to reduce potential errors, ASCII is treated as UTF-8 (i.e. characters 330 | // greater than 0x7f are not rejected). 331 | func charsetReader(charset string, input io.Reader) (io.Reader, error) { 332 | switch strings.ToLower(charset) { 333 | case "ascii": 334 | return input, nil 335 | default: 336 | return nil, fmt.Errorf("can't decode XML document using charset %q", charset) 337 | } 338 | } 339 | 340 | // parseMetaGoImports returns meta imports from the HTML in r. 341 | // Parsing ends at the end of the section or the beginning of the . 342 | func parseMetaGoImports(r io.Reader) (imports []metaImport, err error) { 343 | d := xml.NewDecoder(r) 344 | d.CharsetReader = charsetReader 345 | d.Strict = false 346 | var t xml.Token 347 | for { 348 | t, err = d.RawToken() 349 | if err != nil { 350 | if err == io.EOF || len(imports) > 0 { 351 | err = nil 352 | } 353 | return 354 | } 355 | if e, ok := t.(xml.StartElement); ok && strings.EqualFold(e.Name.Local, "body") { 356 | return 357 | } 358 | if e, ok := t.(xml.EndElement); ok && strings.EqualFold(e.Name.Local, "head") { 359 | return 360 | } 361 | e, ok := t.(xml.StartElement) 362 | if !ok || !strings.EqualFold(e.Name.Local, "meta") { 363 | continue 364 | } 365 | if attrValue(e.Attr, "name") != "go-import" { 366 | continue 367 | } 368 | if f := strings.Fields(attrValue(e.Attr, "content")); len(f) == 3 { 369 | imports = append(imports, metaImport{ 370 | Prefix: f[0], 371 | VCS: f[1], 372 | RepoRoot: f[2], 373 | }) 374 | } 375 | } 376 | } 377 | 378 | // attrValue returns the attribute value for the case-insensitive key 379 | // `name', or the empty string if nothing is found. 380 | func attrValue(attrs []xml.Attr, name string) string { 381 | for _, a := range attrs { 382 | if strings.EqualFold(a.Name.Local, name) { 383 | return a.Value 384 | } 385 | } 386 | return "" 387 | } 388 | --------------------------------------------------------------------------------