├── .travis.yml ├── version.go ├── Changelog.md ├── CONTRIBUTING.md ├── wrapper ├── helpers_test.go ├── generate.go ├── finding.go ├── flags_test.go ├── cycles.go ├── flags.go ├── wrapper.go └── packages.go ├── README.md ├── cmd ├── cyclecheck │ └── cyclecheck.go └── protowrap │ └── protowrap.go └── LICENSE.md /.travis.yml: -------------------------------------------------------------------------------- 1 | language: go 2 | 3 | go: 4 | - 1.6 -------------------------------------------------------------------------------- /version.go: -------------------------------------------------------------------------------- 1 | package goprotowrap 2 | 3 | const Version = "0.2.0" 4 | -------------------------------------------------------------------------------- /Changelog.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | ## v0.2.0 4 | 5 | - Add versions, and `--version` flag. 6 | - Use go1.6 on travis 7 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | Contributing 2 | ============ 3 | 4 | Contributions are welcome! 5 | 6 | If you would like to contribute code to protowrap you can do so 7 | through GitHub by forking the repository and sending a pull request. 8 | 9 | When submitting code, please make every effort to follow existing 10 | conventions and style in order to keep the code as readable as 11 | possible. Please also make sure your code compiles and the tests pass 12 | by running `go test ./...`. The code must also be formatted with `go 13 | fmt`. 14 | 15 | Before your code can be accepted into the project you must also sign the 16 | [Individual Contributor License Agreement (CLA)][1]. 17 | 18 | 19 | [1]: https://spreadsheets.google.com/spreadsheet/viewform?formkey=dDViT2xzUHAwRkI3X3k5Z0lQM091OGc6MQ&ndplr=1 20 | -------------------------------------------------------------------------------- /wrapper/helpers_test.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Square, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | package wrapper 16 | 17 | import "reflect" 18 | 19 | // Equality functions that consider nil equal to empty. 20 | 21 | func mapStringStringEqual(a, b map[string]string) bool { 22 | return len(a) == len(b) && (len(a) == 0 || reflect.DeepEqual(a, b)) 23 | } 24 | 25 | func sliceStringEqual(a, b []string) bool { 26 | return len(a) == len(b) && (len(a) == 0 || reflect.DeepEqual(a, b)) 27 | } 28 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # goprotowrap 2 | 3 | ⚠️ **The Go plugin for `protoc` got smarter: you should no longer need this** 4 | 5 | A package-at-a-time wrapper for protoc, for generating Go protobuf 6 | code. 7 | 8 | [![Build Status](https://travis-ci.org/square/goprotowrap.svg?branch=master)](https://travis-ci.org/square/goprotowrap) 9 | 10 | `protowrap` is a small tool we found helpful when working with 11 | [protocol buffers](https://developers.google.com/protocol-buffers/) in 12 | Go at Square. We're publishing it in the hope that others find it 13 | useful too. Contributions are welcome. 14 | 15 | ## Install 16 | 17 | ```shell 18 | go get -u github.com/square/goprotowrap/cmd/protowrap 19 | ``` 20 | 21 | ## Philosophy 22 | 23 | Unlike other language plugins, the Go 24 | [protobuf plugin](https://github.com/golang/protobuf) expects to be 25 | called separately for each package, ~and given all files in that 26 | package~ (fixed). 27 | 28 | `protowrap` is called instead of `protoc`, and ensures that `.proto` 29 | files are processed one Go package at a time. 30 | 31 | It parses out the flags it understands, passing the rest through to 32 | `protoc` unchanged. 33 | 34 | ## Operation 35 | 36 | - search for all `.proto` files under the same import paths as the 37 | `.proto` file arguments 38 | - call `protoc` to generate FileDescriptorProtos 39 | - inspect the FileDescriptorProtos to deduce package information 40 | - group `.proto` files into packages 41 | - call `protoc` once for each package 42 | 43 | ## TODOs 44 | 45 | - [x] Replace square-specific handling of `go_package` with 46 | recently updated upstream logic. 47 | - [ ] In the initial call to `protoc` for generating 48 | FileDescriptorProtos, pass `.proto` files to `protoc` in batches 49 | instead of all at once. 50 | - [ ] Better tests, especially of the code paths not exercised by our 51 | build process. 52 | 53 | ## License 54 | 55 | Copyright 2016 Square, Inc. 56 | 57 | Licensed under the Apache License, Version 2.0 (the "License"); you 58 | may not use this file except in compliance with the License. You may 59 | obtain a copy of the License at 60 | 61 | http://www.apache.org/licenses/LICENSE-2.0 62 | 63 | Unless required by applicable law or agreed to in writing, software 64 | distributed under the License is distributed on an "AS IS" BASIS, 65 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 66 | implied. See the License for the specific language governing 67 | permissions and limitations under the License. 68 | 69 | 70 | -------------------------------------------------------------------------------- /wrapper/generate.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Square, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // generate.go contains the code that does the actual generation. 16 | 17 | package wrapper 18 | 19 | import ( 20 | "bufio" 21 | "fmt" 22 | "os" 23 | "os/exec" 24 | "regexp" 25 | "sort" 26 | "strings" 27 | ) 28 | 29 | // Generate does the actual generation of protos. 30 | func Generate(pkg *PackageInfo, importDirs []string, protocCommand string, protocFlags []string, printOnly bool) (err error) { 31 | args := protocFlags[0:len(protocFlags):len(protocFlags)] 32 | 33 | files := make([]string, 0, len(pkg.Files)) 34 | for _, f := range pkg.Files { 35 | files = append(files, f.FullPath) 36 | } 37 | sort.Strings(files) 38 | args = append(args, files...) 39 | 40 | if printOnly { 41 | fmt.Printf("%s %s\n", protocCommand, strings.Join(args, " ")) 42 | return nil 43 | } 44 | cmd := exec.Command(protocCommand, args...) 45 | out, err := cmd.CombinedOutput() 46 | if err != nil { 47 | cmdline := fmt.Sprintf("%s %s\n", protocCommand, strings.Join(args, " ")) 48 | return fmt.Errorf("error running %v\n%v\nOutput:\n======\n%s======\n", cmdline, err, out) 49 | } 50 | return nil 51 | } 52 | 53 | var packageRe = regexp.MustCompile(`^package [\p{L}_][\p{L}\p{N}_]*`) 54 | 55 | // CopyAndChangePackage copies file `in` to file `out`, rewriting the 56 | // `package` declaration to `pkg`. 57 | func CopyAndChangePackage(in, out, pkg string) error { 58 | inf, err := os.Open(in) 59 | if err != nil { 60 | return err 61 | } 62 | defer inf.Close() 63 | outf, err := os.Create(out) 64 | if err != nil { 65 | return err 66 | } 67 | defer outf.Close() 68 | scanner := bufio.NewScanner(inf) 69 | matched := false 70 | for scanner.Scan() { 71 | line := scanner.Text() 72 | if !matched && packageRe.MatchString(line) { 73 | matched = true 74 | line = packageRe.ReplaceAllString(line, fmt.Sprintf("package %s", pkg)) 75 | } 76 | fmt.Fprintln(outf, line) 77 | } 78 | if err := scanner.Err(); err != nil { 79 | return err 80 | } 81 | return nil 82 | } 83 | -------------------------------------------------------------------------------- /wrapper/finding.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Square, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // finding.go contains the functions responsible for the building the 16 | // initial list of protobufs to build, and the list of import path 17 | // directories that contain them. 18 | 19 | package wrapper 20 | 21 | import ( 22 | "os" 23 | "path/filepath" 24 | "strings" 25 | ) 26 | 27 | // ProtosBelow returns a slice containing the filenames of all .proto 28 | // files found in or below the given directories. 29 | func ProtosBelow(dirs []string) ([]string, error) { 30 | protos := []string{} 31 | for _, dir := range dirs { 32 | err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error { 33 | if err != nil { 34 | return err 35 | } 36 | if !info.IsDir() && strings.HasSuffix(info.Name(), ".proto") { 37 | protos = append(protos, path) 38 | } 39 | return nil 40 | }) 41 | if err != nil { 42 | return nil, err 43 | } 44 | } 45 | return protos, nil 46 | } 47 | 48 | // ImportDirsUsed returns the set of import directories that contain 49 | // entries in the set of proto files. 50 | func ImportDirsUsed(importDirs []string, protos []string) []string { 51 | used := []string{} 52 | for _, imp := range importDirs { 53 | for _, proto := range protos { 54 | if strings.HasPrefix(proto, imp) { 55 | used = append(used, imp) 56 | break 57 | } 58 | } 59 | } 60 | return used 61 | } 62 | 63 | // Disjoint takes a slice of existing .proto files, and a slice of new 64 | // .proto files. It returns a slice containing the subset of the new 65 | // .proto files with distinct paths not in the first set. 66 | func Disjoint(existing, additional []string) []string { 67 | set := make(map[string]bool, len(existing)+len(additional)) 68 | result := make([]string, 0, len(additional)) 69 | for _, proto := range existing { 70 | set[proto] = true 71 | } 72 | 73 | for _, proto := range additional { 74 | if set[proto] { 75 | continue 76 | } 77 | set[proto] = true 78 | result = append(result, proto) 79 | } 80 | return result 81 | } 82 | -------------------------------------------------------------------------------- /cmd/cyclecheck/cyclecheck.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Square, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // Binary cyclecheck looks for situations where imports in .proto 16 | // files would result in circular package dependencies in the 17 | // generated Go code. 18 | package main 19 | 20 | import ( 21 | "fmt" 22 | "os" 23 | 24 | "github.com/square/goprotowrap" 25 | "github.com/square/goprotowrap/wrapper" 26 | ) 27 | 28 | // customFlags is a map describing flags we add to protoc. true means 29 | // a value is required. false implies boolean. 30 | var customFlags = map[string]bool{ 31 | "print_structure": false, 32 | "protoc_command": true, 33 | "only_specified_files": false, 34 | "version": false, 35 | } 36 | 37 | func usageAndExit(format string, args ...interface{}) { 38 | fmt.Fprintf(os.Stderr, format, args...) 39 | fmt.Fprintf(os.Stderr, "Usage: %s [flags] [protofiles]\n", os.Args[0]) 40 | fmt.Fprintf(os.Stderr, ` --only_specified_files true|false 41 | if true, don't search the nearest import path ancestor for other .proto files 42 | --protoc_command string 43 | command to use to call protoc (default "protoc") 44 | --print_structure 45 | if true, print out computed package structure 46 | `) 47 | os.Exit(1) 48 | } 49 | 50 | func main() { 51 | flags, protocFlags, protos, importDirs, err := wrapper.ParseArgs(os.Args[1:], customFlags) 52 | if err != nil { 53 | usageAndExit("Error: %v\n", err) 54 | } 55 | if flags.Has("version") { 56 | fmt.Println(goprotowrap.Version) 57 | os.Exit(0) 58 | } 59 | if len(importDirs) == 0 { 60 | usageAndExit("Error: at least one import directory (-I) needed\n") 61 | } 62 | 63 | noExpand, err := flags.Bool("only_specified_files", false) 64 | if err != nil { 65 | usageAndExit("Error: %v\n", err) 66 | } 67 | printStructure, err := flags.Bool("print_structure", false) 68 | if err != nil { 69 | usageAndExit("Error: %v\n", err) 70 | } 71 | 72 | w := &wrapper.Wrapper{ 73 | ProtocCommand: flags.String("protoc_command", "protoc"), 74 | ProtocFlags: protocFlags, 75 | ProtoFiles: protos, 76 | ImportDirs: importDirs, 77 | NoExpand: noExpand, 78 | } 79 | err = w.Init() 80 | if err != nil { 81 | fmt.Fprintf(os.Stderr, "Error: %v\n", err) 82 | os.Exit(1) 83 | } 84 | 85 | // Debugging output. 86 | if printStructure { 87 | w.PrintStructure(os.Stdout) 88 | } 89 | 90 | if err := w.CheckCycles(); err != nil { 91 | fmt.Fprintf(os.Stderr, "Error: %v", err) 92 | os.Exit(2) 93 | } 94 | } 95 | -------------------------------------------------------------------------------- /wrapper/flags_test.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Square, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | package wrapper 16 | 17 | import ( 18 | "strings" 19 | "testing" 20 | ) 21 | 22 | func TestParseArgs(t *testing.T) { 23 | // func ParseArgs(args []string, custom map[string]bool) (customFlags map[string]string, protocFlags, protos, importDirs []string, err error) { 24 | 25 | realCustom := map[string]bool{ 26 | "parallelism": true, 27 | "print_structure": false, 28 | "protoc_command": true, 29 | "only_specified_files": false, 30 | } 31 | 32 | tests := map[string]struct { 33 | args string 34 | custom map[string]bool 35 | customFlags map[string]string 36 | protocFlags []string 37 | protos []string 38 | importDirs []string 39 | err bool 40 | }{ 41 | "basic": { 42 | "-I. foo1.proto -I includes foo2.proto --go-square_out=plugins=sake+grpc,import_prefix=foo:output/protos", 43 | realCustom, 44 | nil, 45 | []string{"-I.", "-I", "includes", "--go-square_out=plugins=sake+grpc,import_prefix=foo:output/protos"}, 46 | []string{"foo1.proto", "foo2.proto"}, 47 | []string{".", "includes"}, 48 | false, 49 | }, 50 | "real custom args": { 51 | "-I. foo1.proto --parallelism=3 --only_specified_files=false --print_structure --protoc_command protoc", 52 | realCustom, 53 | map[string]string{ 54 | "parallelism": "3", 55 | "only_specified_files": "false", 56 | "print_structure": "", 57 | "protoc_command": "protoc", 58 | }, 59 | []string{"-I."}, 60 | []string{"foo1.proto"}, 61 | []string{"."}, 62 | false, 63 | }, 64 | "missing flag value": { 65 | "-I. foo1.proto --parallelism=3 --only_specified_files=false --print_structure --protoc_command", 66 | realCustom, 67 | nil, 68 | nil, 69 | nil, 70 | nil, 71 | true, 72 | }, 73 | } 74 | 75 | for name, tt := range tests { 76 | if name[0] == '_' { 77 | continue 78 | } 79 | cf, pf, p, id, err := ParseArgs(strings.Split(tt.args, " "), tt.custom) 80 | if tt.err { 81 | if err == nil { 82 | t.Errorf("%q: want error; got nil", name) 83 | } 84 | continue 85 | } 86 | if err != nil { 87 | t.Errorf("%q: unexpected error: %v", name, err) 88 | continue 89 | } 90 | 91 | if (tt.err && err == nil) || (!tt.err && err != nil) { 92 | t.Errorf("%q: want error=%v; got %v", name, tt.err, err) 93 | } 94 | if !mapStringStringEqual(cf, tt.customFlags) { 95 | t.Errorf("%q: want customFlags=%v; got %v", name, tt.customFlags, cf) 96 | } 97 | if !sliceStringEqual(pf, tt.protocFlags) { 98 | t.Errorf("%q: want protocFlags=%v; got %v", name, tt.protocFlags, pf) 99 | } 100 | if !sliceStringEqual(p, tt.protos) { 101 | t.Errorf("%q: want protos=%v; got %v", name, tt.protos, p) 102 | } 103 | if !sliceStringEqual(id, tt.importDirs) { 104 | t.Errorf("%q: want importDirs=%v; got %v", name, tt.importDirs, id) 105 | } 106 | } 107 | } 108 | -------------------------------------------------------------------------------- /wrapper/cycles.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Square, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | package wrapper 16 | 17 | import ( 18 | "fmt" 19 | "strings" 20 | ) 21 | 22 | // CheckCycles checks for proto import structures that would result in 23 | // Go package cycles. 24 | func (w *Wrapper) CheckCycles() error { 25 | w.sccs = w.tarjan() 26 | cycles := []string{} 27 | for _, scc := range w.sccs { 28 | if len(scc) > 1 { 29 | cycles = append(cycles, w.showComponent(scc)) 30 | } 31 | } 32 | if len(cycles) > 0 { 33 | return fmt.Errorf("cycles found:\n%s\n", strings.Join(cycles, "\n")) 34 | } 35 | return nil 36 | } 37 | 38 | // https://en.wikipedia.org/wiki/Tarjan%27s_SCC_algorithm 39 | func (wr *Wrapper) tarjan() [][]*PackageInfo { 40 | index := 1 41 | sccs := [][]*PackageInfo{} 42 | s := []*PackageInfo{} 43 | 44 | var strongConnect func(*PackageInfo) 45 | strongConnect = func(v *PackageInfo) { 46 | // Set the depth index for v to the smallest unused index. 47 | v.index = index 48 | v.lowlink = index 49 | index++ 50 | s = append(s, v) 51 | v.onStack = true 52 | 53 | // Consider successors of v. 54 | for _, wName := range v.ImportedPackageComputedNames() { 55 | w, ok := wr.allPackages[wName] 56 | if !ok { 57 | panic(fmt.Sprintf("%q not found in %v", wName, wr.allPackages)) 58 | } 59 | if w.index == 0 { 60 | // Successor w has not yet been visited; recurse on it 61 | strongConnect(w) 62 | if w.lowlink < v.lowlink { 63 | v.lowlink = w.lowlink 64 | } 65 | } else { 66 | if w.onStack { 67 | // Successor w is in stack s and hence in the current SCC 68 | if w.index < v.lowlink { 69 | v.lowlink = w.index 70 | } 71 | } 72 | } 73 | } 74 | 75 | // If v is a root node, pop the stack and generate an SCC 76 | if v.lowlink == v.index { 77 | scc := []*PackageInfo{} 78 | var w *PackageInfo 79 | for w != v { 80 | w = s[len(s)-1] 81 | s = s[:len(s)-1] 82 | w.onStack = false 83 | scc = append(scc, w) 84 | } 85 | sccs = append(sccs, scc) 86 | } 87 | } 88 | 89 | for _, pkg := range wr.packagesInOrder() { 90 | if pkg.index == 0 { 91 | strongConnect(pkg) 92 | } 93 | } 94 | 95 | return sccs 96 | } 97 | 98 | // showComponent returns a string describing why a strongly connected 99 | // component is strongly connected. 100 | func (w *Wrapper) showComponent(pkgs []*PackageInfo) string { 101 | result := []string{} 102 | inCycle := map[string]bool{} 103 | for _, pkg := range pkgs { 104 | inCycle[pkg.ComputedPackage] = true 105 | } 106 | for _, pkg := range pkgs { 107 | for _, other := range pkg.ImportedPackageComputedNames() { 108 | if inCycle[other] { 109 | result = append(result, fmt.Sprintf(" %s --> %s", pkg.ComputedPackage, other)) 110 | for _, f := range pkg.Files { 111 | for _, depName := range f.Deps { 112 | dep := w.infos[depName] 113 | if dep.ComputedPackage == other { 114 | result = append(result, fmt.Sprintf(" %s imports %s", f.Name, dep.Name)) 115 | } 116 | } 117 | } 118 | } 119 | } 120 | } 121 | return strings.Join(result, "\n") 122 | } 123 | -------------------------------------------------------------------------------- /cmd/protowrap/protowrap.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Square, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // Binary protoc-wrapper calls protoc, with our plugins, but using 16 | // commandline flags to move things around instead of forking 17 | // protoc-gen-go. 18 | package main 19 | 20 | import ( 21 | "fmt" 22 | "os" 23 | 24 | "github.com/square/goprotowrap" 25 | "github.com/square/goprotowrap/wrapper" 26 | ) 27 | 28 | // customFlags is a map describing flags we add to protoc. true means 29 | // a value is required. false implies boolean. 30 | var customFlags = map[string]bool{ 31 | "parallelism": true, 32 | "print_structure": false, 33 | "protoc_command": true, 34 | "only_specified_files": false, 35 | "print_only": false, 36 | "version": false, 37 | } 38 | 39 | func usageAndExit(format string, args ...interface{}) { 40 | fmt.Fprintf(os.Stderr, format, args...) 41 | fmt.Fprintf(os.Stderr, "Usage: %s [flags] [protofiles]\n", os.Args[0]) 42 | fmt.Fprintf(os.Stderr, ` --only_specified_files true|false 43 | if true, don't search the nearest import path ancestor for other .proto files 44 | --parallelism int 45 | parallelism when generating (default 5) 46 | --protoc_command string 47 | command to use to call protoc (default "protoc") 48 | --print_structure 49 | if true, print out computed package structure 50 | --print_only 51 | if true, print protoc commandlines instead of generating protos 52 | --version 53 | print version and exit 54 | @file 55 | read command line arguments from the named file. Each line of the file 56 | will become a single argument at the position where @file is used. 57 | `) 58 | os.Exit(1) 59 | } 60 | 61 | func main() { 62 | flags, protocFlags, protos, importDirs, err := wrapper.ParseArgs(os.Args[1:], customFlags) 63 | if err != nil { 64 | usageAndExit("Error: %v\n", err) 65 | } 66 | if flags.Has("version") { 67 | fmt.Println(goprotowrap.Version) 68 | os.Exit(0) 69 | } 70 | if len(importDirs) == 0 { 71 | usageAndExit("Error: at least one import directory (-I) needed\n") 72 | } 73 | 74 | noExpand, err := flags.Bool("only_specified_files", false) 75 | if err != nil { 76 | usageAndExit("Error: %v\n", err) 77 | } 78 | parallelism, err := flags.Int("parallelism", 5) 79 | if err != nil { 80 | usageAndExit("Error: %v\n", err) 81 | } 82 | printStructure, err := flags.Bool("print_structure", false) 83 | if err != nil { 84 | usageAndExit("Error: %v\n", err) 85 | } 86 | printOnly, err := flags.Bool("print_only", false) 87 | if err != nil { 88 | usageAndExit("Error: %v\n", err) 89 | } 90 | 91 | w := &wrapper.Wrapper{ 92 | ProtocCommand: flags.String("protoc_command", "protoc"), 93 | ProtocFlags: protocFlags, 94 | ProtoFiles: protos, 95 | ImportDirs: importDirs, 96 | NoExpand: noExpand, 97 | Parallelism: parallelism, 98 | PrintOnly: printOnly, 99 | } 100 | err = w.Init() 101 | if err != nil { 102 | fmt.Fprintf(os.Stderr, "Error: %v\n", err) 103 | os.Exit(1) 104 | } 105 | 106 | // Debugging output. 107 | if printStructure { 108 | w.PrintStructure(os.Stdout) 109 | } 110 | 111 | if err := w.CheckCycles(); err != nil { 112 | fmt.Fprintf(os.Stderr, "Error: %v\n", err) 113 | os.Exit(2) 114 | } 115 | 116 | if err := w.Generate(); err != nil { 117 | fmt.Fprintf(os.Stderr, "Error generating protos: %v\n", err) 118 | os.Exit(1) 119 | } 120 | } 121 | -------------------------------------------------------------------------------- /wrapper/flags.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Square, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // File flags.go contains code to parse protoc-style commandline 16 | // arguments. 17 | 18 | package wrapper 19 | 20 | import ( 21 | "bufio" 22 | "fmt" 23 | "os" 24 | "strconv" 25 | "strings" 26 | ) 27 | 28 | // Flags that take no values. See 29 | // https://github.com/google/protobuf/blob/5e933847/src/google/protobuf/compiler/command_line_interface.cc#L1069 30 | var noValueFlags = map[string]bool{ 31 | "-h": true, 32 | "--help": true, 33 | "--disallow_services": true, 34 | "--include_imports": true, 35 | "--include_source_info": true, 36 | "--decode_raw": true, 37 | "--print_free_field_numbers": true, 38 | } 39 | 40 | // Flag values is a simple map of parsed flag values. A map of string 41 | // to string, with convenience getters. 42 | type FlagValues map[string]string 43 | 44 | // ParseArgs parses protoc-style commandline arguments, splitting them 45 | // into custom flags, protoc flags and input files, and capturing a 46 | // list of import directories. Custom flag names are passed without 47 | // dashes, and are expected to be specified with two dashes. If 48 | // customFlagNames[name] is true, the custom flag expects a value; 49 | // otherwise it can have no value, and will get a value of "". 50 | func ParseArgs(args []string, custom map[string]bool) (customFlags FlagValues, protocFlags, protos, importDirs []string, err error) { 51 | customFlags = make(FlagValues) 52 | 53 | // Support protoc-style argument files starting with '@' 54 | fullArgs := make([]string, 0, len(args)) 55 | for _, arg := range args { 56 | if len(arg) > 0 && arg[0] == '@' { 57 | fileArgs, err := expandArgumentFile(arg[1:]) 58 | if err != nil { 59 | return nil, nil, nil, nil, err 60 | } 61 | fullArgs = append(fullArgs, fileArgs...) 62 | } else { 63 | fullArgs = append(fullArgs, arg) 64 | } 65 | } 66 | 67 | var nextIsFlag, nextIsCustomFlag, nextIsImportDir bool 68 | var customName string 69 | for _, arg := range fullArgs { 70 | // Catch empty, "-" and "--" arguments. See 71 | // https://github.com/google/protobuf/blob/5e933847/src/google/protobuf/compiler/command_line_interface.cc#L1049 72 | if arg == "" || arg == "-" || arg == "--" { 73 | return nil, nil, nil, nil, fmt.Errorf("flag %q not allowed", arg) 74 | } 75 | 76 | if nextIsCustomFlag { 77 | customFlags[customName] = arg 78 | nextIsCustomFlag = false 79 | continue 80 | } 81 | if nextIsFlag { 82 | protocFlags = append(protocFlags, arg) 83 | nextIsFlag = false 84 | if nextIsImportDir { 85 | importDirs = append(importDirs, strings.Split(arg, string(os.PathListSeparator))...) 86 | nextIsImportDir = false 87 | } 88 | continue 89 | } 90 | if noValueFlags[arg] { 91 | protocFlags = append(protocFlags, arg) 92 | continue 93 | } 94 | if arg[0] != '-' { 95 | protos = append(protos, arg) 96 | continue 97 | } 98 | 99 | // Two dashes. Expect "=" or second arg for value. 100 | if arg[1] == '-' { 101 | parts := strings.SplitN(arg[2:], "=", 2) 102 | name := parts[0] 103 | needsValue, isCustom := custom[name] 104 | if isCustom { 105 | if len(parts) == 1 { 106 | if needsValue { 107 | nextIsCustomFlag = true 108 | customName = name 109 | } else { 110 | customFlags[name] = "" 111 | } 112 | } else { 113 | customFlags[name] = parts[1] 114 | } 115 | continue 116 | } 117 | protocFlags = append(protocFlags, arg) 118 | if len(parts) == 1 { 119 | nextIsFlag = true 120 | } 121 | continue 122 | } 123 | 124 | protocFlags = append(protocFlags, arg) 125 | // One dash. Expect single-char flag with value concatenated, or second arg for value. 126 | // Capture import directory (-I) values separately. 127 | if len(arg) == 2 { 128 | nextIsFlag = true 129 | nextIsImportDir = arg[1] == 'I' 130 | continue 131 | } 132 | if arg[1] == 'I' { 133 | importDirs = append(importDirs, strings.Split(arg[2:], string(os.PathListSeparator))...) 134 | } 135 | } 136 | if nextIsFlag || nextIsCustomFlag { 137 | return nil, nil, nil, nil, fmt.Errorf("%q flag with no value", fullArgs[len(fullArgs)-1]) 138 | } 139 | return customFlags, protocFlags, protos, importDirs, nil 140 | } 141 | 142 | // Int returns the integer version of a flag, if set. 143 | func (fv FlagValues) Int(name string, defaultValue int) (int, error) { 144 | value, found := fv[name] 145 | if !found { 146 | return defaultValue, nil 147 | } 148 | i, err := strconv.Atoi(value) 149 | if err != nil { 150 | return 0, fmt.Errorf("flag %q: cannot parse integer from %q", name, value) 151 | } 152 | return i, nil 153 | } 154 | 155 | // Bool returns the boolean version of a flag, if set. 156 | func (fv FlagValues) Bool(name string, defaultValue bool) (bool, error) { 157 | value, found := fv[name] 158 | if !found { 159 | return defaultValue, nil 160 | } 161 | switch value { 162 | case "", "t", "T", "true", "True", "1": 163 | return true, nil 164 | case "f", "F", "false", "False", "0": 165 | return false, nil 166 | } 167 | return false, fmt.Errorf("flag %q: cannot parse boolean from %q", name, value) 168 | } 169 | 170 | // Has returns true if the given flag was specified at all. 171 | func (fv FlagValues) Has(name string) bool { 172 | _, found := fv[name] 173 | return found 174 | } 175 | 176 | // String returns the string version of a flag, if set. 177 | func (fv FlagValues) String(name string, defaultValue string) string { 178 | value, found := fv[name] 179 | if !found { 180 | return defaultValue 181 | } 182 | return value 183 | } 184 | 185 | // expandArgumentFile reads additional command line argument from a file. 186 | func expandArgumentFile(filename string) ([]string, error) { 187 | f, err := os.Open(filename) 188 | if err != nil { 189 | return nil, err 190 | } 191 | defer f.Close() 192 | 193 | var args []string 194 | scanner := bufio.NewScanner(f) 195 | for scanner.Scan() { 196 | args = append(args, scanner.Text()) 197 | } 198 | if err := scanner.Err(); err != nil { 199 | return nil, fmt.Errorf("%s: %w", filename, err) 200 | } 201 | 202 | return args, nil 203 | } 204 | -------------------------------------------------------------------------------- /wrapper/wrapper.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Square, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // Package wrapper implements the actual functionality of wrapping 16 | // protoc. 17 | package wrapper 18 | 19 | import ( 20 | "errors" 21 | "fmt" 22 | "io" 23 | "os" 24 | "sort" 25 | "strings" 26 | "sync" 27 | ) 28 | 29 | // defaultProtocCommand is the default command used to call protoc. 30 | const defaultProtocCommand = "protoc" 31 | 32 | // The wrapper object. 33 | type Wrapper struct { 34 | ProtocCommand string // The command to call to run protoc. 35 | Parallelism int // Number of simultaneous calls to make to protoc when generating. 36 | ProtocFlags []string // Flags to pass to protoc. 37 | ImportDirs []string // Base directories in which .proto files reside. 38 | ProtoFiles []string // The list of .proto files to generate code for. 39 | NoExpand bool // If true, don't search for other protos in import directories. 40 | PrintOnly bool // If true, don't generate: just print the protoc commandlines that would be called. 41 | 42 | allProtos []string // All proto files: those specified, plus those found alongside them. 43 | infos map[string]*FileInfo // A map of filename to FileInfo struct for all proto files we care about in this run. 44 | packages map[string]*PackageInfo // A list of PackageInfo structs for packages containing files we care about. 45 | allPackages map[string]*PackageInfo // A list of PackageInfo structs for all packages. 46 | 47 | initCalled bool // Has Init() been called? 48 | 49 | // Used internally for checking for cycles and topologically sorting 50 | sccs [][]*PackageInfo // Slice of strongly-connected components in the package graph. 51 | } 52 | 53 | // Init must be called before any of the methods that do anything. 54 | func (w *Wrapper) Init() error { 55 | if len(w.ImportDirs) == 0 { 56 | return errors.New("at least one import directory required") 57 | } 58 | for _, importDir := range w.ImportDirs { 59 | stat, err := os.Stat(importDir) 60 | if err != nil { 61 | return fmt.Errorf("Nonexistent import directory: %q", importDir) 62 | } 63 | if !stat.IsDir() { 64 | return fmt.Errorf("Non-directory import directory: %q", importDir) 65 | } 66 | } 67 | 68 | if len(w.ProtoFiles) == 0 { 69 | return errors.New("at least one input .proto file is required") 70 | } 71 | for _, file := range w.ProtoFiles { 72 | if !strings.HasSuffix(file, ".proto") { 73 | return fmt.Errorf("non-proto input file: %q", file) 74 | } 75 | if !w.inImportDir(file) { 76 | return fmt.Errorf("proto file %q must have a lexicographical prefix of one of the import directories", file) 77 | } 78 | if _, err := os.Stat(file); os.IsNotExist(err) { 79 | return fmt.Errorf("input %q does not exist", file) 80 | } 81 | } 82 | 83 | if w.ProtocCommand == "" { 84 | w.ProtocCommand = defaultProtocCommand 85 | } 86 | 87 | // Get the list of actually-used import directories. 88 | dirs := w.importDirsUsed() 89 | 90 | expanded := []string{} 91 | // Unless asked not to, find all proto files with common import directory ancestors. 92 | if !w.NoExpand { 93 | neighbors, err := ProtosBelow(dirs) 94 | if err != nil { 95 | return err 96 | } 97 | expanded = Disjoint(w.ProtoFiles, neighbors) 98 | } 99 | 100 | w.allProtos = make([]string, len(w.ProtoFiles), len(w.ProtoFiles)+len(expanded)) 101 | copy(w.allProtos, w.ProtoFiles) 102 | w.allProtos = append(w.allProtos, expanded...) 103 | var err error 104 | w.infos, err = GetFileInfos(w.ImportDirs, w.allProtos, w.ProtocCommand) 105 | if err != nil { 106 | return fmt.Errorf("cannot get .proto file information: %v", err) 107 | } 108 | 109 | AnnotateFullPaths(w.infos, w.allProtos, w.ImportDirs) 110 | ComputeGoLocations(w.infos) 111 | 112 | neededPackages := map[string]struct{}{} 113 | for _, proto := range w.ProtoFiles { 114 | info, ok := w.infos[FileDescriptorName(proto, w.ImportDirs)] 115 | if !ok { 116 | return fmt.Errorf("missing file info for %q.\n", proto) 117 | } 118 | neededPackages[info.ComputedPackage] = struct{}{} 119 | } 120 | 121 | w.allPackages, err = CollectPackages(w.infos, w.ProtoFiles, w.ImportDirs) 122 | if err != nil { 123 | return fmt.Errorf("cannot collect package information: %v", err) 124 | } 125 | 126 | w.packages = map[string]*PackageInfo{} 127 | for pkgName := range neededPackages { 128 | pkg, ok := w.allPackages[pkgName] 129 | if !ok { 130 | return fmt.Errorf("cannot find package information for %q", pkgName) 131 | } 132 | w.packages[pkgName] = pkg 133 | } 134 | 135 | w.initCalled = true 136 | return nil 137 | } 138 | 139 | // inImportDir returns true if the given file has a lexicographical 140 | // prefix of one of the import directories. 141 | func (w *Wrapper) inImportDir(file string) bool { 142 | for _, imp := range w.ImportDirs { 143 | if strings.HasPrefix(file, imp) { 144 | return true 145 | } 146 | } 147 | return false 148 | } 149 | 150 | // importDirsUsed returns the set of import directories that contain 151 | // entries in the set of proto files. 152 | func (w *Wrapper) importDirsUsed() []string { 153 | used := []string{} 154 | for _, imp := range w.ImportDirs { 155 | for _, proto := range w.ProtoFiles { 156 | if strings.HasPrefix(proto, imp) { 157 | used = append(used, imp) 158 | break 159 | } 160 | } 161 | } 162 | return used 163 | } 164 | 165 | // PrintStructure dumps out the computed structure to the given 166 | // io.Writer. 167 | func (w *Wrapper) PrintStructure(writer io.Writer) { 168 | if !w.initCalled { 169 | fmt.Fprintln(writer, "[Not initialized]") 170 | return 171 | } 172 | // Debugging output. 173 | fmt.Fprintln(writer, "> Structure:") 174 | for _, pkg := range w.packagesInOrder() { 175 | fmt.Fprintf(writer, "> %v\n", pkg.ComputedPackage) 176 | fmt.Fprintln(writer, "> files:") 177 | for _, file := range pkg.Files { 178 | fmt.Fprintf(writer, "> %v (%v)\n", file.Name, file.FullPath) 179 | } 180 | fmt.Fprintln(writer, "> deps:") 181 | for _, file := range pkg.Deps { 182 | if file.FullPath != "" { 183 | fmt.Fprintf(writer, "> %v (%v)\n", file.Name, file.FullPath) 184 | } else { 185 | fmt.Fprintf(writer, "> %v\n", file.Name) 186 | } 187 | } 188 | } 189 | } 190 | 191 | // Generate actually generates the output files. 192 | func (w *Wrapper) Generate() error { 193 | if !w.initCalled { 194 | return errors.New("Init() must be called before Generate()") 195 | } 196 | if w.Parallelism < 1 { 197 | return fmt.Errorf("parallelism cannot be < 1; got %d", w.Parallelism) 198 | } 199 | parallelism := len(w.packages) 200 | if w.Parallelism < parallelism { 201 | parallelism = w.Parallelism 202 | } 203 | 204 | pkgChan := make(chan *PackageInfo) 205 | 206 | errChan := make(chan error, parallelism) 207 | var wg sync.WaitGroup 208 | wg.Add(parallelism) 209 | for i := 0; i < parallelism; i++ { 210 | go func() { 211 | for pkg := range pkgChan { 212 | fmt.Printf("Generating package %s\n", pkg.ComputedPackage) 213 | if err := Generate(pkg, w.ImportDirs, w.ProtocCommand, w.ProtocFlags, w.PrintOnly); err != nil { 214 | errChan <- fmt.Errorf("error generating package %s: %v\n", pkg.ComputedPackage, err) 215 | } 216 | } 217 | wg.Done() 218 | }() 219 | } 220 | 221 | var err error 222 | OUTER: 223 | for _, pkg := range w.packagesInOrder() { 224 | select { 225 | case pkgChan <- pkg: 226 | case err = <-errChan: 227 | break OUTER 228 | } 229 | } 230 | close(pkgChan) 231 | wg.Wait() 232 | select { 233 | case err = <-errChan: 234 | default: 235 | } 236 | return err 237 | } 238 | 239 | // packagesInOrder returns the list of packages, sorted by name. 240 | func (w *Wrapper) packagesInOrder() []*PackageInfo { 241 | result := make([]*PackageInfo, 0, len(w.packages)) 242 | names := make([]string, 0, len(w.packages)) 243 | for name := range w.packages { 244 | names = append(names, name) 245 | } 246 | sort.Strings(names) 247 | for _, name := range names { 248 | result = append(result, w.packages[name]) 249 | } 250 | return result 251 | } 252 | 253 | // allPackagesInOrder returns the list of all packages, sorted by name. 254 | func (w *Wrapper) allPackagesInOrder() []*PackageInfo { 255 | result := make([]*PackageInfo, 0, len(w.allPackages)) 256 | names := make([]string, 0, len(w.allPackages)) 257 | for name := range w.allPackages { 258 | names = append(names, name) 259 | } 260 | sort.Strings(names) 261 | for _, name := range names { 262 | result = append(result, w.allPackages[name]) 263 | } 264 | return result 265 | } 266 | -------------------------------------------------------------------------------- /wrapper/packages.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Square, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // packages.go contains the code needed to figure out what packages we 16 | // want .proto files to generate into. It calls out to protoc to parse 17 | // the .proto files in to a FileDescriptorSet. 18 | 19 | package wrapper 20 | 21 | import ( 22 | "fmt" 23 | "io/ioutil" 24 | "os" 25 | "os/exec" 26 | "path" 27 | "path/filepath" 28 | "strings" 29 | "unicode" 30 | 31 | "github.com/golang/protobuf/proto" 32 | "github.com/golang/protobuf/protoc-gen-go/descriptor" 33 | ) 34 | 35 | // FileInfo is the set of information we need to know about a file from protoc. 36 | type FileInfo struct { 37 | Name string // The Name field of the FileDescriptorProto (import-path-relative) 38 | FullPath string // The full path to the file, as specified on the command-line 39 | Package string // The declared package 40 | GoPackage string // The declared go_package 41 | Deps []string // The names of files imported by this file (import-path-relative) 42 | 43 | // Our final decision for which package this file should generate 44 | // to. In the full form "path;decl" (whether decl is redundant or 45 | // not) as described in github.com/golang/protobuf/issues/139 46 | ComputedPackage string // Our final decision for which package this file should generate to 47 | } 48 | 49 | // PackageDir returns the desired directory location for the given 50 | // file; ComputedPackage, with dots replaced by slashes. 51 | func (f FileInfo) PackageDir() string { 52 | parts := strings.Split(f.ComputedPackage, ".") 53 | return filepath.Join(parts...) 54 | } 55 | 56 | // GoPluginOutputFilename returns the filename the vanilla go protoc 57 | // plugin will use when generating output for this file. 58 | func (f FileInfo) GoPluginOutputFilename() string { 59 | name := f.Name 60 | ext := path.Ext(name) 61 | if ext == ".proto" || ext == ".protodevel" { 62 | name = name[0 : len(name)-len(ext)] 63 | } 64 | return name + ".pb.go" 65 | } 66 | 67 | // PackageInfo collects all the information for a single package. 68 | type PackageInfo struct { 69 | ComputedPackage string 70 | Files []*FileInfo 71 | Deps []*FileInfo 72 | 73 | // Internal use for cycle-checking and topological sorting. 74 | index int 75 | lowlink int 76 | onStack bool 77 | } 78 | 79 | // PackageDir returns the desired directory location for the given 80 | // package; ComputedPackage, with dots replaced by slashes. 81 | func (p PackageInfo) PackageDir() string { 82 | parts := strings.Split(p.ComputedPackage, ".") 83 | return filepath.Join(parts...) 84 | } 85 | 86 | // PackageName returns the desired package name for the given package; 87 | // whatever follows the last dot in ComputedPackage. 88 | func (p PackageInfo) PackageName() string { 89 | parts := strings.Split(p.ComputedPackage, ".") 90 | return parts[len(parts)-1] 91 | } 92 | 93 | //ImportedPackageComputedNames returns the list of packages imported by this 94 | //package. 95 | func (p PackageInfo) ImportedPackageComputedNames() []string { 96 | result := []string{} 97 | seen := map[string]bool{ 98 | p.ComputedPackage: true, 99 | } 100 | for _, d := range p.Deps { 101 | pkg := d.ComputedPackage 102 | if seen[pkg] { 103 | continue 104 | } 105 | seen[pkg] = true 106 | result = append(result, pkg) 107 | } 108 | return result 109 | } 110 | 111 | // GetFileInfos gets the FileInfo struct for every proto passed in. 112 | func GetFileInfos(importPaths []string, protos []string, protocCommand string) (info map[string]*FileInfo, err error) { 113 | if len(importPaths) == 0 { 114 | return nil, fmt.Errorf("GetFileInfos: empty importPaths") 115 | } 116 | if len(protos) == 0 { 117 | return nil, fmt.Errorf("GetFileInfos: empty protos") 118 | } 119 | info = map[string]*FileInfo{} 120 | 121 | var dir string 122 | dir, err = ioutil.TempDir("", "filedescriptors") 123 | if err != nil { 124 | return nil, err 125 | } 126 | defer func() { 127 | if err2 := os.RemoveAll(dir); err != nil && err2 != nil { 128 | err = err2 129 | } 130 | }() 131 | 132 | descriptorFilename := filepath.Join(dir, "all.pb") 133 | 134 | args := []string{} 135 | for _, importPath := range importPaths { 136 | args = append(args, "-I", importPath) 137 | } 138 | args = append(args, "--descriptor_set_out="+descriptorFilename) 139 | 140 | args = append(args, "--include_imports") 141 | 142 | if len(protos) <= 1000 { 143 | // For a small number of protos (arbitrarily picked size), pass files 144 | // to protoc in the command argv 145 | args = append(args, protos...) 146 | } else { 147 | // For a large number of protos, use a file containing the arguments 148 | argfile := filepath.Join(dir, "protoc-args") 149 | if err = ioutil.WriteFile(argfile, []byte(strings.Join(protos, "\n")), 0666); err != nil { 150 | return nil, err 151 | } 152 | args = append(args, "@"+argfile) 153 | } 154 | 155 | fmt.Println("Collecting filedescriptors...") 156 | cmd := exec.Command(protocCommand, args...) 157 | out, err := cmd.CombinedOutput() 158 | if err != nil { 159 | cmdline := fmt.Sprintf("%s %s\n", protocCommand, strings.Join(args, " ")) 160 | return nil, fmt.Errorf("error running %v\n%v\nOutput:\n======\n%s======\n", cmdline, err, out) 161 | } 162 | descriptorSetBytes, err := ioutil.ReadFile(descriptorFilename) 163 | if err != nil { 164 | return nil, err 165 | } 166 | 167 | descriptorSet := &descriptor.FileDescriptorSet{} 168 | err = proto.Unmarshal(descriptorSetBytes, descriptorSet) 169 | if err != nil { 170 | return nil, err 171 | } 172 | 173 | for _, fd := range descriptorSet.File { 174 | fi := &FileInfo{ 175 | Name: fd.GetName(), 176 | Package: fd.GetPackage(), 177 | } 178 | for _, dep := range fd.Dependency { 179 | fi.Deps = append(fi.Deps, dep) 180 | } 181 | fi.GoPackage = fd.Options.GetGoPackage() 182 | info[fi.Name] = fi 183 | } 184 | 185 | return info, nil 186 | } 187 | 188 | // ComputeGoLocations uses the package and go_package information to 189 | // figure out the effective Go location and package. It sets 190 | // ComputedPackage to the full form "path;decl" (whether decl is 191 | // redundant or not) as described in 192 | // github.com/golang/protobuf/issues/139 193 | func ComputeGoLocations(infos map[string]*FileInfo) { 194 | for _, info := range infos { 195 | dir := filepath.Dir(info.Name) 196 | pkg := info.GoPackage 197 | // The presence of a slash implies there's an import path. 198 | slash := strings.LastIndex(pkg, "/") 199 | if slash > 0 { 200 | if strings.Contains(pkg, ";") { 201 | info.ComputedPackage = pkg 202 | continue 203 | } 204 | decl := pkg[slash+1:] 205 | info.ComputedPackage = pkg + ";" + decl 206 | continue 207 | } 208 | if pkg == "" { 209 | pkg = info.Package 210 | } 211 | if pkg == "" { 212 | pkg = baseName(info.Name) 213 | fmt.Fprintf(os.Stderr, "Warning: file %q has no go_package and no package.\n", info.Name) 214 | } 215 | info.ComputedPackage = dir + ";" + strings.Map(badToUnderscore, pkg) 216 | } 217 | } 218 | 219 | // baseName returns the last path element of the name, with the last dotted suffix removed. 220 | func baseName(name string) string { 221 | // First, find the last element 222 | if i := strings.LastIndex(name, "/"); i >= 0 { 223 | name = name[i+1:] 224 | } 225 | // Now drop the suffix 226 | if i := strings.LastIndex(name, "."); i >= 0 { 227 | name = name[0:i] 228 | } 229 | return name 230 | } 231 | 232 | // CollectPackages returns a map of PackageInfos. 233 | func CollectPackages(infos map[string]*FileInfo, protos []string, importDirs []string) (map[string]*PackageInfo, error) { 234 | pkgMap := map[string]*PackageInfo{} 235 | 236 | // Collect into packages. 237 | for _, info := range infos { 238 | packageInfo, ok := pkgMap[info.ComputedPackage] 239 | if !ok { 240 | packageInfo = &PackageInfo{ComputedPackage: info.ComputedPackage} 241 | pkgMap[info.ComputedPackage] = packageInfo 242 | } 243 | packageInfo.Files = append(packageInfo.Files, info) 244 | } 245 | 246 | // Collect deps for each package. 247 | for _, pkg := range pkgMap { 248 | deps := map[string]*FileInfo{} 249 | for _, info := range pkg.Files { 250 | for _, dep := range info.Deps { 251 | deps[dep] = infos[dep] 252 | } 253 | } 254 | for _, dep := range deps { 255 | pkg.Deps = append(pkg.Deps, dep) 256 | } 257 | } 258 | 259 | return pkgMap, nil 260 | } 261 | 262 | // FileDescriptorName computes the import-dir-relative Name that the 263 | // FileDescriptor for a full filename will have. 264 | func FileDescriptorName(protoFile string, importDirs []string) string { 265 | isAbs := path.IsAbs(protoFile) 266 | for _, imp := range importDirs { 267 | // Handle import dirs of "." - the FileDescriptorProtos don't have the "./" prefix. 268 | if imp == "." && !isAbs { 269 | if strings.HasPrefix(protoFile, "./") { 270 | return protoFile[2:] 271 | } 272 | return protoFile 273 | } 274 | if strings.HasPrefix(protoFile, imp) { 275 | name := protoFile[len(imp):] 276 | if strings.HasPrefix(name, "/") && imp != "/" { 277 | return name[1:] 278 | } 279 | return name 280 | } 281 | } 282 | panic(fmt.Sprintf("Unable to find import dir for %q", protoFile)) 283 | } 284 | 285 | // AnnotateFullPaths annotates an existing set of FileInfos with their 286 | // full paths. 287 | func AnnotateFullPaths(infos map[string]*FileInfo, allProtos []string, importDirs []string) { 288 | for _, proto := range allProtos { 289 | name := FileDescriptorName(proto, importDirs) 290 | info, ok := infos[name] 291 | if !ok { 292 | panic(fmt.Sprintf("Unable to find file information for %q", name)) 293 | } 294 | info.FullPath = proto 295 | } 296 | } 297 | 298 | // badToUnderscore is the mapping function used to generate Go names from package names, 299 | // which can be dotted in the input .proto file. It replaces non-identifier characters such as 300 | // dot or dash with underscore. 301 | func badToUnderscore(r rune) rune { 302 | if unicode.IsLetter(r) || unicode.IsDigit(r) || r == '_' { 303 | return r 304 | } 305 | return '_' 306 | } 307 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {yyyy} {name of copyright owner} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | 203 | --------------------------------------------------------------------------------