├── CHANGELOG.md ├── Dockerfile ├── LICENSE.md ├── README.md ├── circle.yml ├── command ├── configparser │ ├── json.go │ ├── parser.go │ ├── raw.go │ └── yaml.go ├── dump.go ├── import.go └── version.go ├── commands.go ├── docker └── entrypoint.sh ├── main.go ├── test ├── json.json ├── json │ └── nested.json ├── test.tar ├── yaml.yaml └── yaml │ └── nested.yaml └── version.go /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Change Log 2 | 3 | ## [0.4.3](https://github.com/lewispeckover/consulator/tree/0.4.3) (2017-04-06) 4 | [Full Changelog](https://github.com/lewispeckover/consulator/compare/0.4.2...0.4.3) 5 | 6 | **Implemented enhancements:** 7 | 8 | - Performance: don't walk the destination kv tree for `import` [\#4](https://github.com/lewispeckover/consulator/issues/4) 9 | 10 | ## [0.4.2](https://github.com/lewispeckover/consulator/tree/0.4.2) (2017-04-02) 11 | [Full Changelog](https://github.com/lewispeckover/consulator/compare/0.4.1...0.4.2) 12 | 13 | ## [0.4.1](https://github.com/lewispeckover/consulator/tree/0.4.1) (2017-04-01) 14 | [Full Changelog](https://github.com/lewispeckover/consulator/compare/0.4.0...0.4.1) 15 | 16 | **Implemented enhancements:** 17 | 18 | - Handle tar archives directly [\#3](https://github.com/lewispeckover/consulator/issues/3) 19 | 20 | ## [0.4.0](https://github.com/lewispeckover/consulator/tree/0.4.0) (2017-03-28) 21 | [Full Changelog](https://github.com/lewispeckover/consulator/compare/0.3.3...0.4.0) 22 | 23 | **Closed issues:** 24 | 25 | - Optional purge / delete [\#2](https://github.com/lewispeckover/consulator/issues/2) 26 | 27 | ## [0.3.3](https://github.com/lewispeckover/consulator/tree/0.3.3) (2017-03-18) 28 | [Full Changelog](https://github.com/lewispeckover/consulator/compare/0.3.2...0.3.3) 29 | 30 | ## [0.3.2](https://github.com/lewispeckover/consulator/tree/0.3.2) (2017-03-17) 31 | [Full Changelog](https://github.com/lewispeckover/consulator/compare/0.3.1...0.3.2) 32 | 33 | ## [0.3.1](https://github.com/lewispeckover/consulator/tree/0.3.1) (2017-03-17) 34 | [Full Changelog](https://github.com/lewispeckover/consulator/compare/0.3.0...0.3.1) 35 | 36 | ## [0.3.0](https://github.com/lewispeckover/consulator/tree/0.3.0) (2017-03-16) 37 | [Full Changelog](https://github.com/lewispeckover/consulator/compare/0.2.1...0.3.0) 38 | 39 | ## [0.2.1](https://github.com/lewispeckover/consulator/tree/0.2.1) (2017-02-25) 40 | [Full Changelog](https://github.com/lewispeckover/consulator/compare/0.2.0...0.2.1) 41 | 42 | ## [0.2.0](https://github.com/lewispeckover/consulator/tree/0.2.0) (2017-02-25) 43 | [Full Changelog](https://github.com/lewispeckover/consulator/compare/0.1.11...0.2.0) 44 | 45 | **Implemented enhancements:** 46 | 47 | - Move to subcommand-driven cli [\#1](https://github.com/lewispeckover/consulator/issues/1) 48 | 49 | ## [0.1.11](https://github.com/lewispeckover/consulator/tree/0.1.11) (2017-02-21) 50 | [Full Changelog](https://github.com/lewispeckover/consulator/compare/0.1.10...0.1.11) 51 | 52 | ## [0.1.10](https://github.com/lewispeckover/consulator/tree/0.1.10) (2017-02-21) 53 | 54 | 55 | \* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)* -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM lewispeckover/base:3.5 2 | COPY ./docker/ / 3 | ENTRYPOINT ["/entrypoint.sh"] 4 | ENV VERSION=0.4.3 5 | ADD https://github.com/lewispeckover/consulator/releases/download/${VERSION}/consulator_${VERSION}_linux_amd64 /bin/consulator 6 | RUN chmod +x /bin/consulator 7 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | #The MIT License (MIT) 2 | 3 | Copyright 2017 Lewis Peckover 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | 7 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # Consulator 3 | 4 | Consulator lets you import and synchronize your KV data from JSON or YAML sources directly to Consul. This allows you to easily version your configuration data, storing it in git or anywhere else you see fit. 5 | 6 | [![CircleCI](https://circleci.com/gh/lewispeckover/consulator/tree/master.svg?style=shield)](https://circleci.com/gh/lewispeckover/consulator/tree/master) 7 | 8 | ## Getting Consulator 9 | 10 | Docker is the easiest way. You can find it on the [Docker Hub](https://hub.docker.com/r/lewispeckover/consulator/). 11 | 12 | 13 | ``` 14 | docker run -it --rm lewispeckover/consulator -help 15 | ``` 16 | 17 | Alternatively, download a binary from the [releases](https://github.com/lewispeckover/consulator/releases/latest) page, or clone the repo and build it yourself. 18 | 19 | ## Running Consulator 20 | 21 | ``` 22 | Usage: consulator [--version] [--help] [] [ ...] 23 | 24 | Available commands are: 25 | dump Dumps parsed config as JSON suitable for use with consul kv import 26 | import Imports data into consul 27 | sync Syncs data into consul (like import, but with deletes) 28 | version Prints the version 29 | 30 | Options: 31 | -glue string 32 | Glue to use for joining array values (default "\n") 33 | -json 34 | Parse stdin as JSON 35 | -prefix string 36 | Key prefix to use for output / Consul import destination 37 | -tar 38 | Parse stdin as a tarball 39 | -yaml 40 | Parse stdin as YAML 41 | 42 | Multiple paths (files or directories) may be provided, they are parsed in order. 43 | This allows you to specify some default values in the first path. 44 | If no paths are provided, stdin is used. In this case, -yaml or -json must be specified. 45 | 46 | The usual Consul client environment variables can be used to configure the connection: 47 | 48 | - CONSUL_HTTP_ADDR 49 | - CONSUL_HTTP_TOKEN 50 | - CONSUL_HTTP_SSL 51 | 52 | Etc. See https://www.consul.io/docs/commands/ for a complete list. 53 | ``` 54 | 55 | 56 | ## Source data 57 | 58 | JSON, YAML, and plain text sources are supported. Data can be loaded from files, piped directly to standard input, or provided as a tarball. Note that Consul KV values are only allowed to be strings, so non-string values are converted where possible. Most significantly, array values are joined with a configurable glue string (default: "\n"). 59 | 60 | Given a `myapp.yaml`: 61 | 62 | ``` 63 | tags: 64 | - production 65 | - web 66 | ``` 67 | or equivalent `myapp.json`: 68 | 69 | ``` 70 | { 71 | "tags": ["production", "web"] 72 | } 73 | ``` 74 | 75 | Running `consulator import -glue=, myapp.yaml` will result in a Consul key `tags` with the value `production,web` 76 | 77 | When a directory is specified as the source, it is scanned for files with extensions .json, .yaml or .yml. Subdirectories and filenames are used to build key prefixes. 78 | 79 | Suppose that the above file is located at `/etc/consuldata/config/myapp.yaml`. When consulator is executed as `consulator import -glue=, /etc/consuldata`, it will result in a Consul key `config/myapp/tags` with value `production,web`. 80 | 81 | ## License 82 | 83 | This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details 84 | 85 | -------------------------------------------------------------------------------- /circle.yml: -------------------------------------------------------------------------------- 1 | --- 2 | deployment: 3 | release: 4 | tag: /.*/ 5 | commands: 6 | - "wget http://www.musl-libc.org/releases/musl-1.1.16.tar.gz && tar -xvf musl-1.1.16.tar.gz" 7 | - "cd musl-1.1.16 && ./configure && make && sudo make install" 8 | - "go get github.com/mitchellh/gox" 9 | - "go get github.com/tcnksm/ghr" 10 | - "CC=/usr/local/musl/bin/musl-gcc gox -osarch=\"linux/amd64\" -ldflags \"-linkmode external -extldflags '-static' -X 'main.Version=$(git describe --tags)' -X 'main.BuildDate=$(date -u '+%Y/%m/%d %H:%M:%S')'\" -output \"dist/consulator_$(git describe --tags)_{{.OS}}_{{.Arch}}\"" 11 | - "ghr -t $GITHUB_TOKEN -u $CIRCLE_PROJECT_USERNAME -r $CIRCLE_PROJECT_REPONAME --replace `git describe --tags` dist/" 12 | - 'curl -H ''Content-Type: application/json'' --data "{\"source_type\": \"Tag\", \"source_name\": \"$(git describe --tags)\"}" -X POST https://registry.hub.docker.com/u/lewispeckover/consulator/trigger/$DOCKER_HUB/' 13 | -------------------------------------------------------------------------------- /command/configparser/json.go: -------------------------------------------------------------------------------- 1 | package configparser 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "io" 7 | "strings" 8 | 9 | "github.com/antonholmquist/jason" 10 | ) 11 | 12 | func parseJson(fp io.Reader, prefix []string, glue string) error { 13 | jsonObj, err := jason.NewObjectFromReader(fp) 14 | if err != nil { 15 | return err 16 | } 17 | j, err := jsonObj.GetObject() 18 | if err != nil { 19 | return err 20 | } 21 | return jsonWalk(prefix, j) 22 | } 23 | 24 | func jsonWalk(prefix []string, obj *jason.Object) error { 25 | for k, v := range obj.Map() { 26 | key := strings.Join(append(prefix, k), "/") 27 | switch v.Interface().(type) { 28 | case string: 29 | data[key] = []byte(fmt.Sprintf("%v", v.Interface())) 30 | case json.Number: 31 | data[key] = []byte(fmt.Sprintf("%v", v.Interface())) 32 | case []interface{}: 33 | // json array 34 | o, _ := v.Array() 35 | val, err := jsonArrayChoose(o) 36 | if err != nil { 37 | return err 38 | } 39 | data[key] = []byte(strings.Join(val, glue)) 40 | case bool: 41 | data[key] = []byte(fmt.Sprintf("%v", v.Interface())) 42 | case nil: 43 | // json nulls 44 | case map[string]interface{}: 45 | // json object 46 | o, _ := v.Object() 47 | if err := jsonWalk(append(prefix, k), o); err != nil { 48 | return err 49 | } 50 | default: 51 | } 52 | } 53 | return nil 54 | } 55 | 56 | func jsonArrayChoose(arr []*jason.Value) (ret []string, err error) { 57 | for _, v := range arr { 58 | switch v.Interface().(type) { 59 | case string: 60 | ret = append(ret, fmt.Sprintf("%v", v.Interface())) 61 | case json.Number: 62 | ret = append(ret, fmt.Sprintf("%v", v.Interface())) 63 | case bool: 64 | ret = append(ret, fmt.Sprintf("%v", v.Interface())) 65 | default: 66 | return ret, fmt.Errorf(fmt.Sprintf("Invalid type %T in array. Only strings, numbers and boolean values are supported.\n", v.Interface())) 67 | } 68 | } 69 | return ret, nil 70 | } 71 | -------------------------------------------------------------------------------- /command/configparser/parser.go: -------------------------------------------------------------------------------- 1 | package configparser 2 | 3 | import ( 4 | "archive/tar" 5 | "io" 6 | "os" 7 | "path/filepath" 8 | "strings" 9 | ) 10 | 11 | var absPath string 12 | var data map[string][]byte 13 | var forceType = "auto" 14 | var glue string 15 | 16 | func Parse(path string, dataDest map[string][]byte, arrayGlue string) error { 17 | absPath, _ = filepath.Abs(path) 18 | data = dataDest 19 | glue = arrayGlue 20 | _, err := os.Stat(absPath) 21 | if err != nil { 22 | return err 23 | } 24 | if forceType == "tar" || strings.HasSuffix(strings.ToLower(path), ".tar") { 25 | fp, err := os.Open(path) 26 | if err != nil { 27 | return err 28 | } 29 | defer fp.Close() 30 | tarReader := tar.NewReader(fp) 31 | for { 32 | header, err := tarReader.Next() 33 | if err == io.EOF { 34 | break 35 | } 36 | if err != nil { 37 | return err 38 | } 39 | // a tar can have some annoying paths 40 | path := strings.TrimPrefix(header.Name, "./") 41 | info := header.FileInfo() 42 | err = fpWalk(path, info, tarReader, nil) 43 | if err != nil && err != filepath.SkipDir { 44 | return err 45 | } 46 | } 47 | } else { 48 | err = filepath.Walk(absPath, walk) 49 | } 50 | return err 51 | } 52 | 53 | func ParseAsJSON(path string, dataDest map[string][]byte, arrayGlue string) error { 54 | forceType = "json" 55 | return Parse(path, dataDest, arrayGlue) 56 | } 57 | 58 | func ParseAsYAML(path string, dataDest map[string][]byte, arrayGlue string) error { 59 | forceType = "yaml" 60 | return Parse(path, dataDest, arrayGlue) 61 | } 62 | 63 | func ParseAsTAR(path string, dataDest map[string][]byte, arrayGlue string) error { 64 | forceType = "tar" 65 | return Parse(path, dataDest, arrayGlue) 66 | } 67 | 68 | func walk(path string, fstat os.FileInfo, err error) error { 69 | fp, err := os.Open(path) 70 | if err != nil { 71 | return err 72 | } 73 | defer fp.Close() 74 | return fpWalk(path, fstat, fp, err) 75 | } 76 | func fpWalk(path string, fstat os.FileInfo, fp io.Reader, err error) error { 77 | // skip .git etc 78 | if fstat.IsDir() && strings.HasPrefix(fstat.Name(), ".") { 79 | return filepath.SkipDir 80 | } 81 | if fstat.Mode().IsDir() { 82 | return nil 83 | } 84 | keyPrefix := strings.Split( 85 | // remove leading '/' 86 | strings.TrimPrefix( 87 | // remove the file extension 88 | strings.TrimSuffix( 89 | // remove the base path that was passed as -path 90 | strings.TrimPrefix(path, absPath), 91 | filepath.Ext(path)), 92 | string(os.PathSeparator)), 93 | string(os.PathSeparator)) 94 | if keyPrefix[0] == "" { 95 | // remove blank token 96 | keyPrefix = []string{} 97 | } 98 | switch { 99 | // skip dotfiles 100 | case strings.HasPrefix(fstat.Name(), "."): 101 | return nil 102 | case strings.HasSuffix(strings.ToLower(path), ".json") || forceType == "json": 103 | err := parseJson(fp, keyPrefix, glue) 104 | if err != nil { 105 | return err 106 | } 107 | case strings.HasSuffix(strings.ToLower(path), ".yml"): 108 | fallthrough 109 | case strings.HasSuffix(strings.ToLower(path), ".yaml") || forceType == "yaml": 110 | // yaml handling based on https://github.com/bronze1man/yaml2json 111 | yamlR, yamlW := io.Pipe() 112 | go func() { 113 | defer yamlW.Close() 114 | yamlToJson(fp, yamlW) 115 | }() 116 | err := parseJson(yamlR, keyPrefix, glue) 117 | if err != nil { 118 | return err 119 | } 120 | //case strings.HasSuffix(strings.ToLower(path), ".properties"): 121 | // TODO: .properties parsing 122 | //case strings.HasSuffix(strings.ToLower(path), ".ini"): 123 | // TODO: .ini parsing 124 | // filenames with no type, or .txt should be handled as raw data 125 | case !strings.Contains(fstat.Name(), ".") || strings.HasSuffix(strings.ToLower(path), ".txt"): 126 | err := parseRaw(fp, keyPrefix, glue) 127 | if err != nil { 128 | return err 129 | } 130 | default: 131 | } 132 | return nil 133 | } 134 | -------------------------------------------------------------------------------- /command/configparser/raw.go: -------------------------------------------------------------------------------- 1 | package configparser 2 | 3 | import ( 4 | "io" 5 | "io/ioutil" 6 | "strings" 7 | ) 8 | 9 | func parseRaw(fp io.Reader, prefix []string, glue string) error { 10 | contents, err := ioutil.ReadAll(fp) 11 | if err == nil { 12 | data[strings.Join(prefix, "/")] = []byte(strings.TrimSuffix(string(contents), "\n")) 13 | } 14 | return err 15 | } 16 | -------------------------------------------------------------------------------- /command/configparser/yaml.go: -------------------------------------------------------------------------------- 1 | package configparser 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | goyaml "gopkg.in/yaml.v2" 7 | "io" 8 | "io/ioutil" 9 | "strconv" 10 | ) 11 | 12 | // yaml handling based on https://github.com/bronze1man/yaml2json 13 | func yamlToJson(in io.Reader, out io.Writer) error { 14 | input, err := ioutil.ReadAll(in) 15 | if err != nil { 16 | return err 17 | } 18 | var data interface{} 19 | err = goyaml.Unmarshal(input, &data) 20 | if err != nil { 21 | return err 22 | } 23 | input = nil 24 | err = yamlWalk(&data) 25 | if err != nil { 26 | return err 27 | } 28 | 29 | output, err := json.Marshal(data) 30 | if err != nil { 31 | return err 32 | } 33 | data = nil 34 | _, err = out.Write(output) 35 | return err 36 | } 37 | 38 | func yamlWalk(pIn *interface{}) (err error) { 39 | switch in := (*pIn).(type) { 40 | case map[interface{}]interface{}: 41 | m := make(map[string]interface{}, len(in)) 42 | for k, v := range in { 43 | if err = yamlWalk(&v); err != nil { 44 | return err 45 | } 46 | var sk string 47 | switch k.(type) { 48 | case string: 49 | sk = k.(string) 50 | case int: 51 | sk = strconv.Itoa(k.(int)) 52 | default: 53 | return fmt.Errorf("type mismatch: expect map key string or int; got: %T", k) 54 | } 55 | m[sk] = v 56 | } 57 | *pIn = m 58 | case []interface{}: 59 | for i := len(in) - 1; i >= 0; i-- { 60 | if err = yamlWalk(&in[i]); err != nil { 61 | return err 62 | } 63 | } 64 | } 65 | return nil 66 | } 67 | -------------------------------------------------------------------------------- /command/dump.go: -------------------------------------------------------------------------------- 1 | package command 2 | 3 | import ( 4 | "bytes" 5 | "encoding/base64" 6 | "encoding/json" 7 | "flag" 8 | "fmt" 9 | "strings" 10 | 11 | "github.com/lewispeckover/consulator/command/configparser" 12 | 13 | "github.com/mitchellh/cli" 14 | ) 15 | 16 | type DumpCommand struct { 17 | Ui cli.Ui 18 | name string 19 | args string 20 | synopsis string 21 | flags *flag.FlagSet 22 | parseAsYAML *bool 23 | parseAsJSON *bool 24 | parseAsTAR *bool 25 | arrayGlue *string 26 | keyPrefix *string 27 | initialised bool 28 | } 29 | 30 | func (c *DumpCommand) init() { 31 | if c.initialised { 32 | return 33 | } 34 | c.name = "consulator dump" 35 | c.args = "[options] [path ...]" 36 | c.synopsis = "Dumps parsed config as JSON suitable for use with consul kv import" 37 | c.flags = flag.NewFlagSet("dump", flag.ContinueOnError) 38 | c.parseAsYAML = c.flags.Bool("yaml", false, "Parse stdin as YAML") 39 | c.parseAsJSON = c.flags.Bool("json", false, "Parse stdin as JSON") 40 | c.parseAsTAR = c.flags.Bool("tar", false, "Parse stdin as a tarball") 41 | c.arrayGlue = c.flags.String("glue", "\n", "Glue to use for joining array values") 42 | c.keyPrefix = c.flags.String("prefix", "", "Key prefix to use for output") 43 | c.flags.Usage = func() { c.Ui.Output(c.Help()) } 44 | c.initialised = true 45 | } 46 | 47 | func (c *DumpCommand) Run(args []string) int { 48 | c.init() 49 | if err := c.flags.Parse(args); err != nil { 50 | return 1 51 | } 52 | if *c.parseAsYAML && *c.parseAsJSON { 53 | c.Ui.Error("Only one input format may be specified") 54 | return 1 55 | } 56 | // clean up the prefix 57 | *c.keyPrefix = strings.TrimSuffix(strings.TrimSpace(*c.keyPrefix), "/") 58 | if *c.keyPrefix != "" { 59 | *c.keyPrefix = *c.keyPrefix + "/" 60 | } 61 | data := make(map[string][]byte) 62 | if c.flags.NArg() == 0 { 63 | switch { 64 | case *c.parseAsYAML: 65 | if err := configparser.ParseAsYAML("/dev/stdin", data, *c.arrayGlue); err != nil { 66 | c.Ui.Error(fmt.Sprintf("Error: %s", err)) 67 | return 1 68 | } 69 | case *c.parseAsJSON: 70 | if err := configparser.ParseAsJSON("/dev/stdin", data, *c.arrayGlue); err != nil { 71 | c.Ui.Error(fmt.Sprintf("Error: %s", err)) 72 | return 1 73 | } 74 | case *c.parseAsTAR: 75 | if err := configparser.ParseAsTAR("/dev/stdin", data, *c.arrayGlue); err != nil { 76 | c.Ui.Error(fmt.Sprintf("Error: %s", err)) 77 | return 1 78 | } 79 | default: 80 | c.Ui.Error("You must specify an input format when using stdin\n") 81 | c.Ui.Error(c.Help()) 82 | return 1 83 | } 84 | } else { 85 | for _, p := range c.flags.Args() { 86 | if err := configparser.Parse(p, data, *c.arrayGlue); err != nil { 87 | c.Ui.Error(fmt.Sprintf("Error: %s", err)) 88 | return 1 89 | } 90 | } 91 | } 92 | 93 | exported := make([]*dumpExportEntry, len(data)) 94 | i := 0 95 | for key, val := range data { 96 | exported[i] = c.toExportEntry(key, val) 97 | i++ 98 | } 99 | json, err := json.MarshalIndent(exported, "", "\t") 100 | if err != nil { 101 | c.Ui.Error(fmt.Sprintf("Error exporting data: %s", err)) 102 | return 1 103 | } 104 | c.Ui.Output(string(json)) 105 | return 0 106 | } 107 | 108 | type dumpExportEntry struct { 109 | Key string `json:"key"` 110 | Flags uint64 `json:"flags"` 111 | Value string `json:"value"` 112 | } 113 | 114 | func (c *DumpCommand) toExportEntry(key string, val []byte) *dumpExportEntry { 115 | return &dumpExportEntry{ 116 | Key: *c.keyPrefix + key, 117 | Flags: 0, 118 | Value: base64.StdEncoding.EncodeToString(val), 119 | } 120 | } 121 | 122 | func (c *DumpCommand) Synopsis() string { 123 | c.init() 124 | return c.synopsis 125 | } 126 | 127 | func (c *DumpCommand) Help() string { 128 | c.init() 129 | flagOut := new(bytes.Buffer) 130 | c.flags.SetOutput(flagOut) 131 | c.flags.PrintDefaults() 132 | c.flags.SetOutput(nil) 133 | return fmt.Sprintf("%s %s\n\n%s\n\nOptions:\n%s", c.name, c.args, c.synopsis, flagOut.String()) 134 | } 135 | -------------------------------------------------------------------------------- /command/import.go: -------------------------------------------------------------------------------- 1 | package command 2 | 3 | import ( 4 | "bytes" 5 | "flag" 6 | "fmt" 7 | "strings" 8 | 9 | "github.com/lewispeckover/consulator/command/configparser" 10 | 11 | "github.com/hashicorp/consul/api" 12 | "github.com/mitchellh/cli" 13 | ) 14 | 15 | type ImportCommand struct { 16 | Ui cli.Ui 17 | name string 18 | args string 19 | synopsis string 20 | flags *flag.FlagSet 21 | parseAsYAML *bool 22 | parseAsJSON *bool 23 | parseAsTAR *bool 24 | arrayGlue *string 25 | keyPrefix *string 26 | initialised bool 27 | Purge bool 28 | } 29 | 30 | func (c *ImportCommand) init() { 31 | if c.initialised { 32 | return 33 | } 34 | if c.Purge { 35 | c.name = "consulator sync" 36 | c.synopsis = "Syncs data into consul (like import, but with deletes)" 37 | } else { 38 | c.name = "consulator import" 39 | c.synopsis = "Imports data into consul" 40 | } 41 | c.args = "[options] [path ...]" 42 | c.flags = flag.NewFlagSet("import", flag.ContinueOnError) 43 | c.parseAsYAML = c.flags.Bool("yaml", false, "Parse stdin as YAML") 44 | c.parseAsJSON = c.flags.Bool("json", false, "Parse stdin as JSON") 45 | c.parseAsTAR = c.flags.Bool("tar", false, "Parse stdin as a tarball") 46 | c.arrayGlue = c.flags.String("glue", "\n", "Glue to use for joining array values") 47 | c.keyPrefix = c.flags.String("prefix", "", "Consul tree to work under") 48 | c.flags.Usage = func() { c.Ui.Output(c.Help()) } 49 | c.initialised = true 50 | } 51 | 52 | func (c *ImportCommand) Run(args []string) int { 53 | c.init() 54 | if err := c.flags.Parse(args); err != nil { 55 | return 1 56 | } 57 | if *c.parseAsYAML && *c.parseAsJSON { 58 | c.Ui.Error("Only one input format may be specified") 59 | return 1 60 | } 61 | // clean up the prefix 62 | *c.keyPrefix = strings.TrimSuffix(strings.TrimSpace(*c.keyPrefix), "/") 63 | if *c.keyPrefix != "" { 64 | *c.keyPrefix = *c.keyPrefix + "/" 65 | } 66 | data := make(map[string][]byte) 67 | if c.flags.NArg() == 0 { 68 | switch { 69 | case *c.parseAsYAML: 70 | if err := configparser.ParseAsYAML("/dev/stdin", data, *c.arrayGlue); err != nil { 71 | c.Ui.Error(fmt.Sprintf("Error: %s", err)) 72 | return 1 73 | } 74 | case *c.parseAsJSON: 75 | if err := configparser.ParseAsJSON("/dev/stdin", data, *c.arrayGlue); err != nil { 76 | c.Ui.Error(fmt.Sprintf("Error: %s", err)) 77 | return 1 78 | } 79 | case *c.parseAsTAR: 80 | if err := configparser.ParseAsTAR("/dev/stdin", data, *c.arrayGlue); err != nil { 81 | c.Ui.Error(fmt.Sprintf("Error: %s", err)) 82 | return 1 83 | } 84 | 85 | default: 86 | c.Ui.Error("You must specify an input format when using stdin\n") 87 | c.Ui.Error(c.Help()) 88 | return 1 89 | } 90 | } else { 91 | for _, p := range c.flags.Args() { 92 | if err := configparser.Parse(p, data, *c.arrayGlue); err != nil { 93 | c.Ui.Error(fmt.Sprintf("Error: %s", err)) 94 | return 1 95 | } 96 | } 97 | } 98 | if err := c.syncConsul(data); err != nil { 99 | c.Ui.Error(fmt.Sprintf("Error: %s", err)) 100 | return 1 101 | } 102 | return 0 103 | } 104 | 105 | func (c *ImportCommand) syncConsul(data map[string][]byte) error { 106 | config := api.DefaultConfig() 107 | client, err := api.NewClient(config) 108 | if err != nil { 109 | return err 110 | } 111 | kv := client.KV() 112 | deleted := 0 113 | updated := 0 114 | if c.Purge { 115 | pairs, _, err := kv.List(*c.keyPrefix, &api.QueryOptions{}) 116 | if err != nil { 117 | return err 118 | } 119 | for _, pair := range pairs { 120 | // if there was a prefix, we need to strip it 121 | relativeKey := strings.TrimPrefix(pair.Key, *c.keyPrefix) 122 | if val, ok := data[relativeKey]; ok { 123 | if bytes.Equal(val, pair.Value) { 124 | delete(data, relativeKey) 125 | } 126 | } else if c.Purge { 127 | _, err := kv.Delete(pair.Key, nil) 128 | if err != nil { 129 | return err 130 | } else { 131 | deleted++ 132 | } 133 | } 134 | } 135 | } 136 | for key, val := range data { 137 | _, err := kv.Put(c.toKVPair(key, val), nil) 138 | if err != nil { 139 | return err 140 | } else { 141 | updated++ 142 | } 143 | } 144 | if c.Purge { 145 | c.Ui.Output(fmt.Sprintf("Sync completed. %d keys deleted, %d keys updated.", deleted, updated)) 146 | } else { 147 | c.Ui.Output(fmt.Sprintf("Import completed. %d keys set.", updated)) 148 | } 149 | return nil 150 | } 151 | 152 | func (c *ImportCommand) toKVPair(key string, val []byte) *api.KVPair { 153 | return &api.KVPair{ 154 | Key: *c.keyPrefix + key, 155 | Flags: 0, 156 | Value: val, 157 | } 158 | } 159 | 160 | func (c *ImportCommand) Synopsis() string { 161 | c.init() 162 | return c.synopsis 163 | } 164 | 165 | func (c *ImportCommand) Help() string { 166 | c.init() 167 | flagOut := new(bytes.Buffer) 168 | c.flags.SetOutput(flagOut) 169 | c.flags.PrintDefaults() 170 | c.flags.SetOutput(nil) 171 | return fmt.Sprintf("%s %s\n\n%s\n\nOptions:\n%s", c.name, c.args, c.synopsis, flagOut.String()) 172 | } 173 | -------------------------------------------------------------------------------- /command/version.go: -------------------------------------------------------------------------------- 1 | package command 2 | 3 | import ( 4 | "bytes" 5 | "fmt" 6 | 7 | "github.com/mitchellh/cli" 8 | ) 9 | 10 | // VersionCommand is a Command implementation that prints the version. 11 | type VersionCommand struct { 12 | BuildDate string 13 | Version string 14 | Ui cli.Ui 15 | } 16 | 17 | func (c *VersionCommand) Help() string { 18 | return "" 19 | } 20 | 21 | func (c *VersionCommand) Run(_ []string) int { 22 | var versionString bytes.Buffer 23 | fmt.Fprintf(&versionString, "Consulator version %s, built %s", c.Version, c.BuildDate) 24 | c.Ui.Output(versionString.String()) 25 | return 0 26 | } 27 | 28 | func (c *VersionCommand) Synopsis() string { 29 | return "Prints the version" 30 | } 31 | -------------------------------------------------------------------------------- /commands.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "os" 5 | 6 | "github.com/lewispeckover/consulator/command" 7 | 8 | "github.com/mitchellh/cli" 9 | ) 10 | 11 | // Commands is the mapping of all the available Serf commands. 12 | var Commands map[string]cli.CommandFactory 13 | 14 | func init() { 15 | ui := &cli.BasicUi{Writer: os.Stdout} 16 | 17 | Commands = map[string]cli.CommandFactory{ 18 | "dump": func() (cli.Command, error) { 19 | return &command.DumpCommand{ 20 | Ui: ui, 21 | }, nil 22 | }, 23 | "import": func() (cli.Command, error) { 24 | return &command.ImportCommand{ 25 | Ui: ui, 26 | }, nil 27 | }, 28 | "sync": func() (cli.Command, error) { 29 | return &command.ImportCommand{ 30 | Ui: ui, 31 | Purge: true, 32 | }, nil 33 | }, 34 | "version": func() (cli.Command, error) { 35 | return &command.VersionCommand{ 36 | Version: Version, 37 | BuildDate: BuildDate, 38 | Ui: ui, 39 | }, nil 40 | }, 41 | } 42 | } 43 | -------------------------------------------------------------------------------- /docker/entrypoint.sh: -------------------------------------------------------------------------------- 1 | #!/bin/dumb-init /bin/sh 2 | set -e 3 | 4 | if [ "$(basename $1 2>/dev/null)" != 'consulator' ]; then 5 | set -- consulator "$@" 6 | fi 7 | 8 | exec "$@" 9 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "io/ioutil" 6 | "log" 7 | "os" 8 | 9 | "github.com/mitchellh/cli" 10 | ) 11 | 12 | func main() { 13 | os.Exit(realMain()) 14 | } 15 | 16 | func realMain() int { 17 | log.SetOutput(ioutil.Discard) 18 | 19 | // Get the command line args. We shortcut "--version" and "-v" to 20 | // just show the version. 21 | args := os.Args[1:] 22 | for _, arg := range args { 23 | if arg == "-v" || arg == "--version" { 24 | newArgs := make([]string, len(args)+1) 25 | newArgs[0] = "version" 26 | copy(newArgs[1:], args) 27 | args = newArgs 28 | break 29 | } 30 | } 31 | 32 | cli := &cli.CLI{ 33 | Args: args, 34 | Commands: Commands, 35 | HelpFunc: cli.BasicHelpFunc("consulator"), 36 | } 37 | 38 | exitCode, err := cli.Run() 39 | if err != nil { 40 | fmt.Fprintf(os.Stderr, "Error executing CLI: %s\n", err.Error()) 41 | return 1 42 | } 43 | 44 | return exitCode 45 | } 46 | -------------------------------------------------------------------------------- /test/json.json: -------------------------------------------------------------------------------- 1 | { 2 | "string": "value", 3 | "int": 1, 4 | "array": ["a", 1, true], 5 | "bool": false 6 | } 7 | -------------------------------------------------------------------------------- /test/json/nested.json: -------------------------------------------------------------------------------- 1 | { 2 | "string": "value", 3 | "int": 1, 4 | "array": ["a", 1, true], 5 | "bool": false 6 | } 7 | -------------------------------------------------------------------------------- /test/test.tar: -------------------------------------------------------------------------------- 1 | ./000755 000765 000024 00000000000 13067765617 013262 5ustar00lewispeckoverstaff000000 000000 ./json/000755 000765 000024 00000000000 13052653617 014221 5ustar00lewispeckoverstaff000000 000000 ./json.json000644 000765 000024 00000000114 13052653164 015105 0ustar00lewispeckoverstaff000000 000000 { 2 | "string": "value", 3 | "int": 1, 4 | "array": ["a", 1, true], 5 | "bool": false 6 | } 7 | ./yaml/000755 000765 000024 00000000000 13054271637 014212 5ustar00lewispeckoverstaff000000 000000 ./yaml.yaml000644 000765 000024 00000000101 13052735372 015065 0ustar00lewispeckoverstaff000000 000000 --- 8 | string: value 9 | int: 1 10 | array: 11 | - a 12 | - 1 13 | - true 14 | bool: false 15 | ./yaml/nested.yaml000644 000765 000024 00000000112 13054271637 016352 0ustar00lewispeckoverstaff000000 000000 --- 16 | string: value 17 | int: 5294967294 18 | array: 19 | - a 20 | - 1 21 | - true 22 | bool: false 23 | ./json/nested.json000644 000765 000024 00000000114 13052653617 016372 0ustar00lewispeckoverstaff000000 000000 { 24 | "string": "value", 25 | "int": 1, 26 | "array": ["a", 1, true], 27 | "bool": false 28 | } 29 | -------------------------------------------------------------------------------- /test/yaml.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | string: value 3 | int: 1 4 | array: 5 | - a 6 | - 1 7 | - true 8 | bool: false 9 | -------------------------------------------------------------------------------- /test/yaml/nested.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | string: value 3 | int: 5294967294 4 | array: 5 | - a 6 | - 1 7 | - true 8 | bool: false 9 | -------------------------------------------------------------------------------- /version.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | // These should be filled in by the compiler. 4 | var Version string 5 | var BuildDate string 6 | --------------------------------------------------------------------------------