├── LICENSE ├── README.md ├── go.mod ├── go.sum ├── protostructure.go ├── protostructure.pb.go ├── protostructure.proto ├── protostructure_test.go ├── type.go └── type_test.go /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Mitchell Hashimoto 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # protostructure [![Godoc](https://godoc.org/github.com/mitchellh/protostructure?status.svg)](https://godoc.org/github.com/mitchellh/protostructure) 2 | 3 | protostructure is a Go library for encoding and decoding a `struct` 4 | _type_ over the wire. 5 | 6 | This library is useful when you want to send arbitrary structures 7 | over protocol buffers for behavior such as configuration decoding 8 | (`encoding/json`, etc.), validation (using packages that use tags), etc. 9 | This works because we can reconstruct the struct type dynamically using 10 | `reflect` including any field tags. 11 | 12 | This library only sends the structure of the struct, not the _value_. 13 | If you want to send the value, you should build your protocol buffer 14 | message in such a way that it encodes that somehow using something 15 | such as JSON. 16 | 17 | ## Installation 18 | 19 | Standard `go get`: 20 | 21 | ``` 22 | $ go get github.com/mitchellh/protostructure 23 | ``` 24 | 25 | ## Usage & Example 26 | 27 | For usage and examples see the [Godoc](http://godoc.org/github.com/mitchellh/protostructure). 28 | 29 | A quick code example is shown below using both the imaginary proto file 30 | and the Go code that uses it. 31 | 32 | ```proto 33 | syntax = "proto3"; 34 | package myapp; 35 | import "protostructure.proto"; 36 | 37 | // Response is an example response structure for an RPC endpoint. 38 | message Response { 39 | protostructure.Struct config = 1; 40 | } 41 | ``` 42 | 43 | ```go 44 | type Config struct { 45 | Name string `json:"name"` 46 | Meta map[string]string `json:"metadata"` 47 | Port []*Port `json:"ports"` 48 | } 49 | 50 | type Port struct { 51 | Number uint `json:"number"` 52 | Desc string `json:"desc"` 53 | } 54 | 55 | // You can encode the structure on one side: 56 | message, err := protostructure.Encode(Config{}) 57 | 58 | // And you can use the structure on the other side. Imagine resp 59 | // is populated using some protobuf RPC such as gRPC. 60 | val, err := protostructure.New(resp.Config) 61 | json.Unmarshal([]byte(`{ 62 | "name": "example", 63 | "meta": { "env": "prod" }, 64 | "ports": [ 65 | { "number": 8080 }, 66 | { "number": 8100, desc: "backup" }, 67 | ] 68 | }`, val) 69 | 70 | // val now holds the same structure dynamically. You can pair with other 71 | // libraries such as https://github.com/go-playground/validator to also 72 | // send validation using this library. 73 | ``` 74 | 75 | ## Limitations 76 | 77 | There are several limitations on the structures that can be encoded. 78 | Some of these limitations are fixable but the effort hasn't been put in 79 | while others are fundamental due to the limitations of Go currently: 80 | 81 | * Circular references are not allowed between any struct types. 82 | * Embedded structs are not supported 83 | * Methods are not preserved, and therefore interface implementation 84 | is not known. This is also an important detail because custom callbacks 85 | such as `UnmarshalJSON` may not work properly. 86 | * Field types cannot be: interfaces, channels, functions 87 | * Certain stdlib types such as `time.Time` currently do not encode well. 88 | 89 | ## But... why? 90 | 91 | The real world use case that led to the creation of this library was 92 | to facilitate decoding and validating configuration for plugins via 93 | [go-plugin](https://github.com/hashicorp/go-plugin), a plugin system for 94 | Go that communicates using [gRPC](https://grpc.io). 95 | 96 | The plugins for this particular program have dynamic configuration structures 97 | that were decoded using an `encoding/json`-like interface (struct tags) and 98 | validated using [`go-playground/validator`](https://github.com/go-playground/validator) 99 | which also uses struct tags. Using protostructure, we can send the configuration 100 | structure across the wire, decode and validate the configuration in the host process, 101 | and report more rich errors that way. 102 | 103 | Another reason we wanted to ship the config structure vs. ship the 104 | config is because the actual language we are using for configuration 105 | is [HCL](https://github.com/hashicorp/hcl) which supports things like 106 | function calls, logic, and more and shipping that runtime across is 107 | much, much more difficult. 108 | 109 | This was extracted into a separate library because the ability to 110 | encode a Go structure (particulary to include tags) seemed more generally 111 | useful, although rare. 112 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/mitchellh/protostructure 2 | 3 | go 1.14 4 | 5 | require ( 6 | github.com/golang/protobuf v1.3.4 7 | github.com/stretchr/testify v1.5.1 8 | ) 9 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8= 2 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 3 | github.com/golang/protobuf v1.3.4 h1:87PNWwrRvUSnqS4dlcBU/ftvOIBep4sYuBLlh6rX2wk= 4 | github.com/golang/protobuf v1.3.4/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= 5 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 6 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 7 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 8 | github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4= 9 | github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= 10 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= 11 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 12 | gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw= 13 | gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 14 | -------------------------------------------------------------------------------- /protostructure.go: -------------------------------------------------------------------------------- 1 | // Package protostructure provides a mechanism for encoding and decoding 2 | // a struct _type_ using protocol buffers. To be clear: this encodes the 3 | // _type_ and not the _value_. 4 | // 5 | // Most importantly, this lets you do things such as transferring a struct 6 | // that supports JSON decoding across a protobuf RPC, and then decoding 7 | // a JSON value directly into it since you have access to things such as 8 | // struct tags from the remote end. 9 | // 10 | // For a pure JSON use case, it may make sense to instead send the JSON 11 | // rather than send the struct type. There are other scenarios where sending 12 | // the type is easier and this library facilitates those use cases. 13 | // 14 | // The primary functions you want to look at are "Encode" and "New". 15 | package protostructure 16 | 17 | import ( 18 | "fmt" 19 | "reflect" 20 | ) 21 | 22 | //go:generate sh -c "protoc ./*.proto --go_out=plugins=grpc:./" 23 | 24 | // Encode converts a struct to a *Struct which implements proto.Message 25 | // and can therefore be sent over the wire. Note that only the _structure_ 26 | // of the struct is encoded and NOT any fields values. 27 | // 28 | // Encoding has a number of limitations: 29 | // 30 | // * Circular references are not allowed between any struct types 31 | // * Embedded structs are not supported 32 | // * Methods are not preserved 33 | // * Field types cannot be: interfaces, channels, functions 34 | // 35 | func Encode(s interface{}) (*Struct, error) { 36 | // If s is a Type already then we use that directly (a code path used 37 | // by protoType but not generally expected for callers). 38 | t, ok := s.(reflect.Type) 39 | if !ok { 40 | t = reflect.TypeOf(s) 41 | } 42 | 43 | // First we need to unwrap any number of layers of interface{} or pointers. 44 | for { 45 | // If we don't have some container type, then we are done. 46 | if t.Kind() != reflect.Ptr && t.Kind() != reflect.Interface { 47 | break 48 | } 49 | 50 | // Unwrap one layer 51 | t = t.Elem() 52 | } 53 | 54 | // We require a struct since that's what we're encoding here. 55 | if k := t.Kind(); k != reflect.Struct { 56 | return nil, fmt.Errorf("encode: requires a struct, got %s", k) 57 | } 58 | 59 | // Build our struct 60 | var result Struct 61 | for i := 0; i < t.NumField(); i++ { 62 | field := t.Field(i) 63 | fieldType, err := protoType(field.Type) 64 | if err != nil { 65 | return nil, err 66 | } 67 | 68 | result.Fields = append(result.Fields, &Struct_Field{ 69 | Name: field.Name, 70 | PkgPath: field.PkgPath, 71 | Tag: string(field.Tag), 72 | Type: fieldType, 73 | }) 74 | } 75 | 76 | return &result, nil 77 | } 78 | 79 | // New returns a pointer to an allocated struct for the structure given 80 | // or an error if there are any invalid fields. 81 | // 82 | // This interface{} value can be used directly in functions such as 83 | // json.Unmarshal, or it can be inspected further as necessary. 84 | func New(s *Struct) (result interface{}, err error) { 85 | // We need to use the recover mechanism because the primary source 86 | // of underlying errors is the stdlib reflect library which just panics 87 | // whenever there is invalid input. 88 | defer func() { 89 | if r := recover(); r != nil { 90 | var ok bool 91 | err, ok = r.(error) 92 | if !ok { 93 | err = fmt.Errorf("%v", r) 94 | } 95 | } 96 | }() 97 | 98 | return reflect.New(reflectType(s)).Interface(), nil 99 | } 100 | -------------------------------------------------------------------------------- /protostructure.pb.go: -------------------------------------------------------------------------------- 1 | // Code generated by protoc-gen-go. DO NOT EDIT. 2 | // source: protostructure.proto 3 | 4 | package protostructure 5 | 6 | import proto "github.com/golang/protobuf/proto" 7 | import fmt "fmt" 8 | import math "math" 9 | 10 | // Reference imports to suppress errors if they are not otherwise used. 11 | var _ = proto.Marshal 12 | var _ = fmt.Errorf 13 | var _ = math.Inf 14 | 15 | // This is a compile-time assertion to ensure that this generated file 16 | // is compatible with the proto package it is being compiled against. 17 | // A compilation error at this line likely means your copy of the 18 | // proto package needs to be updated. 19 | const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package 20 | 21 | // Struct represents a struct type. 22 | // 23 | // This has the following limitations: 24 | // 25 | // * Circular references are not allowed between any struct types 26 | // * Embedded structs are not supported 27 | // * Methods are not preserved 28 | // 29 | type Struct struct { 30 | // fields is the list of fields in the struct 31 | Fields []*Struct_Field `protobuf:"bytes,1,rep,name=fields,proto3" json:"fields,omitempty"` 32 | XXX_NoUnkeyedLiteral struct{} `json:"-"` 33 | XXX_unrecognized []byte `json:"-"` 34 | XXX_sizecache int32 `json:"-"` 35 | } 36 | 37 | func (m *Struct) Reset() { *m = Struct{} } 38 | func (m *Struct) String() string { return proto.CompactTextString(m) } 39 | func (*Struct) ProtoMessage() {} 40 | func (*Struct) Descriptor() ([]byte, []int) { 41 | return fileDescriptor_protostructure_6518de6f1f7e1710, []int{0} 42 | } 43 | func (m *Struct) XXX_Unmarshal(b []byte) error { 44 | return xxx_messageInfo_Struct.Unmarshal(m, b) 45 | } 46 | func (m *Struct) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { 47 | return xxx_messageInfo_Struct.Marshal(b, m, deterministic) 48 | } 49 | func (dst *Struct) XXX_Merge(src proto.Message) { 50 | xxx_messageInfo_Struct.Merge(dst, src) 51 | } 52 | func (m *Struct) XXX_Size() int { 53 | return xxx_messageInfo_Struct.Size(m) 54 | } 55 | func (m *Struct) XXX_DiscardUnknown() { 56 | xxx_messageInfo_Struct.DiscardUnknown(m) 57 | } 58 | 59 | var xxx_messageInfo_Struct proto.InternalMessageInfo 60 | 61 | func (m *Struct) GetFields() []*Struct_Field { 62 | if m != nil { 63 | return m.Fields 64 | } 65 | return nil 66 | } 67 | 68 | // Field is a field type. See reflect.StructField in the Go stdlib 69 | // since the fields in this message match that almost exactly. 70 | type Struct_Field struct { 71 | Name string `protobuf:"bytes,1,opt,name=Name,proto3" json:"Name,omitempty"` 72 | PkgPath string `protobuf:"bytes,2,opt,name=PkgPath,proto3" json:"PkgPath,omitempty"` 73 | Tag string `protobuf:"bytes,3,opt,name=Tag,proto3" json:"Tag,omitempty"` 74 | Type *Type `protobuf:"bytes,4,opt,name=type,proto3" json:"type,omitempty"` 75 | XXX_NoUnkeyedLiteral struct{} `json:"-"` 76 | XXX_unrecognized []byte `json:"-"` 77 | XXX_sizecache int32 `json:"-"` 78 | } 79 | 80 | func (m *Struct_Field) Reset() { *m = Struct_Field{} } 81 | func (m *Struct_Field) String() string { return proto.CompactTextString(m) } 82 | func (*Struct_Field) ProtoMessage() {} 83 | func (*Struct_Field) Descriptor() ([]byte, []int) { 84 | return fileDescriptor_protostructure_6518de6f1f7e1710, []int{0, 0} 85 | } 86 | func (m *Struct_Field) XXX_Unmarshal(b []byte) error { 87 | return xxx_messageInfo_Struct_Field.Unmarshal(m, b) 88 | } 89 | func (m *Struct_Field) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { 90 | return xxx_messageInfo_Struct_Field.Marshal(b, m, deterministic) 91 | } 92 | func (dst *Struct_Field) XXX_Merge(src proto.Message) { 93 | xxx_messageInfo_Struct_Field.Merge(dst, src) 94 | } 95 | func (m *Struct_Field) XXX_Size() int { 96 | return xxx_messageInfo_Struct_Field.Size(m) 97 | } 98 | func (m *Struct_Field) XXX_DiscardUnknown() { 99 | xxx_messageInfo_Struct_Field.DiscardUnknown(m) 100 | } 101 | 102 | var xxx_messageInfo_Struct_Field proto.InternalMessageInfo 103 | 104 | func (m *Struct_Field) GetName() string { 105 | if m != nil { 106 | return m.Name 107 | } 108 | return "" 109 | } 110 | 111 | func (m *Struct_Field) GetPkgPath() string { 112 | if m != nil { 113 | return m.PkgPath 114 | } 115 | return "" 116 | } 117 | 118 | func (m *Struct_Field) GetTag() string { 119 | if m != nil { 120 | return m.Tag 121 | } 122 | return "" 123 | } 124 | 125 | func (m *Struct_Field) GetType() *Type { 126 | if m != nil { 127 | return m.Type 128 | } 129 | return nil 130 | } 131 | 132 | // Type represents a Go type. 133 | type Type struct { 134 | // Types that are valid to be assigned to Type: 135 | // *Type_Primitive 136 | // *Type_Container 137 | // *Type_Struct 138 | Type isType_Type `protobuf_oneof:"type"` 139 | XXX_NoUnkeyedLiteral struct{} `json:"-"` 140 | XXX_unrecognized []byte `json:"-"` 141 | XXX_sizecache int32 `json:"-"` 142 | } 143 | 144 | func (m *Type) Reset() { *m = Type{} } 145 | func (m *Type) String() string { return proto.CompactTextString(m) } 146 | func (*Type) ProtoMessage() {} 147 | func (*Type) Descriptor() ([]byte, []int) { 148 | return fileDescriptor_protostructure_6518de6f1f7e1710, []int{1} 149 | } 150 | func (m *Type) XXX_Unmarshal(b []byte) error { 151 | return xxx_messageInfo_Type.Unmarshal(m, b) 152 | } 153 | func (m *Type) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { 154 | return xxx_messageInfo_Type.Marshal(b, m, deterministic) 155 | } 156 | func (dst *Type) XXX_Merge(src proto.Message) { 157 | xxx_messageInfo_Type.Merge(dst, src) 158 | } 159 | func (m *Type) XXX_Size() int { 160 | return xxx_messageInfo_Type.Size(m) 161 | } 162 | func (m *Type) XXX_DiscardUnknown() { 163 | xxx_messageInfo_Type.DiscardUnknown(m) 164 | } 165 | 166 | var xxx_messageInfo_Type proto.InternalMessageInfo 167 | 168 | type isType_Type interface { 169 | isType_Type() 170 | } 171 | 172 | type Type_Primitive struct { 173 | Primitive *Primitive `protobuf:"bytes,1,opt,name=primitive,proto3,oneof"` 174 | } 175 | 176 | type Type_Container struct { 177 | Container *Container `protobuf:"bytes,2,opt,name=container,proto3,oneof"` 178 | } 179 | 180 | type Type_Struct struct { 181 | Struct *Struct `protobuf:"bytes,3,opt,name=struct,proto3,oneof"` 182 | } 183 | 184 | func (*Type_Primitive) isType_Type() {} 185 | 186 | func (*Type_Container) isType_Type() {} 187 | 188 | func (*Type_Struct) isType_Type() {} 189 | 190 | func (m *Type) GetType() isType_Type { 191 | if m != nil { 192 | return m.Type 193 | } 194 | return nil 195 | } 196 | 197 | func (m *Type) GetPrimitive() *Primitive { 198 | if x, ok := m.GetType().(*Type_Primitive); ok { 199 | return x.Primitive 200 | } 201 | return nil 202 | } 203 | 204 | func (m *Type) GetContainer() *Container { 205 | if x, ok := m.GetType().(*Type_Container); ok { 206 | return x.Container 207 | } 208 | return nil 209 | } 210 | 211 | func (m *Type) GetStruct() *Struct { 212 | if x, ok := m.GetType().(*Type_Struct); ok { 213 | return x.Struct 214 | } 215 | return nil 216 | } 217 | 218 | // XXX_OneofFuncs is for the internal use of the proto package. 219 | func (*Type) XXX_OneofFuncs() (func(msg proto.Message, b *proto.Buffer) error, func(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error), func(msg proto.Message) (n int), []interface{}) { 220 | return _Type_OneofMarshaler, _Type_OneofUnmarshaler, _Type_OneofSizer, []interface{}{ 221 | (*Type_Primitive)(nil), 222 | (*Type_Container)(nil), 223 | (*Type_Struct)(nil), 224 | } 225 | } 226 | 227 | func _Type_OneofMarshaler(msg proto.Message, b *proto.Buffer) error { 228 | m := msg.(*Type) 229 | // type 230 | switch x := m.Type.(type) { 231 | case *Type_Primitive: 232 | b.EncodeVarint(1<<3 | proto.WireBytes) 233 | if err := b.EncodeMessage(x.Primitive); err != nil { 234 | return err 235 | } 236 | case *Type_Container: 237 | b.EncodeVarint(2<<3 | proto.WireBytes) 238 | if err := b.EncodeMessage(x.Container); err != nil { 239 | return err 240 | } 241 | case *Type_Struct: 242 | b.EncodeVarint(3<<3 | proto.WireBytes) 243 | if err := b.EncodeMessage(x.Struct); err != nil { 244 | return err 245 | } 246 | case nil: 247 | default: 248 | return fmt.Errorf("Type.Type has unexpected type %T", x) 249 | } 250 | return nil 251 | } 252 | 253 | func _Type_OneofUnmarshaler(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error) { 254 | m := msg.(*Type) 255 | switch tag { 256 | case 1: // type.primitive 257 | if wire != proto.WireBytes { 258 | return true, proto.ErrInternalBadWireType 259 | } 260 | msg := new(Primitive) 261 | err := b.DecodeMessage(msg) 262 | m.Type = &Type_Primitive{msg} 263 | return true, err 264 | case 2: // type.container 265 | if wire != proto.WireBytes { 266 | return true, proto.ErrInternalBadWireType 267 | } 268 | msg := new(Container) 269 | err := b.DecodeMessage(msg) 270 | m.Type = &Type_Container{msg} 271 | return true, err 272 | case 3: // type.struct 273 | if wire != proto.WireBytes { 274 | return true, proto.ErrInternalBadWireType 275 | } 276 | msg := new(Struct) 277 | err := b.DecodeMessage(msg) 278 | m.Type = &Type_Struct{msg} 279 | return true, err 280 | default: 281 | return false, nil 282 | } 283 | } 284 | 285 | func _Type_OneofSizer(msg proto.Message) (n int) { 286 | m := msg.(*Type) 287 | // type 288 | switch x := m.Type.(type) { 289 | case *Type_Primitive: 290 | s := proto.Size(x.Primitive) 291 | n += 1 // tag and wire 292 | n += proto.SizeVarint(uint64(s)) 293 | n += s 294 | case *Type_Container: 295 | s := proto.Size(x.Container) 296 | n += 1 // tag and wire 297 | n += proto.SizeVarint(uint64(s)) 298 | n += s 299 | case *Type_Struct: 300 | s := proto.Size(x.Struct) 301 | n += 1 // tag and wire 302 | n += proto.SizeVarint(uint64(s)) 303 | n += s 304 | case nil: 305 | default: 306 | panic(fmt.Sprintf("proto: unexpected type %T in oneof", x)) 307 | } 308 | return n 309 | } 310 | 311 | // Primitive is a primitive type such as int, bool, etc. 312 | type Primitive struct { 313 | // kind is the reflect.Kind value for this primitive. This MUST be 314 | // a primitive value. For example, reflect.Ptr would be invalid here. 315 | Kind uint32 `protobuf:"varint,1,opt,name=kind,proto3" json:"kind,omitempty"` 316 | XXX_NoUnkeyedLiteral struct{} `json:"-"` 317 | XXX_unrecognized []byte `json:"-"` 318 | XXX_sizecache int32 `json:"-"` 319 | } 320 | 321 | func (m *Primitive) Reset() { *m = Primitive{} } 322 | func (m *Primitive) String() string { return proto.CompactTextString(m) } 323 | func (*Primitive) ProtoMessage() {} 324 | func (*Primitive) Descriptor() ([]byte, []int) { 325 | return fileDescriptor_protostructure_6518de6f1f7e1710, []int{2} 326 | } 327 | func (m *Primitive) XXX_Unmarshal(b []byte) error { 328 | return xxx_messageInfo_Primitive.Unmarshal(m, b) 329 | } 330 | func (m *Primitive) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { 331 | return xxx_messageInfo_Primitive.Marshal(b, m, deterministic) 332 | } 333 | func (dst *Primitive) XXX_Merge(src proto.Message) { 334 | xxx_messageInfo_Primitive.Merge(dst, src) 335 | } 336 | func (m *Primitive) XXX_Size() int { 337 | return xxx_messageInfo_Primitive.Size(m) 338 | } 339 | func (m *Primitive) XXX_DiscardUnknown() { 340 | xxx_messageInfo_Primitive.DiscardUnknown(m) 341 | } 342 | 343 | var xxx_messageInfo_Primitive proto.InternalMessageInfo 344 | 345 | func (m *Primitive) GetKind() uint32 { 346 | if m != nil { 347 | return m.Kind 348 | } 349 | return 0 350 | } 351 | 352 | // Container represents any "container" type such as a sliec, array, map, etc. 353 | type Container struct { 354 | // kind must be one of: array, map, ptr, slice 355 | Kind uint32 `protobuf:"varint,1,opt,name=kind,proto3" json:"kind,omitempty"` 356 | // elem is the type of the element of this container 357 | Elem *Type `protobuf:"bytes,2,opt,name=elem,proto3" json:"elem,omitempty"` 358 | // key is the type of the key, only if kind == map 359 | Key *Type `protobuf:"bytes,3,opt,name=key,proto3" json:"key,omitempty"` 360 | // count is the number of elements, only if kind == array 361 | Count int32 `protobuf:"varint,4,opt,name=count,proto3" json:"count,omitempty"` 362 | XXX_NoUnkeyedLiteral struct{} `json:"-"` 363 | XXX_unrecognized []byte `json:"-"` 364 | XXX_sizecache int32 `json:"-"` 365 | } 366 | 367 | func (m *Container) Reset() { *m = Container{} } 368 | func (m *Container) String() string { return proto.CompactTextString(m) } 369 | func (*Container) ProtoMessage() {} 370 | func (*Container) Descriptor() ([]byte, []int) { 371 | return fileDescriptor_protostructure_6518de6f1f7e1710, []int{3} 372 | } 373 | func (m *Container) XXX_Unmarshal(b []byte) error { 374 | return xxx_messageInfo_Container.Unmarshal(m, b) 375 | } 376 | func (m *Container) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { 377 | return xxx_messageInfo_Container.Marshal(b, m, deterministic) 378 | } 379 | func (dst *Container) XXX_Merge(src proto.Message) { 380 | xxx_messageInfo_Container.Merge(dst, src) 381 | } 382 | func (m *Container) XXX_Size() int { 383 | return xxx_messageInfo_Container.Size(m) 384 | } 385 | func (m *Container) XXX_DiscardUnknown() { 386 | xxx_messageInfo_Container.DiscardUnknown(m) 387 | } 388 | 389 | var xxx_messageInfo_Container proto.InternalMessageInfo 390 | 391 | func (m *Container) GetKind() uint32 { 392 | if m != nil { 393 | return m.Kind 394 | } 395 | return 0 396 | } 397 | 398 | func (m *Container) GetElem() *Type { 399 | if m != nil { 400 | return m.Elem 401 | } 402 | return nil 403 | } 404 | 405 | func (m *Container) GetKey() *Type { 406 | if m != nil { 407 | return m.Key 408 | } 409 | return nil 410 | } 411 | 412 | func (m *Container) GetCount() int32 { 413 | if m != nil { 414 | return m.Count 415 | } 416 | return 0 417 | } 418 | 419 | func init() { 420 | proto.RegisterType((*Struct)(nil), "protostructure.Struct") 421 | proto.RegisterType((*Struct_Field)(nil), "protostructure.Struct.Field") 422 | proto.RegisterType((*Type)(nil), "protostructure.Type") 423 | proto.RegisterType((*Primitive)(nil), "protostructure.Primitive") 424 | proto.RegisterType((*Container)(nil), "protostructure.Container") 425 | } 426 | 427 | func init() { 428 | proto.RegisterFile("protostructure.proto", fileDescriptor_protostructure_6518de6f1f7e1710) 429 | } 430 | 431 | var fileDescriptor_protostructure_6518de6f1f7e1710 = []byte{ 432 | // 304 bytes of a gzipped FileDescriptorProto 433 | 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x74, 0x90, 0xcd, 0x4a, 0xc4, 0x30, 434 | 0x14, 0x85, 0x27, 0xb6, 0x53, 0xe9, 0x2d, 0x8a, 0x84, 0x22, 0x51, 0x04, 0x4b, 0x17, 0xd2, 0x55, 435 | 0x91, 0xea, 0xc6, 0xad, 0x82, 0xcc, 0x4a, 0x4a, 0x9c, 0x17, 0xa8, 0x9d, 0x38, 0x86, 0x4e, 0x7f, 436 | 0xec, 0xa4, 0x42, 0x9f, 0xc0, 0xe7, 0x71, 0xe3, 0xf3, 0x49, 0x6e, 0x7f, 0xc4, 0x52, 0x77, 0xf7, 437 | 0x5c, 0xbe, 0x93, 0x7b, 0x72, 0xc0, 0xad, 0xea, 0x52, 0x95, 0x7b, 0x55, 0x37, 0xa9, 0x6a, 0x6a, 438 | 0x11, 0xa2, 0xa4, 0xc7, 0x7f, 0xb7, 0xfe, 0x17, 0x01, 0xeb, 0x19, 0x15, 0xbd, 0x05, 0xeb, 0x55, 439 | 0x8a, 0xdd, 0x66, 0xcf, 0x88, 0x67, 0x04, 0x4e, 0x74, 0x11, 0x4e, 0x5e, 0xe8, 0xb8, 0xf0, 0x51, 440 | 0x43, 0xbc, 0x67, 0xcf, 0xdf, 0x61, 0x89, 0x0b, 0x4a, 0xc1, 0x7c, 0x4a, 0x72, 0xc1, 0x88, 0x47, 441 | 0x02, 0x9b, 0xe3, 0x4c, 0x19, 0x1c, 0xc6, 0xd9, 0x36, 0x4e, 0xd4, 0x1b, 0x3b, 0xc0, 0xf5, 0x20, 442 | 0xe9, 0x09, 0x18, 0xeb, 0x64, 0xcb, 0x0c, 0xdc, 0xea, 0x91, 0x06, 0x60, 0xaa, 0xb6, 0x12, 0xcc, 443 | 0xf4, 0x48, 0xe0, 0x44, 0xee, 0xf4, 0xf8, 0xba, 0xad, 0x04, 0x47, 0xc2, 0xff, 0x26, 0x60, 0x6a, 444 | 0x49, 0xef, 0xc0, 0xae, 0x6a, 0x99, 0x4b, 0x25, 0x3f, 0xba, 0xbb, 0x4e, 0x74, 0x36, 0xf5, 0xc5, 445 | 0x03, 0xb0, 0x5a, 0xf0, 0x5f, 0x5a, 0x5b, 0xd3, 0xb2, 0x50, 0x89, 0x2c, 0x44, 0x8d, 0xd9, 0x66, 446 | 0xac, 0x0f, 0x03, 0xa0, 0xad, 0x23, 0x4d, 0xaf, 0xc1, 0xea, 0x18, 0x4c, 0xef, 0x44, 0xa7, 0xf3, 447 | 0x3d, 0xad, 0x16, 0xbc, 0xe7, 0xee, 0xad, 0xee, 0x6b, 0xfe, 0x25, 0xd8, 0x63, 0x1c, 0xdd, 0x57, 448 | 0x26, 0x8b, 0x0d, 0xe6, 0x3e, 0xe2, 0x38, 0xfb, 0x9f, 0x04, 0xec, 0xf1, 0xea, 0x1c, 0xa1, 0x5b, 449 | 0x12, 0x3b, 0x91, 0xf7, 0x91, 0xff, 0x69, 0x49, 0x13, 0xf4, 0x0a, 0x8c, 0x4c, 0xb4, 0x7d, 0xc6, 450 | 0x79, 0x50, 0x03, 0xd4, 0x85, 0x65, 0x5a, 0x36, 0x85, 0xc2, 0xe2, 0x97, 0xbc, 0x13, 0x2f, 0x16, 451 | 0xf2, 0x37, 0x3f, 0x01, 0x00, 0x00, 0xff, 0xff, 0x8c, 0x7e, 0x7c, 0x3a, 0x46, 0x02, 0x00, 0x00, 452 | } 453 | -------------------------------------------------------------------------------- /protostructure.proto: -------------------------------------------------------------------------------- 1 | syntax = "proto3"; 2 | 3 | package protostructure; 4 | 5 | option go_package = "github.com/mitchellh/protostructure"; 6 | 7 | // Struct represents a struct type. 8 | // 9 | // This has the following limitations: 10 | // 11 | // * Circular references are not allowed between any struct types 12 | // * Embedded structs are not supported 13 | // * Methods are not preserved 14 | // 15 | message Struct { 16 | // fields is the list of fields in the struct 17 | repeated Field fields = 1; 18 | 19 | // Field is a field type. See reflect.StructField in the Go stdlib 20 | // since the fields in this message match that almost exactly. 21 | message Field { 22 | string Name = 1; 23 | string PkgPath = 2; 24 | string Tag = 3; 25 | Type type = 4; 26 | } 27 | } 28 | 29 | // Type represents a Go type. 30 | message Type { 31 | oneof type { 32 | Primitive primitive = 1; 33 | Container container = 2; 34 | Struct struct = 3; 35 | 36 | // NOTE(mitchellh): for now, we only allow embedding a full Struct type 37 | // here. If there are a number of fields with the same Struct, then we'll 38 | // repeat ourselves a lot. If this is ever a real problem, we can introduce 39 | // some sort of "reference" mechanism but it wasn't necessary when this 40 | // was first built. 41 | } 42 | } 43 | 44 | // Primitive is a primitive type such as int, bool, etc. 45 | message Primitive { 46 | // kind is the reflect.Kind value for this primitive. This MUST be 47 | // a primitive value. For example, reflect.Ptr would be invalid here. 48 | uint32 kind = 1; 49 | } 50 | 51 | // Container represents any "container" type such as a sliec, array, map, etc. 52 | message Container { 53 | // kind must be one of: array, map, ptr, slice 54 | uint32 kind = 1; 55 | 56 | // elem is the type of the element of this container 57 | Type elem = 2; 58 | 59 | // key is the type of the key, only if kind == map 60 | Type key = 3; 61 | 62 | // count is the number of elements, only if kind == array 63 | int32 count = 4; 64 | } 65 | -------------------------------------------------------------------------------- /protostructure_test.go: -------------------------------------------------------------------------------- 1 | package protostructure 2 | 3 | import ( 4 | "encoding/json" 5 | "testing" 6 | 7 | "github.com/stretchr/testify/require" 8 | ) 9 | 10 | func TestEncode(t *testing.T) { 11 | type nested struct { 12 | Value string `json:"value"` 13 | } 14 | 15 | type test struct { 16 | Value string `json:"v"` 17 | Map map[string]string `json:"map"` 18 | Nested nested `json:"nested"` 19 | NestedPtr *nested `json:"nested_ptr"` 20 | Slice []int `json:"slice"` 21 | SliceString []string `json:"slice_string"` 22 | Array [3]int `json:"array"` 23 | } 24 | 25 | cases := []struct { 26 | Name string // name of the test 27 | Value interface{} // value to encode 28 | Err string // err to expect if any 29 | JSON string // json value to test unmarshal/marshal equality (can be blank) 30 | }{ 31 | { 32 | "major field test", 33 | &test{}, 34 | "", 35 | ` 36 | { 37 | "v": "hello", 38 | "map": { "key": "value" }, 39 | "nested": { "value": "direct" }, 40 | "nested_ptr": { "value": "ptr" }, 41 | "slice": [1, 4, 8], 42 | "slice_string": ["foo", "bar", "baz"], 43 | "array": [1, 2, 3] 44 | }`, 45 | }, 46 | 47 | { 48 | "direct struct (not pointer)", 49 | test{}, 50 | "", 51 | "", 52 | }, 53 | 54 | { 55 | "not a struct", 56 | 12, 57 | "got int", 58 | "", 59 | }, 60 | } 61 | 62 | for _, tt := range cases { 63 | t.Run(tt.Name, func(t *testing.T) { 64 | require := require.New(t) 65 | 66 | s, err := Encode(tt.Value) 67 | if tt.Err != "" { 68 | require.Error(err) 69 | require.Nil(s) 70 | require.Contains(err.Error(), tt.Err) 71 | return 72 | } 73 | require.NoError(err) 74 | require.NotNil(s) 75 | 76 | if tt.JSON == "" { 77 | return 78 | } 79 | 80 | // Unmarhal into real 81 | require.NoError(json.Unmarshal([]byte(tt.JSON), tt.Value)) 82 | 83 | // Unmarshal into dynamic 84 | v, err := New(s) 85 | require.NoError(err) 86 | require.NoError(json.Unmarshal([]byte(tt.JSON), v)) 87 | 88 | // Remarshal both and compare results 89 | marshalReal, err := json.Marshal(tt.Value) 90 | require.NoError(err) 91 | marshalDynamic, err := json.Marshal(v) 92 | require.NoError(err) 93 | require.Equal(string(marshalReal), string(marshalDynamic)) 94 | }) 95 | } 96 | } 97 | -------------------------------------------------------------------------------- /type.go: -------------------------------------------------------------------------------- 1 | package protostructure 2 | 3 | import ( 4 | "fmt" 5 | "reflect" 6 | ) 7 | 8 | // reflectType returns the reflect.Type for a Struct. This will panic if 9 | // there are any invalid values in Struct. This behavior is not ideal but 10 | // is inherited primarily by the underlying "reflect" standard library. 11 | // 12 | // It is recommended that you use a defer with a recover around this function 13 | // in most cases since Struct is probably sent over the wire. 14 | func reflectType(s *Struct) reflect.Type { 15 | var fields []reflect.StructField 16 | for _, field := range s.Fields { 17 | fields = append(fields, reflect.StructField{ 18 | Name: field.Name, 19 | PkgPath: field.PkgPath, 20 | Tag: reflect.StructTag(field.Tag), 21 | Type: goType(field.Type), 22 | }) 23 | } 24 | 25 | return reflect.StructOf(fields) 26 | } 27 | 28 | // goType returns the reflect.Type for a *Type value. 29 | func goType(t *Type) reflect.Type { 30 | switch t := t.Type.(type) { 31 | case *Type_Primitive: 32 | // Look up the type directly by kind 33 | return kindTypes[reflect.Kind(t.Primitive.Kind)] 34 | 35 | case *Type_Struct: 36 | // Build the type 37 | return reflectType(t.Struct) 38 | 39 | case *Type_Container: 40 | switch reflect.Kind(t.Container.Kind) { 41 | case reflect.Map: 42 | return reflect.MapOf(goType(t.Container.Key), goType(t.Container.Elem)) 43 | 44 | case reflect.Ptr: 45 | return reflect.PtrTo(goType(t.Container.Elem)) 46 | 47 | case reflect.Slice: 48 | return reflect.SliceOf(goType(t.Container.Elem)) 49 | 50 | case reflect.Array: 51 | return reflect.ArrayOf(int(t.Container.Count), goType(t.Container.Elem)) 52 | } 53 | } 54 | 55 | panic(fmt.Sprintf("unknown type to decode: %#v", t)) 56 | } 57 | 58 | // protoType takes a reflect.Type can returns the proto *Type value. 59 | // This will return an error if the type given cannot be represented by 60 | // protocol buffer messages. 61 | func protoType(t reflect.Type) (*Type, error) { 62 | switch k := t.Kind(); k { 63 | // Primitives 64 | case reflect.Bool, 65 | reflect.Int, 66 | reflect.Int8, 67 | reflect.Int16, 68 | reflect.Int32, 69 | reflect.Int64, 70 | reflect.Uint, 71 | reflect.Uint8, 72 | reflect.Uint16, 73 | reflect.Uint32, 74 | reflect.Uint64, 75 | reflect.Uintptr, 76 | reflect.Float32, 77 | reflect.Float64, 78 | reflect.Complex64, 79 | reflect.Complex128, 80 | reflect.Interface, 81 | reflect.String: 82 | return &Type{ 83 | Type: &Type_Primitive{ 84 | Primitive: &Primitive{ 85 | Kind: uint32(k), 86 | }, 87 | }, 88 | }, nil 89 | 90 | case reflect.Array: 91 | elem, err := protoType(t.Elem()) 92 | if err != nil { 93 | return nil, err 94 | } 95 | 96 | return &Type{ 97 | Type: &Type_Container{ 98 | Container: &Container{ 99 | Kind: uint32(k), 100 | Elem: elem, 101 | Count: int32(t.Len()), 102 | }, 103 | }, 104 | }, nil 105 | 106 | case reflect.Map: 107 | key, err := protoType(t.Key()) 108 | if err != nil { 109 | return nil, err 110 | } 111 | 112 | elem, err := protoType(t.Elem()) 113 | if err != nil { 114 | return nil, err 115 | } 116 | 117 | return &Type{ 118 | Type: &Type_Container{ 119 | Container: &Container{ 120 | Kind: uint32(k), 121 | Elem: elem, 122 | Key: key, 123 | }, 124 | }, 125 | }, nil 126 | 127 | case reflect.Ptr, reflect.Slice: 128 | elem, err := protoType(t.Elem()) 129 | if err != nil { 130 | return nil, err 131 | } 132 | 133 | return &Type{ 134 | Type: &Type_Container{ 135 | Container: &Container{ 136 | Kind: uint32(k), 137 | Elem: elem, 138 | }, 139 | }, 140 | }, nil 141 | 142 | case reflect.Struct: 143 | elem, err := Encode(t) 144 | if err != nil { 145 | return nil, err 146 | } 147 | 148 | return &Type{ 149 | Type: &Type_Struct{ 150 | Struct: elem, 151 | }, 152 | }, nil 153 | 154 | default: 155 | return nil, fmt.Errorf("encode: cannot encode type: %s (kind = %s)", t.String(), k) 156 | } 157 | } 158 | 159 | // kindTypes is a mapping from reflect.Kind to an equivalent reflect.Type 160 | // representing that kind. This only contains the "primitive" values as 161 | // defined by what a valid value is for a Type_Primitive. 162 | var kindTypes = map[reflect.Kind]reflect.Type{ 163 | reflect.Bool: reflect.TypeOf(false), 164 | reflect.Int: reflect.TypeOf(int(0)), 165 | reflect.Int8: reflect.TypeOf(int8(0)), 166 | reflect.Int16: reflect.TypeOf(int16(0)), 167 | reflect.Int32: reflect.TypeOf(int32(0)), 168 | reflect.Int64: reflect.TypeOf(int64(0)), 169 | reflect.Uint: reflect.TypeOf(uint(0)), 170 | reflect.Uint8: reflect.TypeOf(uint8(0)), 171 | reflect.Uint16: reflect.TypeOf(uint16(0)), 172 | reflect.Uint32: reflect.TypeOf(uint32(0)), 173 | reflect.Uint64: reflect.TypeOf(uint64(0)), 174 | reflect.Uintptr: reflect.TypeOf(uintptr(0)), 175 | reflect.Float32: reflect.TypeOf(float32(0)), 176 | reflect.Float64: reflect.TypeOf(float64(0)), 177 | reflect.Complex64: reflect.TypeOf(complex64(0)), 178 | reflect.Complex128: reflect.TypeOf(complex128(0)), 179 | reflect.Interface: reflect.TypeOf((*interface{})(nil)).Elem(), 180 | reflect.String: reflect.TypeOf(""), 181 | } 182 | -------------------------------------------------------------------------------- /type_test.go: -------------------------------------------------------------------------------- 1 | package protostructure 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestKindTypes(t *testing.T) { 10 | for kind, typ := range kindTypes { 11 | t.Run(kind.String(), func(t *testing.T) { 12 | require.Equal(t, kind, typ.Kind()) 13 | }) 14 | } 15 | } 16 | --------------------------------------------------------------------------------