├── kv ├── doc.go ├── serialize.go ├── kv_test.go └── kv.go ├── examples └── main.go ├── README.md ├── tests └── integration_test.go └── LICENSE /kv/doc.go: -------------------------------------------------------------------------------- 1 | /* 2 | Package kv implements a low-level key/value store backed by Kubernetes config maps. 3 | It supports main operations expected from key/value store such as Put, Get, Delete and List. 4 | Operations are protected by an internal mutex and therefore can be safely used inside a single 5 | node application. 6 | Basics 7 | There are only few things worth to know: key/value database is created based on bucket name so in order 8 | to have multiple configMaps - use different bucket names. Teardown() function will remove configMap entry 9 | completely destroying all entries. 10 | Caveats 11 | Since k8s-kv is based on configMaps which are in turn based on Etcd key/value store - all values have a limitation 12 | of 1MB so each bucket in k8s-kv is limited to that size. To overcome it - create more buckets. 13 | If you have multi-node application that is frequently reading/writing to the same buckets - be aware of race 14 | conditions as it doesn't provide any cross-node locking capabilities. 15 | */ 16 | package kv 17 | -------------------------------------------------------------------------------- /examples/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "os" 6 | "path/filepath" 7 | 8 | "github.com/rusenask/k8s-kv/kv" 9 | 10 | "k8s.io/client-go/kubernetes" 11 | core_v1 "k8s.io/client-go/kubernetes/typed/core/v1" 12 | "k8s.io/client-go/tools/clientcmd" 13 | ) 14 | 15 | func getImplementer() (implementer core_v1.ConfigMapInterface) { 16 | 17 | cfg, err := clientcmd.BuildConfigFromFlags("", filepath.Join(os.Getenv("HOME"), ".kube", "config")) // in your app you could replace it with in-cluster-config 18 | if err != nil { 19 | panic(err) 20 | } 21 | 22 | client, err := kubernetes.NewForConfig(cfg) 23 | if err != nil { 24 | panic(err) 25 | } 26 | 27 | return client.CoreV1().ConfigMaps("default") 28 | } 29 | 30 | func main() { 31 | impl := getImplementer() 32 | 33 | kvdb, err := kv.New(impl, "my-app", "bucket1") 34 | if err != nil { 35 | panic(err) 36 | } 37 | 38 | kvdb.Put("foo", []byte("hello kubernetes world")) 39 | 40 | stored, _ := kvdb.Get("foo") 41 | 42 | fmt.Println(string(stored)) 43 | } 44 | -------------------------------------------------------------------------------- /kv/serialize.go: -------------------------------------------------------------------------------- 1 | package kv 2 | 3 | import ( 4 | "bytes" 5 | "encoding/gob" 6 | "sync" 7 | ) 8 | 9 | var bufferPool = sync.Pool{New: allocBuffer} 10 | 11 | func allocBuffer() interface{} { 12 | return &bytes.Buffer{} 13 | } 14 | 15 | func getBuffer() *bytes.Buffer { 16 | return bufferPool.Get().(*bytes.Buffer) 17 | } 18 | 19 | func releaseBuffer(v *bytes.Buffer) { 20 | v.Reset() 21 | v.Grow(0) 22 | bufferPool.Put(v) 23 | } 24 | 25 | // Serializer - generic serializer interface 26 | type Serializer interface { 27 | Encode(source interface{}) ([]byte, error) 28 | Decode(data []byte, target interface{}) error 29 | } 30 | 31 | // DefaultSerializer - returns default serializer 32 | func DefaultSerializer() Serializer { 33 | return &GobSerializer{} 34 | } 35 | 36 | // GobSerializer - gob based serializer 37 | type GobSerializer struct{} 38 | 39 | // Encode - encodes source into bytes using Gob encoder 40 | func (s *GobSerializer) Encode(source interface{}) ([]byte, error) { 41 | buf := getBuffer() 42 | defer releaseBuffer(buf) 43 | enc := gob.NewEncoder(buf) 44 | err := enc.Encode(source) 45 | if err != nil { 46 | return nil, err 47 | } 48 | return buf.Bytes(), nil 49 | } 50 | 51 | // Decode - decodes given bytes into target struct 52 | func (s *GobSerializer) Decode(data []byte, target interface{}) error { 53 | buf := bytes.NewBuffer(data) 54 | dec := gob.NewDecoder(buf) 55 | return dec.Decode(target) 56 | } 57 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Kubernetes backed KV 2 | 3 | [![GoDoc](https://godoc.org/github.com/rusenask/k8s-kv/kv?status.svg)](https://godoc.org/github.com/rusenask/k8s-kv/kv) 4 | 5 | Use Kubernetes config maps as key/value store! 6 | When to use k8s-kv: 7 | * You have a simple application that has a need to store some configuration and you can't be bothered to set up EBS like volumes or use some fancy external KV store. 8 | * You have a stateless application that suddenly got to store state and you are not into converting 9 | it into full stateless app that will use a proper database. 10 | 11 | When __not to__ use k8s-kv: 12 | * You have a read/write heavy multi-node application (k8s-kv doesn't have cross-app locking). 13 | * You want to store bigger values than 1MB. Even though k8s-kv uses compression for the data stored in bucket - it's wise to not try the limits. It's there because of the limit in Etcd. In this case use something else. 14 | 15 | 16 | ## Basics 17 | 18 | Package API: 19 | 20 | ``` 21 | // Pyt key/value pair into the store 22 | Put(key string, value []byte) error 23 | // Get value of the specified key 24 | Get(key string) (value []byte, err error) 25 | // Delete key/value pair from the store 26 | Delete(key string) error 27 | // List all key/value pairs under specified prefix 28 | List(prefix string) (data map[string][]byte, err error) 29 | // Delete config map (results in deleted data) 30 | Teardown() error 31 | ``` 32 | 33 | ## Caveats 34 | 35 | * Don't be silly, you can't put a lot of stuff here. 36 | 37 | ## Example 38 | 39 | Usage example: 40 | 41 | 1. Get minikube or your favourite k8s environment running. 42 | 43 | 2. In your app you will probably want to use this: https://github.com/kubernetes/client-go/tree/master/examples/in-cluster-client-configuration 44 | 45 | 3. Get ConfigMaps interface and supply it to this lib: 46 | 47 | ``` 48 | package main 49 | 50 | import ( 51 | "fmt" 52 | 53 | "github.com/rusenask/k8s-kv/kv" 54 | 55 | "k8s.io/client-go/kubernetes" 56 | core_v1 "k8s.io/client-go/kubernetes/typed/core/v1" 57 | "k8s.io/client-go/tools/clientcmd" 58 | ) 59 | 60 | // get ConfigMapInterface to access config maps in "default" namespace 61 | func getImplementer() (implementer core_v1.ConfigMapInterface) { 62 | cfg, err := clientcmd.BuildConfigFromFlags("", ".kubeconfig") // in your app you could replace it with in-cluster-config 63 | if err != nil { 64 | panic(err) 65 | } 66 | 67 | client, err := kubernetes.NewForConfig(cfg) 68 | if err != nil { 69 | panic(err) 70 | } 71 | 72 | return client.ConfigMaps("default") 73 | } 74 | 75 | func main() { 76 | impl := getImplementer() 77 | 78 | // getting acces to k8s-kv. "my-app" will become a label 79 | // for this config map, this way it's easier to manage configs 80 | // "bucket1" will be config map's name and represent one entry in config maps list 81 | kvdb, err := kv.New(impl, "my-app", "bucket1") 82 | if err != nil { 83 | panic(err) 84 | } 85 | 86 | // insert a key "foo" with value "hello kubernetes world" 87 | kvdb.Put("foo", []byte("hello kubernetes world")) 88 | 89 | // get value of key "foo" 90 | stored, _ := kvdb.Get("foo") 91 | 92 | fmt.Println(string(stored)) 93 | } 94 | ``` -------------------------------------------------------------------------------- /kv/kv_test.go: -------------------------------------------------------------------------------- 1 | package kv 2 | 3 | import ( 4 | "fmt" 5 | "testing" 6 | 7 | "k8s.io/api/core/v1" 8 | 9 | meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1" 10 | ) 11 | 12 | type fakeImplementer struct { 13 | getcfgMap *v1.ConfigMap 14 | 15 | createdMap *v1.ConfigMap 16 | updatedMap *v1.ConfigMap 17 | 18 | deletedName string 19 | deletedOptions *meta_v1.DeleteOptions 20 | } 21 | 22 | func (i *fakeImplementer) Get(name string, options meta_v1.GetOptions) (*v1.ConfigMap, error) { 23 | return i.getcfgMap, nil 24 | } 25 | 26 | func (i *fakeImplementer) Create(cfgMap *v1.ConfigMap) (*v1.ConfigMap, error) { 27 | i.createdMap = cfgMap 28 | return i.createdMap, nil 29 | } 30 | 31 | func (i *fakeImplementer) Update(cfgMap *v1.ConfigMap) (*v1.ConfigMap, error) { 32 | i.updatedMap = cfgMap 33 | return i.updatedMap, nil 34 | } 35 | 36 | func (i *fakeImplementer) Delete(name string, options *meta_v1.DeleteOptions) error { 37 | i.deletedName = name 38 | i.deletedOptions = options 39 | return nil 40 | } 41 | 42 | func TestGetMap(t *testing.T) { 43 | fi := &fakeImplementer{ 44 | getcfgMap: &v1.ConfigMap{ 45 | Data: map[string]string{ 46 | "foo": "bar", 47 | }, 48 | }, 49 | } 50 | kv, err := New(fi, "app", "b1") 51 | if err != nil { 52 | t.Fatalf("failed to get kv: %s", err) 53 | } 54 | 55 | cfgMap, err := kv.getMap() 56 | if err != nil { 57 | t.Fatalf("failed to get map: %s", err) 58 | } 59 | 60 | if cfgMap.Data["foo"] != "bar" { 61 | t.Errorf("cfgMap.Data is missing expected key") 62 | } 63 | } 64 | 65 | func TestGet(t *testing.T) { 66 | 67 | im := map[string][]byte{ 68 | "foo": []byte("bar"), 69 | } 70 | fi := &fakeImplementer{ 71 | getcfgMap: &v1.ConfigMap{ 72 | Data: map[string]string{}, 73 | }, 74 | } 75 | kv, err := New(fi, "app", "b1") 76 | if err != nil { 77 | t.Fatalf("failed to get kv: %s", err) 78 | } 79 | 80 | cfgMap, _ := kv.getMap() 81 | 82 | kv.saveInternalMap(cfgMap, im) 83 | 84 | val, err := kv.Get("foo") 85 | if err != nil { 86 | t.Fatalf("failed to get key: %s", err) 87 | } 88 | 89 | if string(val) != "bar" { 90 | t.Errorf("expected 'bar' but got: %s", string(val)) 91 | } 92 | } 93 | 94 | func TestUpdate(t *testing.T) { 95 | 96 | im := map[string][]byte{ 97 | "a": []byte("a-val"), 98 | "b": []byte("b-val"), 99 | "c": []byte("c-val"), 100 | "d": []byte("d-val"), 101 | } 102 | 103 | fi := &fakeImplementer{ 104 | getcfgMap: &v1.ConfigMap{ 105 | Data: map[string]string{}, 106 | }, 107 | } 108 | kv, err := New(fi, "app", "b1") 109 | if err != nil { 110 | t.Fatalf("failed to get kv: %s", err) 111 | } 112 | 113 | cfgMap, _ := kv.getMap() 114 | 115 | kv.saveInternalMap(cfgMap, im) 116 | 117 | err = kv.Put("b", []byte("updated")) 118 | if err != nil { 119 | t.Fatalf("failed to get key: %s", err) 120 | } 121 | 122 | updatedIm, err := decodeInternalMap(kv.serializer, fi.updatedMap.Data[dataKey]) 123 | if err != nil { 124 | t.Fatalf("failed to decode internal map: %s", err) 125 | } 126 | 127 | if string(updatedIm["b"]) != "updated" { 128 | t.Errorf("b value was not updated") 129 | } 130 | 131 | } 132 | 133 | func TestEncodeInternal(t *testing.T) { 134 | serializer := DefaultSerializer() 135 | 136 | im := make(map[string][]byte) 137 | 138 | for i := 0; i < 100; i++ { 139 | im[fmt.Sprintf("foo-%d", i)] = []byte(fmt.Sprintf("some important data here %d", i)) 140 | } 141 | 142 | encoded, err := encodeInternalMap(serializer, im) 143 | if err != nil { 144 | t.Fatalf("failed to encode map: %s", err) 145 | } 146 | 147 | decoded, err := decodeInternalMap(serializer, encoded) 148 | if err != nil { 149 | t.Fatalf("failed to decode map: %s", err) 150 | } 151 | 152 | if string(decoded["foo-1"]) != "some important data here 1" { 153 | t.Errorf("expected to find 'some important data here 1' but got: %s", string(decoded["foo-1"])) 154 | } 155 | 156 | } 157 | -------------------------------------------------------------------------------- /tests/integration_test.go: -------------------------------------------------------------------------------- 1 | package tests 2 | 3 | import ( 4 | "fmt" 5 | "testing" 6 | 7 | "github.com/rusenask/k8s-kv/kv" 8 | 9 | "k8s.io/client-go/kubernetes" 10 | core_v1 "k8s.io/client-go/kubernetes/typed/core/v1" 11 | "k8s.io/client-go/tools/clientcmd" 12 | ) 13 | 14 | const clusterConfig = ".kubeconfig" 15 | const testingNamespace = "default" 16 | 17 | func getImplementer(t *testing.T) (implementer core_v1.ConfigMapInterface) { 18 | cfg, err := clientcmd.BuildConfigFromFlags("", clusterConfig) 19 | if err != nil { 20 | t.Fatalf("failed to get config: %s", err) 21 | } 22 | 23 | client, err := kubernetes.NewForConfig(cfg) 24 | if err != nil { 25 | t.Fatalf("failed to create client: %s", err) 26 | } 27 | 28 | return client.ConfigMaps(testingNamespace) 29 | } 30 | 31 | func TestPut(t *testing.T) { 32 | 33 | impl := getImplementer(t) 34 | kv, err := kv.New(impl, "test", "testput") 35 | if err != nil { 36 | t.Fatalf("failed to create kv: %s", err) 37 | } 38 | defer kv.Teardown() 39 | 40 | err = kv.Put("key", []byte("val")) 41 | if err != nil { 42 | t.Errorf("failed to put: %s", err) 43 | } 44 | } 45 | 46 | func TestPutDirectoryKeys(t *testing.T) { 47 | impl := getImplementer(t) 48 | kv, err := kv.New(impl, "test", "testputdirectorykeys") 49 | if err != nil { 50 | t.Fatalf("failed to create kv: %s", err) 51 | } 52 | defer kv.Teardown() 53 | 54 | err = kv.Put("/somedir/key-here", []byte("val")) 55 | if err != nil { 56 | t.Errorf("failed to put: %s", err) 57 | } 58 | 59 | val, err := kv.Get("/somedir/key-here") 60 | if err != nil { 61 | t.Errorf("failed to get key: %s", err) 62 | } 63 | 64 | if string(val) != "val" { 65 | t.Errorf("unexpected return: %s", string(val)) 66 | } 67 | } 68 | 69 | func TestGet(t *testing.T) { 70 | 71 | impl := getImplementer(t) 72 | kv, err := kv.New(impl, "test", "testget") 73 | if err != nil { 74 | t.Fatalf("failed to create kv: %s", err) 75 | } 76 | defer kv.Teardown() 77 | 78 | err = kv.Put("foo", []byte("bar")) 79 | if err != nil { 80 | t.Errorf("failed to put: %s", err) 81 | } 82 | 83 | // getting it back 84 | val, err := kv.Get("foo") 85 | if err != nil { 86 | t.Errorf("failed to get: %s", err) 87 | } 88 | 89 | if string(val) != "bar" { 90 | t.Errorf("expected 'bar' but got: '%s'", string(val)) 91 | } 92 | 93 | } 94 | 95 | func TestDelete(t *testing.T) { 96 | 97 | impl := getImplementer(t) 98 | kvdb, err := kv.New(impl, "test", "testdelete") 99 | if err != nil { 100 | t.Fatalf("failed to create kv: %s", err) 101 | } 102 | defer kvdb.Teardown() 103 | 104 | err = kvdb.Put("foo", []byte("bar")) 105 | if err != nil { 106 | t.Errorf("failed to put: %s", err) 107 | } 108 | 109 | // getting it back 110 | val, err := kvdb.Get("foo") 111 | if err != nil { 112 | t.Errorf("failed to get: %s", err) 113 | } 114 | 115 | if string(val) != "bar" { 116 | t.Errorf("expected 'bar' but got: '%s'", string(val)) 117 | } 118 | 119 | // deleting it 120 | err = kvdb.Delete("foo") 121 | if err != nil { 122 | t.Errorf("got error while deleting: %s", err) 123 | } 124 | 125 | _, err = kvdb.Get("foo") 126 | if err != kv.ErrNotFound { 127 | t.Errorf("expected to get an error on deleted key") 128 | } 129 | } 130 | 131 | func TestList(t *testing.T) { 132 | count := 3 133 | 134 | impl := getImplementer(t) 135 | kv, err := kv.New(impl, "test", "testlist") 136 | if err != nil { 137 | t.Fatalf("failed to create kv: %s", err) 138 | } 139 | defer kv.Teardown() 140 | 141 | for i := 0; i < count; i++ { 142 | err = kv.Put(fmt.Sprint(i), []byte(fmt.Sprintf("bar-%d", i))) 143 | if err != nil { 144 | t.Errorf("failed to put: %s", err) 145 | } 146 | } 147 | 148 | items, err := kv.List("") 149 | if err != nil { 150 | t.Fatalf("failed to list items, error: %s", err) 151 | } 152 | 153 | if len(items) != count { 154 | t.Errorf("expected %d items, got: %d", count, len(items)) 155 | } 156 | 157 | if string(items["0"]) != "bar-0" { 158 | t.Errorf("unexpected value on '0': %s", items["0"]) 159 | } 160 | if string(items["1"]) != "bar-1" { 161 | t.Errorf("unexpected value on '1': %s", items["1"]) 162 | } 163 | if string(items["2"]) != "bar-2" { 164 | t.Errorf("unexpected value on '2': %s", items["2"]) 165 | } 166 | 167 | } 168 | 169 | func TestListPrefix(t *testing.T) { 170 | impl := getImplementer(t) 171 | kv, err := kv.New(impl, "test", "testlistprefix") 172 | if err != nil { 173 | t.Fatalf("failed to create kv: %s", err) 174 | } 175 | defer kv.Teardown() 176 | 177 | err = kv.Put("aaa", []byte("aaa")) 178 | if err != nil { 179 | t.Errorf("failed to put key, error: %s", err) 180 | } 181 | err = kv.Put("aaaaa", []byte("aaa")) 182 | if err != nil { 183 | t.Errorf("failed to put key, error: %s", err) 184 | } 185 | err = kv.Put("aaaaaaa", []byte("aaa")) 186 | if err != nil { 187 | t.Errorf("failed to put key, error: %s", err) 188 | } 189 | 190 | err = kv.Put("bbb", []byte("bbb")) 191 | if err != nil { 192 | t.Errorf("failed to put key, error: %s", err) 193 | } 194 | err = kv.Put("bbbbb", []byte("bbb")) 195 | if err != nil { 196 | t.Errorf("failed to put key, error: %s", err) 197 | } 198 | err = kv.Put("bbbbbbb", []byte("bbb")) 199 | if err != nil { 200 | t.Errorf("failed to put key, error: %s", err) 201 | } 202 | 203 | items, err := kv.List("aaa") 204 | if err != nil { 205 | t.Fatalf("failed to list items, error: %s", err) 206 | } 207 | 208 | if len(items) != 3 { 209 | t.Errorf("expected %d items, got: %d", 3, len(items)) 210 | } 211 | 212 | if string(items["aaa"]) != "aaa" { 213 | t.Errorf("unexpected value on 'aaa': %s", items["aaa"]) 214 | } 215 | if string(items["aaaaa"]) != "aaa" { 216 | t.Errorf("unexpected value on 'aaaaa': %s", items["aaaaa"]) 217 | } 218 | if string(items["aaaaaaa"]) != "aaa" { 219 | t.Errorf("unexpected value on 'aaaaaaa': %s", items["aaaaaaa"]) 220 | } 221 | 222 | } 223 | -------------------------------------------------------------------------------- /kv/kv.go: -------------------------------------------------------------------------------- 1 | package kv 2 | 3 | import ( 4 | "bytes" 5 | "compress/gzip" 6 | "encoding/base64" 7 | "encoding/gob" 8 | "errors" 9 | "io/ioutil" 10 | "strings" 11 | "sync" 12 | 13 | "k8s.io/api/core/v1" 14 | apierrors "k8s.io/apimachinery/pkg/api/errors" 15 | meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1" 16 | ) 17 | 18 | func init() { 19 | gob.Register(&internalMap{}) 20 | } 21 | 22 | type internalMap struct { 23 | Data map[string][]byte 24 | } 25 | 26 | // errors 27 | var ( 28 | ErrNotFound = errors.New("not found") 29 | ) 30 | 31 | var b64 = base64.StdEncoding 32 | 33 | // KVDB generic kv package interface 34 | type KVDB interface { 35 | Put(key string, value []byte) error 36 | Get(key string) (value []byte, err error) 37 | Delete(key string) error 38 | List(prefix string) (data map[string][]byte, err error) 39 | Teardown() error 40 | } 41 | 42 | // KV provides access to key/value store operations such as Put, Get, Delete, List. 43 | // Entry in ConfigMap is created based on bucket name and total size is limited to 1MB per bucket. 44 | // Operations are protected by an internal mutex so it's safe to use in a single node application. 45 | type KV struct { 46 | app string 47 | bucket string 48 | implementer ConfigMapInterface 49 | mu *sync.RWMutex 50 | serializer Serializer 51 | } 52 | 53 | // ConfigMapInterface implements a subset of Kubernetes original ConfigMapInterface to provide 54 | // required operations for k8s-kv. Main purpose of this interface is to enable easier testing. 55 | type ConfigMapInterface interface { 56 | Get(name string, options meta_v1.GetOptions) (*v1.ConfigMap, error) 57 | Create(cfgMap *v1.ConfigMap) (*v1.ConfigMap, error) 58 | Update(cfgMap *v1.ConfigMap) (*v1.ConfigMap, error) 59 | Delete(name string, options *meta_v1.DeleteOptions) error 60 | } 61 | 62 | // New creates a new instance of KV. Requires prepared ConfigMapInterface (provided by go-client), app and bucket names. 63 | // App name is used as a label to make it easier to distinguish different k8s-kv instances created by separate (or the same) 64 | // application. Bucket name is used to give a name to config map. 65 | func New(implementer ConfigMapInterface, app, bucket string) (*KV, error) { 66 | kv := &KV{ 67 | implementer: implementer, 68 | app: app, 69 | bucket: bucket, 70 | mu: &sync.RWMutex{}, 71 | serializer: DefaultSerializer(), 72 | } 73 | 74 | _, err := kv.getMap() 75 | if err != nil { 76 | return nil, err 77 | } 78 | 79 | return kv, nil 80 | 81 | } 82 | 83 | // Teardown deletes configMap for this bucket. All bucket's data is lost. 84 | func (k *KV) Teardown() error { 85 | return k.implementer.Delete(k.bucket, &meta_v1.DeleteOptions{}) 86 | } 87 | 88 | func (k *KV) getMap() (*v1.ConfigMap, error) { 89 | cfgMap, err := k.implementer.Get(k.bucket, meta_v1.GetOptions{}) 90 | if err != nil { 91 | // creating 92 | if apierrors.IsNotFound(err) { 93 | return k.newConfigMapsObject() 94 | } 95 | return nil, err 96 | } 97 | 98 | if cfgMap.Data == nil { 99 | cfgMap.Data = make(map[string]string) 100 | } 101 | 102 | // it's there, nothing to do 103 | return cfgMap, nil 104 | } 105 | 106 | func encodeInternalMap(serializer Serializer, data map[string][]byte) (string, error) { 107 | var im internalMap 108 | im.Data = data 109 | bts, err := serializer.Encode(&im) 110 | if err != nil { 111 | return "", err 112 | } 113 | 114 | var buf bytes.Buffer 115 | w, err := gzip.NewWriterLevel(&buf, gzip.BestCompression) 116 | if err != nil { 117 | return "", err 118 | } 119 | if _, err = w.Write(bts); err != nil { 120 | return "", err 121 | } 122 | w.Close() 123 | 124 | return b64.EncodeToString(buf.Bytes()), nil 125 | } 126 | 127 | func decodeInternalMap(serializer Serializer, data string) (map[string][]byte, error) { 128 | if data == "" { 129 | empty := make(map[string][]byte) 130 | return empty, nil 131 | } 132 | 133 | b, err := b64.DecodeString(data) 134 | if err != nil { 135 | return nil, err 136 | } 137 | 138 | r, err := gzip.NewReader(bytes.NewReader(b)) 139 | if err != nil { 140 | return nil, err 141 | } 142 | decompressed, err := ioutil.ReadAll(r) 143 | if err != nil { 144 | return nil, err 145 | } 146 | 147 | var im internalMap 148 | 149 | err = serializer.Decode(decompressed, &im) 150 | return im.Data, err 151 | } 152 | 153 | const dataKey = "data" 154 | 155 | func (k *KV) newConfigMapsObject() (*v1.ConfigMap, error) { 156 | 157 | var lbs labels 158 | 159 | lbs.init() 160 | 161 | // apply labels 162 | lbs.set("BUCKET", k.bucket) 163 | lbs.set("APP", k.app) 164 | lbs.set("OWNER", "K8S-KV") 165 | 166 | // create and return configmap object 167 | cfgMap := &v1.ConfigMap{ 168 | ObjectMeta: meta_v1.ObjectMeta{ 169 | Name: k.bucket, 170 | Labels: lbs.toMap(), 171 | }, 172 | Data: map[string]string{ 173 | dataKey: "", 174 | }, 175 | } 176 | 177 | cm, err := k.implementer.Create(cfgMap) 178 | if err != nil { 179 | return nil, err 180 | } 181 | 182 | return cm, nil 183 | } 184 | 185 | func (k *KV) saveInternalMap(cfgMap *v1.ConfigMap, im map[string][]byte) error { 186 | encoded, err := encodeInternalMap(k.serializer, im) 187 | if err != nil { 188 | return err 189 | } 190 | 191 | cfgMap.Data[dataKey] = encoded 192 | 193 | return k.saveMap(cfgMap) 194 | } 195 | 196 | func (k *KV) getInternalMap() (*v1.ConfigMap, map[string][]byte, error) { 197 | cfgMap, err := k.getMap() 198 | if err != nil { 199 | return nil, nil, err 200 | } 201 | 202 | im, err := decodeInternalMap(k.serializer, cfgMap.Data[dataKey]) 203 | if err != nil { 204 | return nil, nil, err 205 | } 206 | return cfgMap, im, nil 207 | } 208 | 209 | func (k *KV) saveMap(cfgMap *v1.ConfigMap) error { 210 | _, err := k.implementer.Update(cfgMap) 211 | return err 212 | } 213 | 214 | // Put saves key/value pair into a bucket. Value can be any []byte value (ie: encoded JSON/GOB) 215 | func (k *KV) Put(key string, value []byte) error { 216 | k.mu.Lock() 217 | defer k.mu.Unlock() 218 | 219 | cfgMap, im, err := k.getInternalMap() 220 | if err != nil { 221 | return err 222 | } 223 | 224 | im[key] = value 225 | 226 | return k.saveInternalMap(cfgMap, im) 227 | } 228 | 229 | // Get retrieves value from the key/value store bucket or returns ErrNotFound error if it was not found. 230 | func (k *KV) Get(key string) (value []byte, err error) { 231 | k.mu.RLock() 232 | defer k.mu.RUnlock() 233 | 234 | _, im, err := k.getInternalMap() 235 | if err != nil { 236 | return 237 | } 238 | 239 | val, ok := im[key] 240 | if !ok { 241 | return []byte(""), ErrNotFound 242 | } 243 | 244 | return val, nil 245 | 246 | } 247 | 248 | // Delete removes entry from the KV store bucket. 249 | func (k *KV) Delete(key string) error { 250 | k.mu.Lock() 251 | defer k.mu.Unlock() 252 | 253 | cfgMap, im, err := k.getInternalMap() 254 | if err != nil { 255 | return err 256 | } 257 | 258 | delete(im, key) 259 | 260 | return k.saveInternalMap(cfgMap, im) 261 | } 262 | 263 | // List retrieves all entries that match specific prefix 264 | func (k *KV) List(prefix string) (data map[string][]byte, err error) { 265 | k.mu.RLock() 266 | defer k.mu.RUnlock() 267 | 268 | _, im, err := k.getInternalMap() 269 | if err != nil { 270 | return 271 | } 272 | 273 | data = make(map[string][]byte) 274 | for key, val := range im { 275 | if strings.HasPrefix(key, prefix) { 276 | data[key] = val 277 | } 278 | } 279 | return 280 | } 281 | 282 | // labels is a map of key value pairs to be included as metadata in a configmap object. 283 | type labels map[string]string 284 | 285 | func (lbs *labels) init() { *lbs = labels(make(map[string]string)) } 286 | func (lbs labels) get(key string) string { return lbs[key] } 287 | func (lbs labels) set(key, val string) { lbs[key] = val } 288 | 289 | func (lbs labels) keys() (ls []string) { 290 | for key := range lbs { 291 | ls = append(ls, key) 292 | } 293 | return 294 | } 295 | 296 | func (lbs labels) match(set labels) bool { 297 | for _, key := range set.keys() { 298 | if lbs.get(key) != set.get(key) { 299 | return false 300 | } 301 | } 302 | return true 303 | } 304 | 305 | func (lbs labels) toMap() map[string]string { return lbs } 306 | 307 | func (lbs *labels) fromMap(kvs map[string]string) { 308 | for k, v := range kvs { 309 | lbs.set(k, v) 310 | } 311 | } 312 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright 2017 Karolis Rusenas 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | --------------------------------------------------------------------------------