├── .gitignore ├── LICENSE ├── README.md ├── bucket_test.go ├── buckets.go ├── buckets.jpg ├── buckets_test.go ├── db_test.go ├── doc.go ├── examples ├── README.md ├── items.go ├── post.go ├── prefix.go ├── range.go └── roundtrip.go ├── http_test.go ├── map_test.go ├── prefixscan.go ├── prefixscan_test.go ├── rangescan.go ├── rangescan_test.go ├── scanner.go └── util.go /.gitignore: -------------------------------------------------------------------------------- 1 | *.DS_Store 2 | *.zip 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 J. Voigt 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of 6 | this software and associated documentation files (the "Software"), to deal in 7 | the Software without restriction, including without limitation the rights to 8 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 9 | the Software, and to permit persons to whom the Software is furnished to do so, 10 | subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 17 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 18 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 19 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 20 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # buckets 2 | 3 | [![GoDoc](https://godoc.org/github.com/joyrexus/buckets?status.svg)](https://godoc.org/github.com/joyrexus/buckets) 4 | [![License](https://img.shields.io/badge/license-MIT-brightgreen.svg)](LICENSE) 5 | [![Report Card](https://goreportcard.com/badge/github.com/joyrexus/buckets)](https://goreportcard.com/report/github.com/joyrexus/buckets) 6 | 7 | 8 | A simple key/value store based on [Bolt](https://github.com/boltdb/bolt). 9 | 10 | ![buckets](buckets.jpg) 11 | 12 | In the parlance of [key/value stores](https://en.wikipedia.org/wiki/Key-value_database), a "bucket" is a collection of unique keys that are associated with values. A buckets database is a set of buckets. The underlying datastore is represented by a single file on disk. 13 | 14 | Note that buckets is just an extension of Bolt, providing a `Bucket` type with some nifty convenience methods for operating on the items (key/value pairs) within instances of it. It streamlines simple transactions (a single put, get, or delete) and working with subsets of items within a bucket (via prefix and range scans). 15 | 16 | For example, here's how you put an item in a bucket and get it back out. (Note we're omitting proper error handling here.) 17 | 18 | ```go 19 | // Open a buckets database. 20 | bx, _ := buckets.Open("data.db") 21 | defer bx.Close() 22 | 23 | // Create a new `things` bucket. 24 | things, _ := bx.New([]byte("things")) 25 | 26 | // Put key/value into the `things` bucket. 27 | key, value := []byte("A"), []byte("alpha") 28 | things.Put(key, value) 29 | 30 | // Read value back in a different read-only transaction. 31 | got, _ := things.Get(key) 32 | 33 | fmt.Printf("The value of %q in `things` is %q\n", key, got) 34 | ``` 35 | 36 | Output: 37 | 38 | The value of "A" in `things` is "alpha" 39 | 40 | 41 | ## Overview 42 | 43 | As noted above, buckets is a wrapper for Bolt, streamlining [basic transactions](https://github.com/boltdb/bolt#transactions). If you're unfamiliar with Bolt, check out the [README](https://github.com/boltdb/bolt#resources) and [intro articles](https://github.com/boltdb/bolt#resources). 44 | 45 | A buckets/bolt database contains a set of buckets. What's a bucket? It's basically just an [associative array](https://en.wikipedia.org/wiki/Associative_array), mapping keys to values. For simplicity, we say that a bucket *contains* key/values pairs and we refer to these k/v pairs as "items". You use buckets for storing and retrieving such items. 46 | 47 | Since Bolt stores keys in [byte-sorted order](https://github.com/boltdb/bolt#iterating-over-keys), we can take advantage of this sorted key namespace for fast prefix and range scanning of keys. In particular, it gives us a way to easily retrieve a subset of items. (See the `PrefixItems` and `RangeItems` methods, described below.) 48 | 49 | 50 | #### Read/write transactions 51 | 52 | * [`Put(k, v)`](https://godoc.org/github.com/joyrexus/buckets#Bucket.Put) - save/update item 53 | * [`PutNX(k, v)`](https://godoc.org/github.com/joyrexus/buckets#Bucket.Put) - save item if key does not exist 54 | * [`Delete(k)`](https://godoc.org/github.com/joyrexus/buckets#Bucket.Delete) - delete item 55 | * [`Insert(items)`](https://godoc.org/github.com/joyrexus/buckets#Bucket.Insert) - save/update items (k/v pairs) 56 | * [`InsertNX(items)`](https://godoc.org/github.com/joyrexus/buckets#Bucket.Insert) - for each item (k/v pair), save item if key does not exist 57 | 58 | 59 | #### Read-only transactions 60 | 61 | * [`Get(k)`](https://godoc.org/github.com/joyrexus/buckets#Bucket.Get) - get value 62 | * [`Items()`](https://godoc.org/github.com/joyrexus/buckets#Bucket.Items) - get list of items (k/v pairs) 63 | * [`PrefixItems(pre)`](https://godoc.org/github.com/joyrexus/buckets#Bucket.PrefixItems) - get list of items with key prefix 64 | * [`RangeItems(min, max)`](https://godoc.org/github.com/joyrexus/buckets#Bucket.RangeItems) - get list of items within key range 65 | * [`Map(func)`](https://godoc.org/github.com/joyrexus/buckets#Bucket.Map) - apply func to each item 66 | * [`MapPrefix(func, pre)`](https://godoc.org/github.com/joyrexus/buckets#Bucket.MapPrefix) - apply func to each item with key prefix 67 | * [`MapRange(func, min, max)`](https://godoc.org/github.com/joyrexus/buckets#Bucket.MapRange) - apply a func to each item within key range 68 | 69 | 70 | ## Getting Started 71 | 72 | Use `go get github.com/joyrexus/buckets` to install and see the [docs](https://godoc.org/github.com/joyrexus/buckets) for details. 73 | 74 | To open a database, use `buckets.Open()`: 75 | 76 | ```go 77 | package main 78 | 79 | import ( 80 | "log" 81 | 82 | "github.com/joyrexus/buckets" 83 | ) 84 | 85 | func main() { 86 | bx, err := buckets.Open("my.db") 87 | if err != nil { 88 | log.Fatal(err) 89 | } 90 | defer bx.Close() 91 | 92 | ... 93 | } 94 | ``` 95 | 96 | Note that buckets obtains a file lock on the data file so multiple processes cannot open the same database at the same time. 97 | 98 | 99 | ## Examples 100 | 101 | The docs contain numerous [examples](https://godoc.org/github.com/joyrexus/buckets#pkg-examples) demonstrating basic usage. 102 | 103 | See also the [examples](examples) directory for standalone examples, demonstrating use of buckets for persistence in a web service context. 104 | -------------------------------------------------------------------------------- /bucket_test.go: -------------------------------------------------------------------------------- 1 | package buckets_test 2 | 3 | import ( 4 | "bytes" 5 | "fmt" 6 | "os" 7 | "testing" 8 | 9 | "github.com/joyrexus/buckets" 10 | ) 11 | 12 | // Ensure that we can create and delete a bucket. 13 | func TestBucket(t *testing.T) { 14 | bx := NewTestDB() 15 | defer bx.Close() 16 | 17 | _, err := bx.New([]byte("things")) 18 | if err != nil { 19 | t.Error(err.Error()) 20 | } 21 | 22 | if err := bx.Delete([]byte("things")); err != nil { 23 | t.Error(err.Error()) 24 | } 25 | } 26 | 27 | // Ensure we can put an item in a bucket. 28 | func TestPut(t *testing.T) { 29 | bx := NewTestDB() 30 | defer bx.Close() 31 | 32 | things, err := bx.New([]byte("things")) 33 | if err != nil { 34 | t.Error(err.Error()) 35 | } 36 | 37 | key, value := []byte("A"), []byte("alpha") 38 | 39 | // Put key/value into the `things` bucket. 40 | if err := things.Put(key, value); err != nil { 41 | t.Error(err.Error()) 42 | } 43 | 44 | // Read value back in a different read-only transaction. 45 | got, err := things.Get(key) 46 | if err != nil && !bytes.Equal(got, value) { 47 | t.Error(err.Error()) 48 | } 49 | } 50 | 51 | // Show we can put an item in a bucket and get it back out. 52 | func ExampleBucket_Put() { 53 | bx, _ := buckets.Open(tempfile()) 54 | defer os.Remove(bx.Path()) 55 | defer bx.Close() 56 | 57 | // Create a new `things` bucket. 58 | bucket := []byte("things") 59 | things, _ := bx.New(bucket) 60 | 61 | // Put key/value into the `things` bucket. 62 | key, value := []byte("A"), []byte("alpha") 63 | if err := things.Put(key, value); err != nil { 64 | fmt.Printf("could not insert item: %v", err) 65 | } 66 | 67 | // Read value back in a different read-only transaction. 68 | got, _ := things.Get(key) 69 | 70 | fmt.Printf("The value of %q in `%s` is %q\n", key, bucket, got) 71 | 72 | // Output: 73 | // The value of "A" in `things` is "alpha" 74 | } 75 | 76 | // Ensure we don't overwrite existing items when using PutNX. 77 | func TestPutNX(t *testing.T) { 78 | bx := NewTestDB() 79 | defer bx.Close() 80 | 81 | things, err := bx.New([]byte("things")) 82 | if err != nil { 83 | t.Error(err.Error()) 84 | } 85 | 86 | key := []byte("A") 87 | a, b := []byte("alpha"), []byte("beta") 88 | 89 | // Put key/a-value into the `things` bucket. 90 | if err := things.PutNX(key, a); err != nil { 91 | t.Error(err.Error()) 92 | } 93 | 94 | // Read value back in a different read-only transaction. 95 | got, err := things.Get(key) 96 | if err != nil && !bytes.Equal(got, a) { 97 | t.Error(err.Error()) 98 | } 99 | 100 | // Try putting key/b-value into the `things` bucket. 101 | if err := things.PutNX(key, b); err != nil { 102 | t.Error(err.Error()) 103 | } 104 | 105 | // Value for key should still be a, not b. 106 | got, err = things.Get(key) 107 | if err != nil && !bytes.Equal(got, a) { 108 | t.Error(err.Error()) 109 | } 110 | } 111 | 112 | // Show we don't overwrite existing values when using PutNX. 113 | func ExampleBucket_PutNX() { 114 | bx, _ := buckets.Open(tempfile()) 115 | defer os.Remove(bx.Path()) 116 | defer bx.Close() 117 | 118 | // Create a new `things` bucket. 119 | bucket := []byte("things") 120 | things, _ := bx.New(bucket) 121 | 122 | // Put key/value into the `things` bucket. 123 | key, value := []byte("A"), []byte("alpha") 124 | if err := things.Put(key, value); err != nil { 125 | fmt.Printf("could not insert item: %v", err) 126 | } 127 | 128 | // Read value back in a different read-only transaction. 129 | got, _ := things.Get(key) 130 | 131 | fmt.Printf("The value of %q in `%s` is %q\n", key, bucket, got) 132 | 133 | // Try putting another value with same key. 134 | things.PutNX(key, []byte("beta")) 135 | 136 | // Read value back in a different read-only transaction. 137 | got, _ = things.Get(key) 138 | 139 | fmt.Printf("The value of %q in `%s` is still %q\n", key, bucket, got) 140 | 141 | // Output: 142 | // The value of "A" in `things` is "alpha" 143 | // The value of "A" in `things` is still "alpha" 144 | } 145 | 146 | // Ensure that a bucket that gets a non-existent key returns nil. 147 | func TestGetMissing(t *testing.T) { 148 | bx := NewTestDB() 149 | defer bx.Close() 150 | 151 | things, err := bx.New([]byte("things")) 152 | if err != nil { 153 | t.Error(err.Error()) 154 | } 155 | 156 | key := []byte("missing") 157 | if got, _ := things.Get(key); got != nil { 158 | t.Errorf("not expecting value for key %q: got %q", key, got) 159 | } 160 | } 161 | 162 | // Ensure that we can delete stuff in a bucket. 163 | func TestDelete(t *testing.T) { 164 | bx := NewTestDB() 165 | defer bx.Close() 166 | 167 | things, err := bx.New([]byte("things")) 168 | if err != nil { 169 | t.Error(err.Error()) 170 | } 171 | 172 | k, v := []byte("foo"), []byte("bar") 173 | if err = things.Put(k, v); err != nil { 174 | t.Error(err.Error()) 175 | } 176 | 177 | if err = things.Delete(k); err != nil { 178 | t.Error(err.Error()) 179 | } 180 | } 181 | 182 | // Ensure we can insert items into a bucket and get them back out. 183 | func TestInsert(t *testing.T) { 184 | bx := NewTestDB() 185 | defer bx.Close() 186 | 187 | paths, err := bx.New([]byte("paths")) 188 | 189 | // k, v pairs to put in `paths` bucket 190 | items := []struct { 191 | Key, Value []byte 192 | }{ 193 | {[]byte("foo/"), []byte("foo")}, 194 | {[]byte("foo/bar/"), []byte("bar")}, 195 | {[]byte("foo/bar/baz/"), []byte("baz")}, 196 | {[]byte("food/"), []byte("")}, 197 | {[]byte("good/"), []byte("")}, 198 | {[]byte("goo/"), []byte("")}, 199 | } 200 | 201 | err = paths.Insert(items) 202 | if err != nil { 203 | t.Error(err.Error()) 204 | } 205 | 206 | gotItems, err := paths.Items() 207 | if err != nil { 208 | t.Error(err.Error()) 209 | } 210 | 211 | // expected k/v mapping 212 | expected := map[string][]byte{ 213 | "foo/": []byte("foo"), 214 | "foo/bar/": []byte("bar"), 215 | "foo/bar/baz/": []byte("baz"), 216 | "food/": []byte(""), 217 | "good/": []byte(""), 218 | "goo/": []byte(""), 219 | } 220 | 221 | for _, item := range gotItems { 222 | want := expected[string(item.Key)] 223 | if !bytes.Equal(item.Value, want) { 224 | t.Errorf("got %v, want %v", item.Value, want) 225 | } 226 | } 227 | } 228 | 229 | // Show we can insert items into a bucket and get them back out. 230 | func ExampleBucket_Insert() { 231 | bx, _ := buckets.Open(tempfile()) 232 | defer os.Remove(bx.Path()) 233 | defer bx.Close() 234 | 235 | letters, _ := bx.New([]byte("letters")) 236 | 237 | // Setup items to insert in `letters` bucket. 238 | items := []struct { 239 | Key, Value []byte 240 | }{ 241 | {[]byte("A"), []byte("alpha")}, 242 | {[]byte("B"), []byte("beta")}, 243 | {[]byte("C"), []byte("gamma")}, 244 | } 245 | 246 | // Insert items into `letters` bucket. 247 | if err := letters.Insert(items); err != nil { 248 | fmt.Println("could not insert items!") 249 | } 250 | 251 | // Get items back out in separate read-only transaction. 252 | results, _ := letters.Items() 253 | 254 | for _, item := range results { 255 | fmt.Printf("%s -> %s\n", item.Key, item.Value) 256 | } 257 | 258 | // Output: 259 | // A -> alpha 260 | // B -> beta 261 | // C -> gamma 262 | } 263 | 264 | // Ensure we can safely insert items into a bucket without overwriting 265 | // existing items. 266 | func TestInsertNX(t *testing.T) { 267 | bx := NewTestDB() 268 | defer bx.Close() 269 | 270 | bk, err := bx.New([]byte("test")) 271 | 272 | // Put k/v into the `bk` bucket. 273 | k, v := []byte("A"), []byte("alpha") 274 | if err := bk.Put(k, v); err != nil { 275 | t.Error(err.Error()) 276 | } 277 | 278 | // k/v pairs to put-if-not-exists 279 | items := []struct { 280 | Key, Value []byte 281 | }{ 282 | {[]byte("A"), []byte("ALPHA")}, // key exists, so don't update 283 | {[]byte("B"), []byte("beta")}, 284 | {[]byte("C"), []byte("gamma")}, 285 | } 286 | 287 | err = bk.InsertNX(items) 288 | if err != nil { 289 | t.Error(err.Error()) 290 | } 291 | 292 | gotItems, err := bk.Items() 293 | if err != nil { 294 | t.Error(err.Error()) 295 | } 296 | 297 | // expected items 298 | expected := []struct { 299 | Key, Value []byte 300 | }{ 301 | {[]byte("A"), []byte("alpha")}, // existing value not updated 302 | {[]byte("B"), []byte("beta")}, 303 | {[]byte("C"), []byte("gamma")}, 304 | } 305 | 306 | for i, got := range gotItems { 307 | want := expected[i] 308 | if !bytes.Equal(got.Value, want.Value) { 309 | t.Errorf("key %q: got %v, want %v", got.Key, got.Value, want.Value) 310 | } 311 | } 312 | } 313 | 314 | // Ensure that we can get items for all keys with a given prefix. 315 | func TestPrefixItems(t *testing.T) { 316 | bx := NewTestDB() 317 | defer bx.Close() 318 | 319 | // Create a new things bucket. 320 | things, err := bx.New([]byte("things")) 321 | if err != nil { 322 | t.Error(err.Error()) 323 | } 324 | 325 | // Setup items to insert. 326 | items := []struct { 327 | Key, Value []byte 328 | }{ 329 | {[]byte("A"), []byte("1")}, // `A` prefix match 330 | {[]byte("AA"), []byte("2")}, // match 331 | {[]byte("AAA"), []byte("3")}, // match 332 | {[]byte("AAB"), []byte("2")}, // match 333 | {[]byte("B"), []byte("O")}, 334 | {[]byte("BA"), []byte("0")}, 335 | {[]byte("BAA"), []byte("0")}, 336 | } 337 | 338 | // Insert 'em. 339 | if err := things.Insert(items); err != nil { 340 | t.Error(err.Error()) 341 | } 342 | 343 | // Now get each item whose key starts with "A". 344 | prefix := []byte("A") 345 | 346 | // Expected items for keys with prefix "A". 347 | expected := []struct { 348 | Key, Value []byte 349 | }{ 350 | {[]byte("A"), []byte("1")}, 351 | {[]byte("AA"), []byte("2")}, 352 | {[]byte("AAA"), []byte("3")}, 353 | {[]byte("AAB"), []byte("2")}, 354 | } 355 | 356 | results, err := things.PrefixItems(prefix) 357 | if err != nil { 358 | t.Error(err.Error()) 359 | } 360 | 361 | for i, want := range expected { 362 | got := results[i] 363 | if !bytes.Equal(got.Key, want.Key) { 364 | t.Errorf("got %v, want %v", got.Key, want.Key) 365 | } 366 | if !bytes.Equal(got.Value, want.Value) { 367 | t.Errorf("got %v, want %v", got.Value, want.Value) 368 | } 369 | } 370 | } 371 | 372 | // Show that we can get items for all keys with a given prefix. 373 | func ExampleBucket_PrefixItems() { 374 | bx, _ := buckets.Open(tempfile()) 375 | defer os.Remove(bx.Path()) 376 | defer bx.Close() 377 | 378 | // Create a new things bucket. 379 | things, _ := bx.New([]byte("things")) 380 | 381 | // Setup items to insert. 382 | items := []struct { 383 | Key, Value []byte 384 | }{ 385 | {[]byte("A"), []byte("1")}, // `A` prefix match 386 | {[]byte("AA"), []byte("2")}, // match 387 | {[]byte("AAA"), []byte("3")}, // match 388 | {[]byte("AAB"), []byte("2")}, // match 389 | {[]byte("B"), []byte("O")}, 390 | {[]byte("BA"), []byte("0")}, 391 | {[]byte("BAA"), []byte("0")}, 392 | } 393 | 394 | // Insert 'em. 395 | if err := things.Insert(items); err != nil { 396 | fmt.Printf("could not insert items in `things` bucket: %v\n", err) 397 | } 398 | 399 | // Now get items whose key starts with "A". 400 | prefix := []byte("A") 401 | 402 | results, err := things.PrefixItems(prefix) 403 | if err != nil { 404 | fmt.Printf("could not get items with prefix %q: %v\n", prefix, err) 405 | } 406 | 407 | for _, item := range results { 408 | fmt.Printf("%s -> %s\n", item.Key, item.Value) 409 | } 410 | // Output: 411 | // A -> 1 412 | // AA -> 2 413 | // AAA -> 3 414 | // AAB -> 2 415 | } 416 | 417 | // Ensure we can get items for all keys within a given range. 418 | func TestRangeItems(t *testing.T) { 419 | bx := NewTestDB() 420 | defer bx.Close() 421 | 422 | years, err := bx.New([]byte("years")) 423 | if err != nil { 424 | t.Error(err.Error()) 425 | } 426 | 427 | // Setup items to insert in `years` bucket 428 | items := []struct { 429 | Key, Value []byte 430 | }{ 431 | {[]byte("1970"), []byte("70")}, 432 | {[]byte("1975"), []byte("75")}, 433 | {[]byte("1980"), []byte("80")}, 434 | {[]byte("1985"), []byte("85")}, 435 | {[]byte("1990"), []byte("90")}, // min = 1990 436 | {[]byte("1995"), []byte("95")}, // min < 1995 < max 437 | {[]byte("2000"), []byte("00")}, // max = 2000 438 | {[]byte("2005"), []byte("05")}, 439 | {[]byte("2010"), []byte("10")}, 440 | } 441 | 442 | // Insert 'em. 443 | if err := years.Insert(items); err != nil { 444 | t.Error(err.Error()) 445 | } 446 | 447 | // Now get each item whose key is in the 1990 to 2000 range. 448 | min := []byte("1990") 449 | max := []byte("2000") 450 | 451 | // Expected items within time range: 1990 <= key <= 2000. 452 | expected := []struct { 453 | Key, Value []byte 454 | }{ 455 | {[]byte("1990"), []byte("90")}, // min = 1990 456 | {[]byte("1995"), []byte("95")}, // min < 1995 < max 457 | {[]byte("2000"), []byte("00")}, // max = 2000 458 | } 459 | 460 | // Get items for keys within min/max range. 461 | results, err := years.RangeItems(min, max) 462 | if err != nil { 463 | t.Error(err.Error()) 464 | } 465 | 466 | for i, want := range expected { 467 | got := results[i] 468 | if !bytes.Equal(got.Key, want.Key) { 469 | t.Errorf("got %v, want %v", got.Key, want.Key) 470 | } 471 | if !bytes.Equal(got.Value, want.Value) { 472 | t.Errorf("got %v, want %v", got.Value, want.Value) 473 | } 474 | } 475 | } 476 | 477 | // Show that we get items for keys within a given range. 478 | func ExampleBucket_RangeItems() { 479 | bx, _ := buckets.Open(tempfile()) 480 | defer os.Remove(bx.Path()) 481 | defer bx.Close() 482 | 483 | // Create a new bucket named "years". 484 | years, _ := bx.New([]byte("years")) 485 | 486 | // Setup items to insert in `years` bucket 487 | items := []struct { 488 | Key, Value []byte 489 | }{ 490 | {[]byte("1970"), []byte("70")}, 491 | {[]byte("1975"), []byte("75")}, 492 | {[]byte("1980"), []byte("80")}, 493 | {[]byte("1985"), []byte("85")}, 494 | {[]byte("1990"), []byte("90")}, // min = 1990 495 | {[]byte("1995"), []byte("95")}, // min < 1995 < max 496 | {[]byte("2000"), []byte("00")}, // max = 2000 497 | {[]byte("2005"), []byte("05")}, 498 | {[]byte("2010"), []byte("10")}, 499 | } 500 | 501 | // Insert 'em. 502 | if err := years.Insert(items); err != nil { 503 | fmt.Printf("could not insert items in `years` bucket: %v\n", err) 504 | } 505 | 506 | // Time range: 1990 <= key <= 2000. 507 | min := []byte("1990") 508 | max := []byte("2000") 509 | 510 | results, err := years.RangeItems(min, max) 511 | if err != nil { 512 | fmt.Printf("could not get items within range: %v\n", err) 513 | } 514 | 515 | for _, item := range results { 516 | fmt.Printf("%s -> %s\n", item.Key, item.Value) 517 | } 518 | // Output: 519 | // 1990 -> 90 520 | // 1995 -> 95 521 | // 2000 -> 00 522 | } 523 | -------------------------------------------------------------------------------- /buckets.go: -------------------------------------------------------------------------------- 1 | package buckets 2 | 3 | import ( 4 | "bytes" 5 | "fmt" 6 | "time" 7 | 8 | "github.com/boltdb/bolt" 9 | ) 10 | 11 | // A DB is a bolt database with convenience methods for working with buckets. 12 | // 13 | // A DB embeds the exposed bolt.DB methods. 14 | type DB struct { 15 | *bolt.DB 16 | } 17 | 18 | // Open creates/opens a buckets database at the specified path. 19 | func Open(path string) (*DB, error) { 20 | config := &bolt.Options{Timeout: 1 * time.Second} 21 | db, err := bolt.Open(path, 0600, config) 22 | if err != nil { 23 | return nil, fmt.Errorf("couldn't open %s: %s", path, err) 24 | } 25 | return &DB{db}, nil 26 | } 27 | 28 | // New creates/opens a named bucket. 29 | func (db *DB) New(name []byte) (*Bucket, error) { 30 | err := db.Update(func(tx *bolt.Tx) error { 31 | _, err := tx.CreateBucketIfNotExists(name) 32 | if err != nil { 33 | return err 34 | } 35 | return nil 36 | }) 37 | if err != nil { 38 | return nil, err 39 | } 40 | return &Bucket{db, name}, nil 41 | } 42 | 43 | // Delete removes the named bucket. 44 | func (db *DB) Delete(name []byte) error { 45 | return db.Update(func(tx *bolt.Tx) error { 46 | return tx.DeleteBucket(name) 47 | }) 48 | } 49 | 50 | /* -- ITEM -- */ 51 | 52 | // An Item holds a key/value pair. 53 | type Item struct { 54 | Key []byte 55 | Value []byte 56 | } 57 | 58 | /* -- BUCKET-- */ 59 | 60 | // Bucket represents a collection of key/value pairs inside the database. 61 | type Bucket struct { 62 | db *DB 63 | Name []byte 64 | } 65 | 66 | // Put inserts value `v` with key `k`. 67 | func (bk *Bucket) Put(k, v []byte) error { 68 | return bk.db.Update(func(tx *bolt.Tx) error { 69 | return tx.Bucket(bk.Name).Put(k, v) 70 | }) 71 | } 72 | 73 | // PutNX (put-if-not-exists) inserts value `v` with key `k` 74 | // if key doesn't exist. 75 | func (bk *Bucket) PutNX(k, v []byte) error { 76 | v, err := bk.Get(k) 77 | if v != nil || err != nil { 78 | return err 79 | } 80 | return bk.db.Update(func(tx *bolt.Tx) error { 81 | return tx.Bucket(bk.Name).Put(k, v) 82 | }) 83 | } 84 | 85 | // Insert iterates over a slice of k/v pairs, putting each item in 86 | // the bucket as part of a single transaction. For large insertions, 87 | // be sure to pre-sort your items (by Key in byte-sorted order), which 88 | // will result in much more efficient insertion times and storage costs. 89 | func (bk *Bucket) Insert(items []struct{ Key, Value []byte }) error { 90 | return bk.db.Update(func(tx *bolt.Tx) error { 91 | for _, item := range items { 92 | tx.Bucket(bk.Name).Put(item.Key, item.Value) 93 | } 94 | return nil 95 | }) 96 | } 97 | 98 | // InsertNX (insert-if-not-exists) iterates over a slice of k/v pairs, 99 | // putting each item in the bucket as part of a single transaction. 100 | // Unlike Insert, however, InsertNX will not update the value for an 101 | // existing key. 102 | func (bk *Bucket) InsertNX(items []struct{ Key, Value []byte }) error { 103 | return bk.db.Update(func(tx *bolt.Tx) error { 104 | for _, item := range items { 105 | v, _ := bk.Get(item.Key) 106 | if v == nil { 107 | tx.Bucket(bk.Name).Put(item.Key, item.Value) 108 | } 109 | } 110 | return nil 111 | }) 112 | } 113 | 114 | // Delete removes key `k`. 115 | func (bk *Bucket) Delete(k []byte) error { 116 | return bk.db.Update(func(tx *bolt.Tx) error { 117 | return tx.Bucket(bk.Name).Delete(k) 118 | }) 119 | } 120 | 121 | // Get retrieves the value for key `k`. 122 | func (bk *Bucket) Get(k []byte) (value []byte, err error) { 123 | err = bk.db.View(func(tx *bolt.Tx) error { 124 | v := tx.Bucket(bk.Name).Get(k) 125 | if v != nil { 126 | value = make([]byte, len(v)) 127 | copy(value, v) 128 | } 129 | return nil 130 | }) 131 | return value, err 132 | } 133 | 134 | // Items returns a slice of key/value pairs. Each k/v pair in the slice 135 | // is of type Item (`struct{ Key, Value []byte }`). 136 | func (bk *Bucket) Items() (items []Item, err error) { 137 | return items, bk.db.View(func(tx *bolt.Tx) error { 138 | c := tx.Bucket(bk.Name).Cursor() 139 | var key, value []byte 140 | for k, v := c.First(); k != nil; k, v = c.Next() { 141 | if v != nil { 142 | key = make([]byte, len(k)) 143 | copy(key, k) 144 | value = make([]byte, len(v)) 145 | copy(value, v) 146 | items = append(items, Item{key, value}) 147 | } 148 | } 149 | return nil 150 | }) 151 | } 152 | 153 | // PrefixItems returns a slice of key/value pairs for all keys with 154 | // a given prefix. Each k/v pair in the slice is of type Item 155 | // (`struct{ Key, Value []byte }`). 156 | func (bk *Bucket) PrefixItems(pre []byte) (items []Item, err error) { 157 | err = bk.db.View(func(tx *bolt.Tx) error { 158 | c := tx.Bucket(bk.Name).Cursor() 159 | var key, value []byte 160 | for k, v := c.Seek(pre); bytes.HasPrefix(k, pre); k, v = c.Next() { 161 | if v != nil { 162 | key = make([]byte, len(k)) 163 | copy(key, k) 164 | value = make([]byte, len(v)) 165 | copy(value, v) 166 | items = append(items, Item{key, value}) 167 | } 168 | } 169 | return nil 170 | }) 171 | return items, err 172 | } 173 | 174 | // RangeItems returns a slice of key/value pairs for all keys within 175 | // a given range. Each k/v pair in the slice is of type Item 176 | // (`struct{ Key, Value []byte }`). 177 | func (bk *Bucket) RangeItems(min []byte, max []byte) (items []Item, err error) { 178 | err = bk.db.View(func(tx *bolt.Tx) error { 179 | c := tx.Bucket(bk.Name).Cursor() 180 | var key, value []byte 181 | for k, v := c.Seek(min); isBefore(k, max); k, v = c.Next() { 182 | if v != nil { 183 | key = make([]byte, len(k)) 184 | copy(key, k) 185 | value = make([]byte, len(v)) 186 | copy(value, v) 187 | items = append(items, Item{key, value}) 188 | } 189 | } 190 | return nil 191 | }) 192 | return items, err 193 | } 194 | 195 | // Map applies `do` on each key/value pair. 196 | func (bk *Bucket) Map(do func(k, v []byte) error) error { 197 | return bk.db.View(func(tx *bolt.Tx) error { 198 | return tx.Bucket(bk.Name).ForEach(do) 199 | }) 200 | } 201 | 202 | // MapPrefix applies `do` on each k/v pair of keys with prefix. 203 | func (bk *Bucket) MapPrefix(do func(k, v []byte) error, pre []byte) error { 204 | return bk.db.View(func(tx *bolt.Tx) error { 205 | c := tx.Bucket(bk.Name).Cursor() 206 | for k, v := c.Seek(pre); bytes.HasPrefix(k, pre); k, v = c.Next() { 207 | do(k, v) 208 | } 209 | return nil 210 | }) 211 | } 212 | 213 | // MapRange applies `do` on each k/v pair of keys within range. 214 | func (bk *Bucket) MapRange(do func(k, v []byte) error, min, max []byte) error { 215 | return bk.db.View(func(tx *bolt.Tx) error { 216 | c := tx.Bucket(bk.Name).Cursor() 217 | for k, v := c.Seek(min); isBefore(k, max); k, v = c.Next() { 218 | do(k, v) 219 | } 220 | return nil 221 | }) 222 | } 223 | 224 | // NewPrefixScanner initializes a new prefix scanner. 225 | func (bk *Bucket) NewPrefixScanner(pre []byte) *PrefixScanner { 226 | return &PrefixScanner{bk.db, bk.Name, pre} 227 | } 228 | 229 | // NewRangeScanner initializes a new range scanner. It takes a `min` and a 230 | // `max` key for specifying the range paramaters. 231 | func (bk *Bucket) NewRangeScanner(min, max []byte) *RangeScanner { 232 | return &RangeScanner{bk.db, bk.Name, min, max} 233 | } 234 | -------------------------------------------------------------------------------- /buckets.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/joyrexus/buckets/95fcbf1aabe464641d7180b55a99650486f1edf9/buckets.jpg -------------------------------------------------------------------------------- /buckets_test.go: -------------------------------------------------------------------------------- 1 | package buckets_test 2 | 3 | import ( 4 | "io/ioutil" 5 | "log" 6 | "os" 7 | 8 | "github.com/joyrexus/buckets" 9 | ) 10 | 11 | type TestDB struct { 12 | *buckets.DB 13 | } 14 | 15 | // NewTestDB returns a TestDB using a temporary path. 16 | func NewTestDB() *TestDB { 17 | bx, err := buckets.Open(tempfile()) 18 | if err != nil { 19 | log.Fatalf("cannot open buckets database: %s", err) 20 | } 21 | // Return wrapped type. 22 | return &TestDB{bx} 23 | } 24 | 25 | // Close and delete buckets database. 26 | func (db *TestDB) Close() { 27 | defer os.Remove(db.Path()) 28 | db.DB.Close() 29 | } 30 | 31 | // tempfile returns a temporary file path. 32 | func tempfile() string { 33 | f, err := ioutil.TempFile("", "bolt-") 34 | if err != nil { 35 | log.Fatalf("Could not create temp file: %s", err) 36 | } 37 | if err := f.Close(); err != nil { 38 | log.Fatal(err) 39 | } 40 | if err := os.Remove(f.Name()); err != nil { 41 | log.Fatal(err) 42 | } 43 | return f.Name() 44 | } 45 | -------------------------------------------------------------------------------- /db_test.go: -------------------------------------------------------------------------------- 1 | package buckets_test 2 | 3 | import ( 4 | "os" 5 | "testing" 6 | 7 | "github.com/joyrexus/buckets" 8 | ) 9 | 10 | // Ensure we can open/close a buckets db. 11 | func TestOpen(t *testing.T) { 12 | bx, err := buckets.Open(tempfile()) 13 | if err != nil { 14 | t.Error(err.Error()) 15 | } 16 | defer os.Remove(bx.Path()) 17 | defer bx.Close() 18 | } 19 | -------------------------------------------------------------------------------- /doc.go: -------------------------------------------------------------------------------- 1 | /* 2 | 3 | Package buckets provides a simplified interface to a Bolt database. 4 | 5 | A buckets DB is a Bolt database, but it allows you to easily create new bucket instances. The database is represented by a single file on disk. A bucket is a collection of unique keys that are associated with values. 6 | 7 | The Bucket type has nifty convenience methods for operating on key/value pairs within it. It streamlines simple transactions (a single put, get, or delete) and working with subsets of items within a bucket (via prefix and range scans). It's not designed to handle more complex or batch transactions. For such cases, use the standard techniques offered by Bolt. 8 | 9 | --- 10 | 11 | What is bolt? 12 | 13 | "Bolt implements a low-level key/value store in pure Go. It supports 14 | fully serializable transactions, ACID semantics, and lock-free MVCC with 15 | multiple readers and a single writer. Bolt can be used for projects that 16 | want a simple data store without the need to add large dependencies such as 17 | Postgres or MySQL." 18 | 19 | See https://github.com/boltdb/bolt for important details. 20 | 21 | */ 22 | package buckets 23 | -------------------------------------------------------------------------------- /examples/README.md: -------------------------------------------------------------------------------- 1 | # Examples 2 | 3 | This directory contains standalone examples demonstrating the use of buckets 4 | for persistence in a web server context. 5 | 6 | * [`post.go`](post.go) - sets up an http server that [stores raw json payloads](https://github.com/joyrexus/buckets/blob/master/examples/post.go#L140-L141) sent via http POST requests. 7 | 8 | * [`roundtrip.go`](roundtrip.go) - extends the previous example by appropriately [handling](https://github.com/joyrexus/buckets/blob/master/examples/roundtrip.go#L39-L42) http requests sent to the same route with different methods (GET or POST). 9 | 10 | * [`prefix.go`](prefix.go) - extends the previous example to demonstrate [prefix scanning](https://github.com/joyrexus/buckets/blob/master/examples/prefix.go#L128-L135). 11 | 12 | * [`range.go`](range.go) - extends the previous example to demonstrate [range scanning](https://github.com/joyrexus/buckets/blob/master/examples/range.go#L171-L174). 13 | 14 | * [`items.go`](range.go) - variant of the previous example, demonstrating another way to get items with a given key prefix: viz., using [`Bucket.PrefixItems`](https://github.com/joyrexus/buckets/blob/master/examples/items.go#L238-L252) and [`Bucket.RangeItems`](https://github.com/joyrexus/buckets/blob/master/examples/items.go#L208-L218). 15 | -------------------------------------------------------------------------------- /examples/items.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bytes" 5 | "encoding/json" 6 | "fmt" 7 | "io/ioutil" 8 | "log" 9 | "net/http" 10 | "net/http/httptest" 11 | "os" 12 | "strconv" 13 | "strings" 14 | "time" 15 | 16 | "github.com/joyrexus/buckets" 17 | "github.com/julienschmidt/httprouter" 18 | ) 19 | 20 | const verbose = false // if `true` you'll see log output 21 | 22 | func main() { 23 | // Open a buckets database. 24 | bx, err := buckets.Open(tempFilePath()) 25 | if err != nil { 26 | log.Fatalf("couldn't open db: %v", err) 27 | } 28 | 29 | // Delete and close the db when done. 30 | defer os.Remove(bx.Path()) 31 | defer bx.Close() 32 | 33 | // Create a bucket for storing todos. 34 | bucket, err := bx.New([]byte("todos")) 35 | if err != nil { 36 | log.Fatalf("couldn't create todos bucket: %v", err) 37 | } 38 | 39 | // Initialize our controller for handling specific routes. 40 | control := NewController(bucket) 41 | 42 | // Create and setup our router. 43 | router := httprouter.New() 44 | router.POST("/day/:day", control.post) 45 | router.GET("/day/:day", control.getDayTasks) 46 | router.GET("/weekend", control.getWeekendTasks) 47 | router.GET("/weekdays", control.getWeekdayTasks) 48 | 49 | // Start our web server. 50 | srv := httptest.NewServer(router) 51 | defer srv.Close() 52 | 53 | // Setup daily todos for client to post. 54 | posts := []*Todo{ 55 | {Day: "mon", Task: "milk cows"}, 56 | {Day: "mon", Task: "feed cows"}, 57 | {Day: "mon", Task: "wash cows"}, 58 | {Day: "tue", Task: "wash laundry"}, 59 | {Day: "tue", Task: "fold laundry"}, 60 | {Day: "tue", Task: "iron laundry"}, 61 | {Day: "wed", Task: "flip burgers"}, 62 | {Day: "thu", Task: "join army"}, 63 | {Day: "fri", Task: "kill time"}, 64 | {Day: "sat", Task: "have beer"}, 65 | {Day: "sat", Task: "make merry"}, 66 | {Day: "sun", Task: "take aspirin"}, 67 | {Day: "sun", Task: "pray quietly"}, 68 | } 69 | 70 | // Create our client. 71 | client := new(Client) 72 | 73 | // Use our client to post each daily todo. 74 | for _, todo := range posts { 75 | url := srv.URL + "/day/" + todo.Day 76 | if err := client.post(url, todo); err != nil { 77 | fmt.Printf("client post error: %v", err) 78 | } 79 | } 80 | 81 | // Now, let's try retrieving the persisted todos. 82 | 83 | // Get a list of tasks for each day. 84 | week := []string{"mon", "tue", "wed", "thu", "fri", "sat", "sun"} 85 | fmt.Println("daily tasks ...") 86 | for _, day := range week { 87 | url := srv.URL + "/day/" + day 88 | tasks, err := client.get(url) 89 | if err != nil { 90 | fmt.Printf("client get error: %v", err) 91 | } 92 | fmt.Printf(" %s: %s\n", day, tasks) 93 | } 94 | // Output: 95 | // daily tasks ... 96 | // mon: milk cows, feed cows, wash cows 97 | // tue: wash laundry, fold laundry, iron laundry 98 | // wed: flip burgers 99 | // thu: join army 100 | // fri: kill time 101 | // sat: have beer, make merry 102 | // sun: take aspirin, pray quietly 103 | 104 | // Get a list of combined tasks for weekdays. 105 | tasks, err := client.get(srv.URL + "/weekdays") 106 | if err != nil { 107 | fmt.Printf("client get error: %v", err) 108 | } 109 | fmt.Printf("\nweekday tasks: %s\n", tasks) 110 | // Output: 111 | // weekday tasks: milk cows, feed cows, wash cows, wash laundry, 112 | // fold laundry, iron laundry, flip burgers, join army, kill time 113 | 114 | // Get a list of combined tasks for the weekend. 115 | tasks, err = client.get(srv.URL + "/weekend") 116 | if err != nil { 117 | fmt.Printf("client get error: %v", err) 118 | } 119 | fmt.Printf("\nweekend tasks: %s\n", tasks) 120 | // Output: 121 | // weekend tasks: have beer, make merry, take aspirin, pray quietly 122 | } 123 | 124 | /* -- MODELS --*/ 125 | 126 | // A Todo models a daily task. 127 | type Todo struct { 128 | Task string // task to be done 129 | Day string // day to do task 130 | Created time.Time // when created 131 | } 132 | 133 | // Encode marshals a Todo into a buffer. 134 | func (todo *Todo) Encode() (*bytes.Buffer, error) { 135 | b, err := json.Marshal(todo) 136 | if err != nil { 137 | return &bytes.Buffer{}, err 138 | } 139 | return bytes.NewBuffer(b), nil 140 | } 141 | 142 | // A TaskList is a list of tasks for a particular day. 143 | type TaskList struct { 144 | When string 145 | Tasks []string 146 | } 147 | 148 | /* -- CONTROLLER -- */ 149 | 150 | // NewController initializes a new instance of our controller. 151 | // It provides handler methods for our router. 152 | func NewController(bk *buckets.Bucket) *Controller { 153 | // map of days to integers 154 | daynum := map[string]int{ 155 | "mon": 1, // monday is the first day of the week 156 | "tue": 2, 157 | "wed": 3, 158 | "thu": 4, 159 | "fri": 5, 160 | "sat": 6, 161 | "sun": 7, 162 | } 163 | return &Controller{bk, daynum} 164 | } 165 | 166 | // Controller handles requests for todo items. The items are stored 167 | // in a todos bucket. The request URLs are used as bucket keys and the 168 | // raw json payload as values. 169 | // 170 | // Note that since we're using `httprouter` (abbreviated as `mux` when 171 | // imported) as our router, each method is a `httprouter.Handle` rather 172 | // than a `http.HandlerFunc`. 173 | type Controller struct { 174 | todos *buckets.Bucket 175 | daynum map[string]int 176 | } 177 | 178 | // getWeekendTasks handles get requests for `/weekend`, returning the 179 | // combined task list for saturday and sunday. 180 | // 181 | // Note how we utilize the RangeItems method, which makes it easy 182 | // to get items in our todos bucket with keys in a certain range 183 | // (6 <= key < 8), viz., the items for sat and sun. 184 | func (c *Controller) getWeekendTasks(w http.ResponseWriter, r *http.Request, 185 | _ httprouter.Params) { 186 | 187 | // Get todo items within the weekend range. 188 | items, err := c.todos.RangeItems([]byte("6"), []byte("8")) 189 | if err != nil { 190 | http.Error(w, err.Error(), 500) 191 | } 192 | 193 | // Generate a list of tasks based on todo items retrieved. 194 | taskList := &TaskList{"weekend", []string{}} 195 | 196 | for _, item := range items { 197 | todo, err := decode(item.Value) 198 | if err != nil { 199 | http.Error(w, err.Error(), 500) 200 | } 201 | taskList.Tasks = append(taskList.Tasks, todo.Task) 202 | } 203 | 204 | w.Header().Set("Content-Type", "application/json") 205 | json.NewEncoder(w).Encode(taskList) 206 | } 207 | 208 | // getWeekdayTasks handles get requests for `/weekdays`, returning the 209 | // combined task list for monday through friday. 210 | // 211 | // Note how we utilize the RangeItems method, which makes it easy 212 | // to get items in our todos bucket with keys in a certain range 213 | // (1 <= key < 6), viz., the items for mon through fri. 214 | func (c *Controller) getWeekdayTasks(w http.ResponseWriter, r *http.Request, 215 | _ httprouter.Params) { 216 | 217 | // Get todo items within the weekday range. 218 | items, err := c.todos.RangeItems([]byte("1"), []byte("6")) 219 | if err != nil { 220 | http.Error(w, err.Error(), 500) 221 | } 222 | 223 | // Generate a list of tasks based on todo items retrieved. 224 | taskList := &TaskList{"weekdays", []string{}} 225 | 226 | for _, item := range items { 227 | todo, err := decode(item.Value) 228 | if err != nil { 229 | http.Error(w, err.Error(), 500) 230 | } 231 | taskList.Tasks = append(taskList.Tasks, todo.Task) 232 | } 233 | 234 | w.Header().Set("Content-Type", "application/json") 235 | json.NewEncoder(w).Encode(taskList) 236 | } 237 | 238 | // getDayTasks handles get requests for `/:day`, returning a particular 239 | // day's task list. 240 | // 241 | // Note how we utilize the PrefixItems method for the day requested (as 242 | // indicated in the route's `day` parameter). This makes it easy to get 243 | // items in our todos bucket with a certain prefix, viz. those with the 244 | // prefix representing the requested day. 245 | func (c *Controller) getDayTasks(w http.ResponseWriter, r *http.Request, 246 | p httprouter.Params) { 247 | 248 | // Get todo items for the day requested. 249 | day := p.ByName("day") 250 | num := c.daynum[day] 251 | pre := []byte(strconv.Itoa(num)) // daynum prefix to use 252 | items, err := c.todos.PrefixItems(pre) 253 | if err != nil { 254 | http.Error(w, err.Error(), 500) 255 | } 256 | 257 | // Generate a list of tasks based on todo items retrieved. 258 | taskList := &TaskList{day, []string{}} 259 | 260 | for _, item := range items { 261 | todo, err := decode(item.Value) 262 | if err != nil { 263 | http.Error(w, err.Error(), 500) 264 | } 265 | taskList.Tasks = append(taskList.Tasks, todo.Task) 266 | } 267 | 268 | w.Header().Set("Content-Type", "application/json") 269 | json.NewEncoder(w).Encode(taskList) 270 | } 271 | 272 | // post handles post requests to create a daily todo item. 273 | // 274 | // 275 | func (c *Controller) post(w http.ResponseWriter, r *http.Request, 276 | p httprouter.Params) { 277 | 278 | // Read request body's json payload into buffer. 279 | b, err := ioutil.ReadAll(r.Body) 280 | todo, err := decode(b) 281 | if err != nil { 282 | http.Error(w, err.Error(), 500) 283 | } 284 | 285 | // Use the day number + creation time as key. 286 | day := p.ByName("day") 287 | num := c.daynum[day] // number of day of week 288 | created := todo.Created.Format(time.RFC3339Nano) 289 | key := fmt.Sprintf("%d/%s", num, created) 290 | 291 | // Put key/buffer into todos bucket. 292 | if err := c.todos.Put([]byte(key), b); err != nil { 293 | http.Error(w, err.Error(), 500) 294 | return 295 | } 296 | if verbose { 297 | log.Printf("server: %s: %v", key, todo.Task) 298 | } 299 | 300 | w.Header().Set("Content-Type", "text/plain") 301 | fmt.Fprintf(w, "put todo for %s: %s\n", key, todo) 302 | } 303 | 304 | /* -- CLIENT -- */ 305 | 306 | // Client is our http client for sending requests. 307 | type Client struct{} 308 | 309 | // post sends a post request with a json payload. 310 | func (c *Client) post(url string, todo *Todo) error { 311 | todo.Created = time.Now() 312 | bodyType := "application/json" 313 | body, err := todo.Encode() 314 | if err != nil { 315 | return err 316 | } 317 | resp, err := http.Post(url, bodyType, body) 318 | if err != nil { 319 | return err 320 | } 321 | if verbose { 322 | log.Printf("client: %s\n", resp.Status) 323 | } 324 | return nil 325 | } 326 | 327 | // get sends get requests and expects responses to be a json-encoded 328 | // task list. 329 | func (c *Client) get(url string) (string, error) { 330 | resp, err := http.Get(url) 331 | if err != nil { 332 | return "", err 333 | } 334 | defer resp.Body.Close() 335 | 336 | taskList := new(TaskList) 337 | if err = json.NewDecoder(resp.Body).Decode(taskList); err != nil { 338 | return "", err 339 | } 340 | return strings.Join(taskList.Tasks, ", "), nil 341 | } 342 | 343 | /* -- UTILITY FUNCTIONS, &c. -- */ 344 | 345 | // decode unmarshals a json-encoded byteslice into a Todo. 346 | func decode(b []byte) (*Todo, error) { 347 | todo := new(Todo) 348 | if err := json.Unmarshal(b, todo); err != nil { 349 | return &Todo{}, err 350 | } 351 | return todo, nil 352 | } 353 | 354 | // tempFilePath returns a temporary file path. 355 | func tempFilePath() string { 356 | f, _ := ioutil.TempFile("", "bolt-") 357 | if err := f.Close(); err != nil { 358 | log.Fatal(err) 359 | } 360 | if err := os.Remove(f.Name()); err != nil { 361 | log.Fatal(err) 362 | } 363 | return f.Name() 364 | } 365 | -------------------------------------------------------------------------------- /examples/post.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bytes" 5 | "encoding/json" 6 | "fmt" 7 | "io/ioutil" 8 | "log" 9 | "net/http" 10 | "net/http/httptest" 11 | "os" 12 | "reflect" 13 | 14 | "github.com/joyrexus/buckets" 15 | ) 16 | 17 | const verbose = false // if `true` you'll see log output 18 | 19 | func main() { 20 | // Open a buckets database. 21 | bx, err := buckets.Open(tempFilePath()) 22 | if err != nil { 23 | log.Fatalf("couldn't open db: %v", err) 24 | } 25 | 26 | // Delete and close the db when done. 27 | defer os.Remove(bx.Path()) 28 | defer bx.Close() 29 | 30 | // Create a bucket for storing todos. 31 | todos, err := bx.New([]byte("todos")) 32 | if err != nil { 33 | log.Fatalf("couldn't create todos bucket: %v", err) 34 | } 35 | 36 | // Start our web server. 37 | handler := service{todos} 38 | srv := httptest.NewServer(handler) 39 | defer srv.Close() 40 | 41 | // Daily todos to post. 42 | posts := map[string]*Todo{ 43 | "/mon": {Day: "mon", Task: "milk cows"}, 44 | "/tue": {Day: "tue", Task: "fold laundry"}, 45 | "/wed": {Day: "wed", Task: "flip burgers"}, 46 | "/thu": {Day: "thu", Task: "join army"}, 47 | "/fri": {Day: "fri", Task: "kill time"}, 48 | "/sat": {Day: "sat", Task: "make merry"}, 49 | "/sun": {Day: "sun", Task: "pray quietly"}, 50 | } 51 | 52 | // Make a series of post requests to our server. 53 | for path, todo := range posts { 54 | url := srv.URL + path 55 | bodyType := "application/json" 56 | body, err := todo.Encode() 57 | if err != nil { 58 | log.Print(err) 59 | } 60 | resp, err := http.Post(url, bodyType, body) 61 | if err != nil { 62 | log.Print(err) 63 | } 64 | if verbose { 65 | log.Printf("client: %s\n", resp.Status) 66 | } 67 | } 68 | 69 | // Test that each encoded todo sent to the server was 70 | // in fact stored in the todos bucket. 71 | for route, want := range posts { 72 | // Get encoded todo sent to route. 73 | encoded, err := todos.Get([]byte(route)) 74 | if err != nil { 75 | log.Fatalf("todo bucket is missing entry for %s: %v", route, err) 76 | } 77 | got, err := decode(encoded) 78 | if err != nil { 79 | log.Fatalf("could not decode entry for %s: %v", route, err) 80 | } 81 | if got.Task != want.Task { 82 | log.Fatalf("%s: got %v, want %v", route, got.Task, want.Task) 83 | } 84 | if !reflect.DeepEqual(got, want) { 85 | log.Fatalf("%s: got %v, want %v", route, got, want) 86 | } 87 | } 88 | 89 | // Show the encoded todos now stored in the todos bucket. 90 | do := func(k, v []byte) error { 91 | todo, err := decode(v) 92 | if err != nil { 93 | log.Print(err) 94 | } 95 | fmt.Printf("%s: %s\n", k, todo.Task) 96 | return nil 97 | } 98 | todos.Map(do) 99 | 100 | // Output: 101 | // /fri: kill time 102 | // /mon: milk cows 103 | // /sat: make merry 104 | // /sun: pray quietly 105 | // /thu: join army 106 | // /tue: fold laundry 107 | // /wed: flip burgers 108 | } 109 | 110 | // Todo holds a task description and the day of week in which to do it. 111 | type Todo struct { 112 | Task string 113 | Day string 114 | } 115 | 116 | // Encode marshals a Todo into a buffer. 117 | func (todo *Todo) Encode() (*bytes.Buffer, error) { 118 | b, err := json.Marshal(todo) 119 | if err != nil { 120 | return &bytes.Buffer{}, err 121 | } 122 | return bytes.NewBuffer(b), nil 123 | } 124 | 125 | // This service handles post requests, storing them in a todos bucket. 126 | // The URLs are used as keys and the json-encoded payload as values. 127 | type service struct { 128 | todos *buckets.Bucket 129 | } 130 | 131 | func (s service) ServeHTTP(w http.ResponseWriter, r *http.Request) { 132 | key := []byte(r.URL.String()) 133 | 134 | // Read the request body's json payload into a byteslice. 135 | b, err := ioutil.ReadAll(r.Body) 136 | todo, err := decode(b) 137 | if err != nil { 138 | http.Error(w, err.Error(), 500) 139 | } 140 | 141 | // Put key/json into todos bucket. 142 | if err := s.todos.Put(key, b); err != nil { 143 | http.Error(w, err.Error(), 500) 144 | return 145 | } 146 | 147 | if verbose { 148 | log.Printf("server: %s: %v", key, todo) 149 | } 150 | 151 | w.Header().Set("Content-Type", "text/plain") 152 | fmt.Fprintf(w, "put todo for %s: %s\n", key, todo) 153 | } 154 | 155 | /* -- UTILITY FUNCTIONS -- */ 156 | 157 | // decode unmarshals a json-encoded byteslice into a Todo. 158 | func decode(b []byte) (*Todo, error) { 159 | todo := new(Todo) 160 | if err := json.Unmarshal(b, todo); err != nil { 161 | return todo, err 162 | } 163 | return todo, nil 164 | } 165 | 166 | // tempFilePath returns a temporary file path. 167 | func tempFilePath() string { 168 | f, _ := ioutil.TempFile("", "bolt-") 169 | if err := f.Close(); err != nil { 170 | log.Fatal(err) 171 | } 172 | if err := os.Remove(f.Name()); err != nil { 173 | log.Fatal(err) 174 | } 175 | return f.Name() 176 | } 177 | -------------------------------------------------------------------------------- /examples/prefix.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bytes" 5 | "encoding/json" 6 | "fmt" 7 | "io/ioutil" 8 | "log" 9 | "net/http" 10 | "net/http/httptest" 11 | "os" 12 | "strings" 13 | "time" 14 | 15 | "github.com/joyrexus/buckets" 16 | "github.com/julienschmidt/httprouter" 17 | ) 18 | 19 | const verbose = false // if `true` you'll see log output 20 | 21 | func main() { 22 | // Open a buckets database. 23 | bx, err := buckets.Open(tempFilePath()) 24 | if err != nil { 25 | log.Fatalf("couldn't open db: %v", err) 26 | } 27 | 28 | // Delete and close the db when done. 29 | defer os.Remove(bx.Path()) 30 | defer bx.Close() 31 | 32 | // Create a bucket for storing todos. 33 | bucket, err := bx.New([]byte("todos")) 34 | if err != nil { 35 | log.Fatalf("couldn't create todos bucket: %v", err) 36 | } 37 | 38 | // Initialize our controller for handling specific routes. 39 | control := NewController(bucket) 40 | 41 | // Create and setup our router. 42 | router := httprouter.New() 43 | router.GET("/:day", control.get) 44 | router.POST("/:day", control.post) 45 | 46 | // Start our web server. 47 | srv := httptest.NewServer(router) 48 | defer srv.Close() 49 | 50 | // Daily todos for client to post. 51 | posts := []*Todo{ 52 | {Day: "mon", Task: "milk cows"}, 53 | {Day: "mon", Task: "feed cows"}, 54 | {Day: "mon", Task: "wash cows"}, 55 | {Day: "tue", Task: "wash laundry"}, 56 | {Day: "tue", Task: "fold laundry"}, 57 | {Day: "tue", Task: "iron laundry"}, 58 | {Day: "wed", Task: "flip burgers"}, 59 | {Day: "thu", Task: "join army"}, 60 | {Day: "fri", Task: "kill time"}, 61 | {Day: "sat", Task: "have beer"}, 62 | {Day: "sat", Task: "make merry"}, 63 | {Day: "sun", Task: "take aspirin"}, 64 | {Day: "sun", Task: "pray quietly"}, 65 | } 66 | 67 | // Create our client. 68 | client := new(Client) 69 | 70 | // Have our client post each daily todo. 71 | for _, todo := range posts { 72 | url := srv.URL + "/" + todo.Day 73 | if err := client.post(url, todo); err != nil { 74 | fmt.Printf("client post error: %v", err) 75 | } 76 | } 77 | 78 | // Have our client get a list of tasks for each day. 79 | week := []string{"mon", "tue", "wed", "thu", "fri", "sat", "sun"} 80 | for _, day := range week { 81 | url := srv.URL + "/" + day 82 | tasks, err := client.get(url) 83 | if err != nil { 84 | fmt.Printf("client get error: %v", err) 85 | } 86 | fmt.Printf("%s: %s\n", day, tasks) 87 | } 88 | 89 | // Output: 90 | // mon: milk cows, feed cows, wash cows 91 | // tue: wash laundry, fold laundry, iron laundry 92 | // wed: flip burgers 93 | // thu: join army 94 | // fri: kill time 95 | // sat: have beer, make merry 96 | // sun: take aspirin, pray quietly 97 | } 98 | 99 | /* -- MODELS --*/ 100 | 101 | // A Todo models a daily task. 102 | type Todo struct { 103 | Task string // task to be done 104 | Day string // day to do task 105 | Created time.Time // when created 106 | } 107 | 108 | // Encode marshals a Todo into a buffer. 109 | func (todo *Todo) Encode() (*bytes.Buffer, error) { 110 | b, err := json.Marshal(todo) 111 | if err != nil { 112 | return &bytes.Buffer{}, err 113 | } 114 | return bytes.NewBuffer(b), nil 115 | } 116 | 117 | // A TaskList is a list of tasks for a particular day. 118 | type TaskList struct { 119 | Day string 120 | Tasks []string 121 | } 122 | 123 | /* -- CONTROLLER -- */ 124 | 125 | // NewController initializes a new instance of our controller. 126 | // It provides handler methods for our router. 127 | func NewController(bk *buckets.Bucket) *Controller { 128 | prefix := map[string]*buckets.PrefixScanner{ 129 | "/mon": bk.NewPrefixScanner([]byte("/mon")), 130 | "/tue": bk.NewPrefixScanner([]byte("/tue")), 131 | "/wed": bk.NewPrefixScanner([]byte("/wed")), 132 | "/thu": bk.NewPrefixScanner([]byte("/thu")), 133 | "/fri": bk.NewPrefixScanner([]byte("/fri")), 134 | "/sat": bk.NewPrefixScanner([]byte("/sat")), 135 | "/sun": bk.NewPrefixScanner([]byte("/sun")), 136 | } 137 | return &Controller{bk, prefix} 138 | } 139 | 140 | // Controller handles requests for todo items. The items are stored 141 | // in a todos bucket. The request URLs are used as bucket keys and the 142 | // raw json payload as values. 143 | // 144 | // Note that since we're using `httprouter` (abbreviated as `mux` when 145 | // imported) as our router, each method is a `httprouter.Handle` rather 146 | // than a `http.HandlerFunc`. 147 | type Controller struct { 148 | todos *buckets.Bucket 149 | prefix map[string]*buckets.PrefixScanner 150 | } 151 | 152 | // get handles get requests for a particular day, returning the day's 153 | // task list. 154 | func (c *Controller) get(w http.ResponseWriter, r *http.Request, 155 | _ httprouter.Params) { 156 | 157 | day := r.URL.String() 158 | items, err := c.prefix[day].Items() 159 | if err != nil { 160 | http.Error(w, err.Error(), 500) 161 | } 162 | 163 | taskList := &TaskList{day, []string{}} 164 | 165 | for _, item := range items { 166 | todo, err := decode(item.Value) 167 | if err != nil { 168 | http.Error(w, err.Error(), 500) 169 | } 170 | taskList.Tasks = append(taskList.Tasks, todo.Task) 171 | } 172 | 173 | w.Header().Set("Content-Type", "application/json") 174 | json.NewEncoder(w).Encode(taskList) 175 | } 176 | 177 | // post handles post requests to create a daily todo item. 178 | func (c *Controller) post(w http.ResponseWriter, r *http.Request, 179 | _ httprouter.Params) { 180 | 181 | // Read request body's json payload into buffer. 182 | b, err := ioutil.ReadAll(r.Body) 183 | todo, err := decode(b) 184 | if err != nil { 185 | http.Error(w, err.Error(), 500) 186 | } 187 | 188 | // Use the day (url path) + creation time as key. 189 | key := fmt.Sprintf("%s/%s", r.URL, todo.Created.Format(time.RFC3339Nano)) 190 | 191 | // Put key/buffer into todos bucket. 192 | if err := c.todos.Put([]byte(key), b); err != nil { 193 | http.Error(w, err.Error(), 500) 194 | return 195 | } 196 | 197 | if verbose { 198 | log.Printf("server: %s: %v", key, todo.Task) 199 | } 200 | 201 | w.Header().Set("Content-Type", "text/plain") 202 | fmt.Fprintf(w, "put todo for %s: %s\n", key, todo) 203 | } 204 | 205 | /* -- CLIENT -- */ 206 | 207 | // Client is our http client for sending requests. 208 | type Client struct{} 209 | 210 | // post sends a post request with a json payload. 211 | func (c *Client) post(url string, todo *Todo) error { 212 | todo.Created = time.Now() 213 | bodyType := "application/json" 214 | body, err := todo.Encode() 215 | if err != nil { 216 | return err 217 | } 218 | resp, err := http.Post(url, bodyType, body) 219 | if err != nil { 220 | return err 221 | } 222 | if verbose { 223 | log.Printf("client: %s\n", resp.Status) 224 | } 225 | return nil 226 | } 227 | 228 | // get sends get requests and expects responses to be a json-encoded 229 | // task list. 230 | func (c *Client) get(url string) (string, error) { 231 | resp, err := http.Get(url) 232 | if err != nil { 233 | return "", err 234 | } 235 | defer resp.Body.Close() 236 | 237 | taskList := new(TaskList) 238 | if err = json.NewDecoder(resp.Body).Decode(taskList); err != nil { 239 | return "", err 240 | } 241 | return strings.Join(taskList.Tasks, ", "), nil 242 | } 243 | 244 | /* -- UTILITY FUNCTIONS -- */ 245 | 246 | // decode unmarshals a json-encoded byteslice into a Todo. 247 | func decode(b []byte) (*Todo, error) { 248 | todo := new(Todo) 249 | if err := json.Unmarshal(b, todo); err != nil { 250 | return &Todo{}, err 251 | } 252 | return todo, nil 253 | } 254 | 255 | // tempFilePath returns a temporary file path. 256 | func tempFilePath() string { 257 | f, _ := ioutil.TempFile("", "bolt-") 258 | if err := f.Close(); err != nil { 259 | log.Fatal(err) 260 | } 261 | if err := os.Remove(f.Name()); err != nil { 262 | log.Fatal(err) 263 | } 264 | return f.Name() 265 | } 266 | -------------------------------------------------------------------------------- /examples/range.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bytes" 5 | "encoding/json" 6 | "fmt" 7 | "io/ioutil" 8 | "log" 9 | "net/http" 10 | "net/http/httptest" 11 | "os" 12 | "strings" 13 | "time" 14 | 15 | "github.com/joyrexus/buckets" 16 | "github.com/julienschmidt/httprouter" 17 | ) 18 | 19 | const verbose = false // if `true` you'll see log output 20 | 21 | func main() { 22 | // Open a buckets database. 23 | bx, err := buckets.Open(tempFilePath()) 24 | if err != nil { 25 | log.Fatalf("couldn't open db: %v", err) 26 | } 27 | 28 | // Delete and close the db when done. 29 | defer os.Remove(bx.Path()) 30 | defer bx.Close() 31 | 32 | // Create a bucket for storing todos. 33 | bucket, err := bx.New([]byte("todos")) 34 | if err != nil { 35 | log.Fatalf("couldn't create todos bucket: %v", err) 36 | } 37 | 38 | // Initialize our controller for handling specific routes. 39 | control := NewController(bucket) 40 | 41 | // Create and setup our router. 42 | router := httprouter.New() 43 | router.POST("/day/:day", control.post) 44 | router.GET("/day/:day", control.getDayTasks) 45 | router.GET("/weekend", control.getWeekendTasks) 46 | router.GET("/weekdays", control.getWeekdayTasks) 47 | 48 | // Start our web server. 49 | srv := httptest.NewServer(router) 50 | defer srv.Close() 51 | 52 | // Setup daily todos for client to post. 53 | posts := []*Todo{ 54 | {Day: "mon", Task: "milk cows"}, 55 | {Day: "mon", Task: "feed cows"}, 56 | {Day: "mon", Task: "wash cows"}, 57 | {Day: "tue", Task: "wash laundry"}, 58 | {Day: "tue", Task: "fold laundry"}, 59 | {Day: "tue", Task: "iron laundry"}, 60 | {Day: "wed", Task: "flip burgers"}, 61 | {Day: "thu", Task: "join army"}, 62 | {Day: "fri", Task: "kill time"}, 63 | {Day: "sat", Task: "have beer"}, 64 | {Day: "sat", Task: "make merry"}, 65 | {Day: "sun", Task: "take aspirin"}, 66 | {Day: "sun", Task: "pray quietly"}, 67 | } 68 | 69 | // Create our client. 70 | client := new(Client) 71 | 72 | // Use our client to post each daily todo. 73 | for _, todo := range posts { 74 | url := srv.URL + "/day/" + todo.Day 75 | if err := client.post(url, todo); err != nil { 76 | fmt.Printf("client post error: %v", err) 77 | } 78 | } 79 | 80 | // Now, let's try retrieving the persisted todos. 81 | 82 | // Get a list of tasks for each day. 83 | week := []string{"mon", "tue", "wed", "thu", "fri", "sat", "sun"} 84 | fmt.Println("daily tasks ...") 85 | for _, day := range week { 86 | url := srv.URL + "/day/" + day 87 | tasks, err := client.get(url) 88 | if err != nil { 89 | fmt.Printf("client get error: %v", err) 90 | } 91 | fmt.Printf(" %s: %s\n", day, tasks) 92 | } 93 | // Output: 94 | // daily tasks ... 95 | // mon: milk cows, feed cows, wash cows 96 | // tue: wash laundry, fold laundry, iron laundry 97 | // wed: flip burgers 98 | // thu: join army 99 | // fri: kill time 100 | // sat: have beer, make merry 101 | // sun: take aspirin, pray quietly 102 | 103 | // Get a list of combined tasks for weekdays. 104 | tasks, err := client.get(srv.URL + "/weekdays") 105 | if err != nil { 106 | fmt.Printf("client get error: %v", err) 107 | } 108 | fmt.Printf("\nweekday tasks: %s\n", tasks) 109 | // Output: 110 | // weekday tasks: milk cows, feed cows, wash cows, wash laundry, 111 | // fold laundry, iron laundry, flip burgers, join army, kill time 112 | 113 | // Get a list of combined tasks for the weekend. 114 | tasks, err = client.get(srv.URL + "/weekend") 115 | if err != nil { 116 | fmt.Printf("client get error: %v", err) 117 | } 118 | fmt.Printf("\nweekend tasks: %s\n", tasks) 119 | // Output: 120 | // weekend tasks: have beer, make merry, take aspirin, pray quietly 121 | } 122 | 123 | /* -- MODELS --*/ 124 | 125 | // A Todo models a daily task. 126 | type Todo struct { 127 | Task string // task to be done 128 | Day string // day to do task 129 | Created time.Time // when created 130 | } 131 | 132 | // Encode marshals a Todo into a buffer. 133 | func (todo *Todo) Encode() (*bytes.Buffer, error) { 134 | b, err := json.Marshal(todo) 135 | if err != nil { 136 | return &bytes.Buffer{}, err 137 | } 138 | return bytes.NewBuffer(b), nil 139 | } 140 | 141 | // A TaskList is a list of tasks for a particular day. 142 | type TaskList struct { 143 | When string 144 | Tasks []string 145 | } 146 | 147 | /* -- CONTROLLER -- */ 148 | 149 | // NewController initializes a new instance of our controller. 150 | // It provides handler methods for our router. 151 | func NewController(bk *buckets.Bucket) *Controller { 152 | // map of days to integers 153 | daynum := map[string]int{ 154 | "mon": 1, // monday is the first day of the week 155 | "tue": 2, 156 | "wed": 3, 157 | "thu": 4, 158 | "fri": 5, 159 | "sat": 6, 160 | "sun": 7, 161 | } 162 | // map of scanners for iterating over keys subsets of keys 163 | scan := map[string]buckets.Scanner{ 164 | "mon": bk.NewPrefixScanner([]byte("1")), 165 | "tue": bk.NewPrefixScanner([]byte("2")), 166 | "wed": bk.NewPrefixScanner([]byte("3")), 167 | "thu": bk.NewPrefixScanner([]byte("4")), 168 | "fri": bk.NewPrefixScanner([]byte("5")), 169 | "sat": bk.NewPrefixScanner([]byte("6")), 170 | "sun": bk.NewPrefixScanner([]byte("7")), 171 | // weekdays are mon to fri: 1 <= key < 6. 172 | "weekday": bk.NewRangeScanner([]byte("1"), []byte("6")), 173 | // weekends are sat to sun: 6 <= key < 8. 174 | "weekend": bk.NewRangeScanner([]byte("6"), []byte("8")), 175 | } 176 | return &Controller{bk, daynum, scan} 177 | } 178 | 179 | // Controller handles requests for todo items. The items are stored 180 | // in a todos bucket. The request URLs are used as bucket keys and the 181 | // raw json payload as values. 182 | // 183 | // Note that since we're using `httprouter` (abbreviated as `mux` when 184 | // imported) as our router, each method is a `httprouter.Handle` rather 185 | // than a `http.HandlerFunc`. 186 | type Controller struct { 187 | todos *buckets.Bucket 188 | daynum map[string]int 189 | scan map[string]buckets.Scanner 190 | } 191 | 192 | // getWeekendTasks handles get requests for `/weekend`, returning the 193 | // combined task list for saturday and sunday. 194 | // 195 | // Note how we utilize the `weekend` range scanner, which makes it easy 196 | // to iterate over keys in our todos bucket within a certain range, 197 | // viz. those keys from saturday (day number 6) to sunday (7). 198 | func (c *Controller) getWeekendTasks(w http.ResponseWriter, r *http.Request, 199 | _ httprouter.Params) { 200 | 201 | // Get todo items within the weekend range. 202 | items, err := c.scan["weekend"].Items() 203 | if err != nil { 204 | http.Error(w, err.Error(), 500) 205 | } 206 | 207 | // Generate a list of tasks based on todo items retrieved. 208 | taskList := &TaskList{"weekend", []string{}} 209 | 210 | for _, item := range items { 211 | todo, err := decode(item.Value) 212 | if err != nil { 213 | http.Error(w, err.Error(), 500) 214 | } 215 | taskList.Tasks = append(taskList.Tasks, todo.Task) 216 | } 217 | 218 | w.Header().Set("Content-Type", "application/json") 219 | json.NewEncoder(w).Encode(taskList) 220 | } 221 | 222 | // getWeekdayTasks handles get requests for `/weekdays`, returning the 223 | // combined task list for monday through friday. 224 | // 225 | // Note how we utilize the `weekday` range scanner, which makes it easy 226 | // to iterate over keys in our todos bucket within a certain range, 227 | // viz. those keys from monday (day number 1) to friday (5). 228 | func (c *Controller) getWeekdayTasks(w http.ResponseWriter, r *http.Request, 229 | _ httprouter.Params) { 230 | 231 | // Get todo items within the weekday range. 232 | items, err := c.scan["weekday"].Items() 233 | if err != nil { 234 | http.Error(w, err.Error(), 500) 235 | } 236 | 237 | // Generate a list of tasks based on todo items retrieved. 238 | taskList := &TaskList{"weekdays", []string{}} 239 | 240 | for _, item := range items { 241 | todo, err := decode(item.Value) 242 | if err != nil { 243 | http.Error(w, err.Error(), 500) 244 | } 245 | taskList.Tasks = append(taskList.Tasks, todo.Task) 246 | } 247 | 248 | w.Header().Set("Content-Type", "application/json") 249 | json.NewEncoder(w).Encode(taskList) 250 | } 251 | 252 | // getDayTasks handles get requests for `/:day`, returning a particular 253 | // day's task list. 254 | // 255 | // Note how we utilize the prefix scanner for the day requested (as indicated 256 | // in the route's `day` parameter). This makes it easy to iterate over keys 257 | // in our todos bucket with a certain prefix, viz. those with the prefix 258 | // representing the requested day. 259 | func (c *Controller) getDayTasks(w http.ResponseWriter, r *http.Request, 260 | p httprouter.Params) { 261 | 262 | // Get todo items for the day requested. 263 | day := p.ByName("day") 264 | items, err := c.scan[day].Items() 265 | if err != nil { 266 | http.Error(w, err.Error(), 500) 267 | } 268 | 269 | // Generate a list of tasks based on todo items retrieved. 270 | taskList := &TaskList{day, []string{}} 271 | 272 | for _, item := range items { 273 | todo, err := decode(item.Value) 274 | if err != nil { 275 | http.Error(w, err.Error(), 500) 276 | } 277 | taskList.Tasks = append(taskList.Tasks, todo.Task) 278 | } 279 | 280 | w.Header().Set("Content-Type", "application/json") 281 | json.NewEncoder(w).Encode(taskList) 282 | } 283 | 284 | // post handles post requests to create a daily todo item. 285 | // 286 | // 287 | func (c *Controller) post(w http.ResponseWriter, r *http.Request, 288 | p httprouter.Params) { 289 | 290 | // Read request body's json payload into buffer. 291 | b, err := ioutil.ReadAll(r.Body) 292 | todo, err := decode(b) 293 | if err != nil { 294 | http.Error(w, err.Error(), 500) 295 | } 296 | 297 | // Use the day number + creation time as key. 298 | day := p.ByName("day") 299 | num := c.daynum[day] // number of day of week 300 | created := todo.Created.Format(time.RFC3339Nano) 301 | key := fmt.Sprintf("%d/%s", num, created) 302 | 303 | // Put key/buffer into todos bucket. 304 | if err := c.todos.Put([]byte(key), b); err != nil { 305 | http.Error(w, err.Error(), 500) 306 | return 307 | } 308 | if verbose { 309 | log.Printf("server: %s: %v", key, todo.Task) 310 | } 311 | 312 | w.Header().Set("Content-Type", "text/plain") 313 | fmt.Fprintf(w, "put todo for %s: %s\n", key, todo) 314 | } 315 | 316 | /* -- CLIENT -- */ 317 | 318 | // Client is our http client for sending requests. 319 | type Client struct{} 320 | 321 | // post sends a post request with a json payload. 322 | func (c *Client) post(url string, todo *Todo) error { 323 | todo.Created = time.Now() 324 | bodyType := "application/json" 325 | body, err := todo.Encode() 326 | if err != nil { 327 | return err 328 | } 329 | resp, err := http.Post(url, bodyType, body) 330 | if err != nil { 331 | return err 332 | } 333 | if verbose { 334 | log.Printf("client: %s\n", resp.Status) 335 | } 336 | return nil 337 | } 338 | 339 | // get sends get requests and expects responses to be a json-encoded 340 | // task list. 341 | func (c *Client) get(url string) (string, error) { 342 | resp, err := http.Get(url) 343 | if err != nil { 344 | return "", err 345 | } 346 | defer resp.Body.Close() 347 | 348 | taskList := new(TaskList) 349 | if err = json.NewDecoder(resp.Body).Decode(taskList); err != nil { 350 | return "", err 351 | } 352 | return strings.Join(taskList.Tasks, ", "), nil 353 | } 354 | 355 | /* -- UTILITY FUNCTIONS, &c. -- */ 356 | 357 | // decode unmarshals a json-encoded byteslice into a Todo. 358 | func decode(b []byte) (*Todo, error) { 359 | todo := new(Todo) 360 | if err := json.Unmarshal(b, todo); err != nil { 361 | return &Todo{}, err 362 | } 363 | return todo, nil 364 | } 365 | 366 | // tempFilePath returns a temporary file path. 367 | func tempFilePath() string { 368 | f, _ := ioutil.TempFile("", "bolt-") 369 | if err := f.Close(); err != nil { 370 | log.Fatal(err) 371 | } 372 | if err := os.Remove(f.Name()); err != nil { 373 | log.Fatal(err) 374 | } 375 | return f.Name() 376 | } 377 | -------------------------------------------------------------------------------- /examples/roundtrip.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bytes" 5 | "encoding/json" 6 | "fmt" 7 | "io/ioutil" 8 | "log" 9 | "net/http" 10 | "net/http/httptest" 11 | "os" 12 | 13 | "github.com/joyrexus/buckets" 14 | mux "github.com/julienschmidt/httprouter" 15 | ) 16 | 17 | const verbose = false // if `true` you'll see log output 18 | 19 | func main() { 20 | // Open a buckets database. 21 | bx, err := buckets.Open(tempFilePath()) 22 | if err != nil { 23 | log.Fatalf("couldn't open db: %v", err) 24 | } 25 | 26 | // Delete and close the db when done. 27 | defer os.Remove(bx.Path()) 28 | defer bx.Close() 29 | 30 | // Create a bucket for storing todos. 31 | bucket, err := bx.New([]byte("todos")) 32 | if err != nil { 33 | log.Fatalf("couldn't create todos bucket: %v", err) 34 | } 35 | 36 | // Create our service for handling routes. 37 | service := NewService(bucket) 38 | 39 | // Create and setup our router. 40 | router := mux.New() 41 | router.GET("/:day", service.get) 42 | router.POST("/:day", service.post) 43 | 44 | // Start our web server. 45 | srv := httptest.NewServer(router) 46 | defer srv.Close() 47 | 48 | // Daily todos for client to post. 49 | posts := map[string]*Todo{ 50 | "/mon": {Day: "mon", Task: "milk cows"}, 51 | "/tue": {Day: "tue", Task: "fold laundry"}, 52 | "/wed": {Day: "wed", Task: "flip burgers"}, 53 | "/thu": {Day: "thu", Task: "join army"}, 54 | "/fri": {Day: "fri", Task: "kill time"}, 55 | "/sat": {Day: "sat", Task: "make merry"}, 56 | "/sun": {Day: "sun", Task: "pray quietly"}, 57 | } 58 | 59 | // Create our client. 60 | client := new(Client) 61 | 62 | for path, todo := range posts { 63 | url := srv.URL + path 64 | if err := client.post(url, todo); err != nil { 65 | fmt.Printf("client post error: %v", err) 66 | } 67 | } 68 | 69 | for path := range posts { 70 | url := srv.URL + path 71 | task, err := client.get(url) 72 | if err != nil { 73 | fmt.Printf("client get error: %v", err) 74 | } 75 | fmt.Printf("%s: %s\n", path, task) 76 | } 77 | 78 | // Output: 79 | // /mon: milk cows 80 | // /tue: fold laundry 81 | // /wed: flip burgers 82 | // /thu: join army 83 | // /fri: kill time 84 | // /sat: make merry 85 | // /sun: pray quietly 86 | } 87 | 88 | /* -- MODELS --*/ 89 | 90 | // Todo holds a task description and the day of week in which to do it. 91 | type Todo struct { 92 | Task string 93 | Day string 94 | } 95 | 96 | // Encode marshals a Todo into a buffer. 97 | func (todo *Todo) Encode() (*bytes.Buffer, error) { 98 | b, err := json.Marshal(todo) 99 | if err != nil { 100 | return &bytes.Buffer{}, err 101 | } 102 | return bytes.NewBuffer(b), nil 103 | } 104 | 105 | /* -- SERVICE -- */ 106 | 107 | // NewService initializes a new instance of our service. 108 | func NewService(bk *buckets.Bucket) *Service { 109 | return &Service{bk} 110 | } 111 | 112 | // Service handles requests for todo items. The items are stored 113 | // in a todos bucket. The request URLs are used as bucket keys and the 114 | // raw json payload as values. 115 | // 116 | // In MVC parlance, our service would be called a "controller". We use 117 | // it to define "handle" methods for our router. Note that since we're using 118 | // `httprouter` (abbreviated as `mux` when imported) as our router, each 119 | // service method is a `httprouter.Handle` rather than a `http.HandlerFunc`. 120 | type Service struct { 121 | todos *buckets.Bucket 122 | } 123 | 124 | // get handles get requests for a daily todo item. 125 | func (s *Service) get(w http.ResponseWriter, r *http.Request, _ mux.Params) { 126 | key := []byte(r.URL.String()) 127 | value, err := s.todos.Get(key) 128 | if err != nil { 129 | http.Error(w, err.Error(), 500) 130 | } 131 | w.Header().Set("Content-Type", "application/json") 132 | w.Write(value) 133 | } 134 | 135 | // post handles post requests to create a daily todo item. 136 | func (s *Service) post(w http.ResponseWriter, r *http.Request, _ mux.Params) { 137 | // Read request body's json payload into buffer. 138 | b, err := ioutil.ReadAll(r.Body) 139 | todo, err := decode(b) 140 | if err != nil { 141 | http.Error(w, err.Error(), 500) 142 | } 143 | 144 | // Use the url path as key. 145 | key := []byte(r.URL.String()) 146 | 147 | // Put key/buffer into todos bucket. 148 | if err := s.todos.Put(key, b); err != nil { 149 | http.Error(w, err.Error(), 500) 150 | return 151 | } 152 | 153 | if verbose { 154 | log.Printf("server: %s: %v", key, todo.Task) 155 | } 156 | 157 | w.Header().Set("Content-Type", "text/plain") 158 | fmt.Fprintf(w, "put todo for %s: %s\n", key, todo) 159 | } 160 | 161 | /* -- CLIENT -- */ 162 | 163 | // Client is our http client for sending requests. 164 | type Client struct{} 165 | 166 | // post sends a post request with a json payload. 167 | func (c *Client) post(url string, todo *Todo) error { 168 | bodyType := "application/json" 169 | body, err := todo.Encode() 170 | if err != nil { 171 | return err 172 | } 173 | resp, err := http.Post(url, bodyType, body) 174 | if err != nil { 175 | return err 176 | } 177 | if verbose { 178 | log.Printf("client: %s\n", resp.Status) 179 | } 180 | return nil 181 | } 182 | 183 | // get sends get requests and expects responses to be a json-encoded todo item. 184 | func (c *Client) get(url string) (string, error) { 185 | resp, err := http.Get(url) 186 | if err != nil { 187 | return "", err 188 | } 189 | defer resp.Body.Close() 190 | 191 | todo := new(Todo) 192 | if err = json.NewDecoder(resp.Body).Decode(todo); err != nil { 193 | return "", err 194 | } 195 | return todo.Task, nil 196 | } 197 | 198 | /* -- UTILITY FUNCTIONS -- */ 199 | 200 | // decode unmarshals a json-encoded byteslice into a Todo. 201 | func decode(b []byte) (*Todo, error) { 202 | todo := new(Todo) 203 | if err := json.Unmarshal(b, todo); err != nil { 204 | return &Todo{}, err 205 | } 206 | return todo, nil 207 | } 208 | 209 | // tempFilePath returns a temporary file path. 210 | func tempFilePath() string { 211 | f, _ := ioutil.TempFile("", "bolt-") 212 | if err := f.Close(); err != nil { 213 | log.Fatal(err) 214 | } 215 | if err := os.Remove(f.Name()); err != nil { 216 | log.Fatal(err) 217 | } 218 | return f.Name() 219 | } 220 | -------------------------------------------------------------------------------- /http_test.go: -------------------------------------------------------------------------------- 1 | package buckets_test 2 | 3 | import ( 4 | "encoding/binary" 5 | "fmt" 6 | "io/ioutil" 7 | "log" 8 | "math/rand" 9 | "net/http" 10 | "net/http/httptest" 11 | "os" 12 | 13 | "github.com/joyrexus/buckets" 14 | ) 15 | 16 | // Set this to see how the counts are actually updated. 17 | const verbose = false 18 | 19 | // Counter updates a the hits bucket for every URL path requested. 20 | type counter struct { 21 | hits *buckets.Bucket 22 | } 23 | 24 | // Our handler communicates the new count from a successful database 25 | // transaction. 26 | func (c counter) ServeHTTP(rw http.ResponseWriter, req *http.Request) { 27 | key := []byte(req.URL.String()) 28 | 29 | // Decode handles key not found for us. 30 | value, _ := c.hits.Get(key) 31 | count := decode(value) + 1 32 | 33 | if err := c.hits.Put(key, encode(count)); err != nil { 34 | http.Error(rw, err.Error(), 500) 35 | return 36 | } 37 | 38 | if verbose { 39 | log.Printf("server: %s: %d", req.URL.String(), count) 40 | } 41 | 42 | // Reply with the new count . 43 | rw.Header().Set("Content-Type", "application/octet-stream") 44 | fmt.Fprintf(rw, "%d\n", count) 45 | } 46 | 47 | func client(id int, base string, paths []string) error { 48 | // Process paths in random order. 49 | rng := rand.New(rand.NewSource(int64(id))) 50 | permutation := rng.Perm(len(paths)) 51 | 52 | for i := range paths { 53 | path := paths[permutation[i]] 54 | resp, err := http.Get(base + path) 55 | if err != nil { 56 | return err 57 | } 58 | defer resp.Body.Close() 59 | buf, err := ioutil.ReadAll(resp.Body) 60 | if err != nil { 61 | return err 62 | } 63 | if verbose { 64 | log.Printf("client: %s: %s", path, buf) 65 | } 66 | } 67 | return nil 68 | } 69 | 70 | func ExampleBucket() { 71 | // Open the database. 72 | bx, _ := buckets.Open(tempfile()) 73 | defer os.Remove(bx.Path()) 74 | defer bx.Close() 75 | 76 | // Create a hits bucket 77 | hits, _ := bx.New([]byte("hits")) 78 | 79 | // Start our web server 80 | count := counter{hits} 81 | srv := httptest.NewServer(count) 82 | defer srv.Close() 83 | 84 | // Get every path multiple times. 85 | paths := []string{ 86 | "/foo", 87 | "/bar", 88 | "/baz", 89 | "/quux", 90 | "/thud", 91 | "/xyzzy", 92 | } 93 | for id := 0; id < 10; id++ { 94 | if err := client(id, srv.URL, paths); err != nil { 95 | fmt.Printf("client error: %v", err) 96 | } 97 | } 98 | 99 | // Check the final result 100 | do := func(k, v []byte) error { 101 | fmt.Printf("hits to %s: %d\n", k, decode(v)) 102 | return nil 103 | } 104 | hits.Map(do) 105 | // outputs ... 106 | // hits to /bar: 10 107 | // hits to /baz: 10 108 | // hits to /foo: 10 109 | // hits to /quux: 10 110 | // hits to /thud: 10 111 | // hits to /xyzzy: 10 112 | 113 | // Output: 114 | // hits to /bar: 10 115 | // hits to /baz: 10 116 | // hits to /foo: 10 117 | // hits to /quux: 10 118 | // hits to /thud: 10 119 | // hits to /xyzzy: 10 120 | } 121 | 122 | // encode marshals a counter. 123 | func encode(n uint64) []byte { 124 | buf := make([]byte, 8) 125 | binary.BigEndian.PutUint64(buf, n) 126 | return buf 127 | } 128 | 129 | // decode unmarshals a counter. Nil buffers are decoded as 0. 130 | func decode(buf []byte) uint64 { 131 | if buf == nil { 132 | return 0 133 | } 134 | return binary.BigEndian.Uint64(buf) 135 | } 136 | -------------------------------------------------------------------------------- /map_test.go: -------------------------------------------------------------------------------- 1 | package buckets_test 2 | 3 | import ( 4 | "bytes" 5 | "fmt" 6 | "os" 7 | "testing" 8 | 9 | "github.com/joyrexus/buckets" 10 | ) 11 | 12 | // Ensure that we can apply functions to each k/v pair. 13 | func TestMap(t *testing.T) { 14 | bx := NewTestDB() 15 | defer bx.Close() 16 | 17 | // Create a new bucket. 18 | letters, err := bx.New([]byte("letters")) 19 | if err != nil { 20 | t.Error(err.Error()) 21 | } 22 | 23 | // Setup items to insert. 24 | items := []struct { 25 | Key, Value []byte 26 | }{ 27 | {[]byte("A"), []byte("alpha")}, 28 | {[]byte("B"), []byte("beta")}, 29 | {[]byte("C"), []byte("gamma")}, 30 | } 31 | 32 | // Insert items into `letters` bucket. 33 | if err := letters.Insert(items); err != nil { 34 | fmt.Println("could not insert items!") 35 | } 36 | 37 | // Setup slice of items to collect results. 38 | type item struct { 39 | Key, Value []byte 40 | } 41 | results := []item{} 42 | 43 | // Anon func to apply to each item in bucket. 44 | // Here, we're just going to collect the items just inserted. 45 | do := func(k, v []byte) error { 46 | results = append(results, item{k, v}) 47 | return nil 48 | } 49 | 50 | // Now map the `do` function over each item. 51 | if err := letters.Map(do); err != nil { 52 | t.Error(err.Error()) 53 | } 54 | 55 | // Finally, check to see if our results match the originally 56 | // inserted items. 57 | for i, want := range items { 58 | got := results[i] 59 | if !bytes.Equal(got.Key, want.Key) { 60 | t.Errorf("got %v, want %v", got.Key, want.Key) 61 | } 62 | if !bytes.Equal(got.Value, want.Value) { 63 | t.Errorf("got %v, want %v", got.Value, want.Value) 64 | } 65 | } 66 | } 67 | 68 | // Ensure that we can apply a function to the k/v pairs 69 | // of keys with a given prefix. 70 | func TestMapPrefix(t *testing.T) { 71 | bx := NewTestDB() 72 | defer bx.Close() 73 | 74 | // Create a new things bucket. 75 | things, err := bx.New([]byte("things")) 76 | if err != nil { 77 | t.Error(err.Error()) 78 | } 79 | 80 | // Setup items to insert. 81 | items := []struct { 82 | Key, Value []byte 83 | }{ 84 | {[]byte("A"), []byte("1")}, // `A` prefix match 85 | {[]byte("AA"), []byte("2")}, // match 86 | {[]byte("AAA"), []byte("3")}, // match 87 | {[]byte("AAB"), []byte("2")}, // match 88 | {[]byte("B"), []byte("O")}, 89 | {[]byte("BA"), []byte("0")}, 90 | {[]byte("BAA"), []byte("0")}, 91 | } 92 | 93 | // Insert 'em. 94 | if err := things.Insert(items); err != nil { 95 | t.Error(err.Error()) 96 | } 97 | 98 | // Now collect each item whose key starts with "A". 99 | prefix := []byte("A") 100 | 101 | // Expected items for keys with prefix "A". 102 | expected := []struct { 103 | Key, Value []byte 104 | }{ 105 | {[]byte("A"), []byte("1")}, 106 | {[]byte("AA"), []byte("2")}, 107 | {[]byte("AAA"), []byte("3")}, 108 | {[]byte("AAB"), []byte("2")}, 109 | } 110 | 111 | // Setup slice of items to collect results. 112 | type item struct { 113 | Key, Value []byte 114 | } 115 | results := []item{} 116 | 117 | // Anon func to map over matched keys. 118 | do := func(k, v []byte) error { 119 | results = append(results, item{k, v}) 120 | return nil 121 | } 122 | 123 | if err := things.MapPrefix(do, prefix); err != nil { 124 | t.Error(err.Error()) 125 | } 126 | 127 | for i, want := range expected { 128 | got := results[i] 129 | if !bytes.Equal(got.Key, want.Key) { 130 | t.Errorf("got %v, want %v", got.Key, want.Key) 131 | } 132 | if !bytes.Equal(got.Value, want.Value) { 133 | t.Errorf("got %v, want %v", got.Value, want.Value) 134 | } 135 | } 136 | } 137 | 138 | // Show that we can apply a function to the k/v pairs 139 | // of keys with a given prefix. 140 | func ExampleBucket_MapPrefix() { 141 | bx, _ := buckets.Open(tempfile()) 142 | defer os.Remove(bx.Path()) 143 | defer bx.Close() 144 | 145 | // Create a new things bucket. 146 | things, _ := bx.New([]byte("things")) 147 | 148 | // Setup items to insert. 149 | items := []struct { 150 | Key, Value []byte 151 | }{ 152 | {[]byte("A"), []byte("1")}, // `A` prefix match 153 | {[]byte("AA"), []byte("2")}, // match 154 | {[]byte("AAA"), []byte("3")}, // match 155 | {[]byte("AAB"), []byte("2")}, // match 156 | {[]byte("B"), []byte("O")}, 157 | {[]byte("BA"), []byte("0")}, 158 | {[]byte("BAA"), []byte("0")}, 159 | } 160 | 161 | // Insert 'em. 162 | if err := things.Insert(items); err != nil { 163 | fmt.Printf("could not insert items in `things` bucket: %v\n", err) 164 | } 165 | 166 | // Now collect each item whose key starts with "A". 167 | prefix := []byte("A") 168 | 169 | // Setup slice of items. 170 | type item struct { 171 | Key, Value []byte 172 | } 173 | results := []item{} 174 | 175 | // Anon func to map over matched keys. 176 | do := func(k, v []byte) error { 177 | results = append(results, item{k, v}) 178 | return nil 179 | } 180 | 181 | if err := things.MapPrefix(do, prefix); err != nil { 182 | fmt.Printf("could not map items with prefix %s: %v\n", prefix, err) 183 | } 184 | 185 | for _, item := range results { 186 | fmt.Printf("%s -> %s\n", item.Key, item.Value) 187 | } 188 | // Output: 189 | // A -> 1 190 | // AA -> 2 191 | // AAA -> 3 192 | // AAB -> 2 193 | } 194 | 195 | // Ensure we can apply functions to the k/v pairs 196 | // of keys within a given range. 197 | func TestMapRange(t *testing.T) { 198 | bx := NewTestDB() 199 | defer bx.Close() 200 | 201 | years, err := bx.New([]byte("years")) 202 | if err != nil { 203 | t.Error(err.Error()) 204 | } 205 | 206 | // Setup items to insert in `years` bucket 207 | items := []struct { 208 | Key, Value []byte 209 | }{ 210 | {[]byte("1970"), []byte("70")}, 211 | {[]byte("1975"), []byte("75")}, 212 | {[]byte("1980"), []byte("80")}, 213 | {[]byte("1985"), []byte("85")}, 214 | {[]byte("1990"), []byte("90")}, // min = 1990 215 | {[]byte("1995"), []byte("95")}, // min < 1995 < max 216 | {[]byte("2000"), []byte("00")}, // max = 2000 217 | {[]byte("2005"), []byte("05")}, 218 | {[]byte("2010"), []byte("10")}, 219 | } 220 | 221 | // Insert 'em. 222 | if err := years.Insert(items); err != nil { 223 | t.Error(err.Error()) 224 | } 225 | 226 | // Time range to map over. 227 | min := []byte("1990") 228 | max := []byte("2000") 229 | 230 | // Expected items within time range: 1990 <= key <= 2000. 231 | expected := []struct { 232 | Key, Value []byte 233 | }{ 234 | {[]byte("1990"), []byte("90")}, // min = 1990 235 | {[]byte("1995"), []byte("95")}, // min < 1995 < max 236 | {[]byte("2000"), []byte("00")}, // max = 2000 237 | } 238 | 239 | // Setup slice of items to collect results. 240 | type item struct { 241 | Key, Value []byte 242 | } 243 | results := []item{} 244 | 245 | // Anon func to map over matched keys. 246 | do := func(k, v []byte) error { 247 | results = append(results, item{k, v}) 248 | return nil 249 | } 250 | 251 | if err := years.MapRange(do, min, max); err != nil { 252 | t.Error(err.Error()) 253 | } 254 | 255 | for i, want := range expected { 256 | got := results[i] 257 | if !bytes.Equal(got.Key, want.Key) { 258 | t.Errorf("got %v, want %v", got.Key, want.Key) 259 | } 260 | if !bytes.Equal(got.Value, want.Value) { 261 | t.Errorf("got %v, want %v", got.Value, want.Value) 262 | } 263 | } 264 | } 265 | 266 | // Show that we can apply a function to the k/v pairs 267 | // of keys within a given range. 268 | func ExampleBucket_MapRange() { 269 | bx, _ := buckets.Open(tempfile()) 270 | defer os.Remove(bx.Path()) 271 | defer bx.Close() 272 | 273 | // Delete any existing bucket named "years". 274 | bx.Delete([]byte("years")) 275 | 276 | // Create a new bucket named "years". 277 | years, _ := bx.New([]byte("years")) 278 | 279 | // Setup items to insert in `years` bucket 280 | items := []struct { 281 | Key, Value []byte 282 | }{ 283 | {[]byte("1970"), []byte("70")}, 284 | {[]byte("1975"), []byte("75")}, 285 | {[]byte("1980"), []byte("80")}, 286 | {[]byte("1985"), []byte("85")}, 287 | {[]byte("1990"), []byte("90")}, // min = 1990 288 | {[]byte("1995"), []byte("95")}, // min < 1995 < max 289 | {[]byte("2000"), []byte("00")}, // max = 2000 290 | {[]byte("2005"), []byte("05")}, 291 | {[]byte("2010"), []byte("10")}, 292 | } 293 | 294 | // Insert 'em. 295 | if err := years.Insert(items); err != nil { 296 | fmt.Printf("could not insert items in `years` bucket: %v\n", err) 297 | } 298 | 299 | // Time range to map over: 1990 <= key <= 2000. 300 | min := []byte("1990") 301 | max := []byte("2000") 302 | 303 | // Setup slice of items to collect results. 304 | type item struct { 305 | Key, Value []byte 306 | } 307 | results := []item{} 308 | 309 | // Anon func to map over matched keys. 310 | do := func(k, v []byte) error { 311 | results = append(results, item{k, v}) 312 | return nil 313 | } 314 | 315 | if err := years.MapRange(do, min, max); err != nil { 316 | fmt.Printf("could not map items within range: %v\n", err) 317 | } 318 | 319 | for _, item := range results { 320 | fmt.Printf("%s -> %s\n", item.Key, item.Value) 321 | } 322 | // Output: 323 | // 1990 -> 90 324 | // 1995 -> 95 325 | // 2000 -> 00 326 | } 327 | -------------------------------------------------------------------------------- /prefixscan.go: -------------------------------------------------------------------------------- 1 | package buckets 2 | 3 | import ( 4 | "bytes" 5 | 6 | "github.com/boltdb/bolt" 7 | ) 8 | 9 | // A PrefixScanner scans a bucket for keys with a given prefix. 10 | type PrefixScanner struct { 11 | db *DB 12 | BucketName []byte 13 | Prefix []byte 14 | } 15 | 16 | // Map applies `do` on each key/value pair for keys with prefix. 17 | func (ps *PrefixScanner) Map(do func(k, v []byte) error) error { 18 | pre := ps.Prefix 19 | return ps.db.View(func(tx *bolt.Tx) error { 20 | c := tx.Bucket(ps.BucketName).Cursor() 21 | for k, v := c.Seek(pre); bytes.HasPrefix(k, pre); k, _ = c.Next() { 22 | do(k, v) 23 | } 24 | return nil 25 | }) 26 | } 27 | 28 | // Count returns a count of the keys with prefix. 29 | func (ps *PrefixScanner) Count() (count int, err error) { 30 | pre := ps.Prefix 31 | err = ps.db.View(func(tx *bolt.Tx) error { 32 | c := tx.Bucket(ps.BucketName).Cursor() 33 | for k, _ := c.Seek(pre); bytes.HasPrefix(k, pre); k, _ = c.Next() { 34 | count++ 35 | } 36 | return nil 37 | }) 38 | if err != nil { 39 | return count, err 40 | } 41 | return count, err 42 | } 43 | 44 | // Keys returns a slice of keys with prefix. 45 | func (ps *PrefixScanner) Keys() (keys [][]byte, err error) { 46 | pre := ps.Prefix 47 | err = ps.db.View(func(tx *bolt.Tx) error { 48 | c := tx.Bucket(ps.BucketName).Cursor() 49 | for k, _ := c.Seek(pre); bytes.HasPrefix(k, pre); k, _ = c.Next() { 50 | keys = append(keys, k) 51 | } 52 | return nil 53 | }) 54 | if err != nil { 55 | return nil, err 56 | } 57 | return keys, err 58 | } 59 | 60 | // Values returns a slice of values for keys with prefix. 61 | func (ps *PrefixScanner) Values() (values [][]byte, err error) { 62 | pre := ps.Prefix 63 | err = ps.db.View(func(tx *bolt.Tx) error { 64 | c := tx.Bucket(ps.BucketName).Cursor() 65 | for k, v := c.Seek(pre); bytes.HasPrefix(k, pre); k, v = c.Next() { 66 | values = append(values, v) 67 | } 68 | return nil 69 | }) 70 | if err != nil { 71 | return nil, err 72 | } 73 | return values, err 74 | } 75 | 76 | // Items returns a slice of key/value pairs for keys with prefix. 77 | func (ps *PrefixScanner) Items() (items []Item, err error) { 78 | pre := ps.Prefix 79 | err = ps.db.View(func(tx *bolt.Tx) error { 80 | c := tx.Bucket(ps.BucketName).Cursor() 81 | for k, v := c.Seek(pre); bytes.HasPrefix(k, pre); k, v = c.Next() { 82 | items = append(items, Item{k, v}) 83 | } 84 | return nil 85 | }) 86 | if err != nil { 87 | return nil, err 88 | } 89 | return items, err 90 | } 91 | 92 | // ItemMapping returns a map of key/value pairs for keys with prefix. 93 | // This only works with buckets whose keys are byte-sliced strings. 94 | func (ps *PrefixScanner) ItemMapping() (map[string][]byte, error) { 95 | pre := ps.Prefix 96 | items := make(map[string][]byte) 97 | err := ps.db.View(func(tx *bolt.Tx) error { 98 | c := tx.Bucket(ps.BucketName).Cursor() 99 | for k, v := c.Seek(pre); bytes.HasPrefix(k, pre); k, v = c.Next() { 100 | items[string(k)] = v 101 | } 102 | return nil 103 | }) 104 | if err != nil { 105 | return nil, err 106 | } 107 | return items, err 108 | } 109 | -------------------------------------------------------------------------------- /prefixscan_test.go: -------------------------------------------------------------------------------- 1 | package buckets_test 2 | 3 | import ( 4 | "bytes" 5 | "testing" 6 | ) 7 | 8 | // Ensure we can scan prefixes. 9 | func TestPrefixScanner(t *testing.T) { 10 | bx := NewTestDB() 11 | defer bx.Close() 12 | 13 | paths, err := bx.New([]byte("paths")) 14 | 15 | // items to put in `paths` bucket 16 | pathItems := []struct { 17 | Key, Value []byte 18 | }{ 19 | {[]byte("f/"), []byte("")}, 20 | {[]byte("fo/"), []byte("")}, 21 | {[]byte("foo/"), []byte("foo")}, 22 | {[]byte("foo/bar/"), []byte("bar")}, 23 | {[]byte("foo/bar/baz/"), []byte("baz")}, 24 | {[]byte("food/"), []byte("")}, 25 | {[]byte("good/"), []byte("")}, 26 | {[]byte("goo/"), []byte("")}, 27 | } 28 | 29 | if err = paths.Insert(pathItems); err != nil { 30 | t.Error(err.Error()) 31 | } 32 | 33 | foo := paths.NewPrefixScanner([]byte("foo/")) 34 | 35 | // expected items in `foo` 36 | wantItems := []struct { 37 | Key, Value []byte 38 | }{ 39 | {[]byte("foo/"), []byte("foo")}, 40 | {[]byte("foo/bar/"), []byte("bar")}, 41 | {[]byte("foo/bar/baz/"), []byte("baz")}, 42 | } 43 | 44 | // expected count of items in range 45 | want := len(wantItems) 46 | 47 | got, err := foo.Count() 48 | if err != nil { 49 | t.Error(err.Error()) 50 | } 51 | 52 | if got != want { 53 | t.Errorf("got %v, want %v", got, want) 54 | } 55 | 56 | // get keys for paths with `foo` prefix 57 | keys, err := foo.Keys() 58 | if err != nil { 59 | t.Error(err.Error()) 60 | } 61 | 62 | for i, want := range wantItems { 63 | if got := keys[i]; !bytes.Equal(got, want.Key) { 64 | t.Errorf("got %s, want %s", got, want.Key) 65 | } 66 | } 67 | 68 | // get values for paths with `foo` prefix 69 | values, err := foo.Values() 70 | if err != nil { 71 | t.Error(err.Error()) 72 | } 73 | 74 | for i, want := range wantItems { 75 | if got := values[i]; !bytes.Equal(got, want.Value) { 76 | t.Errorf("got %s, want %s", got, want.Value) 77 | } 78 | } 79 | 80 | // get k/v pairs for keys with `foo` prefix 81 | items, err := foo.Items() 82 | 83 | for i, want := range wantItems { 84 | got := items[i] 85 | if !bytes.Equal(got.Key, want.Key) { 86 | t.Errorf("got %s, want %s", got.Key, want.Key) 87 | } 88 | if !bytes.Equal(got.Value, want.Value) { 89 | t.Errorf("got %s, want %s", got.Value, want.Value) 90 | } 91 | } 92 | 93 | // expected mapping 94 | wantMapping := map[string][]byte{ 95 | "foo/": []byte("foo"), 96 | "foo/bar/": []byte("bar"), 97 | "foo/bar/baz/": []byte("baz"), 98 | } 99 | 100 | // get mapping of k/v pairs for keys with `foo` prefix 101 | gotMapping, err := foo.ItemMapping() 102 | if err != nil { 103 | t.Error(err.Error()) 104 | } 105 | 106 | for key, want := range wantMapping { 107 | got, ok := gotMapping[key] 108 | if ok == false { 109 | t.Errorf("missing wanted key: %s", key) 110 | } 111 | if !bytes.Equal(got, want) { 112 | t.Errorf("got %s, want %s", got, want) 113 | } 114 | } 115 | 116 | if err = bx.Delete([]byte("paths")); err != nil { 117 | t.Error(err.Error()) 118 | } 119 | } 120 | -------------------------------------------------------------------------------- /rangescan.go: -------------------------------------------------------------------------------- 1 | package buckets 2 | 3 | import "github.com/boltdb/bolt" 4 | 5 | // A RangeScanner scans a bucket for keys within a given range. 6 | type RangeScanner struct { 7 | db *DB 8 | BucketName []byte 9 | Min []byte 10 | Max []byte 11 | } 12 | 13 | // Map applies `do` on each key/value pair for keys within range. 14 | func (rs *RangeScanner) Map(do func(k, v []byte) error) error { 15 | return rs.db.View(func(tx *bolt.Tx) error { 16 | c := tx.Bucket(rs.BucketName).Cursor() 17 | for k, v := c.Seek(rs.Min); isBefore(k, rs.Max); k, v = c.Next() { 18 | do(k, v) 19 | } 20 | return nil 21 | }) 22 | } 23 | 24 | // Count returns a count of the keys within the range. 25 | func (rs *RangeScanner) Count() (count int, err error) { 26 | err = rs.db.View(func(tx *bolt.Tx) error { 27 | c := tx.Bucket(rs.BucketName).Cursor() 28 | for k, _ := c.Seek(rs.Min); isBefore(k, rs.Max); k, _ = c.Next() { 29 | count++ 30 | } 31 | return nil 32 | }) 33 | if err != nil { 34 | return count, err 35 | } 36 | return count, err 37 | } 38 | 39 | // Keys returns a slice of keys within the range. 40 | func (rs *RangeScanner) Keys() (keys [][]byte, err error) { 41 | err = rs.db.View(func(tx *bolt.Tx) error { 42 | c := tx.Bucket(rs.BucketName).Cursor() 43 | for k, _ := c.Seek(rs.Min); isBefore(k, rs.Max); k, _ = c.Next() { 44 | keys = append(keys, k) 45 | } 46 | return nil 47 | }) 48 | if err != nil { 49 | return nil, err 50 | } 51 | return keys, err 52 | } 53 | 54 | // Values returns a slice of values for keys within the range. 55 | func (rs *RangeScanner) Values() (values [][]byte, err error) { 56 | err = rs.db.View(func(tx *bolt.Tx) error { 57 | c := tx.Bucket(rs.BucketName).Cursor() 58 | for k, v := c.Seek(rs.Min); isBefore(k, rs.Max); k, v = c.Next() { 59 | values = append(values, v) 60 | } 61 | return nil 62 | }) 63 | if err != nil { 64 | return nil, err 65 | } 66 | return values, err 67 | } 68 | 69 | // Items returns a slice of key/value pairs for keys within the range. 70 | // Note that the returned slice contains elements of type Item. 71 | func (rs *RangeScanner) Items() (items []Item, err error) { 72 | err = rs.db.View(func(tx *bolt.Tx) error { 73 | c := tx.Bucket(rs.BucketName).Cursor() 74 | for k, v := c.Seek(rs.Min); isBefore(k, rs.Max); k, v = c.Next() { 75 | items = append(items, Item{k, v}) 76 | } 77 | return nil 78 | }) 79 | if err != nil { 80 | return nil, err 81 | } 82 | return items, err 83 | } 84 | 85 | // ItemMapping returns a map of key/value pairs for keys within the range. 86 | // This only works with buckets whose keys are byte-sliced strings. 87 | func (rs *RangeScanner) ItemMapping() (map[string][]byte, error) { 88 | items := make(map[string][]byte) 89 | err := rs.db.View(func(tx *bolt.Tx) error { 90 | c := tx.Bucket(rs.BucketName).Cursor() 91 | for k, v := c.Seek(rs.Min); isBefore(k, rs.Max); k, v = c.Next() { 92 | items[string(k)] = v 93 | } 94 | return nil 95 | }) 96 | if err != nil { 97 | return nil, err 98 | } 99 | return items, err 100 | } 101 | -------------------------------------------------------------------------------- /rangescan_test.go: -------------------------------------------------------------------------------- 1 | package buckets_test 2 | 3 | import ( 4 | "bytes" 5 | "testing" 6 | ) 7 | 8 | // Ensures we can scan ranges. 9 | func TestRangeScanner(t *testing.T) { 10 | bx := NewTestDB() 11 | defer bx.Close() 12 | 13 | years, err := bx.New([]byte("years")) 14 | if err != nil { 15 | t.Error(err.Error()) 16 | } 17 | 18 | // items to put in `years` bucket 19 | yearItems := []struct { 20 | Key, Value []byte 21 | }{ 22 | {[]byte("1970"), []byte("70")}, 23 | {[]byte("1975"), []byte("75")}, 24 | {[]byte("1980"), []byte("80")}, 25 | {[]byte("1985"), []byte("85")}, 26 | {[]byte("1990"), []byte("90")}, // min = 1990 27 | {[]byte("1995"), []byte("95")}, // min < 1995 < max 28 | {[]byte("2000"), []byte("00")}, // max = 2000 29 | {[]byte("2005"), []byte("05")}, 30 | {[]byte("2010"), []byte("10")}, 31 | } 32 | 33 | // insert items into `years` bucket 34 | if err = years.Insert(yearItems); err != nil { 35 | t.Error(err.Error()) 36 | } 37 | 38 | // time range to scan over 39 | min := []byte("1990") 40 | max := []byte("2000") 41 | 42 | nineties := years.NewRangeScanner(min, max) 43 | 44 | // expected count of items in range 45 | wantCount := 3 46 | 47 | // expected items 48 | wantItems := []struct { 49 | Key []byte 50 | Value []byte 51 | }{ 52 | {[]byte("1990"), []byte("90")}, 53 | {[]byte("1995"), []byte("95")}, 54 | {[]byte("2000"), []byte("00")}, 55 | } 56 | 57 | count, err := nineties.Count() 58 | if err != nil { 59 | t.Error(err.Error()) 60 | } 61 | if count != wantCount { 62 | t.Errorf("got %v, want %v", count, wantCount) 63 | } 64 | 65 | keys, err := nineties.Keys() 66 | if err != nil { 67 | t.Error(err.Error()) 68 | } 69 | 70 | for i, want := range wantItems { 71 | if got := keys[i]; !bytes.Equal(got, want.Key) { 72 | t.Errorf("got %s, want %s", got, want.Key) 73 | } 74 | } 75 | 76 | values, err := nineties.Values() 77 | if err != nil { 78 | t.Error(err.Error()) 79 | } 80 | 81 | for i, want := range wantItems { 82 | if got := values[i]; !bytes.Equal(got, want.Value) { 83 | t.Errorf("got %s, want %s", got, want.Value) 84 | } 85 | } 86 | 87 | // get k/v pairs for keys within range (1995 <= year <= 2000) 88 | items, err := nineties.Items() 89 | if err != nil { 90 | t.Error(err.Error()) 91 | } 92 | 93 | for i, want := range wantItems { 94 | got := items[i] 95 | if !bytes.Equal(got.Key, want.Key) { 96 | t.Errorf("got %s, want %s", got.Key, want.Key) 97 | } 98 | if !bytes.Equal(got.Value, want.Value) { 99 | t.Errorf("got %s, want %s", got.Value, want.Value) 100 | } 101 | } 102 | 103 | // expected mapping 104 | wantMapping := map[string][]byte{ 105 | "1990": []byte("90"), 106 | "1995": []byte("95"), 107 | "2000": []byte("00"), 108 | } 109 | 110 | // get mapping of k/v pairs for keys within range (1995 <= year <= 2000) 111 | gotMapping, err := nineties.ItemMapping() 112 | if err != nil { 113 | t.Error(err.Error()) 114 | } 115 | 116 | for key, want := range wantMapping { 117 | got, ok := gotMapping[key] 118 | if ok == false { 119 | t.Errorf("missing wanted key: %s", key) 120 | } 121 | if !bytes.Equal(got, want) { 122 | t.Errorf("got %s, want %s", got, want) 123 | } 124 | } 125 | 126 | if err = bx.Delete([]byte("years")); err != nil { 127 | t.Error(err.Error()) 128 | } 129 | } 130 | -------------------------------------------------------------------------------- /scanner.go: -------------------------------------------------------------------------------- 1 | package buckets 2 | 3 | // A Scanner implements methods for scanning a subset of keys 4 | // in a bucket and retrieving data from or about those keys. 5 | type Scanner interface { 6 | // Map applies a func on each key/value pair scanned. 7 | Map(func(k, v []byte) error) error 8 | // Count returns a count of the scanned keys. 9 | Count() (int, error) 10 | // Keys returns a slice of the scanned keys. 11 | Keys() ([][]byte, error) 12 | // Values returns a slice of values from scanned keys. 13 | Values() ([][]byte, error) 14 | // Items returns a slice of k/v pairs from scanned keys. 15 | Items() ([]Item, error) 16 | // ItemMapping returns a mapping of k/v pairs from scanned keys. 17 | ItemMapping() (map[string][]byte, error) 18 | } 19 | -------------------------------------------------------------------------------- /util.go: -------------------------------------------------------------------------------- 1 | package buckets 2 | 3 | import "bytes" 4 | 5 | // isBefore checks whether `key` comes before `max`. 6 | func isBefore(key, max []byte) bool { 7 | return key != nil && bytes.Compare(key, max) <= 0 8 | } 9 | --------------------------------------------------------------------------------