├── .github
    └── workflows
    │   └── go.yml
├── LICENSE
├── README.md
├── buntdb.go
├── buntdb_test.go
├── go.mod
├── go.sum
└── logo.png


/.github/workflows/go.yml:
--------------------------------------------------------------------------------
 1 | name: Go
 2 | 
 3 | on:
 4 |   push:
 5 |     branches: [ master ]
 6 |   pull_request:
 7 |     branches: [ master ]
 8 | 
 9 | jobs:
10 | 
11 |   build:
12 |     name: Build
13 |     runs-on: ubuntu-latest
14 |     steps:
15 | 
16 |     - name: Set up Go 1.x
17 |       uses: actions/setup-go@v2
18 |       with:
19 |         go-version: ^1.13
20 | 
21 |     - name: Check out code into the Go module directory
22 |       uses: actions/checkout@v2
23 | 
24 |     - name: Get dependencies
25 |       run: |
26 |         go get -v -t -d ./...
27 |         if [ -f Gopkg.toml ]; then
28 |             curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
29 |             dep ensure
30 |         fi
31 | 
32 |     - name: Build
33 |       run: go build -v .
34 | 
35 |     - name: Test
36 |       run: go test -v .
37 | 


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
 1 | The MIT License (MIT)
 2 | 
 3 | Copyright (c) 2016 Josh Baker
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of
 6 | this software and associated documentation files (the "Software"), to deal in
 7 | the Software without restriction, including without limitation the rights to
 8 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
 9 | the Software, and to permit persons to whom the Software is furnished to do so,
10 | subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
17 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
18 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
19 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
20 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
21 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | <p align="center">
  2 | <img
  3 |     src="logo.png"
  4 |     width="307" height="150" border="0" alt="BuntDB">
  5 | <br>
  6 | <a href="https://godoc.org/github.com/tidwall/buntdb"><img src="https://img.shields.io/badge/go-documentation-blue.svg?style=flat-square" alt="Godoc"></a>
  7 | <a href="https://github.com/tidwall/buntdb/blob/master/LICENSE"><img src="https://img.shields.io/github/license/tidwall/buntdb.svg?style=flat-square" alt="LICENSE"></a>
  8 | </p>
  9 | 
 10 | BuntDB is a low-level, in-memory, key/value store in pure Go.
 11 | It persists to disk, is ACID compliant, and uses locking for multiple
 12 | readers and a single writer. It supports custom indexes and geospatial
 13 | data. It's ideal for projects that need a dependable database and favor
 14 | speed over data size.
 15 | 
 16 | Features
 17 | ========
 18 | 
 19 | - In-memory database for [fast reads and writes](#performance)
 20 | - Embeddable with a [simple API](https://godoc.org/github.com/tidwall/buntdb)
 21 | - [Spatial indexing](#spatial-indexes) for up to 20 dimensions; Useful for Geospatial data
 22 | - Index fields inside [JSON](#json-indexes) documents
 23 | - [Collate i18n Indexes](#collate-i18n-indexes) using the optional [collate package](https://github.com/tidwall/collate)
 24 | - Create [custom indexes](#custom-indexes) for any data type
 25 | - Support for [multi value indexes](#multi-value-index); Similar to a SQL multi column index
 26 | - [Built-in types](#built-in-types) that are easy to get up & running; String, Uint, Int, Float
 27 | - Flexible [iteration](#iterating) of data; ascending, descending, and ranges
 28 | - [Durable append-only file](#append-only-file) format for persistence
 29 | - Option to evict old items with an [expiration](#data-expiration) TTL
 30 | - ACID semantics with locking [transactions](#transactions) that support rollbacks
 31 | 
 32 | 
 33 | Getting Started
 34 | ===============
 35 | 
 36 | ## Installing
 37 | 
 38 | To start using BuntDB, install Go and run `go get`:
 39 | 
 40 | ```sh
 41 | $ go get -u github.com/tidwall/buntdb
 42 | ```
 43 | 
 44 | This will retrieve the library.
 45 | 
 46 | 
 47 | ## Opening a database
 48 | 
 49 | The primary object in BuntDB is a `DB`. To open or create your
 50 | database, use the `buntdb.Open()` function:
 51 | 
 52 | ```go
 53 | package main
 54 | 
 55 | import (
 56 | 	"log"
 57 | 
 58 | 	"github.com/tidwall/buntdb"
 59 | )
 60 | 
 61 | func main() {
 62 | 	// Open the data.db file. It will be created if it doesn't exist.
 63 | 	db, err := buntdb.Open("data.db")
 64 | 	if err != nil {
 65 | 		log.Fatal(err)
 66 | 	}
 67 | 	defer db.Close()
 68 | 
 69 | 	...
 70 | }
 71 | ```
 72 | 
 73 | It's also possible to open a database that does not persist to disk by using `:memory:` as the path of the file.
 74 | 
 75 | ```go
 76 | buntdb.Open(":memory:") // Open a file that does not persist to disk.
 77 | ```
 78 | 
 79 | ## Transactions
 80 | All reads and writes must be performed from inside a transaction. BuntDB can have one write transaction opened at a time, but can have many concurrent read transactions. Each transaction maintains a stable view of the database. In other words, once a transaction has begun, the data for that transaction cannot be changed by other transactions.
 81 | 
 82 | Transactions run in a function that exposes a `Tx` object, which represents the transaction state. While inside a transaction, all database operations should be performed using this object. You should never access the origin `DB` object while inside a transaction. Doing so may have side-effects, such as blocking your application.
 83 | 
 84 | When a transaction fails, it will roll back, and revert all changes that occurred to the database during that transaction. There's a single return value that you can use to close the transaction. For read/write transactions, returning an error this way will force the transaction to roll back. When a read/write transaction succeeds all changes are persisted to disk.
 85 | 
 86 | ### Read-only Transactions
 87 | A read-only transaction should be used when you don't need to make changes to the data. The advantage of a read-only transaction is that there can be many running concurrently.
 88 | 
 89 | ```go
 90 | err := db.View(func(tx *buntdb.Tx) error {
 91 | 	...
 92 | 	return nil
 93 | })
 94 | ```
 95 | 
 96 | ### Read/write Transactions
 97 | A read/write transaction is used when you need to make changes to your data. There can only be one read/write transaction running at a time. So make sure you close it as soon as you are done with it.
 98 | 
 99 | ```go
100 | err := db.Update(func(tx *buntdb.Tx) error {
101 | 	...
102 | 	return nil
103 | })
104 | ```
105 | 
106 | ## Setting and getting key/values
107 | 
108 | To set a value you must open a read/write transaction:
109 | 
110 | ```go
111 | err := db.Update(func(tx *buntdb.Tx) error {
112 | 	_, _, err := tx.Set("mykey", "myvalue", nil)
113 | 	return err
114 | })
115 | ```
116 | 
117 | 
118 | To get the value:
119 | 
120 | ```go
121 | err := db.View(func(tx *buntdb.Tx) error {
122 | 	val, err := tx.Get("mykey")
123 | 	if err != nil{
124 | 		return err
125 | 	}
126 | 	fmt.Printf("value is %s\n", val)
127 | 	return nil
128 | })
129 | ```
130 | 
131 | Getting non-existent values will cause an `ErrNotFound` error.
132 | 
133 | ### Iterating
134 | All keys/value pairs are ordered in the database by the key. To iterate over the keys:
135 | 
136 | ```go
137 | err := db.View(func(tx *buntdb.Tx) error {
138 | 	err := tx.Ascend("", func(key, value string) bool {
139 | 		fmt.Printf("key: %s, value: %s\n", key, value)
140 | 		return true // continue iteration
141 | 	})
142 | 	return err
143 | })
144 | ```
145 | 
146 | There is also `AscendGreaterOrEqual`, `AscendLessThan`, `AscendRange`, `AscendEqual`, `Descend`, `DescendLessOrEqual`, `DescendGreaterThan`, `DescendRange`, and `DescendEqual`. Please see the [documentation](https://godoc.org/github.com/tidwall/buntdb) for more information on these functions.
147 | 
148 | 
149 | ## Custom Indexes
150 | Initially all data is stored in a single [B-tree](https://en.wikipedia.org/wiki/B-tree) with each item having one key and one value. All of these items are ordered by the key. This is great for quickly getting a value from a key or [iterating](#iterating) over the keys. Feel free to peruse the [B-tree implementation](https://github.com/tidwall/btree).
151 | 
152 | You can also create custom indexes that allow for ordering and [iterating](#iterating) over values. A custom index also uses a B-tree, but it's more flexible because it allows for custom ordering.
153 | 
154 | For example, let's say you want to create an index for ordering names:
155 | 
156 | ```go
157 | db.CreateIndex("names", "*", buntdb.IndexString)
158 | ```
159 | 
160 | This will create an index named `names` which stores and sorts all values. The second parameter is a pattern that is used to filter on keys. A `*` wildcard argument means that we want to accept all keys. `IndexString` is a built-in function that performs case-insensitive ordering on the values
161 | 
162 | Now you can add various names:
163 | 
164 | ```go
165 | db.Update(func(tx *buntdb.Tx) error {
166 | 	tx.Set("user:0:name", "tom", nil)
167 | 	tx.Set("user:1:name", "Randi", nil)
168 | 	tx.Set("user:2:name", "jane", nil)
169 | 	tx.Set("user:4:name", "Janet", nil)
170 | 	tx.Set("user:5:name", "Paula", nil)
171 | 	tx.Set("user:6:name", "peter", nil)
172 | 	tx.Set("user:7:name", "Terri", nil)
173 | 	return nil
174 | })
175 | ```
176 | 
177 | Finally you can iterate over the index:
178 | 
179 | ```go
180 | db.View(func(tx *buntdb.Tx) error {
181 | 	tx.Ascend("names", func(key, val string) bool {
182 | 	fmt.Printf(buf, "%s %s\n", key, val)
183 | 		return true
184 | 	})
185 | 	return nil
186 | })
187 | ```
188 | The output should be:
189 | ```
190 | user:2:name jane
191 | user:4:name Janet
192 | user:5:name Paula
193 | user:6:name peter
194 | user:1:name Randi
195 | user:7:name Terri
196 | user:0:name tom
197 | ```
198 | 
199 | The pattern parameter can be used to filter on keys like this:
200 | 
201 | ```go
202 | db.CreateIndex("names", "user:*", buntdb.IndexString)
203 | ```
204 | 
205 | Now only items with keys that have the prefix `user:` will be added to the `names` index.
206 | 
207 | 
208 | ### Built-in types
209 | Along with `IndexString`, there is also `IndexInt`, `IndexUint`, and `IndexFloat`.
210 | These are built-in types for indexing. You can choose to use these or create your own.
211 | 
212 | So to create an index that is numerically ordered on an age key, we could use:
213 | 
214 | ```go
215 | db.CreateIndex("ages", "user:*:age", buntdb.IndexInt)
216 | ```
217 | 
218 | And then add values:
219 | 
220 | ```go
221 | db.Update(func(tx *buntdb.Tx) error {
222 | 	tx.Set("user:0:age", "35", nil)
223 | 	tx.Set("user:1:age", "49", nil)
224 | 	tx.Set("user:2:age", "13", nil)
225 | 	tx.Set("user:4:age", "63", nil)
226 | 	tx.Set("user:5:age", "8", nil)
227 | 	tx.Set("user:6:age", "3", nil)
228 | 	tx.Set("user:7:age", "16", nil)
229 | 	return nil
230 | })
231 | ```
232 | 
233 | ```go
234 | db.View(func(tx *buntdb.Tx) error {
235 | 	tx.Ascend("ages", func(key, val string) bool {
236 | 	fmt.Printf(buf, "%s %s\n", key, val)
237 | 		return true
238 | 	})
239 | 	return nil
240 | })
241 | ```
242 | 
243 | The output should be:
244 | ```
245 | user:6:age 3
246 | user:5:age 8
247 | user:2:age 13
248 | user:7:age 16
249 | user:0:age 35
250 | user:1:age 49
251 | user:4:age 63
252 | ```
253 | 
254 | ## Spatial Indexes
255 | BuntDB has support for spatial indexes by storing rectangles in an [R-tree](https://en.wikipedia.org/wiki/R-tree). An R-tree is organized in a similar manner as a [B-tree](https://en.wikipedia.org/wiki/B-tree), and both are balanced trees. But, an R-tree is special because it can operate on data that is in multiple dimensions. This is super handy for Geospatial applications.
256 | 
257 | To create a spatial index use the `CreateSpatialIndex` function:
258 | 
259 | ```go
260 | db.CreateSpatialIndex("fleet", "fleet:*:pos", buntdb.IndexRect)
261 | ```
262 | 
263 | Then `IndexRect` is a built-in function that converts rect strings to a format that the R-tree can use. It's easy to use this function out of the box, but you might find it better to create a custom one that renders from a different format, such as [Well-known text](https://en.wikipedia.org/wiki/Well-known_text) or [GeoJSON](http://geojson.org/).
264 | 
265 | To add some lon,lat points to the `fleet` index:
266 | 
267 | ```go
268 | db.Update(func(tx *buntdb.Tx) error {
269 | 	tx.Set("fleet:0:pos", "[-115.567 33.532]", nil)
270 | 	tx.Set("fleet:1:pos", "[-116.671 35.735]", nil)
271 | 	tx.Set("fleet:2:pos", "[-113.902 31.234]", nil)
272 | 	return nil
273 | })
274 | ```
275 | 
276 | And then you can run the `Intersects` function on the index:
277 | 
278 | ```go
279 | db.View(func(tx *buntdb.Tx) error {
280 | 	tx.Intersects("fleet", "[-117 30],[-112 36]", func(key, val string) bool {
281 | 		...
282 | 		return true
283 | 	})
284 | 	return nil
285 | })
286 | ```
287 | 
288 | This will get all three positions.
289 | 
290 | ### k-Nearest Neighbors
291 | 
292 | Use the `Nearby` function to get all the positions in order of nearest to farthest :
293 | 
294 | ```go
295 | db.View(func(tx *buntdb.Tx) error {
296 | 	tx.Nearby("fleet", "[-113 33]", func(key, val string, dist float64) bool {
297 | 		...
298 | 		return true
299 | 	})
300 | 	return nil
301 | })
302 | ```
303 | 
304 | ### Spatial bracket syntax
305 | 
306 | The bracket syntax `[-117 30],[-112 36]` is unique to BuntDB, and it's how the built-in rectangles are processed. But, you are not limited to this syntax. Whatever Rect function you choose to use during `CreateSpatialIndex` will be used to process the parameter, in this case it's `IndexRect`.
307 | 
308 | - **2D rectangle:** `[10 15],[20 25]`
309 | *Min XY: "10x15", Max XY: "20x25"*
310 | 
311 | - **3D rectangle:** `[10 15 12],[20 25 18]`
312 | *Min XYZ: "10x15x12", Max XYZ: "20x25x18"*
313 | 
314 | - **2D point:** `[10 15]`
315 | *XY: "10x15"*
316 | 
317 | - **LonLat point:** `[-112.2693 33.5123]`
318 | *LatLon: "33.5123 -112.2693"*
319 | 
320 | - **LonLat bounding box:** `[-112.26 33.51],[-112.18 33.67]`
321 | *Min LatLon: "33.51 -112.26", Max LatLon: "33.67 -112.18"*
322 | 
323 | **Notice:** The longitude is the Y axis and is on the left, and latitude is the X axis and is on the right.
324 | 
325 | You can also represent `Infinity` by using `-inf` and `+inf`.
326 | For example, you might have the following points (`[X Y M]` where XY is a point and M is a timestamp):
327 | ```
328 | [3 9 1]
329 | [3 8 2]
330 | [4 8 3]
331 | [4 7 4]
332 | [5 7 5]
333 | [5 6 6]
334 | ```
335 | 
336 | You can then do a search for all points with `M` between 2-4 by calling `Intersects`.
337 | 
338 | ```go
339 | tx.Intersects("points", "[-inf -inf 2],[+inf +inf 4]", func(key, val string) bool {
340 | 	println(val)
341 | 	return true
342 | })
343 | ```
344 | 
345 | Which will return:
346 | 
347 | ```
348 | [3 8 2]
349 | [4 8 3]
350 | [4 7 4]
351 | ```
352 | 
353 | ## JSON Indexes
354 | Indexes can be created on individual fields inside JSON documents. BuntDB uses [GJSON](https://github.com/tidwall/gjson) under the hood.
355 | 
356 | For example:
357 | 
358 | ```go
359 | package main
360 | 
361 | import (
362 | 	"fmt"
363 | 
364 | 	"github.com/tidwall/buntdb"
365 | )
366 | 
367 | func main() {
368 | 	db, _ := buntdb.Open(":memory:")
369 | 	db.CreateIndex("last_name", "*", buntdb.IndexJSON("name.last"))
370 | 	db.CreateIndex("age", "*", buntdb.IndexJSON("age"))
371 | 	db.Update(func(tx *buntdb.Tx) error {
372 | 		tx.Set("1", `{"name":{"first":"Tom","last":"Johnson"},"age":38}`, nil)
373 | 		tx.Set("2", `{"name":{"first":"Janet","last":"Prichard"},"age":47}`, nil)
374 | 		tx.Set("3", `{"name":{"first":"Carol","last":"Anderson"},"age":52}`, nil)
375 | 		tx.Set("4", `{"name":{"first":"Alan","last":"Cooper"},"age":28}`, nil)
376 | 		return nil
377 | 	})
378 | 	db.View(func(tx *buntdb.Tx) error {
379 | 		fmt.Println("Order by last name")
380 | 		tx.Ascend("last_name", func(key, value string) bool {
381 | 			fmt.Printf("%s: %s\n", key, value)
382 | 			return true
383 | 		})
384 | 		fmt.Println("Order by age")
385 | 		tx.Ascend("age", func(key, value string) bool {
386 | 			fmt.Printf("%s: %s\n", key, value)
387 | 			return true
388 | 		})
389 | 		fmt.Println("Order by age range 30-50")
390 | 		tx.AscendRange("age", `{"age":30}`, `{"age":50}`, func(key, value string) bool {
391 | 			fmt.Printf("%s: %s\n", key, value)
392 | 			return true
393 | 		})
394 | 		return nil
395 | 	})
396 | }
397 | ```
398 | 
399 | Results:
400 | 
401 | ```
402 | Order by last name
403 | 3: {"name":{"first":"Carol","last":"Anderson"},"age":52}
404 | 4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
405 | 1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
406 | 2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
407 | 
408 | Order by age
409 | 4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
410 | 1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
411 | 2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
412 | 3: {"name":{"first":"Carol","last":"Anderson"},"age":52}
413 | 
414 | Order by age range 30-50
415 | 1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
416 | 2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
417 | ```
418 | 
419 | ## Multi Value Index
420 | With BuntDB it's possible to join multiple values on a single index.
421 | This is similar to a [multi column index](http://dev.mysql.com/doc/refman/5.7/en/multiple-column-indexes.html) in a traditional SQL database.
422 | 
423 | In this example we are creating a multi value index on "name.last" and "age":
424 | 
425 | ```go
426 | db, _ := buntdb.Open(":memory:")
427 | db.CreateIndex("last_name_age", "*", buntdb.IndexJSON("name.last"), buntdb.IndexJSON("age"))
428 | db.Update(func(tx *buntdb.Tx) error {
429 | 	tx.Set("1", `{"name":{"first":"Tom","last":"Johnson"},"age":38}`, nil)
430 | 	tx.Set("2", `{"name":{"first":"Janet","last":"Prichard"},"age":47}`, nil)
431 | 	tx.Set("3", `{"name":{"first":"Carol","last":"Anderson"},"age":52}`, nil)
432 | 	tx.Set("4", `{"name":{"first":"Alan","last":"Cooper"},"age":28}`, nil)
433 | 	tx.Set("5", `{"name":{"first":"Sam","last":"Anderson"},"age":51}`, nil)
434 | 	tx.Set("6", `{"name":{"first":"Melinda","last":"Prichard"},"age":44}`, nil)
435 | 	return nil
436 | })
437 | db.View(func(tx *buntdb.Tx) error {
438 | 	tx.Ascend("last_name_age", func(key, value string) bool {
439 | 		fmt.Printf("%s: %s\n", key, value)
440 | 		return true
441 | 	})
442 | 	return nil
443 | })
444 | 
445 | // Output:
446 | // 5: {"name":{"first":"Sam","last":"Anderson"},"age":51}
447 | // 3: {"name":{"first":"Carol","last":"Anderson"},"age":52}
448 | // 4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
449 | // 1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
450 | // 6: {"name":{"first":"Melinda","last":"Prichard"},"age":44}
451 | // 2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
452 | ```
453 | 
454 | ## Descending Ordered Index
455 | Any index can be put in descending order by wrapping it's less function with `buntdb.Desc`.
456 | 
457 | ```go
458 | db.CreateIndex("last_name_age", "*",
459 |     buntdb.IndexJSON("name.last"),
460 |     buntdb.Desc(buntdb.IndexJSON("age")),
461 | )
462 | ```
463 | 
464 | This will create a multi value index where the last name is ascending and the age is descending.
465 | 
466 | ## Collate i18n Indexes
467 | 
468 | Using the external [collate package](https://github.com/tidwall/collate) it's possible to create
469 | indexes that are sorted by the specified language. This is similar to the [SQL COLLATE keyword](https://msdn.microsoft.com/en-us/library/ms174596.aspx) found in traditional databases.
470 | 
471 | To install:
472 | 
473 | ```
474 | go get -u github.com/tidwall/collate
475 | ```
476 | 
477 | For example:
478 | 
479 | ```go
480 | import "github.com/tidwall/collate"
481 | 
482 | // To sort case-insensitive in French.
483 | db.CreateIndex("name", "*", collate.IndexString("FRENCH_CI"))
484 | 
485 | // To specify that numbers should sort numerically ("2" < "12")
486 | // and use a comma to represent a decimal point.
487 | db.CreateIndex("amount", "*", collate.IndexString("FRENCH_NUM"))
488 | ```
489 | 
490 | There's also support for Collation on JSON indexes:
491 | 
492 | ```go
493 | db.CreateIndex("last_name", "*", collate.IndexJSON("CHINESE_CI", "name.last"))
494 | ```
495 | 
496 | Check out the [collate project](https://github.com/tidwall/collate) for more information.
497 | 
498 | ## Data Expiration
499 | Items can be automatically evicted by using the `SetOptions` object in the `Set` function to set a `TTL`.
500 | 
501 | ```go
502 | db.Update(func(tx *buntdb.Tx) error {
503 | 	tx.Set("mykey", "myval", &buntdb.SetOptions{Expires:true, TTL:time.Second})
504 | 	return nil
505 | })
506 | ```
507 | 
508 | Now `mykey` will automatically be deleted after one second. You can remove the TTL by setting the value again with the same key/value, but with the options parameter set to nil.
509 | 
510 | ## Delete while iterating
511 | BuntDB does not currently support deleting a key while in the process of iterating.
512 | As a workaround you'll need to delete keys following the completion of the iterator.
513 | 
514 | ```go
515 | var delkeys []string
516 | tx.AscendKeys("object:*", func(k, v string) bool {
517 | 	if someCondition(k) == true {
518 | 		delkeys = append(delkeys, k)
519 | 	}
520 | 	return true // continue
521 | })
522 | for _, k := range delkeys {
523 | 	if _, err = tx.Delete(k); err != nil {
524 | 		return err
525 | 	}
526 | }
527 | ```
528 | 
529 | ## Append-only File
530 | 
531 | BuntDB uses an AOF (append-only file) which is a log of all database changes that occur from operations like `Set()` and `Delete()`.
532 | 
533 | The format of this file looks like:
534 | ```
535 | set key:1 value1
536 | set key:2 value2
537 | set key:1 value3
538 | del key:2
539 | ...
540 | ```
541 | 
542 | When the database opens again, it will read back the aof file and process each command in exact order.
543 | This read process happens one time when the database opens.
544 | From there on the file is only appended.
545 | 
546 | As you may guess this log file can grow large over time.
547 | There's a background routine that automatically shrinks the log file when it gets too large.
548 | There is also a `Shrink()` function which will rewrite the aof file so that it contains only the items in the database.
549 | The shrink operation does not lock up the database so read and write transactions can continue while shrinking is in process.
550 | 
551 | ### Durability and fsync
552 | 
553 | By default BuntDB executes an `fsync` once every second on the [aof file](#append-only-file). Which simply means that there's a chance that up to one second of data might be lost. If you need higher durability then there's an optional database config setting `Config.SyncPolicy` which can be set to `Always`.
554 | 
555 | The `Config.SyncPolicy` has the following options:
556 | 
557 | - `Never` - fsync is managed by the operating system, less safe
558 | - `EverySecond` - fsync every second, fast and safer, this is the default
559 | - `Always` - fsync after every write, very durable, slower
560 | 
561 | ## Config
562 | 
563 | Here are some configuration options that can be use to change various behaviors of the database.
564 | 
565 | - **SyncPolicy** adjusts how often the data is synced to disk. This value can be Never, EverySecond, or Always. Default is EverySecond.
566 | - **AutoShrinkPercentage** is used by the background process to trigger a shrink of the aof file when the size of the file is larger than the percentage of the result of the previous shrunk file. For example, if this value is 100, and the last shrink process resulted in a 100mb file, then the new aof file must be 200mb before a shrink is triggered. Default is 100.
567 | - **AutoShrinkMinSize** defines the minimum size of the aof file before an automatic shrink can occur. Default is 32MB.
568 | - **AutoShrinkDisabled** turns off automatic background shrinking. Default is false.
569 | 
570 | To update the configuration you should call `ReadConfig` followed by `SetConfig`. For example:
571 | 
572 | ```go
573 | 
574 | var config buntdb.Config
575 | if err := db.ReadConfig(&config); err != nil{
576 | 	log.Fatal(err)
577 | }
578 | if err := db.SetConfig(config); err != nil{
579 | 	log.Fatal(err)
580 | }
581 | ```
582 | 
583 | ## Performance
584 | 
585 | How fast is BuntDB?
586 | 
587 | Here are some example [benchmarks](https://github.com/tidwall/raft-buntdb#raftstore-performance-comparison) when using BuntDB in a Raft Store implementation.
588 | 
589 | You can also run the standard Go benchmark tool from the project root directory:
590 | 
591 | ```
592 | go test --bench=.
593 | ```
594 | 
595 | ### BuntDB-Benchmark
596 | 
597 | There's a [custom utility](https://github.com/tidwall/buntdb-benchmark) that was created specifically for benchmarking BuntDB.
598 | 
599 | *These are the results from running the benchmarks on a MacBook Pro 15" 2.8 GHz Intel Core i7:*
600 | 
601 | ```
602 | $ buntdb-benchmark -q
603 | GET: 4609604.74 operations per second
604 | SET: 248500.33 operations per second
605 | ASCEND_100: 2268998.79 operations per second
606 | ASCEND_200: 1178388.14 operations per second
607 | ASCEND_400: 679134.20 operations per second
608 | ASCEND_800: 348445.55 operations per second
609 | DESCEND_100: 2313821.69 operations per second
610 | DESCEND_200: 1292738.38 operations per second
611 | DESCEND_400: 675258.76 operations per second
612 | DESCEND_800: 337481.67 operations per second
613 | SPATIAL_SET: 134824.60 operations per second
614 | SPATIAL_INTERSECTS_100: 939491.47 operations per second
615 | SPATIAL_INTERSECTS_200: 561590.40 operations per second
616 | SPATIAL_INTERSECTS_400: 306951.15 operations per second
617 | SPATIAL_INTERSECTS_800: 159673.91 operations per second
618 | ```
619 | 
620 | To install this utility:
621 | 
622 | ```
623 | go get github.com/tidwall/buntdb-benchmark
624 | ```
625 | 
626 | 
627 | 
628 | ## Contact
629 | Josh Baker [@tidwall](http://twitter.com/tidwall)
630 | 
631 | ## License
632 | 
633 | BuntDB source code is available under the MIT [License](/LICENSE).
634 | 


--------------------------------------------------------------------------------
/buntdb.go:
--------------------------------------------------------------------------------
   1 | // Package buntdb implements a low-level in-memory key/value store in pure Go.
   2 | // It persists to disk, is ACID compliant, and uses locking for multiple
   3 | // readers and a single writer. Bunt is ideal for projects that need a
   4 | // dependable database, and favor speed over data size.
   5 | package buntdb
   6 | 
   7 | import (
   8 | 	"bufio"
   9 | 	"errors"
  10 | 	"fmt"
  11 | 	"io"
  12 | 	"os"
  13 | 	"runtime"
  14 | 	"sort"
  15 | 	"strconv"
  16 | 	"strings"
  17 | 	"sync"
  18 | 	"time"
  19 | 
  20 | 	"github.com/tidwall/btree"
  21 | 	"github.com/tidwall/gjson"
  22 | 	"github.com/tidwall/grect"
  23 | 	"github.com/tidwall/match"
  24 | 	"github.com/tidwall/rtred"
  25 | )
  26 | 
  27 | var (
  28 | 	// ErrTxNotWritable is returned when performing a write operation on a
  29 | 	// read-only transaction.
  30 | 	ErrTxNotWritable = errors.New("tx not writable")
  31 | 
  32 | 	// ErrTxClosed is returned when committing or rolling back a transaction
  33 | 	// that has already been committed or rolled back.
  34 | 	ErrTxClosed = errors.New("tx closed")
  35 | 
  36 | 	// ErrNotFound is returned when an item or index is not in the database.
  37 | 	ErrNotFound = errors.New("not found")
  38 | 
  39 | 	// ErrInvalid is returned when the database file is an invalid format.
  40 | 	ErrInvalid = errors.New("invalid database")
  41 | 
  42 | 	// ErrDatabaseClosed is returned when the database is closed.
  43 | 	ErrDatabaseClosed = errors.New("database closed")
  44 | 
  45 | 	// ErrIndexExists is returned when an index already exists in the database.
  46 | 	ErrIndexExists = errors.New("index exists")
  47 | 
  48 | 	// ErrInvalidOperation is returned when an operation cannot be completed.
  49 | 	ErrInvalidOperation = errors.New("invalid operation")
  50 | 
  51 | 	// ErrInvalidSyncPolicy is returned for an invalid SyncPolicy value.
  52 | 	ErrInvalidSyncPolicy = errors.New("invalid sync policy")
  53 | 
  54 | 	// ErrShrinkInProcess is returned when a shrink operation is in-process.
  55 | 	ErrShrinkInProcess = errors.New("shrink is in-process")
  56 | 
  57 | 	// ErrPersistenceActive is returned when post-loading data from an database
  58 | 	// not opened with Open(":memory:").
  59 | 	ErrPersistenceActive = errors.New("persistence active")
  60 | 
  61 | 	// ErrTxIterating is returned when Set or Delete are called while iterating.
  62 | 	ErrTxIterating = errors.New("tx is iterating")
  63 | )
  64 | 
  65 | const useAbsEx = true
  66 | 
  67 | // DB represents a collection of key-value pairs that persist on disk.
  68 | // Transactions are used for all forms of data access to the DB.
  69 | type DB struct {
  70 | 	mu        sync.RWMutex      // the gatekeeper for all fields
  71 | 	file      *os.File          // the underlying file
  72 | 	buf       []byte            // a buffer to write to
  73 | 	keys      *btree.BTree      // a tree of all item ordered by key
  74 | 	exps      *btree.BTree      // a tree of items ordered by expiration
  75 | 	idxs      map[string]*index // the index trees.
  76 | 	insIdxs   []*index          // a reuse buffer for gathering indexes
  77 | 	flushes   int               // a count of the number of disk flushes
  78 | 	closed    bool              // set when the database has been closed
  79 | 	config    Config            // the database configuration
  80 | 	persist   bool              // do we write to disk
  81 | 	shrinking bool              // when an aof shrink is in-process.
  82 | 	lastaofsz int               // the size of the last shrink aof size
  83 | }
  84 | 
  85 | // SyncPolicy represents how often data is synced to disk.
  86 | type SyncPolicy int
  87 | 
  88 | const (
  89 | 	// Never is used to disable syncing data to disk.
  90 | 	// The faster and less safe method.
  91 | 	Never SyncPolicy = 0
  92 | 	// EverySecond is used to sync data to disk every second.
  93 | 	// It's pretty fast and you can lose 1 second of data if there
  94 | 	// is a disaster.
  95 | 	// This is the recommended setting.
  96 | 	EverySecond = 1
  97 | 	// Always is used to sync data after every write to disk.
  98 | 	// Slow. Very safe.
  99 | 	Always = 2
 100 | )
 101 | 
 102 | // Config represents database configuration options. These
 103 | // options are used to change various behaviors of the database.
 104 | type Config struct {
 105 | 	// SyncPolicy adjusts how often the data is synced to disk.
 106 | 	// This value can be Never, EverySecond, or Always.
 107 | 	// The default is EverySecond.
 108 | 	SyncPolicy SyncPolicy
 109 | 
 110 | 	// AutoShrinkPercentage is used by the background process to trigger
 111 | 	// a shrink of the aof file when the size of the file is larger than the
 112 | 	// percentage of the result of the previous shrunk file.
 113 | 	// For example, if this value is 100, and the last shrink process
 114 | 	// resulted in a 100mb file, then the new aof file must be 200mb before
 115 | 	// a shrink is triggered.
 116 | 	AutoShrinkPercentage int
 117 | 
 118 | 	// AutoShrinkMinSize defines the minimum size of the aof file before
 119 | 	// an automatic shrink can occur.
 120 | 	AutoShrinkMinSize int
 121 | 
 122 | 	// AutoShrinkDisabled turns off automatic background shrinking
 123 | 	AutoShrinkDisabled bool
 124 | 
 125 | 	// OnExpired is used to custom handle the deletion option when a key
 126 | 	// has been expired.
 127 | 	OnExpired func(keys []string)
 128 | 
 129 | 	// OnExpiredSync will be called inside the same transaction that is
 130 | 	// performing the deletion of expired items. If OnExpired is present then
 131 | 	// this callback will not be called. If this callback is present, then the
 132 | 	// deletion of the timeed-out item is the explicit responsibility of this
 133 | 	// callback.
 134 | 	OnExpiredSync func(key, value string, tx *Tx) error
 135 | }
 136 | 
 137 | // exctx is a simple b-tree context for ordering by expiration.
 138 | type exctx struct {
 139 | 	db *DB
 140 | }
 141 | 
 142 | // Open opens a database at the provided path.
 143 | // If the file does not exist then it will be created automatically.
 144 | func Open(path string) (*DB, error) {
 145 | 	db := &DB{}
 146 | 	// initialize trees and indexes
 147 | 	db.keys = btreeNew(lessCtx(nil))
 148 | 	db.exps = btreeNew(lessCtx(&exctx{db}))
 149 | 	db.idxs = make(map[string]*index)
 150 | 	// initialize default configuration
 151 | 	db.config = Config{
 152 | 		SyncPolicy:           EverySecond,
 153 | 		AutoShrinkPercentage: 100,
 154 | 		AutoShrinkMinSize:    32 * 1024 * 1024,
 155 | 	}
 156 | 	// turn off persistence for pure in-memory
 157 | 	db.persist = path != ":memory:"
 158 | 	if db.persist {
 159 | 		var err error
 160 | 		// hardcoding 0666 as the default mode.
 161 | 		db.file, err = os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0666)
 162 | 		if err != nil {
 163 | 			return nil, err
 164 | 		}
 165 | 		// load the database from disk
 166 | 		if err := db.load(); err != nil {
 167 | 			// close on error, ignore close error
 168 | 			_ = db.file.Close()
 169 | 			return nil, err
 170 | 		}
 171 | 	}
 172 | 	// start the background manager.
 173 | 	go db.backgroundManager()
 174 | 	return db, nil
 175 | }
 176 | 
 177 | // Close releases all database resources.
 178 | // All transactions must be closed before closing the database.
 179 | func (db *DB) Close() error {
 180 | 	db.mu.Lock()
 181 | 	defer db.mu.Unlock()
 182 | 	if db.closed {
 183 | 		return ErrDatabaseClosed
 184 | 	}
 185 | 	db.closed = true
 186 | 	if db.persist {
 187 | 		db.file.Sync() // do a sync but ignore the error
 188 | 		if err := db.file.Close(); err != nil {
 189 | 			return err
 190 | 		}
 191 | 	}
 192 | 	// Let's release all references to nil. This will help both with debugging
 193 | 	// late usage panics and it provides a hint to the garbage collector
 194 | 	db.keys, db.exps, db.idxs, db.file = nil, nil, nil, nil
 195 | 	return nil
 196 | }
 197 | 
 198 | // Save writes a snapshot of the database to a writer. This operation blocks all
 199 | // writes, but not reads. This can be used for snapshots and backups for pure
 200 | // in-memory databases using the ":memory:". Database that persist to disk
 201 | // can be snapshotted by simply copying the database file.
 202 | func (db *DB) Save(wr io.Writer) error {
 203 | 	var err error
 204 | 	db.mu.RLock()
 205 | 	defer db.mu.RUnlock()
 206 | 	// use a buffered writer and flush every 4MB
 207 | 	var buf []byte
 208 | 	now := time.Now()
 209 | 	// iterated through every item in the database and write to the buffer
 210 | 	btreeAscend(db.keys, func(item interface{}) bool {
 211 | 		dbi := item.(*dbItem)
 212 | 		buf = dbi.writeSetTo(buf, now)
 213 | 		if len(buf) > 1024*1024*4 {
 214 | 			// flush when buffer is over 4MB
 215 | 			_, err = wr.Write(buf)
 216 | 			if err != nil {
 217 | 				return false
 218 | 			}
 219 | 			buf = buf[:0]
 220 | 		}
 221 | 		return true
 222 | 	})
 223 | 	if err != nil {
 224 | 		return err
 225 | 	}
 226 | 	// one final flush
 227 | 	if len(buf) > 0 {
 228 | 		_, err = wr.Write(buf)
 229 | 		if err != nil {
 230 | 			return err
 231 | 		}
 232 | 	}
 233 | 	return nil
 234 | }
 235 | 
 236 | // Load loads commands from reader. This operation blocks all reads and writes.
 237 | // Note that this can only work for fully in-memory databases opened with
 238 | // Open(":memory:").
 239 | func (db *DB) Load(rd io.Reader) error {
 240 | 	db.mu.Lock()
 241 | 	defer db.mu.Unlock()
 242 | 	if db.persist {
 243 | 		// cannot load into databases that persist to disk
 244 | 		return ErrPersistenceActive
 245 | 	}
 246 | 	_, err := db.readLoad(rd, time.Now())
 247 | 	return err
 248 | }
 249 | 
 250 | // index represents a b-tree or r-tree index and also acts as the
 251 | // b-tree/r-tree context for itself.
 252 | type index struct {
 253 | 	btr     *btree.BTree                           // contains the items
 254 | 	rtr     *rtred.RTree                           // contains the items
 255 | 	name    string                                 // name of the index
 256 | 	pattern string                                 // a required key pattern
 257 | 	less    func(a, b string) bool                 // less comparison function
 258 | 	rect    func(item string) (min, max []float64) // rect from string function
 259 | 	db      *DB                                    // the origin database
 260 | 	opts    IndexOptions                           // index options
 261 | }
 262 | 
 263 | // match matches the pattern to the key
 264 | func (idx *index) match(key string) bool {
 265 | 	if idx.pattern == "*" {
 266 | 		return true
 267 | 	}
 268 | 	if idx.opts.CaseInsensitiveKeyMatching {
 269 | 		for i := 0; i < len(key); i++ {
 270 | 			if key[i] >= 'A' && key[i] <= 'Z' {
 271 | 				key = strings.ToLower(key)
 272 | 				break
 273 | 			}
 274 | 		}
 275 | 	}
 276 | 	return match.Match(key, idx.pattern)
 277 | }
 278 | 
 279 | // clearCopy creates a copy of the index, but with an empty dataset.
 280 | func (idx *index) clearCopy() *index {
 281 | 	// copy the index meta information
 282 | 	nidx := &index{
 283 | 		name:    idx.name,
 284 | 		pattern: idx.pattern,
 285 | 		db:      idx.db,
 286 | 		less:    idx.less,
 287 | 		rect:    idx.rect,
 288 | 		opts:    idx.opts,
 289 | 	}
 290 | 	// initialize with empty trees
 291 | 	if nidx.less != nil {
 292 | 		nidx.btr = btreeNew(lessCtx(nidx))
 293 | 	}
 294 | 	if nidx.rect != nil {
 295 | 		nidx.rtr = rtred.New(nidx)
 296 | 	}
 297 | 	return nidx
 298 | }
 299 | 
 300 | // rebuild rebuilds the index
 301 | func (idx *index) rebuild() {
 302 | 	// initialize trees
 303 | 	if idx.less != nil {
 304 | 		idx.btr = btreeNew(lessCtx(idx))
 305 | 	}
 306 | 	if idx.rect != nil {
 307 | 		idx.rtr = rtred.New(idx)
 308 | 	}
 309 | 	// iterate through all keys and fill the index
 310 | 	btreeAscend(idx.db.keys, func(item interface{}) bool {
 311 | 		dbi := item.(*dbItem)
 312 | 		if !idx.match(dbi.key) {
 313 | 			// does not match the pattern, continue
 314 | 			return true
 315 | 		}
 316 | 		if idx.less != nil {
 317 | 			idx.btr.Set(dbi)
 318 | 		}
 319 | 		if idx.rect != nil {
 320 | 			idx.rtr.Insert(dbi)
 321 | 		}
 322 | 		return true
 323 | 	})
 324 | }
 325 | 
 326 | // CreateIndex builds a new index and populates it with items.
 327 | // The items are ordered in an b-tree and can be retrieved using the
 328 | // Ascend* and Descend* methods.
 329 | // An error will occur if an index with the same name already exists.
 330 | //
 331 | // When a pattern is provided, the index will be populated with
 332 | // keys that match the specified pattern. This is a very simple pattern
 333 | // match where '*' matches on any number characters and '?' matches on
 334 | // any one character.
 335 | // The less function compares if string 'a' is less than string 'b'.
 336 | // It allows for indexes to create custom ordering. It's possible
 337 | // that the strings may be textual or binary. It's up to the provided
 338 | // less function to handle the content format and comparison.
 339 | // There are some default less function that can be used such as
 340 | // IndexString, IndexBinary, etc.
 341 | func (db *DB) CreateIndex(name, pattern string,
 342 | 	less ...func(a, b string) bool) error {
 343 | 	return db.Update(func(tx *Tx) error {
 344 | 		return tx.CreateIndex(name, pattern, less...)
 345 | 	})
 346 | }
 347 | 
 348 | // ReplaceIndex builds a new index and populates it with items.
 349 | // The items are ordered in an b-tree and can be retrieved using the
 350 | // Ascend* and Descend* methods.
 351 | // If a previous index with the same name exists, that index will be deleted.
 352 | func (db *DB) ReplaceIndex(name, pattern string,
 353 | 	less ...func(a, b string) bool) error {
 354 | 	return db.Update(func(tx *Tx) error {
 355 | 		err := tx.CreateIndex(name, pattern, less...)
 356 | 		if err != nil {
 357 | 			if err == ErrIndexExists {
 358 | 				err := tx.DropIndex(name)
 359 | 				if err != nil {
 360 | 					return err
 361 | 				}
 362 | 				return tx.CreateIndex(name, pattern, less...)
 363 | 			}
 364 | 			return err
 365 | 		}
 366 | 		return nil
 367 | 	})
 368 | }
 369 | 
 370 | // CreateSpatialIndex builds a new index and populates it with items.
 371 | // The items are organized in an r-tree and can be retrieved using the
 372 | // Intersects method.
 373 | // An error will occur if an index with the same name already exists.
 374 | //
 375 | // The rect function converts a string to a rectangle. The rectangle is
 376 | // represented by two arrays, min and max. Both arrays may have a length
 377 | // between 1 and 20, and both arrays must match in length. A length of 1 is a
 378 | // one dimensional rectangle, and a length of 4 is a four dimension rectangle.
 379 | // There is support for up to 20 dimensions.
 380 | // The values of min must be less than the values of max at the same dimension.
 381 | // Thus min[0] must be less-than-or-equal-to max[0].
 382 | // The IndexRect is a default function that can be used for the rect
 383 | // parameter.
 384 | func (db *DB) CreateSpatialIndex(name, pattern string,
 385 | 	rect func(item string) (min, max []float64)) error {
 386 | 	return db.Update(func(tx *Tx) error {
 387 | 		return tx.CreateSpatialIndex(name, pattern, rect)
 388 | 	})
 389 | }
 390 | 
 391 | // ReplaceSpatialIndex builds a new index and populates it with items.
 392 | // The items are organized in an r-tree and can be retrieved using the
 393 | // Intersects method.
 394 | // If a previous index with the same name exists, that index will be deleted.
 395 | func (db *DB) ReplaceSpatialIndex(name, pattern string,
 396 | 	rect func(item string) (min, max []float64)) error {
 397 | 	return db.Update(func(tx *Tx) error {
 398 | 		err := tx.CreateSpatialIndex(name, pattern, rect)
 399 | 		if err != nil {
 400 | 			if err == ErrIndexExists {
 401 | 				err := tx.DropIndex(name)
 402 | 				if err != nil {
 403 | 					return err
 404 | 				}
 405 | 				return tx.CreateSpatialIndex(name, pattern, rect)
 406 | 			}
 407 | 			return err
 408 | 		}
 409 | 		return nil
 410 | 	})
 411 | }
 412 | 
 413 | // DropIndex removes an index.
 414 | func (db *DB) DropIndex(name string) error {
 415 | 	return db.Update(func(tx *Tx) error {
 416 | 		return tx.DropIndex(name)
 417 | 	})
 418 | }
 419 | 
 420 | // Indexes returns a list of index names.
 421 | func (db *DB) Indexes() ([]string, error) {
 422 | 	var names []string
 423 | 	var err = db.View(func(tx *Tx) error {
 424 | 		var err error
 425 | 		names, err = tx.Indexes()
 426 | 		return err
 427 | 	})
 428 | 	return names, err
 429 | }
 430 | 
 431 | // ReadConfig returns the database configuration.
 432 | func (db *DB) ReadConfig(config *Config) error {
 433 | 	db.mu.RLock()
 434 | 	defer db.mu.RUnlock()
 435 | 	if db.closed {
 436 | 		return ErrDatabaseClosed
 437 | 	}
 438 | 	*config = db.config
 439 | 	return nil
 440 | }
 441 | 
 442 | // SetConfig updates the database configuration.
 443 | func (db *DB) SetConfig(config Config) error {
 444 | 	db.mu.Lock()
 445 | 	defer db.mu.Unlock()
 446 | 	if db.closed {
 447 | 		return ErrDatabaseClosed
 448 | 	}
 449 | 	switch config.SyncPolicy {
 450 | 	default:
 451 | 		return ErrInvalidSyncPolicy
 452 | 	case Never, EverySecond, Always:
 453 | 	}
 454 | 	db.config = config
 455 | 	return nil
 456 | }
 457 | 
 458 | // insertIntoDatabase performs inserts an item in to the database and updates
 459 | // all indexes. If a previous item with the same key already exists, that item
 460 | // will be replaced with the new one, and return the previous item.
 461 | func (db *DB) insertIntoDatabase(item *dbItem) *dbItem {
 462 | 	var pdbi *dbItem
 463 | 	// Generate a list of indexes that this item will be inserted in to.
 464 | 	idxs := db.insIdxs
 465 | 	for _, idx := range db.idxs {
 466 | 		if idx.match(item.key) {
 467 | 			idxs = append(idxs, idx)
 468 | 		}
 469 | 	}
 470 | 	prev := db.keys.Set(item)
 471 | 	if prev != nil {
 472 | 		// A previous item was removed from the keys tree. Let's
 473 | 		// fully delete this item from all indexes.
 474 | 		pdbi = prev.(*dbItem)
 475 | 		if pdbi.opts != nil && pdbi.opts.ex {
 476 | 			// Remove it from the expires tree.
 477 | 			db.exps.Delete(pdbi)
 478 | 		}
 479 | 		for _, idx := range idxs {
 480 | 			if idx.btr != nil {
 481 | 				// Remove it from the btree index.
 482 | 				idx.btr.Delete(pdbi)
 483 | 			}
 484 | 			if idx.rtr != nil {
 485 | 				// Remove it from the rtree index.
 486 | 				idx.rtr.Remove(pdbi)
 487 | 			}
 488 | 		}
 489 | 	}
 490 | 	if item.opts != nil && item.opts.ex {
 491 | 		// The new item has eviction options. Add it to the
 492 | 		// expires tree
 493 | 		db.exps.Set(item)
 494 | 	}
 495 | 	for i, idx := range idxs {
 496 | 		if idx.btr != nil {
 497 | 			// Add new item to btree index.
 498 | 			idx.btr.Set(item)
 499 | 		}
 500 | 		if idx.rtr != nil {
 501 | 			// Add new item to rtree index.
 502 | 			idx.rtr.Insert(item)
 503 | 		}
 504 | 		// clear the index
 505 | 		idxs[i] = nil
 506 | 	}
 507 | 	// reuse the index list slice
 508 | 	db.insIdxs = idxs[:0]
 509 | 	// we must return the previous item to the caller.
 510 | 	return pdbi
 511 | }
 512 | 
 513 | // deleteFromDatabase removes and item from the database and indexes. The input
 514 | // item must only have the key field specified thus "&dbItem{key: key}" is all
 515 | // that is needed to fully remove the item with the matching key. If an item
 516 | // with the matching key was found in the database, it will be removed and
 517 | // returned to the caller. A nil return value means that the item was not
 518 | // found in the database
 519 | func (db *DB) deleteFromDatabase(item *dbItem) *dbItem {
 520 | 	var pdbi *dbItem
 521 | 	prev := db.keys.Delete(item)
 522 | 	if prev != nil {
 523 | 		pdbi = prev.(*dbItem)
 524 | 		if pdbi.opts != nil && pdbi.opts.ex {
 525 | 			// Remove it from the exipres tree.
 526 | 			db.exps.Delete(pdbi)
 527 | 		}
 528 | 		for _, idx := range db.idxs {
 529 | 			if !idx.match(pdbi.key) {
 530 | 				continue
 531 | 			}
 532 | 			if idx.btr != nil {
 533 | 				// Remove it from the btree index.
 534 | 				idx.btr.Delete(pdbi)
 535 | 			}
 536 | 			if idx.rtr != nil {
 537 | 				// Remove it from the rtree index.
 538 | 				idx.rtr.Remove(pdbi)
 539 | 			}
 540 | 		}
 541 | 	}
 542 | 	return pdbi
 543 | }
 544 | 
 545 | // backgroundManager runs continuously in the background and performs various
 546 | // operations such as removing expired items and syncing to disk.
 547 | func (db *DB) backgroundManager() {
 548 | 	flushes := 0
 549 | 	t := time.NewTicker(time.Second)
 550 | 	defer t.Stop()
 551 | 	for range t.C {
 552 | 		var shrink bool
 553 | 		// Open a standard view. This will take a full lock of the
 554 | 		// database thus allowing for access to anything we need.
 555 | 		var onExpired func([]string)
 556 | 		var expired []*dbItem
 557 | 		var onExpiredSync func(key, value string, tx *Tx) error
 558 | 		err := db.Update(func(tx *Tx) error {
 559 | 			onExpired = db.config.OnExpired
 560 | 			if onExpired == nil {
 561 | 				onExpiredSync = db.config.OnExpiredSync
 562 | 			}
 563 | 			if db.persist && !db.config.AutoShrinkDisabled {
 564 | 				pos, err := db.file.Seek(0, 1)
 565 | 				if err != nil {
 566 | 					return err
 567 | 				}
 568 | 				aofsz := int(pos)
 569 | 				if aofsz > db.config.AutoShrinkMinSize {
 570 | 					prc := float64(db.config.AutoShrinkPercentage) / 100.0
 571 | 					shrink = aofsz > db.lastaofsz+int(float64(db.lastaofsz)*prc)
 572 | 				}
 573 | 			}
 574 | 			// produce a list of expired items that need removing
 575 | 			btreeAscendLessThan(db.exps, &dbItem{
 576 | 				opts: &dbItemOpts{ex: true, exat: time.Now()},
 577 | 			}, func(item interface{}) bool {
 578 | 				expired = append(expired, item.(*dbItem))
 579 | 				return true
 580 | 			})
 581 | 			if onExpired == nil && onExpiredSync == nil {
 582 | 				for _, itm := range expired {
 583 | 					if _, err := tx.Delete(itm.key); err != nil {
 584 | 						// it's ok to get a "not found" because the
 585 | 						// 'Delete' method reports "not found" for
 586 | 						// expired items.
 587 | 						if err != ErrNotFound {
 588 | 							return err
 589 | 						}
 590 | 					}
 591 | 				}
 592 | 			} else if onExpiredSync != nil {
 593 | 				for _, itm := range expired {
 594 | 					if err := onExpiredSync(itm.key, itm.val, tx); err != nil {
 595 | 						return err
 596 | 					}
 597 | 				}
 598 | 			}
 599 | 			return nil
 600 | 		})
 601 | 		if err == ErrDatabaseClosed {
 602 | 			break
 603 | 		}
 604 | 
 605 | 		// send expired event, if needed
 606 | 		if onExpired != nil && len(expired) > 0 {
 607 | 			keys := make([]string, 0, 32)
 608 | 			for _, itm := range expired {
 609 | 				keys = append(keys, itm.key)
 610 | 			}
 611 | 			onExpired(keys)
 612 | 		}
 613 | 
 614 | 		// execute a disk sync, if needed
 615 | 		func() {
 616 | 			db.mu.Lock()
 617 | 			defer db.mu.Unlock()
 618 | 			if db.persist && db.config.SyncPolicy == EverySecond &&
 619 | 				flushes != db.flushes {
 620 | 				_ = db.file.Sync()
 621 | 				flushes = db.flushes
 622 | 			}
 623 | 		}()
 624 | 		if shrink {
 625 | 			if err = db.Shrink(); err != nil {
 626 | 				if err == ErrDatabaseClosed {
 627 | 					break
 628 | 				}
 629 | 			}
 630 | 		}
 631 | 	}
 632 | }
 633 | 
 634 | // Shrink will make the database file smaller by removing redundant
 635 | // log entries. This operation does not block the database.
 636 | func (db *DB) Shrink() error {
 637 | 	db.mu.Lock()
 638 | 	if db.closed {
 639 | 		db.mu.Unlock()
 640 | 		return ErrDatabaseClosed
 641 | 	}
 642 | 	if !db.persist {
 643 | 		// The database was opened with ":memory:" as the path.
 644 | 		// There is no persistence, and no need to do anything here.
 645 | 		db.mu.Unlock()
 646 | 		return nil
 647 | 	}
 648 | 	if db.shrinking {
 649 | 		// The database is already in the process of shrinking.
 650 | 		db.mu.Unlock()
 651 | 		return ErrShrinkInProcess
 652 | 	}
 653 | 	db.shrinking = true
 654 | 	defer func() {
 655 | 		db.mu.Lock()
 656 | 		db.shrinking = false
 657 | 		db.mu.Unlock()
 658 | 	}()
 659 | 	fname := db.file.Name()
 660 | 	tmpname := fname + ".tmp"
 661 | 	// the endpos is used to return to the end of the file when we are
 662 | 	// finished writing all of the current items.
 663 | 	endpos, err := db.file.Seek(0, 2)
 664 | 	if err != nil {
 665 | 		return err
 666 | 	}
 667 | 	db.mu.Unlock()
 668 | 	time.Sleep(time.Second / 4) // wait just a bit before starting
 669 | 	f, err := os.Create(tmpname)
 670 | 	if err != nil {
 671 | 		return err
 672 | 	}
 673 | 	defer func() {
 674 | 		_ = f.Close()
 675 | 		_ = os.RemoveAll(tmpname)
 676 | 	}()
 677 | 
 678 | 	// we are going to read items in as chunks as to not hold up the database
 679 | 	// for too long.
 680 | 	var buf []byte
 681 | 	pivot := ""
 682 | 	done := false
 683 | 	for !done {
 684 | 		err := func() error {
 685 | 			db.mu.RLock()
 686 | 			defer db.mu.RUnlock()
 687 | 			if db.closed {
 688 | 				return ErrDatabaseClosed
 689 | 			}
 690 | 			done = true
 691 | 			var n int
 692 | 			now := time.Now()
 693 | 			btreeAscendGreaterOrEqual(db.keys, &dbItem{key: pivot},
 694 | 				func(item interface{}) bool {
 695 | 					dbi := item.(*dbItem)
 696 | 					// 1000 items or 64MB buffer
 697 | 					if n > 1000 || len(buf) > 64*1024*1024 {
 698 | 						pivot = dbi.key
 699 | 						done = false
 700 | 						return false
 701 | 					}
 702 | 					buf = dbi.writeSetTo(buf, now)
 703 | 					n++
 704 | 					return true
 705 | 				},
 706 | 			)
 707 | 			if len(buf) > 0 {
 708 | 				if _, err := f.Write(buf); err != nil {
 709 | 					return err
 710 | 				}
 711 | 				buf = buf[:0]
 712 | 			}
 713 | 			return nil
 714 | 		}()
 715 | 		if err != nil {
 716 | 			return err
 717 | 		}
 718 | 	}
 719 | 	// We reached this far so all of the items have been written to a new tmp
 720 | 	// There's some more work to do by appending the new line from the aof
 721 | 	// to the tmp file and finally swap the files out.
 722 | 	return func() error {
 723 | 		// We're wrapping this in a function to get the benefit of a defered
 724 | 		// lock/unlock.
 725 | 		db.mu.Lock()
 726 | 		defer db.mu.Unlock()
 727 | 		if db.closed {
 728 | 			return ErrDatabaseClosed
 729 | 		}
 730 | 		// We are going to open a new version of the aof file so that we do
 731 | 		// not change the seek position of the previous. This may cause a
 732 | 		// problem in the future if we choose to use syscall file locking.
 733 | 		aof, err := os.Open(fname)
 734 | 		if err != nil {
 735 | 			return err
 736 | 		}
 737 | 		defer func() { _ = aof.Close() }()
 738 | 		if _, err := aof.Seek(endpos, 0); err != nil {
 739 | 			return err
 740 | 		}
 741 | 		// Just copy all of the new commands that have occurred since we
 742 | 		// started the shrink process.
 743 | 		if _, err := io.Copy(f, aof); err != nil {
 744 | 			return err
 745 | 		}
 746 | 		// Close all files
 747 | 		if err := aof.Close(); err != nil {
 748 | 			return err
 749 | 		}
 750 | 		if err := f.Close(); err != nil {
 751 | 			return err
 752 | 		}
 753 | 		if err := db.file.Close(); err != nil {
 754 | 			return err
 755 | 		}
 756 | 		// Any failures below here are really bad. So just panic.
 757 | 		if err := renameFile(tmpname, fname); err != nil {
 758 | 			panicErr(err)
 759 | 		}
 760 | 		db.file, err = os.OpenFile(fname, os.O_CREATE|os.O_RDWR, 0666)
 761 | 		if err != nil {
 762 | 			panicErr(err)
 763 | 		}
 764 | 		pos, err := db.file.Seek(0, 2)
 765 | 		if err != nil {
 766 | 			return err
 767 | 		}
 768 | 		db.lastaofsz = int(pos)
 769 | 		return nil
 770 | 	}()
 771 | }
 772 | 
 773 | func panicErr(err error) error {
 774 | 	panic(fmt.Errorf("buntdb: %w", err))
 775 | }
 776 | 
 777 | func renameFile(src, dest string) error {
 778 | 	var err error
 779 | 	if err = os.Rename(src, dest); err != nil {
 780 | 		if runtime.GOOS == "windows" {
 781 | 			if err = os.Remove(dest); err == nil {
 782 | 				err = os.Rename(src, dest)
 783 | 			}
 784 | 		}
 785 | 	}
 786 | 	return err
 787 | }
 788 | 
 789 | // readLoad reads from the reader and loads commands into the database.
 790 | // modTime is the modified time of the reader, should be no greater than
 791 | // the current time.Now().
 792 | // Returns the number of bytes of the last command read and the error if any.
 793 | func (db *DB) readLoad(rd io.Reader, modTime time.Time) (n int64, err error) {
 794 | 	defer func() {
 795 | 		if err == io.EOF {
 796 | 			err = io.ErrUnexpectedEOF
 797 | 		}
 798 | 	}()
 799 | 	totalSize := int64(0)
 800 | 	data := make([]byte, 4096)
 801 | 	parts := make([]string, 0, 8)
 802 | 	r := bufio.NewReader(rd)
 803 | 	for {
 804 | 		// peek at the first byte. If it's a 'nul' control character then
 805 | 		// ignore it and move to the next byte.
 806 | 		c, err := r.ReadByte()
 807 | 		if err != nil {
 808 | 			if err == io.EOF {
 809 | 				err = nil
 810 | 			}
 811 | 			return totalSize, err
 812 | 		}
 813 | 		if c == 0 {
 814 | 			// ignore nul control characters
 815 | 			n += 1
 816 | 			continue
 817 | 		}
 818 | 		if err := r.UnreadByte(); err != nil {
 819 | 			return totalSize, err
 820 | 		}
 821 | 
 822 | 		// read a single command.
 823 | 		// first we should read the number of parts that the of the command
 824 | 		cmdByteSize := int64(0)
 825 | 		line, err := r.ReadBytes('\n')
 826 | 		if err != nil {
 827 | 			return totalSize, err
 828 | 		}
 829 | 		if line[0] != '*' {
 830 | 			return totalSize, ErrInvalid
 831 | 		}
 832 | 		cmdByteSize += int64(len(line))
 833 | 
 834 | 		// convert the string number to and int
 835 | 		var n int
 836 | 		if len(line) == 4 && line[len(line)-2] == '\r' {
 837 | 			if line[1] < '0' || line[1] > '9' {
 838 | 				return totalSize, ErrInvalid
 839 | 			}
 840 | 			n = int(line[1] - '0')
 841 | 		} else {
 842 | 			if len(line) < 5 || line[len(line)-2] != '\r' {
 843 | 				return totalSize, ErrInvalid
 844 | 			}
 845 | 			for i := 1; i < len(line)-2; i++ {
 846 | 				if line[i] < '0' || line[i] > '9' {
 847 | 					return totalSize, ErrInvalid
 848 | 				}
 849 | 				n = n*10 + int(line[i]-'0')
 850 | 			}
 851 | 		}
 852 | 		// read each part of the command.
 853 | 		parts = parts[:0]
 854 | 		for i := 0; i < n; i++ {
 855 | 			// read the number of bytes of the part.
 856 | 			line, err := r.ReadBytes('\n')
 857 | 			if err != nil {
 858 | 				return totalSize, err
 859 | 			}
 860 | 			if line[0] != '$' {
 861 | 				return totalSize, ErrInvalid
 862 | 			}
 863 | 			cmdByteSize += int64(len(line))
 864 | 			// convert the string number to and int
 865 | 			var n int
 866 | 			if len(line) == 4 && line[len(line)-2] == '\r' {
 867 | 				if line[1] < '0' || line[1] > '9' {
 868 | 					return totalSize, ErrInvalid
 869 | 				}
 870 | 				n = int(line[1] - '0')
 871 | 			} else {
 872 | 				if len(line) < 5 || line[len(line)-2] != '\r' {
 873 | 					return totalSize, ErrInvalid
 874 | 				}
 875 | 				for i := 1; i < len(line)-2; i++ {
 876 | 					if line[i] < '0' || line[i] > '9' {
 877 | 						return totalSize, ErrInvalid
 878 | 					}
 879 | 					n = n*10 + int(line[i]-'0')
 880 | 				}
 881 | 			}
 882 | 			// resize the read buffer
 883 | 			if len(data) < n+2 {
 884 | 				dataln := len(data)
 885 | 				for dataln < n+2 {
 886 | 					dataln *= 2
 887 | 				}
 888 | 				data = make([]byte, dataln)
 889 | 			}
 890 | 			if _, err = io.ReadFull(r, data[:n+2]); err != nil {
 891 | 				return totalSize, err
 892 | 			}
 893 | 			if data[n] != '\r' || data[n+1] != '\n' {
 894 | 				return totalSize, ErrInvalid
 895 | 			}
 896 | 			// copy string
 897 | 			parts = append(parts, string(data[:n]))
 898 | 			cmdByteSize += int64(n + 2)
 899 | 		}
 900 | 		// finished reading the command
 901 | 
 902 | 		if len(parts) == 0 {
 903 | 			continue
 904 | 		}
 905 | 		if (parts[0][0] == 's' || parts[0][0] == 'S') &&
 906 | 			(parts[0][1] == 'e' || parts[0][1] == 'E') &&
 907 | 			(parts[0][2] == 't' || parts[0][2] == 'T') {
 908 | 			// SET
 909 | 			if len(parts) < 3 || len(parts) == 4 || len(parts) > 5 {
 910 | 				return totalSize, ErrInvalid
 911 | 			}
 912 | 			if len(parts) == 5 {
 913 | 				arg := strings.ToLower(parts[3])
 914 | 				if arg != "ex" && arg != "ae" {
 915 | 					return totalSize, ErrInvalid
 916 | 				}
 917 | 				ex, err := strconv.ParseInt(parts[4], 10, 64)
 918 | 				if err != nil {
 919 | 					return totalSize, err
 920 | 				}
 921 | 				var exat time.Time
 922 | 				now := time.Now()
 923 | 				if arg == "ex" {
 924 | 					dur := (time.Duration(ex) * time.Second) - now.Sub(modTime)
 925 | 					exat = now.Add(dur)
 926 | 				} else {
 927 | 					exat = time.Unix(ex, 0)
 928 | 				}
 929 | 				if exat.After(now) {
 930 | 					db.insertIntoDatabase(&dbItem{
 931 | 						key: parts[1],
 932 | 						val: parts[2],
 933 | 						opts: &dbItemOpts{
 934 | 							ex:   true,
 935 | 							exat: exat,
 936 | 						},
 937 | 					})
 938 | 				} else {
 939 | 					db.deleteFromDatabase(&dbItem{
 940 | 						key: parts[1],
 941 | 					})
 942 | 				}
 943 | 			} else {
 944 | 				db.insertIntoDatabase(&dbItem{key: parts[1], val: parts[2]})
 945 | 			}
 946 | 		} else if (parts[0][0] == 'd' || parts[0][0] == 'D') &&
 947 | 			(parts[0][1] == 'e' || parts[0][1] == 'E') &&
 948 | 			(parts[0][2] == 'l' || parts[0][2] == 'L') {
 949 | 			// DEL
 950 | 			if len(parts) != 2 {
 951 | 				return totalSize, ErrInvalid
 952 | 			}
 953 | 			db.deleteFromDatabase(&dbItem{key: parts[1]})
 954 | 		} else if (parts[0][0] == 'f' || parts[0][0] == 'F') &&
 955 | 			strings.ToLower(parts[0]) == "flushdb" {
 956 | 			db.keys = btreeNew(lessCtx(nil))
 957 | 			db.exps = btreeNew(lessCtx(&exctx{db}))
 958 | 			db.idxs = make(map[string]*index)
 959 | 		} else {
 960 | 			return totalSize, ErrInvalid
 961 | 		}
 962 | 		totalSize += cmdByteSize
 963 | 	}
 964 | }
 965 | 
 966 | // load reads entries from the append only database file and fills the database.
 967 | // The file format uses the Redis append only file format, which is and a series
 968 | // of RESP commands. For more information on RESP please read
 969 | // http://redis.io/topics/protocol. The only supported RESP commands are DEL and
 970 | // SET.
 971 | func (db *DB) load() error {
 972 | 	fi, err := db.file.Stat()
 973 | 	if err != nil {
 974 | 		return err
 975 | 	}
 976 | 	n, err := db.readLoad(db.file, fi.ModTime())
 977 | 	if err != nil {
 978 | 		if err == io.ErrUnexpectedEOF {
 979 | 			// The db file has ended mid-command, which is allowed but the
 980 | 			// data file should be truncated to the end of the last valid
 981 | 			// command
 982 | 			if err := db.file.Truncate(n); err != nil {
 983 | 				return err
 984 | 			}
 985 | 		} else {
 986 | 			return err
 987 | 		}
 988 | 	}
 989 | 	if _, err := db.file.Seek(n, 0); err != nil {
 990 | 		return err
 991 | 	}
 992 | 	var estaofsz int
 993 | 	db.keys.Walk(func(items []interface{}) {
 994 | 		for _, v := range items {
 995 | 			estaofsz += v.(*dbItem).estAOFSetSize()
 996 | 		}
 997 | 	})
 998 | 	db.lastaofsz += estaofsz
 999 | 	return nil
1000 | }
1001 | 
1002 | // managed calls a block of code that is fully contained in a transaction.
1003 | // This method is intended to be wrapped by Update and View
1004 | func (db *DB) managed(writable bool, fn func(tx *Tx) error) (err error) {
1005 | 	var tx *Tx
1006 | 	tx, err = db.Begin(writable)
1007 | 	if err != nil {
1008 | 		return
1009 | 	}
1010 | 	defer func() {
1011 | 		if err != nil {
1012 | 			// The caller returned an error. We must rollback.
1013 | 			_ = tx.Rollback()
1014 | 			return
1015 | 		}
1016 | 		if writable {
1017 | 			// Everything went well. Lets Commit()
1018 | 			err = tx.Commit()
1019 | 		} else {
1020 | 			// read-only transaction can only roll back.
1021 | 			err = tx.Rollback()
1022 | 		}
1023 | 	}()
1024 | 	tx.funcd = true
1025 | 	defer func() {
1026 | 		tx.funcd = false
1027 | 	}()
1028 | 	err = fn(tx)
1029 | 	return
1030 | }
1031 | 
1032 | // View executes a function within a managed read-only transaction.
1033 | // When a non-nil error is returned from the function that error will be return
1034 | // to the caller of View().
1035 | //
1036 | // Executing a manual commit or rollback from inside the function will result
1037 | // in a panic.
1038 | func (db *DB) View(fn func(tx *Tx) error) error {
1039 | 	return db.managed(false, fn)
1040 | }
1041 | 
1042 | // Update executes a function within a managed read/write transaction.
1043 | // The transaction has been committed when no error is returned.
1044 | // In the event that an error is returned, the transaction will be rolled back.
1045 | // When a non-nil error is returned from the function, the transaction will be
1046 | // rolled back and the that error will be return to the caller of Update().
1047 | //
1048 | // Executing a manual commit or rollback from inside the function will result
1049 | // in a panic.
1050 | func (db *DB) Update(fn func(tx *Tx) error) error {
1051 | 	return db.managed(true, fn)
1052 | }
1053 | 
1054 | // get return an item or nil if not found.
1055 | func (db *DB) get(key string) *dbItem {
1056 | 	item := db.keys.Get(&dbItem{key: key})
1057 | 	if item != nil {
1058 | 		return item.(*dbItem)
1059 | 	}
1060 | 	return nil
1061 | }
1062 | 
1063 | // Tx represents a transaction on the database. This transaction can either be
1064 | // read-only or read/write. Read-only transactions can be used for retrieving
1065 | // values for keys and iterating through keys and values. Read/write
1066 | // transactions can set and delete keys.
1067 | //
1068 | // All transactions must be committed or rolled-back when done.
1069 | type Tx struct {
1070 | 	db       *DB             // the underlying database.
1071 | 	writable bool            // when false mutable operations fail.
1072 | 	funcd    bool            // when true Commit and Rollback panic.
1073 | 	wc       *txWriteContext // context for writable transactions.
1074 | }
1075 | 
1076 | type txWriteContext struct {
1077 | 	// rollback when deleteAll is called
1078 | 	rbkeys *btree.BTree      // a tree of all item ordered by key
1079 | 	rbexps *btree.BTree      // a tree of items ordered by expiration
1080 | 	rbidxs map[string]*index // the index trees.
1081 | 
1082 | 	rollbackItems   map[string]*dbItem // details for rolling back tx.
1083 | 	commitItems     map[string]*dbItem // details for committing tx.
1084 | 	itercount       int                // stack of iterators
1085 | 	rollbackIndexes map[string]*index  // details for dropped indexes.
1086 | }
1087 | 
1088 | // DeleteAll deletes all items from the database.
1089 | func (tx *Tx) DeleteAll() error {
1090 | 	if tx.db == nil {
1091 | 		return ErrTxClosed
1092 | 	} else if !tx.writable {
1093 | 		return ErrTxNotWritable
1094 | 	} else if tx.wc.itercount > 0 {
1095 | 		return ErrTxIterating
1096 | 	}
1097 | 
1098 | 	// check to see if we've already deleted everything
1099 | 	if tx.wc.rbkeys == nil {
1100 | 		// we need to backup the live data in case of a rollback.
1101 | 		tx.wc.rbkeys = tx.db.keys
1102 | 		tx.wc.rbexps = tx.db.exps
1103 | 		tx.wc.rbidxs = tx.db.idxs
1104 | 	}
1105 | 
1106 | 	// now reset the live database trees
1107 | 	tx.db.keys = btreeNew(lessCtx(nil))
1108 | 	tx.db.exps = btreeNew(lessCtx(&exctx{tx.db}))
1109 | 	tx.db.idxs = make(map[string]*index)
1110 | 
1111 | 	// finally re-create the indexes
1112 | 	for name, idx := range tx.wc.rbidxs {
1113 | 		tx.db.idxs[name] = idx.clearCopy()
1114 | 	}
1115 | 
1116 | 	// always clear out the commits
1117 | 	tx.wc.commitItems = make(map[string]*dbItem)
1118 | 
1119 | 	return nil
1120 | }
1121 | 
1122 | // Begin opens a new transaction.
1123 | // Multiple read-only transactions can be opened at the same time but there can
1124 | // only be one read/write transaction at a time. Attempting to open a read/write
1125 | // transactions while another one is in progress will result in blocking until
1126 | // the current read/write transaction is completed.
1127 | //
1128 | // All transactions must be closed by calling Commit() or Rollback() when done.
1129 | func (db *DB) Begin(writable bool) (*Tx, error) {
1130 | 	tx := &Tx{
1131 | 		db:       db,
1132 | 		writable: writable,
1133 | 	}
1134 | 	tx.lock()
1135 | 	if db.closed {
1136 | 		tx.unlock()
1137 | 		return nil, ErrDatabaseClosed
1138 | 	}
1139 | 	if writable {
1140 | 		// writable transactions have a writeContext object that
1141 | 		// contains information about changes to the database.
1142 | 		tx.wc = &txWriteContext{}
1143 | 		tx.wc.rollbackItems = make(map[string]*dbItem)
1144 | 		tx.wc.rollbackIndexes = make(map[string]*index)
1145 | 		if db.persist {
1146 | 			tx.wc.commitItems = make(map[string]*dbItem)
1147 | 		}
1148 | 	}
1149 | 	return tx, nil
1150 | }
1151 | 
1152 | // lock locks the database based on the transaction type.
1153 | func (tx *Tx) lock() {
1154 | 	if tx.writable {
1155 | 		tx.db.mu.Lock()
1156 | 	} else {
1157 | 		tx.db.mu.RLock()
1158 | 	}
1159 | }
1160 | 
1161 | // unlock unlocks the database based on the transaction type.
1162 | func (tx *Tx) unlock() {
1163 | 	if tx.writable {
1164 | 		tx.db.mu.Unlock()
1165 | 	} else {
1166 | 		tx.db.mu.RUnlock()
1167 | 	}
1168 | }
1169 | 
1170 | // rollbackInner handles the underlying rollback logic.
1171 | // Intended to be called from Commit() and Rollback().
1172 | func (tx *Tx) rollbackInner() {
1173 | 	// rollback the deleteAll if needed
1174 | 	if tx.wc.rbkeys != nil {
1175 | 		tx.db.keys = tx.wc.rbkeys
1176 | 		tx.db.idxs = tx.wc.rbidxs
1177 | 		tx.db.exps = tx.wc.rbexps
1178 | 	}
1179 | 	for key, item := range tx.wc.rollbackItems {
1180 | 		tx.db.deleteFromDatabase(&dbItem{key: key})
1181 | 		if item != nil {
1182 | 			// When an item is not nil, we will need to reinsert that item
1183 | 			// into the database overwriting the current one.
1184 | 			tx.db.insertIntoDatabase(item)
1185 | 		}
1186 | 	}
1187 | 	for name, idx := range tx.wc.rollbackIndexes {
1188 | 		delete(tx.db.idxs, name)
1189 | 		if idx != nil {
1190 | 			// When an index is not nil, we will need to rebuilt that index
1191 | 			// this could be an expensive process if the database has many
1192 | 			// items or the index is complex.
1193 | 			tx.db.idxs[name] = idx
1194 | 			idx.rebuild()
1195 | 		}
1196 | 	}
1197 | }
1198 | 
1199 | // Commit writes all changes to disk.
1200 | // An error is returned when a write error occurs, or when a Commit() is called
1201 | // from a read-only transaction.
1202 | func (tx *Tx) Commit() error {
1203 | 	if tx.funcd {
1204 | 		panic("managed tx commit not allowed")
1205 | 	}
1206 | 	if tx.db == nil {
1207 | 		return ErrTxClosed
1208 | 	} else if !tx.writable {
1209 | 		return ErrTxNotWritable
1210 | 	}
1211 | 	var err error
1212 | 	if tx.db.persist && (len(tx.wc.commitItems) > 0 || tx.wc.rbkeys != nil) {
1213 | 		tx.db.buf = tx.db.buf[:0]
1214 | 		// write a flushdb if a deleteAll was called.
1215 | 		if tx.wc.rbkeys != nil {
1216 | 			tx.db.buf = append(tx.db.buf, "*1\r\n$7\r\nflushdb\r\n"...)
1217 | 		}
1218 | 		now := time.Now()
1219 | 		// Each committed record is written to disk
1220 | 		for key, item := range tx.wc.commitItems {
1221 | 			if item == nil {
1222 | 				tx.db.buf = (&dbItem{key: key}).writeDeleteTo(tx.db.buf)
1223 | 			} else {
1224 | 				tx.db.buf = item.writeSetTo(tx.db.buf, now)
1225 | 			}
1226 | 		}
1227 | 		// Flushing the buffer only once per transaction.
1228 | 		// If this operation fails then the write did failed and we must
1229 | 		// rollback.
1230 | 		var n int
1231 | 		n, err = tx.db.file.Write(tx.db.buf)
1232 | 		if err != nil {
1233 | 			if n > 0 {
1234 | 				// There was a partial write to disk.
1235 | 				// We are possibly out of disk space.
1236 | 				// Delete the partially written bytes from the data file by
1237 | 				// seeking to the previously known position and performing
1238 | 				// a truncate operation.
1239 | 				// At this point a syscall failure is fatal and the process
1240 | 				// should be killed to avoid corrupting the file.
1241 | 				pos, err := tx.db.file.Seek(-int64(n), 1)
1242 | 				if err != nil {
1243 | 					panicErr(err)
1244 | 				}
1245 | 				if err := tx.db.file.Truncate(pos); err != nil {
1246 | 					panicErr(err)
1247 | 				}
1248 | 			}
1249 | 			tx.rollbackInner()
1250 | 		}
1251 | 		if tx.db.config.SyncPolicy == Always {
1252 | 			_ = tx.db.file.Sync()
1253 | 		}
1254 | 		// Increment the number of flushes. The background syncing uses this.
1255 | 		tx.db.flushes++
1256 | 	}
1257 | 	// Unlock the database and allow for another writable transaction.
1258 | 	tx.unlock()
1259 | 	// Clear the db field to disable this transaction from future use.
1260 | 	tx.db = nil
1261 | 	return err
1262 | }
1263 | 
1264 | // Rollback closes the transaction and reverts all mutable operations that
1265 | // were performed on the transaction such as Set() and Delete().
1266 | //
1267 | // Read-only transactions can only be rolled back, not committed.
1268 | func (tx *Tx) Rollback() error {
1269 | 	if tx.funcd {
1270 | 		panic("managed tx rollback not allowed")
1271 | 	}
1272 | 	if tx.db == nil {
1273 | 		return ErrTxClosed
1274 | 	}
1275 | 	// The rollback func does the heavy lifting.
1276 | 	if tx.writable {
1277 | 		tx.rollbackInner()
1278 | 	}
1279 | 	// unlock the database for more transactions.
1280 | 	tx.unlock()
1281 | 	// Clear the db field to disable this transaction from future use.
1282 | 	tx.db = nil
1283 | 	return nil
1284 | }
1285 | 
1286 | // dbItemOpts holds various meta information about an item.
1287 | type dbItemOpts struct {
1288 | 	ex   bool      // does this item expire?
1289 | 	exat time.Time // when does this item expire?
1290 | }
1291 | type dbItem struct {
1292 | 	key, val string      // the binary key and value
1293 | 	opts     *dbItemOpts // optional meta information
1294 | 	keyless  bool        // keyless item for scanning
1295 | }
1296 | 
1297 | // estIntSize returns the string representions size.
1298 | // Has the same result as len(strconv.Itoa(x)).
1299 | func estIntSize(x int) int {
1300 | 	n := 1
1301 | 	if x < 0 {
1302 | 		n++
1303 | 		x *= -1
1304 | 	}
1305 | 	for x >= 10 {
1306 | 		n++
1307 | 		x /= 10
1308 | 	}
1309 | 	return n
1310 | }
1311 | 
1312 | func estArraySize(count int) int {
1313 | 	return 1 + estIntSize(count) + 2
1314 | }
1315 | 
1316 | func estBulkStringSize(s string) int {
1317 | 	return 1 + estIntSize(len(s)) + 2 + len(s) + 2
1318 | }
1319 | 
1320 | // estAOFSetSize returns an estimated number of bytes that this item will use
1321 | // when stored in the aof file.
1322 | func (dbi *dbItem) estAOFSetSize() int {
1323 | 	var n int
1324 | 	if dbi.opts != nil && dbi.opts.ex {
1325 | 		n += estArraySize(5)
1326 | 		n += estBulkStringSize("set")
1327 | 		n += estBulkStringSize(dbi.key)
1328 | 		n += estBulkStringSize(dbi.val)
1329 | 		n += estBulkStringSize("ex")
1330 | 		n += estBulkStringSize("99") // estimate two byte bulk string
1331 | 	} else {
1332 | 		n += estArraySize(3)
1333 | 		n += estBulkStringSize("set")
1334 | 		n += estBulkStringSize(dbi.key)
1335 | 		n += estBulkStringSize(dbi.val)
1336 | 	}
1337 | 	return n
1338 | }
1339 | 
1340 | func appendArray(buf []byte, count int) []byte {
1341 | 	buf = append(buf, '*')
1342 | 	buf = strconv.AppendInt(buf, int64(count), 10)
1343 | 	buf = append(buf, '\r', '\n')
1344 | 	return buf
1345 | }
1346 | 
1347 | func appendBulkString(buf []byte, s string) []byte {
1348 | 	buf = append(buf, '$')
1349 | 	buf = strconv.AppendInt(buf, int64(len(s)), 10)
1350 | 	buf = append(buf, '\r', '\n')
1351 | 	buf = append(buf, s...)
1352 | 	buf = append(buf, '\r', '\n')
1353 | 	return buf
1354 | }
1355 | 
1356 | // writeSetTo writes an item as a single SET record to the a bufio Writer.
1357 | func (dbi *dbItem) writeSetTo(buf []byte, now time.Time) []byte {
1358 | 	if dbi.opts != nil && dbi.opts.ex {
1359 | 		buf = appendArray(buf, 5)
1360 | 		buf = appendBulkString(buf, "set")
1361 | 		buf = appendBulkString(buf, dbi.key)
1362 | 		buf = appendBulkString(buf, dbi.val)
1363 | 		if useAbsEx {
1364 | 			ex := dbi.opts.exat.Unix()
1365 | 			buf = appendBulkString(buf, "ae")
1366 | 			buf = appendBulkString(buf, strconv.FormatUint(uint64(ex), 10))
1367 | 		} else {
1368 | 			ex := dbi.opts.exat.Sub(now) / time.Second
1369 | 			buf = appendBulkString(buf, "ex")
1370 | 			buf = appendBulkString(buf, strconv.FormatUint(uint64(ex), 10))
1371 | 		}
1372 | 	} else {
1373 | 		buf = appendArray(buf, 3)
1374 | 		buf = appendBulkString(buf, "set")
1375 | 		buf = appendBulkString(buf, dbi.key)
1376 | 		buf = appendBulkString(buf, dbi.val)
1377 | 	}
1378 | 	return buf
1379 | }
1380 | 
1381 | // writeSetTo writes an item as a single DEL record to the a bufio Writer.
1382 | func (dbi *dbItem) writeDeleteTo(buf []byte) []byte {
1383 | 	buf = appendArray(buf, 2)
1384 | 	buf = appendBulkString(buf, "del")
1385 | 	buf = appendBulkString(buf, dbi.key)
1386 | 	return buf
1387 | }
1388 | 
1389 | // expired evaluates id the item has expired. This will always return false when
1390 | // the item does not have `opts.ex` set to true.
1391 | func (dbi *dbItem) expired() bool {
1392 | 	return dbi.opts != nil && dbi.opts.ex && time.Now().After(dbi.opts.exat)
1393 | }
1394 | 
1395 | // MaxTime from http://stackoverflow.com/questions/25065055#32620397
1396 | // This is a long time in the future. It's an imaginary number that is
1397 | // used for b-tree ordering.
1398 | var maxTime = time.Unix(1<<63-62135596801, 999999999)
1399 | 
1400 | // expiresAt will return the time when the item will expire. When an item does
1401 | // not expire `maxTime` is used.
1402 | func (dbi *dbItem) expiresAt() time.Time {
1403 | 	if dbi.opts == nil || !dbi.opts.ex {
1404 | 		return maxTime
1405 | 	}
1406 | 	return dbi.opts.exat
1407 | }
1408 | 
1409 | // Less determines if a b-tree item is less than another. This is required
1410 | // for ordering, inserting, and deleting items from a b-tree. It's important
1411 | // to note that the ctx parameter is used to help with determine which
1412 | // formula to use on an item. Each b-tree should use a different ctx when
1413 | // sharing the same item.
1414 | func (dbi *dbItem) Less(dbi2 *dbItem, ctx interface{}) bool {
1415 | 	switch ctx := ctx.(type) {
1416 | 	case *exctx:
1417 | 		// The expires b-tree formula
1418 | 		if dbi2.expiresAt().After(dbi.expiresAt()) {
1419 | 			return true
1420 | 		}
1421 | 		if dbi.expiresAt().After(dbi2.expiresAt()) {
1422 | 			return false
1423 | 		}
1424 | 	case *index:
1425 | 		if ctx.less != nil {
1426 | 			// Using an index
1427 | 			if ctx.less(dbi.val, dbi2.val) {
1428 | 				return true
1429 | 			}
1430 | 			if ctx.less(dbi2.val, dbi.val) {
1431 | 				return false
1432 | 			}
1433 | 		}
1434 | 	}
1435 | 	// Always fall back to the key comparison. This creates absolute uniqueness.
1436 | 	if dbi.keyless {
1437 | 		return false
1438 | 	} else if dbi2.keyless {
1439 | 		return true
1440 | 	}
1441 | 	return dbi.key < dbi2.key
1442 | }
1443 | 
1444 | func lessCtx(ctx interface{}) func(a, b interface{}) bool {
1445 | 	return func(a, b interface{}) bool {
1446 | 		return a.(*dbItem).Less(b.(*dbItem), ctx)
1447 | 	}
1448 | }
1449 | 
1450 | // Rect converts a string to a rectangle.
1451 | // An invalid rectangle will cause a panic.
1452 | func (dbi *dbItem) Rect(ctx interface{}) (min, max []float64) {
1453 | 	switch ctx := ctx.(type) {
1454 | 	case *index:
1455 | 		return ctx.rect(dbi.val)
1456 | 	}
1457 | 	return nil, nil
1458 | }
1459 | 
1460 | // SetOptions represents options that may be included with the Set() command.
1461 | type SetOptions struct {
1462 | 	// Expires indicates that the Set() key-value will expire
1463 | 	Expires bool
1464 | 	// TTL is how much time the key-value will exist in the database
1465 | 	// before being evicted. The Expires field must also be set to true.
1466 | 	// TTL stands for Time-To-Live.
1467 | 	TTL time.Duration
1468 | }
1469 | 
1470 | // GetLess returns the less function for an index. This is handy for
1471 | // doing ad-hoc compares inside a transaction.
1472 | // Returns ErrNotFound if the index is not found or there is no less
1473 | // function bound to the index
1474 | func (tx *Tx) GetLess(index string) (func(a, b string) bool, error) {
1475 | 	if tx.db == nil {
1476 | 		return nil, ErrTxClosed
1477 | 	}
1478 | 	idx, ok := tx.db.idxs[index]
1479 | 	if !ok || idx.less == nil {
1480 | 		return nil, ErrNotFound
1481 | 	}
1482 | 	return idx.less, nil
1483 | }
1484 | 
1485 | // GetRect returns the rect function for an index. This is handy for
1486 | // doing ad-hoc searches inside a transaction.
1487 | // Returns ErrNotFound if the index is not found or there is no rect
1488 | // function bound to the index
1489 | func (tx *Tx) GetRect(index string) (func(s string) (min, max []float64),
1490 | 	error) {
1491 | 	if tx.db == nil {
1492 | 		return nil, ErrTxClosed
1493 | 	}
1494 | 	idx, ok := tx.db.idxs[index]
1495 | 	if !ok || idx.rect == nil {
1496 | 		return nil, ErrNotFound
1497 | 	}
1498 | 	return idx.rect, nil
1499 | }
1500 | 
1501 | // Set inserts or replaces an item in the database based on the key.
1502 | // The opt params may be used for additional functionality such as forcing
1503 | // the item to be evicted at a specified time. When the return value
1504 | // for err is nil the operation succeeded. When the return value of
1505 | // replaced is true, then the operaton replaced an existing item whose
1506 | // value will be returned through the previousValue variable.
1507 | // The results of this operation will not be available to other
1508 | // transactions until the current transaction has successfully committed.
1509 | //
1510 | // Only a writable transaction can be used with this operation.
1511 | // This operation is not allowed during iterations such as Ascend* & Descend*.
1512 | func (tx *Tx) Set(key, value string, opts *SetOptions) (previousValue string,
1513 | 	replaced bool, err error) {
1514 | 	if tx.db == nil {
1515 | 		return "", false, ErrTxClosed
1516 | 	} else if !tx.writable {
1517 | 		return "", false, ErrTxNotWritable
1518 | 	} else if tx.wc.itercount > 0 {
1519 | 		return "", false, ErrTxIterating
1520 | 	}
1521 | 	item := &dbItem{key: key, val: value}
1522 | 	if opts != nil {
1523 | 		if opts.Expires {
1524 | 			// The caller is requesting that this item expires. Convert the
1525 | 			// TTL to an absolute time and bind it to the item.
1526 | 			item.opts = &dbItemOpts{ex: true, exat: time.Now().Add(opts.TTL)}
1527 | 		}
1528 | 	}
1529 | 	// Insert the item into the keys tree.
1530 | 	prev := tx.db.insertIntoDatabase(item)
1531 | 
1532 | 	// insert into the rollback map if there has not been a deleteAll.
1533 | 	if tx.wc.rbkeys == nil {
1534 | 		if prev == nil {
1535 | 			// An item with the same key did not previously exist. Let's
1536 | 			// create a rollback entry with a nil value. A nil value indicates
1537 | 			// that the entry should be deleted on rollback. When the value is
1538 | 			// *not* nil, that means the entry should be reverted.
1539 | 			if _, ok := tx.wc.rollbackItems[key]; !ok {
1540 | 				tx.wc.rollbackItems[key] = nil
1541 | 			}
1542 | 		} else {
1543 | 			// A previous item already exists in the database. Let's create a
1544 | 			// rollback entry with the item as the value. We need to check the
1545 | 			// map to see if there isn't already an item that matches the
1546 | 			// same key.
1547 | 			if _, ok := tx.wc.rollbackItems[key]; !ok {
1548 | 				tx.wc.rollbackItems[key] = prev
1549 | 			}
1550 | 			if !prev.expired() {
1551 | 				previousValue, replaced = prev.val, true
1552 | 			}
1553 | 		}
1554 | 	}
1555 | 	// For commits we simply assign the item to the map. We use this map to
1556 | 	// write the entry to disk.
1557 | 	if tx.db.persist {
1558 | 		tx.wc.commitItems[key] = item
1559 | 	}
1560 | 	return previousValue, replaced, nil
1561 | }
1562 | 
1563 | // Get returns a value for a key. If the item does not exist or if the item
1564 | // has expired then ErrNotFound is returned. If ignoreExpired is true, then
1565 | // the found value will be returned even if it is expired.
1566 | func (tx *Tx) Get(key string, ignoreExpired ...bool) (val string, err error) {
1567 | 	if tx.db == nil {
1568 | 		return "", ErrTxClosed
1569 | 	}
1570 | 	var ignore bool
1571 | 	if len(ignoreExpired) != 0 {
1572 | 		ignore = ignoreExpired[0]
1573 | 	}
1574 | 	item := tx.db.get(key)
1575 | 	if item == nil || (item.expired() && !ignore) {
1576 | 		// The item does not exists or has expired. Let's assume that
1577 | 		// the caller is only interested in items that have not expired.
1578 | 		return "", ErrNotFound
1579 | 	}
1580 | 	return item.val, nil
1581 | }
1582 | 
1583 | // Delete removes an item from the database based on the item's key. If the item
1584 | // does not exist or if the item has expired then ErrNotFound is returned.
1585 | //
1586 | // Only a writable transaction can be used for this operation.
1587 | // This operation is not allowed during iterations such as Ascend* & Descend*.
1588 | func (tx *Tx) Delete(key string) (val string, err error) {
1589 | 	if tx.db == nil {
1590 | 		return "", ErrTxClosed
1591 | 	} else if !tx.writable {
1592 | 		return "", ErrTxNotWritable
1593 | 	} else if tx.wc.itercount > 0 {
1594 | 		return "", ErrTxIterating
1595 | 	}
1596 | 	item := tx.db.deleteFromDatabase(&dbItem{key: key})
1597 | 	if item == nil {
1598 | 		return "", ErrNotFound
1599 | 	}
1600 | 	// create a rollback entry if there has not been a deleteAll call.
1601 | 	if tx.wc.rbkeys == nil {
1602 | 		if _, ok := tx.wc.rollbackItems[key]; !ok {
1603 | 			tx.wc.rollbackItems[key] = item
1604 | 		}
1605 | 	}
1606 | 	if tx.db.persist {
1607 | 		tx.wc.commitItems[key] = nil
1608 | 	}
1609 | 	// Even though the item has been deleted, we still want to check
1610 | 	// if it has expired. An expired item should not be returned.
1611 | 	if item.expired() {
1612 | 		// The item exists in the tree, but has expired. Let's assume that
1613 | 		// the caller is only interested in items that have not expired.
1614 | 		return "", ErrNotFound
1615 | 	}
1616 | 	return item.val, nil
1617 | }
1618 | 
1619 | // TTL returns the remaining time-to-live for an item.
1620 | // A negative duration will be returned for items that do not have an
1621 | // expiration.
1622 | func (tx *Tx) TTL(key string) (time.Duration, error) {
1623 | 	if tx.db == nil {
1624 | 		return 0, ErrTxClosed
1625 | 	}
1626 | 	item := tx.db.get(key)
1627 | 	if item == nil {
1628 | 		return 0, ErrNotFound
1629 | 	} else if item.opts == nil || !item.opts.ex {
1630 | 		return -1, nil
1631 | 	}
1632 | 	dur := time.Until(item.opts.exat)
1633 | 	if dur < 0 {
1634 | 		return 0, ErrNotFound
1635 | 	}
1636 | 	return dur, nil
1637 | }
1638 | 
1639 | // scan iterates through a specified index and calls user-defined iterator
1640 | // function for each item encountered.
1641 | // The desc param indicates that the iterator should descend.
1642 | // The gt param indicates that there is a greaterThan limit.
1643 | // The lt param indicates that there is a lessThan limit.
1644 | // The index param tells the scanner to use the specified index tree. An
1645 | // empty string for the index means to scan the keys, not the values.
1646 | // The start and stop params are the greaterThan, lessThan limits. For
1647 | // descending order, these will be lessThan, greaterThan.
1648 | // An error will be returned if the tx is closed or the index is not found.
1649 | func (tx *Tx) scan(desc, gt, lt bool, index, start, stop string,
1650 | 	iterator func(key, value string) bool) error {
1651 | 	if tx.db == nil {
1652 | 		return ErrTxClosed
1653 | 	}
1654 | 	// wrap a btree specific iterator around the user-defined iterator.
1655 | 	iter := func(item interface{}) bool {
1656 | 		dbi := item.(*dbItem)
1657 | 		if dbi.expired() {
1658 | 			return true
1659 | 		}
1660 | 		return iterator(dbi.key, dbi.val)
1661 | 	}
1662 | 	var tr *btree.BTree
1663 | 	if index == "" {
1664 | 		// empty index means we will use the keys tree.
1665 | 		tr = tx.db.keys
1666 | 	} else {
1667 | 		idx := tx.db.idxs[index]
1668 | 		if idx == nil {
1669 | 			// index was not found. return error
1670 | 			return ErrNotFound
1671 | 		}
1672 | 		tr = idx.btr
1673 | 		if tr == nil {
1674 | 			return nil
1675 | 		}
1676 | 	}
1677 | 	// create some limit items
1678 | 	var itemA, itemB *dbItem
1679 | 	if gt || lt {
1680 | 		if index == "" {
1681 | 			itemA = &dbItem{key: start}
1682 | 			itemB = &dbItem{key: stop}
1683 | 		} else {
1684 | 			itemA = &dbItem{val: start}
1685 | 			itemB = &dbItem{val: stop}
1686 | 			if desc {
1687 | 				itemA.keyless = true
1688 | 				itemB.keyless = true
1689 | 			}
1690 | 		}
1691 | 	}
1692 | 	// execute the scan on the underlying tree.
1693 | 	if tx.wc != nil {
1694 | 		tx.wc.itercount++
1695 | 		defer func() {
1696 | 			tx.wc.itercount--
1697 | 		}()
1698 | 	}
1699 | 	if desc {
1700 | 		if gt {
1701 | 			if lt {
1702 | 				btreeDescendRange(tr, itemA, itemB, iter)
1703 | 			} else {
1704 | 				btreeDescendGreaterThan(tr, itemA, iter)
1705 | 			}
1706 | 		} else if lt {
1707 | 			btreeDescendLessOrEqual(tr, itemA, iter)
1708 | 		} else {
1709 | 			btreeDescend(tr, iter)
1710 | 		}
1711 | 	} else {
1712 | 		if gt {
1713 | 			if lt {
1714 | 				btreeAscendRange(tr, itemA, itemB, iter)
1715 | 			} else {
1716 | 				btreeAscendGreaterOrEqual(tr, itemA, iter)
1717 | 			}
1718 | 		} else if lt {
1719 | 			btreeAscendLessThan(tr, itemA, iter)
1720 | 		} else {
1721 | 			btreeAscend(tr, iter)
1722 | 		}
1723 | 	}
1724 | 	return nil
1725 | }
1726 | 
1727 | // Match returns true if the specified key matches the pattern. This is a very
1728 | // simple pattern matcher where '*' matches on any number characters and '?'
1729 | // matches on any one character.
1730 | func Match(key, pattern string) bool {
1731 | 	return match.Match(key, pattern)
1732 | }
1733 | 
1734 | // AscendKeys allows for iterating through keys based on the specified pattern.
1735 | func (tx *Tx) AscendKeys(pattern string,
1736 | 	iterator func(key, value string) bool) error {
1737 | 	if pattern == "" {
1738 | 		return nil
1739 | 	}
1740 | 	if pattern[0] == '*' {
1741 | 		if pattern == "*" {
1742 | 			return tx.Ascend("", iterator)
1743 | 		}
1744 | 		return tx.Ascend("", func(key, value string) bool {
1745 | 			if match.Match(key, pattern) {
1746 | 				if !iterator(key, value) {
1747 | 					return false
1748 | 				}
1749 | 			}
1750 | 			return true
1751 | 		})
1752 | 	}
1753 | 	min, max := match.Allowable(pattern)
1754 | 	return tx.AscendGreaterOrEqual("", min, func(key, value string) bool {
1755 | 		if key > max {
1756 | 			return false
1757 | 		}
1758 | 		if match.Match(key, pattern) {
1759 | 			if !iterator(key, value) {
1760 | 				return false
1761 | 			}
1762 | 		}
1763 | 		return true
1764 | 	})
1765 | }
1766 | 
1767 | // DescendKeys allows for iterating through keys based on the specified pattern.
1768 | func (tx *Tx) DescendKeys(pattern string,
1769 | 	iterator func(key, value string) bool) error {
1770 | 	if pattern == "" {
1771 | 		return nil
1772 | 	}
1773 | 	if pattern[0] == '*' {
1774 | 		if pattern == "*" {
1775 | 			return tx.Descend("", iterator)
1776 | 		}
1777 | 		return tx.Descend("", func(key, value string) bool {
1778 | 			if match.Match(key, pattern) {
1779 | 				if !iterator(key, value) {
1780 | 					return false
1781 | 				}
1782 | 			}
1783 | 			return true
1784 | 		})
1785 | 	}
1786 | 	min, max := match.Allowable(pattern)
1787 | 	return tx.DescendLessOrEqual("", max, func(key, value string) bool {
1788 | 		if key < min {
1789 | 			return false
1790 | 		}
1791 | 		if match.Match(key, pattern) {
1792 | 			if !iterator(key, value) {
1793 | 				return false
1794 | 			}
1795 | 		}
1796 | 		return true
1797 | 	})
1798 | }
1799 | 
1800 | // Ascend calls the iterator for every item in the database within the range
1801 | // [first, last], until iterator returns false.
1802 | // When an index is provided, the results will be ordered by the item values
1803 | // as specified by the less() function of the defined index.
1804 | // When an index is not provided, the results will be ordered by the item key.
1805 | // An invalid index will return an error.
1806 | func (tx *Tx) Ascend(index string,
1807 | 	iterator func(key, value string) bool) error {
1808 | 	return tx.scan(false, false, false, index, "", "", iterator)
1809 | }
1810 | 
1811 | // AscendGreaterOrEqual calls the iterator for every item in the database within
1812 | // the range [pivot, last], until iterator returns false.
1813 | // When an index is provided, the results will be ordered by the item values
1814 | // as specified by the less() function of the defined index.
1815 | // When an index is not provided, the results will be ordered by the item key.
1816 | // An invalid index will return an error.
1817 | func (tx *Tx) AscendGreaterOrEqual(index, pivot string,
1818 | 	iterator func(key, value string) bool) error {
1819 | 	return tx.scan(false, true, false, index, pivot, "", iterator)
1820 | }
1821 | 
1822 | // AscendLessThan calls the iterator for every item in the database within the
1823 | // range [first, pivot), until iterator returns false.
1824 | // When an index is provided, the results will be ordered by the item values
1825 | // as specified by the less() function of the defined index.
1826 | // When an index is not provided, the results will be ordered by the item key.
1827 | // An invalid index will return an error.
1828 | func (tx *Tx) AscendLessThan(index, pivot string,
1829 | 	iterator func(key, value string) bool) error {
1830 | 	return tx.scan(false, false, true, index, pivot, "", iterator)
1831 | }
1832 | 
1833 | // AscendRange calls the iterator for every item in the database within
1834 | // the range [greaterOrEqual, lessThan), until iterator returns false.
1835 | // When an index is provided, the results will be ordered by the item values
1836 | // as specified by the less() function of the defined index.
1837 | // When an index is not provided, the results will be ordered by the item key.
1838 | // An invalid index will return an error.
1839 | func (tx *Tx) AscendRange(index, greaterOrEqual, lessThan string,
1840 | 	iterator func(key, value string) bool) error {
1841 | 	return tx.scan(
1842 | 		false, true, true, index, greaterOrEqual, lessThan, iterator,
1843 | 	)
1844 | }
1845 | 
1846 | // Descend calls the iterator for every item in the database within the range
1847 | // [last, first], until iterator returns false.
1848 | // When an index is provided, the results will be ordered by the item values
1849 | // as specified by the less() function of the defined index.
1850 | // When an index is not provided, the results will be ordered by the item key.
1851 | // An invalid index will return an error.
1852 | func (tx *Tx) Descend(index string,
1853 | 	iterator func(key, value string) bool) error {
1854 | 	return tx.scan(true, false, false, index, "", "", iterator)
1855 | }
1856 | 
1857 | // DescendGreaterThan calls the iterator for every item in the database within
1858 | // the range [last, pivot), until iterator returns false.
1859 | // When an index is provided, the results will be ordered by the item values
1860 | // as specified by the less() function of the defined index.
1861 | // When an index is not provided, the results will be ordered by the item key.
1862 | // An invalid index will return an error.
1863 | func (tx *Tx) DescendGreaterThan(index, pivot string,
1864 | 	iterator func(key, value string) bool) error {
1865 | 	return tx.scan(true, true, false, index, pivot, "", iterator)
1866 | }
1867 | 
1868 | // DescendLessOrEqual calls the iterator for every item in the database within
1869 | // the range [pivot, first], until iterator returns false.
1870 | // When an index is provided, the results will be ordered by the item values
1871 | // as specified by the less() function of the defined index.
1872 | // When an index is not provided, the results will be ordered by the item key.
1873 | // An invalid index will return an error.
1874 | func (tx *Tx) DescendLessOrEqual(index, pivot string,
1875 | 	iterator func(key, value string) bool) error {
1876 | 	return tx.scan(true, false, true, index, pivot, "", iterator)
1877 | }
1878 | 
1879 | // DescendRange calls the iterator for every item in the database within
1880 | // the range [lessOrEqual, greaterThan), until iterator returns false.
1881 | // When an index is provided, the results will be ordered by the item values
1882 | // as specified by the less() function of the defined index.
1883 | // When an index is not provided, the results will be ordered by the item key.
1884 | // An invalid index will return an error.
1885 | func (tx *Tx) DescendRange(index, lessOrEqual, greaterThan string,
1886 | 	iterator func(key, value string) bool) error {
1887 | 	return tx.scan(
1888 | 		true, true, true, index, lessOrEqual, greaterThan, iterator,
1889 | 	)
1890 | }
1891 | 
1892 | // AscendEqual calls the iterator for every item in the database that equals
1893 | // pivot, until iterator returns false.
1894 | // When an index is provided, the results will be ordered by the item values
1895 | // as specified by the less() function of the defined index.
1896 | // When an index is not provided, the results will be ordered by the item key.
1897 | // An invalid index will return an error.
1898 | func (tx *Tx) AscendEqual(index, pivot string,
1899 | 	iterator func(key, value string) bool) error {
1900 | 	var err error
1901 | 	var less func(a, b string) bool
1902 | 	if index != "" {
1903 | 		less, err = tx.GetLess(index)
1904 | 		if err != nil {
1905 | 			return err
1906 | 		}
1907 | 	}
1908 | 	return tx.AscendGreaterOrEqual(index, pivot, func(key, value string) bool {
1909 | 		if less == nil {
1910 | 			if key != pivot {
1911 | 				return false
1912 | 			}
1913 | 		} else if less(pivot, value) {
1914 | 			return false
1915 | 		}
1916 | 		return iterator(key, value)
1917 | 	})
1918 | }
1919 | 
1920 | // DescendEqual calls the iterator for every item in the database that equals
1921 | // pivot, until iterator returns false.
1922 | // When an index is provided, the results will be ordered by the item values
1923 | // as specified by the less() function of the defined index.
1924 | // When an index is not provided, the results will be ordered by the item key.
1925 | // An invalid index will return an error.
1926 | func (tx *Tx) DescendEqual(index, pivot string,
1927 | 	iterator func(key, value string) bool) error {
1928 | 	var err error
1929 | 	var less func(a, b string) bool
1930 | 	if index != "" {
1931 | 		less, err = tx.GetLess(index)
1932 | 		if err != nil {
1933 | 			return err
1934 | 		}
1935 | 	}
1936 | 	return tx.DescendLessOrEqual(index, pivot, func(key, value string) bool {
1937 | 		if less == nil {
1938 | 			if key != pivot {
1939 | 				return false
1940 | 			}
1941 | 		} else if less(value, pivot) {
1942 | 			return false
1943 | 		}
1944 | 		return iterator(key, value)
1945 | 	})
1946 | }
1947 | 
1948 | // rect is used by Intersects and Nearby
1949 | type rect struct {
1950 | 	min, max []float64
1951 | }
1952 | 
1953 | func (r *rect) Rect(ctx interface{}) (min, max []float64) {
1954 | 	return r.min, r.max
1955 | }
1956 | 
1957 | // Nearby searches for rectangle items that are nearby a target rect.
1958 | // All items belonging to the specified index will be returned in order of
1959 | // nearest to farthest.
1960 | // The specified index must have been created by AddIndex() and the target
1961 | // is represented by the rect string. This string will be processed by the
1962 | // same bounds function that was passed to the CreateSpatialIndex() function.
1963 | // An invalid index will return an error.
1964 | // The dist param is the distance of the bounding boxes. In the case of
1965 | // simple 2D points, it's the distance of the two 2D points squared.
1966 | func (tx *Tx) Nearby(index, bounds string,
1967 | 	iterator func(key, value string, dist float64) bool) error {
1968 | 	if tx.db == nil {
1969 | 		return ErrTxClosed
1970 | 	}
1971 | 	if index == "" {
1972 | 		// cannot search on keys tree. just return nil.
1973 | 		return nil
1974 | 	}
1975 | 	// // wrap a rtree specific iterator around the user-defined iterator.
1976 | 	iter := func(item rtred.Item, dist float64) bool {
1977 | 		dbi := item.(*dbItem)
1978 | 		return iterator(dbi.key, dbi.val, dist)
1979 | 	}
1980 | 	idx := tx.db.idxs[index]
1981 | 	if idx == nil {
1982 | 		// index was not found. return error
1983 | 		return ErrNotFound
1984 | 	}
1985 | 	if idx.rtr == nil {
1986 | 		// not an r-tree index. just return nil
1987 | 		return nil
1988 | 	}
1989 | 	// execute the nearby search
1990 | 	var min, max []float64
1991 | 	if idx.rect != nil {
1992 | 		min, max = idx.rect(bounds)
1993 | 	}
1994 | 	// set the center param to false, which uses the box dist calc.
1995 | 	idx.rtr.KNN(&rect{min, max}, false, iter)
1996 | 	return nil
1997 | }
1998 | 
1999 | // Intersects searches for rectangle items that intersect a target rect.
2000 | // The specified index must have been created by AddIndex() and the target
2001 | // is represented by the rect string. This string will be processed by the
2002 | // same bounds function that was passed to the CreateSpatialIndex() function.
2003 | // An invalid index will return an error.
2004 | func (tx *Tx) Intersects(index, bounds string,
2005 | 	iterator func(key, value string) bool) error {
2006 | 	if tx.db == nil {
2007 | 		return ErrTxClosed
2008 | 	}
2009 | 	if index == "" {
2010 | 		// cannot search on keys tree. just return nil.
2011 | 		return nil
2012 | 	}
2013 | 	// wrap a rtree specific iterator around the user-defined iterator.
2014 | 	iter := func(item rtred.Item) bool {
2015 | 		dbi := item.(*dbItem)
2016 | 		return iterator(dbi.key, dbi.val)
2017 | 	}
2018 | 	idx := tx.db.idxs[index]
2019 | 	if idx == nil {
2020 | 		// index was not found. return error
2021 | 		return ErrNotFound
2022 | 	}
2023 | 	if idx.rtr == nil {
2024 | 		// not an r-tree index. just return nil
2025 | 		return nil
2026 | 	}
2027 | 	// execute the search
2028 | 	var min, max []float64
2029 | 	if idx.rect != nil {
2030 | 		min, max = idx.rect(bounds)
2031 | 	}
2032 | 	idx.rtr.Search(&rect{min, max}, iter)
2033 | 	return nil
2034 | }
2035 | 
2036 | // Len returns the number of items in the database
2037 | func (tx *Tx) Len() (int, error) {
2038 | 	if tx.db == nil {
2039 | 		return 0, ErrTxClosed
2040 | 	}
2041 | 	return tx.db.keys.Len(), nil
2042 | }
2043 | 
2044 | // IndexOptions provides an index with additional features or
2045 | // alternate functionality.
2046 | type IndexOptions struct {
2047 | 	// CaseInsensitiveKeyMatching allow for case-insensitive
2048 | 	// matching on keys when setting key/values.
2049 | 	CaseInsensitiveKeyMatching bool
2050 | }
2051 | 
2052 | // CreateIndex builds a new index and populates it with items.
2053 | // The items are ordered in an b-tree and can be retrieved using the
2054 | // Ascend* and Descend* methods.
2055 | // An error will occur if an index with the same name already exists.
2056 | //
2057 | // When a pattern is provided, the index will be populated with
2058 | // keys that match the specified pattern. This is a very simple pattern
2059 | // match where '*' matches on any number characters and '?' matches on
2060 | // any one character.
2061 | // The less function compares if string 'a' is less than string 'b'.
2062 | // It allows for indexes to create custom ordering. It's possible
2063 | // that the strings may be textual or binary. It's up to the provided
2064 | // less function to handle the content format and comparison.
2065 | // There are some default less function that can be used such as
2066 | // IndexString, IndexBinary, etc.
2067 | func (tx *Tx) CreateIndex(name, pattern string,
2068 | 	less ...func(a, b string) bool) error {
2069 | 	return tx.createIndex(name, pattern, less, nil, nil)
2070 | }
2071 | 
2072 | // CreateIndexOptions is the same as CreateIndex except that it allows
2073 | // for additional options.
2074 | func (tx *Tx) CreateIndexOptions(name, pattern string,
2075 | 	opts *IndexOptions,
2076 | 	less ...func(a, b string) bool) error {
2077 | 	return tx.createIndex(name, pattern, less, nil, opts)
2078 | }
2079 | 
2080 | // CreateSpatialIndex builds a new index and populates it with items.
2081 | // The items are organized in an r-tree and can be retrieved using the
2082 | // Intersects method.
2083 | // An error will occur if an index with the same name already exists.
2084 | //
2085 | // The rect function converts a string to a rectangle. The rectangle is
2086 | // represented by two arrays, min and max. Both arrays may have a length
2087 | // between 1 and 20, and both arrays must match in length. A length of 1 is a
2088 | // one dimensional rectangle, and a length of 4 is a four dimension rectangle.
2089 | // There is support for up to 20 dimensions.
2090 | // The values of min must be less than the values of max at the same dimension.
2091 | // Thus min[0] must be less-than-or-equal-to max[0].
2092 | // The IndexRect is a default function that can be used for the rect
2093 | // parameter.
2094 | func (tx *Tx) CreateSpatialIndex(name, pattern string,
2095 | 	rect func(item string) (min, max []float64)) error {
2096 | 	return tx.createIndex(name, pattern, nil, rect, nil)
2097 | }
2098 | 
2099 | // CreateSpatialIndexOptions is the same as CreateSpatialIndex except that
2100 | // it allows for additional options.
2101 | func (tx *Tx) CreateSpatialIndexOptions(name, pattern string,
2102 | 	opts *IndexOptions,
2103 | 	rect func(item string) (min, max []float64)) error {
2104 | 	return tx.createIndex(name, pattern, nil, rect, nil)
2105 | }
2106 | 
2107 | // createIndex is called by CreateIndex() and CreateSpatialIndex()
2108 | func (tx *Tx) createIndex(name string, pattern string,
2109 | 	lessers []func(a, b string) bool,
2110 | 	rect func(item string) (min, max []float64),
2111 | 	opts *IndexOptions,
2112 | ) error {
2113 | 	if tx.db == nil {
2114 | 		return ErrTxClosed
2115 | 	} else if !tx.writable {
2116 | 		return ErrTxNotWritable
2117 | 	} else if tx.wc.itercount > 0 {
2118 | 		return ErrTxIterating
2119 | 	}
2120 | 	if name == "" {
2121 | 		// cannot create an index without a name.
2122 | 		// an empty name index is designated for the main "keys" tree.
2123 | 		return ErrIndexExists
2124 | 	}
2125 | 	// check if an index with that name already exists.
2126 | 	if _, ok := tx.db.idxs[name]; ok {
2127 | 		// index with name already exists. error.
2128 | 		return ErrIndexExists
2129 | 	}
2130 | 	// genreate a less function
2131 | 	var less func(a, b string) bool
2132 | 	switch len(lessers) {
2133 | 	default:
2134 | 		// multiple less functions specified.
2135 | 		// create a compound less function.
2136 | 		less = func(a, b string) bool {
2137 | 			for i := 0; i < len(lessers)-1; i++ {
2138 | 				if lessers[i](a, b) {
2139 | 					return true
2140 | 				}
2141 | 				if lessers[i](b, a) {
2142 | 					return false
2143 | 				}
2144 | 			}
2145 | 			return lessers[len(lessers)-1](a, b)
2146 | 		}
2147 | 	case 0:
2148 | 		// no less function
2149 | 	case 1:
2150 | 		less = lessers[0]
2151 | 	}
2152 | 	var sopts IndexOptions
2153 | 	if opts != nil {
2154 | 		sopts = *opts
2155 | 	}
2156 | 	if sopts.CaseInsensitiveKeyMatching {
2157 | 		pattern = strings.ToLower(pattern)
2158 | 	}
2159 | 	// intialize new index
2160 | 	idx := &index{
2161 | 		name:    name,
2162 | 		pattern: pattern,
2163 | 		less:    less,
2164 | 		rect:    rect,
2165 | 		db:      tx.db,
2166 | 		opts:    sopts,
2167 | 	}
2168 | 	idx.rebuild()
2169 | 	// save the index
2170 | 	tx.db.idxs[name] = idx
2171 | 	if tx.wc.rbkeys == nil {
2172 | 		// store the index in the rollback map.
2173 | 		if _, ok := tx.wc.rollbackIndexes[name]; !ok {
2174 | 			// we use nil to indicate that the index should be removed upon
2175 | 			// rollback.
2176 | 			tx.wc.rollbackIndexes[name] = nil
2177 | 		}
2178 | 	}
2179 | 	return nil
2180 | }
2181 | 
2182 | // DropIndex removes an index.
2183 | func (tx *Tx) DropIndex(name string) error {
2184 | 	if tx.db == nil {
2185 | 		return ErrTxClosed
2186 | 	} else if !tx.writable {
2187 | 		return ErrTxNotWritable
2188 | 	} else if tx.wc.itercount > 0 {
2189 | 		return ErrTxIterating
2190 | 	}
2191 | 	if name == "" {
2192 | 		// cannot drop the default "keys" index
2193 | 		return ErrInvalidOperation
2194 | 	}
2195 | 	idx, ok := tx.db.idxs[name]
2196 | 	if !ok {
2197 | 		return ErrNotFound
2198 | 	}
2199 | 	// delete from the map.
2200 | 	// this is all that is needed to delete an index.
2201 | 	delete(tx.db.idxs, name)
2202 | 	if tx.wc.rbkeys == nil {
2203 | 		// store the index in the rollback map.
2204 | 		if _, ok := tx.wc.rollbackIndexes[name]; !ok {
2205 | 			// we use a non-nil copy of the index without the data to indicate
2206 | 			// that the index should be rebuilt upon rollback.
2207 | 			tx.wc.rollbackIndexes[name] = idx.clearCopy()
2208 | 		}
2209 | 	}
2210 | 	return nil
2211 | }
2212 | 
2213 | // Indexes returns a list of index names.
2214 | func (tx *Tx) Indexes() ([]string, error) {
2215 | 	if tx.db == nil {
2216 | 		return nil, ErrTxClosed
2217 | 	}
2218 | 	names := make([]string, 0, len(tx.db.idxs))
2219 | 	for name := range tx.db.idxs {
2220 | 		names = append(names, name)
2221 | 	}
2222 | 	sort.Strings(names)
2223 | 	return names, nil
2224 | }
2225 | 
2226 | // Rect is helper function that returns a string representation
2227 | // of a rect. IndexRect() is the reverse function and can be used
2228 | // to generate a rect from a string.
2229 | func Rect(min, max []float64) string {
2230 | 	r := grect.Rect{Min: min, Max: max}
2231 | 	return r.String()
2232 | }
2233 | 
2234 | // Point is a helper function that converts a series of float64s
2235 | // to a rectangle for a spatial index.
2236 | func Point(coords ...float64) string {
2237 | 	return Rect(coords, coords)
2238 | }
2239 | 
2240 | // IndexRect is a helper function that converts string to a rect.
2241 | // Rect() is the reverse function and can be used to generate a string
2242 | // from a rect.
2243 | func IndexRect(a string) (min, max []float64) {
2244 | 	r := grect.Get(a)
2245 | 	return r.Min, r.Max
2246 | }
2247 | 
2248 | // IndexString is a helper function that return true if 'a' is less than 'b'.
2249 | // This is a case-insensitive comparison. Use the IndexBinary() for comparing
2250 | // case-sensitive strings.
2251 | func IndexString(a, b string) bool {
2252 | 	for i := 0; i < len(a) && i < len(b); i++ {
2253 | 		if a[i] >= 'A' && a[i] <= 'Z' {
2254 | 			if b[i] >= 'A' && b[i] <= 'Z' {
2255 | 				// both are uppercase, do nothing
2256 | 				if a[i] < b[i] {
2257 | 					return true
2258 | 				} else if a[i] > b[i] {
2259 | 					return false
2260 | 				}
2261 | 			} else {
2262 | 				// a is uppercase, convert a to lowercase
2263 | 				if a[i]+32 < b[i] {
2264 | 					return true
2265 | 				} else if a[i]+32 > b[i] {
2266 | 					return false
2267 | 				}
2268 | 			}
2269 | 		} else if b[i] >= 'A' && b[i] <= 'Z' {
2270 | 			// b is uppercase, convert b to lowercase
2271 | 			if a[i] < b[i]+32 {
2272 | 				return true
2273 | 			} else if a[i] > b[i]+32 {
2274 | 				return false
2275 | 			}
2276 | 		} else {
2277 | 			// neither are uppercase
2278 | 			if a[i] < b[i] {
2279 | 				return true
2280 | 			} else if a[i] > b[i] {
2281 | 				return false
2282 | 			}
2283 | 		}
2284 | 	}
2285 | 	return len(a) < len(b)
2286 | }
2287 | 
2288 | // IndexBinary is a helper function that returns true if 'a' is less than 'b'.
2289 | // This compares the raw binary of the string.
2290 | func IndexBinary(a, b string) bool {
2291 | 	return a < b
2292 | }
2293 | 
2294 | // IndexInt is a helper function that returns true if 'a' is less than 'b'.
2295 | func IndexInt(a, b string) bool {
2296 | 	ia, _ := strconv.ParseInt(a, 10, 64)
2297 | 	ib, _ := strconv.ParseInt(b, 10, 64)
2298 | 	return ia < ib
2299 | }
2300 | 
2301 | // IndexUint is a helper function that returns true if 'a' is less than 'b'.
2302 | // This compares uint64s that are added to the database using the
2303 | // Uint() conversion function.
2304 | func IndexUint(a, b string) bool {
2305 | 	ia, _ := strconv.ParseUint(a, 10, 64)
2306 | 	ib, _ := strconv.ParseUint(b, 10, 64)
2307 | 	return ia < ib
2308 | }
2309 | 
2310 | // IndexFloat is a helper function that returns true if 'a' is less than 'b'.
2311 | // This compares float64s that are added to the database using the
2312 | // Float() conversion function.
2313 | func IndexFloat(a, b string) bool {
2314 | 	ia, _ := strconv.ParseFloat(a, 64)
2315 | 	ib, _ := strconv.ParseFloat(b, 64)
2316 | 	return ia < ib
2317 | }
2318 | 
2319 | // IndexJSON provides for the ability to create an index on any JSON field.
2320 | // When the field is a string, the comparison will be case-insensitive.
2321 | // It returns a helper function used by CreateIndex.
2322 | func IndexJSON(path string) func(a, b string) bool {
2323 | 	return func(a, b string) bool {
2324 | 		return gjson.Get(a, path).Less(gjson.Get(b, path), false)
2325 | 	}
2326 | }
2327 | 
2328 | // IndexJSONCaseSensitive provides for the ability to create an index on
2329 | // any JSON field.
2330 | // When the field is a string, the comparison will be case-sensitive.
2331 | // It returns a helper function used by CreateIndex.
2332 | func IndexJSONCaseSensitive(path string) func(a, b string) bool {
2333 | 	return func(a, b string) bool {
2334 | 		return gjson.Get(a, path).Less(gjson.Get(b, path), true)
2335 | 	}
2336 | }
2337 | 
2338 | // Desc is a helper function that changes the order of an index.
2339 | func Desc(less func(a, b string) bool) func(a, b string) bool {
2340 | 	return func(a, b string) bool { return less(b, a) }
2341 | }
2342 | 
2343 | //// Wrappers around btree Ascend/Descend
2344 | 
2345 | func bLT(tr *btree.BTree, a, b interface{}) bool { return tr.Less(a, b) }
2346 | func bGT(tr *btree.BTree, a, b interface{}) bool { return tr.Less(b, a) }
2347 | 
2348 | // func bLTE(tr *btree.BTree, a, b interface{}) bool { return !tr.Less(b, a) }
2349 | // func bGTE(tr *btree.BTree, a, b interface{}) bool { return !tr.Less(a, b) }
2350 | 
2351 | // Ascend
2352 | 
2353 | func btreeAscend(tr *btree.BTree, iter func(item interface{}) bool) {
2354 | 	tr.Ascend(nil, iter)
2355 | }
2356 | 
2357 | func btreeAscendLessThan(tr *btree.BTree, pivot interface{},
2358 | 	iter func(item interface{}) bool,
2359 | ) {
2360 | 	tr.Ascend(nil, func(item interface{}) bool {
2361 | 		return bLT(tr, item, pivot) && iter(item)
2362 | 	})
2363 | }
2364 | 
2365 | func btreeAscendGreaterOrEqual(tr *btree.BTree, pivot interface{},
2366 | 	iter func(item interface{}) bool,
2367 | ) {
2368 | 	tr.Ascend(pivot, iter)
2369 | }
2370 | 
2371 | func btreeAscendRange(tr *btree.BTree, greaterOrEqual, lessThan interface{},
2372 | 	iter func(item interface{}) bool,
2373 | ) {
2374 | 	tr.Ascend(greaterOrEqual, func(item interface{}) bool {
2375 | 		return bLT(tr, item, lessThan) && iter(item)
2376 | 	})
2377 | }
2378 | 
2379 | // Descend
2380 | 
2381 | func btreeDescend(tr *btree.BTree, iter func(item interface{}) bool) {
2382 | 	tr.Descend(nil, iter)
2383 | }
2384 | 
2385 | func btreeDescendGreaterThan(tr *btree.BTree, pivot interface{},
2386 | 	iter func(item interface{}) bool,
2387 | ) {
2388 | 	tr.Descend(nil, func(item interface{}) bool {
2389 | 		return bGT(tr, item, pivot) && iter(item)
2390 | 	})
2391 | }
2392 | 
2393 | func btreeDescendRange(tr *btree.BTree, lessOrEqual, greaterThan interface{},
2394 | 	iter func(item interface{}) bool,
2395 | ) {
2396 | 	tr.Descend(lessOrEqual, func(item interface{}) bool {
2397 | 		return bGT(tr, item, greaterThan) && iter(item)
2398 | 	})
2399 | }
2400 | 
2401 | func btreeDescendLessOrEqual(tr *btree.BTree, pivot interface{},
2402 | 	iter func(item interface{}) bool,
2403 | ) {
2404 | 	tr.Descend(pivot, iter)
2405 | }
2406 | 
2407 | func btreeNew(less func(a, b interface{}) bool) *btree.BTree {
2408 | 	// Using NewNonConcurrent because we're managing our own locks.
2409 | 	return btree.NewNonConcurrent(less)
2410 | }
2411 | 


--------------------------------------------------------------------------------
/buntdb_test.go:
--------------------------------------------------------------------------------
   1 | package buntdb
   2 | 
   3 | import (
   4 | 	"bytes"
   5 | 	"errors"
   6 | 	"fmt"
   7 | 	"io/ioutil"
   8 | 	"math/rand"
   9 | 	"os"
  10 | 	"strconv"
  11 | 	"strings"
  12 | 	"sync"
  13 | 	"testing"
  14 | 	"time"
  15 | 
  16 | 	"github.com/tidwall/assert"
  17 | 	"github.com/tidwall/lotsa"
  18 | )
  19 | 
  20 | func testOpen(t testing.TB) *DB {
  21 | 	if err := os.RemoveAll("data.db"); err != nil {
  22 | 		t.Fatal(err)
  23 | 	}
  24 | 	return testReOpen(t, nil)
  25 | }
  26 | func testReOpen(t testing.TB, db *DB) *DB {
  27 | 	return testReOpenDelay(t, db, 0)
  28 | }
  29 | 
  30 | func testReOpenDelay(t testing.TB, db *DB, dur time.Duration) *DB {
  31 | 	if db != nil {
  32 | 		if err := db.Close(); err != nil {
  33 | 			t.Fatal(err)
  34 | 		}
  35 | 	}
  36 | 	time.Sleep(dur)
  37 | 	db, err := Open("data.db")
  38 | 	if err != nil {
  39 | 		t.Fatal(err)
  40 | 	}
  41 | 	return db
  42 | }
  43 | 
  44 | func testClose(db *DB) {
  45 | 	_ = db.Close()
  46 | 	_ = os.RemoveAll("data.db")
  47 | }
  48 | 
  49 | func TestBackgroundOperations(t *testing.T) {
  50 | 	db := testOpen(t)
  51 | 	defer testClose(db)
  52 | 	for i := 0; i < 1000; i++ {
  53 | 		if err := db.Update(func(tx *Tx) error {
  54 | 			for j := 0; j < 200; j++ {
  55 | 				if _, _, err := tx.Set(fmt.Sprintf("hello%d", j), "planet", nil); err != nil {
  56 | 					return err
  57 | 				}
  58 | 			}
  59 | 			if _, _, err := tx.Set("hi", "world", &SetOptions{Expires: true, TTL: time.Second / 2}); err != nil {
  60 | 				return err
  61 | 			}
  62 | 			return nil
  63 | 		}); err != nil {
  64 | 			t.Fatal(err)
  65 | 		}
  66 | 	}
  67 | 	n := 0
  68 | 	err := db.View(func(tx *Tx) error {
  69 | 		var err error
  70 | 		n, err = tx.Len()
  71 | 		return err
  72 | 	})
  73 | 	if err != nil {
  74 | 		t.Fatal(err)
  75 | 	}
  76 | 	if n != 201 {
  77 | 		t.Fatalf("expecting '%v', got '%v'", 201, n)
  78 | 	}
  79 | 	time.Sleep(time.Millisecond * 1500)
  80 | 	db = testReOpen(t, db)
  81 | 	defer testClose(db)
  82 | 	n = 0
  83 | 	err = db.View(func(tx *Tx) error {
  84 | 		var err error
  85 | 		n, err = tx.Len()
  86 | 		return err
  87 | 	})
  88 | 	if err != nil {
  89 | 		t.Fatal(err)
  90 | 	}
  91 | 	if n != 200 {
  92 | 		t.Fatalf("expecting '%v', got '%v'", 200, n)
  93 | 	}
  94 | }
  95 | func TestSaveLoad(t *testing.T) {
  96 | 	db, _ := Open(":memory:")
  97 | 	defer db.Close()
  98 | 	if err := db.Update(func(tx *Tx) error {
  99 | 		for i := 0; i < 20; i++ {
 100 | 			_, _, err := tx.Set(fmt.Sprintf("key:%d", i), fmt.Sprintf("planet:%d", i), nil)
 101 | 			if err != nil {
 102 | 				return err
 103 | 			}
 104 | 		}
 105 | 		return nil
 106 | 	}); err != nil {
 107 | 		t.Fatal(err)
 108 | 	}
 109 | 	f, err := os.Create("temp.db")
 110 | 	if err != nil {
 111 | 		t.Fatal(err)
 112 | 	}
 113 | 	defer func() {
 114 | 		f.Close()
 115 | 		os.RemoveAll("temp.db")
 116 | 	}()
 117 | 	if err := db.Save(f); err != nil {
 118 | 		t.Fatal(err)
 119 | 	}
 120 | 	if err := f.Close(); err != nil {
 121 | 		t.Fatal(err)
 122 | 	}
 123 | 	db.Close()
 124 | 	db, _ = Open(":memory:")
 125 | 	defer db.Close()
 126 | 	f, err = os.Open("temp.db")
 127 | 	if err != nil {
 128 | 		t.Fatal(err)
 129 | 	}
 130 | 	defer f.Close()
 131 | 	if err := db.Load(f); err != nil {
 132 | 		t.Fatal(err)
 133 | 	}
 134 | 	if err := db.View(func(tx *Tx) error {
 135 | 		for i := 0; i < 20; i++ {
 136 | 			ex := fmt.Sprintf("planet:%d", i)
 137 | 			val, err := tx.Get(fmt.Sprintf("key:%d", i))
 138 | 			if err != nil {
 139 | 				return err
 140 | 			}
 141 | 			if ex != val {
 142 | 				t.Fatalf("expected %s, got %s", ex, val)
 143 | 			}
 144 | 		}
 145 | 		return nil
 146 | 	}); err != nil {
 147 | 		t.Fatal(err)
 148 | 	}
 149 | }
 150 | 
 151 | func TestMutatingIterator(t *testing.T) {
 152 | 	db := testOpen(t)
 153 | 	defer testClose(db)
 154 | 	count := 1000
 155 | 	if err := db.CreateIndex("ages", "user:*:age", IndexInt); err != nil {
 156 | 		t.Fatal(err)
 157 | 	}
 158 | 
 159 | 	for i := 0; i < 10; i++ {
 160 | 		if err := db.Update(func(tx *Tx) error {
 161 | 			for j := 0; j < count; j++ {
 162 | 				key := fmt.Sprintf("user:%d:age", j)
 163 | 				val := fmt.Sprintf("%d", rand.Intn(100))
 164 | 				if _, _, err := tx.Set(key, val, nil); err != nil {
 165 | 					return err
 166 | 				}
 167 | 			}
 168 | 			return nil
 169 | 		}); err != nil {
 170 | 			t.Fatal(err)
 171 | 		}
 172 | 
 173 | 		if err := db.Update(func(tx *Tx) error {
 174 | 			return tx.Ascend("ages", func(key, val string) bool {
 175 | 				_, err := tx.Delete(key)
 176 | 				if err != ErrTxIterating {
 177 | 					t.Fatal("should not be able to call Delete while iterating.")
 178 | 				}
 179 | 				_, _, err = tx.Set(key, "", nil)
 180 | 				if err != ErrTxIterating {
 181 | 					t.Fatal("should not be able to call Set while iterating.")
 182 | 				}
 183 | 				return true
 184 | 			})
 185 | 		}); err != nil {
 186 | 			t.Fatal(err)
 187 | 		}
 188 | 	}
 189 | }
 190 | 
 191 | func TestCaseInsensitiveIndex(t *testing.T) {
 192 | 	db := testOpen(t)
 193 | 	defer testClose(db)
 194 | 	count := 1000
 195 | 	if err := db.Update(func(tx *Tx) error {
 196 | 		opts := &IndexOptions{
 197 | 			CaseInsensitiveKeyMatching: true,
 198 | 		}
 199 | 		return tx.CreateIndexOptions("ages", "User:*:age", opts, IndexInt)
 200 | 	}); err != nil {
 201 | 		t.Fatal(err)
 202 | 	}
 203 | 
 204 | 	if err := db.Update(func(tx *Tx) error {
 205 | 		for j := 0; j < count; j++ {
 206 | 			key := fmt.Sprintf("user:%d:age", j)
 207 | 			val := fmt.Sprintf("%d", rand.Intn(100))
 208 | 			if _, _, err := tx.Set(key, val, nil); err != nil {
 209 | 				return err
 210 | 			}
 211 | 		}
 212 | 		return nil
 213 | 	}); err != nil {
 214 | 		t.Fatal(err)
 215 | 	}
 216 | 
 217 | 	if err := db.View(func(tx *Tx) error {
 218 | 		var vals []string
 219 | 		err := tx.Ascend("ages", func(key, value string) bool {
 220 | 			vals = append(vals, value)
 221 | 			return true
 222 | 		})
 223 | 		if err != nil {
 224 | 			return err
 225 | 		}
 226 | 		if len(vals) != count {
 227 | 			return fmt.Errorf("expected '%v', got '%v'", count, len(vals))
 228 | 		}
 229 | 		return nil
 230 | 	}); err != nil {
 231 | 		t.Fatal(err)
 232 | 	}
 233 | 
 234 | }
 235 | 
 236 | func TestIndexTransaction(t *testing.T) {
 237 | 	db := testOpen(t)
 238 | 	defer testClose(db)
 239 | 	var errFine = errors.New("this is fine")
 240 | 	ascend := func(tx *Tx, index string) ([]string, error) {
 241 | 		var vals []string
 242 | 		if err := tx.Ascend(index, func(key, val string) bool {
 243 | 			vals = append(vals, key, val)
 244 | 			return true
 245 | 		}); err != nil {
 246 | 			return nil, err
 247 | 		}
 248 | 		return vals, nil
 249 | 	}
 250 | 	ascendEqual := func(tx *Tx, index string, vals []string) error {
 251 | 		vals2, err := ascend(tx, index)
 252 | 		if err != nil {
 253 | 			return err
 254 | 		}
 255 | 		if len(vals) != len(vals2) {
 256 | 			return errors.New("invalid size match")
 257 | 		}
 258 | 		for i := 0; i < len(vals); i++ {
 259 | 			if vals[i] != vals2[i] {
 260 | 				return errors.New("invalid order")
 261 | 			}
 262 | 		}
 263 | 		return nil
 264 | 	}
 265 | 	// test creating an index and adding items
 266 | 	if err := db.Update(func(tx *Tx) error {
 267 | 		tx.Set("1", "3", nil)
 268 | 		tx.Set("2", "2", nil)
 269 | 		tx.Set("3", "1", nil)
 270 | 		if err := tx.CreateIndex("idx1", "*", IndexInt); err != nil {
 271 | 			return err
 272 | 		}
 273 | 		if err := ascendEqual(tx, "idx1", []string{"3", "1", "2", "2", "1", "3"}); err != nil {
 274 | 			return err
 275 | 		}
 276 | 		return nil
 277 | 	}); err != nil {
 278 | 		t.Fatal(err)
 279 | 	}
 280 | 
 281 | 	// test to see if the items persisted from previous transaction
 282 | 	// test add item.
 283 | 	// test force rollback.
 284 | 	if err := db.Update(func(tx *Tx) error {
 285 | 		if err := ascendEqual(tx, "idx1", []string{"3", "1", "2", "2", "1", "3"}); err != nil {
 286 | 			return err
 287 | 		}
 288 | 		tx.Set("4", "0", nil)
 289 | 		if err := ascendEqual(tx, "idx1", []string{"4", "0", "3", "1", "2", "2", "1", "3"}); err != nil {
 290 | 			return err
 291 | 		}
 292 | 		return errFine
 293 | 	}); err != errFine {
 294 | 		t.Fatalf("expected '%v', got '%v'", errFine, err)
 295 | 	}
 296 | 
 297 | 	// test to see if the rollback happened
 298 | 	if err := db.View(func(tx *Tx) error {
 299 | 		if err := ascendEqual(tx, "idx1", []string{"3", "1", "2", "2", "1", "3"}); err != nil {
 300 | 			return err
 301 | 		}
 302 | 		return nil
 303 | 	}); err != nil {
 304 | 		t.Fatalf("expected '%v', got '%v'", nil, err)
 305 | 	}
 306 | 
 307 | 	// del item, drop index, rollback
 308 | 	if err := db.Update(func(tx *Tx) error {
 309 | 		if err := tx.DropIndex("idx1"); err != nil {
 310 | 			return err
 311 | 		}
 312 | 		return errFine
 313 | 	}); err != errFine {
 314 | 		t.Fatalf("expected '%v', got '%v'", errFine, err)
 315 | 	}
 316 | 
 317 | 	// test to see if the rollback happened
 318 | 	if err := db.View(func(tx *Tx) error {
 319 | 		if err := ascendEqual(tx, "idx1", []string{"3", "1", "2", "2", "1", "3"}); err != nil {
 320 | 			return err
 321 | 		}
 322 | 		return nil
 323 | 	}); err != nil {
 324 | 		t.Fatalf("expected '%v', got '%v'", nil, err)
 325 | 	}
 326 | 
 327 | 	various := func(reterr error) error {
 328 | 		// del item 3, add index 2, add item 4, test index 1 and 2.
 329 | 		// flushdb, test index 1 and 2.
 330 | 		// add item 1 and 2, add index 2 and 3, test index 2 and 3
 331 | 		return db.Update(func(tx *Tx) error {
 332 | 			tx.Delete("3")
 333 | 			tx.CreateIndex("idx2", "*", IndexInt)
 334 | 			tx.Set("4", "0", nil)
 335 | 			if err := ascendEqual(tx, "idx1", []string{"4", "0", "2", "2", "1", "3"}); err != nil {
 336 | 				return fmt.Errorf("err: %v", err)
 337 | 			}
 338 | 			if err := ascendEqual(tx, "idx2", []string{"4", "0", "2", "2", "1", "3"}); err != nil {
 339 | 				return fmt.Errorf("err: %v", err)
 340 | 			}
 341 | 			tx.DeleteAll()
 342 | 			if err := ascendEqual(tx, "idx1", []string{}); err != nil {
 343 | 				return fmt.Errorf("err: %v", err)
 344 | 			}
 345 | 			if err := ascendEqual(tx, "idx2", []string{}); err != nil {
 346 | 				return fmt.Errorf("err: %v", err)
 347 | 			}
 348 | 			tx.Set("1", "3", nil)
 349 | 			tx.Set("2", "2", nil)
 350 | 			tx.CreateIndex("idx1", "*", IndexInt)
 351 | 			tx.CreateIndex("idx2", "*", IndexInt)
 352 | 			if err := ascendEqual(tx, "idx1", []string{"2", "2", "1", "3"}); err != nil {
 353 | 				return fmt.Errorf("err: %v", err)
 354 | 			}
 355 | 			if err := ascendEqual(tx, "idx2", []string{"2", "2", "1", "3"}); err != nil {
 356 | 				return fmt.Errorf("err: %v", err)
 357 | 			}
 358 | 			return reterr
 359 | 		})
 360 | 	}
 361 | 	// various rollback
 362 | 	if err := various(errFine); err != errFine {
 363 | 		t.Fatalf("expected '%v', got '%v'", errFine, err)
 364 | 	}
 365 | 	// test to see if the rollback happened
 366 | 	if err := db.View(func(tx *Tx) error {
 367 | 		if err := ascendEqual(tx, "idx1", []string{"3", "1", "2", "2", "1", "3"}); err != nil {
 368 | 			return fmt.Errorf("err: %v", err)
 369 | 		}
 370 | 		if err := ascendEqual(tx, "idx2", []string{"3", "1", "2", "2", "1", "3"}); err != ErrNotFound {
 371 | 			return fmt.Errorf("err: %v", err)
 372 | 		}
 373 | 		return nil
 374 | 	}); err != nil {
 375 | 		t.Fatalf("expected '%v', got '%v'", nil, err)
 376 | 	}
 377 | 
 378 | 	// various commit
 379 | 	if err := various(nil); err != nil {
 380 | 		t.Fatalf("expected '%v', got '%v'", nil, err)
 381 | 	}
 382 | 
 383 | 	// test to see if the commit happened
 384 | 	if err := db.View(func(tx *Tx) error {
 385 | 		if err := ascendEqual(tx, "idx1", []string{"2", "2", "1", "3"}); err != nil {
 386 | 			return fmt.Errorf("err: %v", err)
 387 | 		}
 388 | 		if err := ascendEqual(tx, "idx2", []string{"2", "2", "1", "3"}); err != nil {
 389 | 			return fmt.Errorf("err: %v", err)
 390 | 		}
 391 | 		return nil
 392 | 	}); err != nil {
 393 | 		t.Fatalf("expected '%v', got '%v'", nil, err)
 394 | 	}
 395 | }
 396 | 
 397 | func TestDeleteAll(t *testing.T) {
 398 | 	db := testOpen(t)
 399 | 	defer testClose(db)
 400 | 
 401 | 	db.Update(func(tx *Tx) error {
 402 | 		tx.Set("hello1", "planet1", nil)
 403 | 		tx.Set("hello2", "planet2", nil)
 404 | 		tx.Set("hello3", "planet3", nil)
 405 | 		return nil
 406 | 	})
 407 | 	db.CreateIndex("all", "*", IndexString)
 408 | 	db.Update(func(tx *Tx) error {
 409 | 		tx.Set("hello1", "planet1.1", nil)
 410 | 		tx.DeleteAll()
 411 | 		tx.Set("bb", "11", nil)
 412 | 		tx.Set("aa", "**", nil)
 413 | 		tx.Delete("aa")
 414 | 		tx.Set("aa", "22", nil)
 415 | 		return nil
 416 | 	})
 417 | 	var res string
 418 | 	var res2 string
 419 | 	db.View(func(tx *Tx) error {
 420 | 		tx.Ascend("", func(key, val string) bool {
 421 | 			res += key + ":" + val + "\n"
 422 | 			return true
 423 | 		})
 424 | 		tx.Ascend("all", func(key, val string) bool {
 425 | 			res2 += key + ":" + val + "\n"
 426 | 			return true
 427 | 		})
 428 | 		return nil
 429 | 	})
 430 | 	if res != "aa:22\nbb:11\n" {
 431 | 		t.Fatal("fail")
 432 | 	}
 433 | 	if res2 != "bb:11\naa:22\n" {
 434 | 		t.Fatal("fail")
 435 | 	}
 436 | 	db = testReOpen(t, db)
 437 | 	defer testClose(db)
 438 | 	res = ""
 439 | 	res2 = ""
 440 | 	db.CreateIndex("all", "*", IndexString)
 441 | 	db.View(func(tx *Tx) error {
 442 | 		tx.Ascend("", func(key, val string) bool {
 443 | 			res += key + ":" + val + "\n"
 444 | 			return true
 445 | 		})
 446 | 		tx.Ascend("all", func(key, val string) bool {
 447 | 			res2 += key + ":" + val + "\n"
 448 | 			return true
 449 | 		})
 450 | 		return nil
 451 | 	})
 452 | 	if res != "aa:22\nbb:11\n" {
 453 | 		t.Fatal("fail")
 454 | 	}
 455 | 	if res2 != "bb:11\naa:22\n" {
 456 | 		t.Fatal("fail")
 457 | 	}
 458 | 	db.Update(func(tx *Tx) error {
 459 | 		tx.Set("1", "1", nil)
 460 | 		tx.Set("2", "2", nil)
 461 | 		tx.Set("3", "3", nil)
 462 | 		tx.Set("4", "4", nil)
 463 | 		return nil
 464 | 	})
 465 | 	err := db.Update(func(tx *Tx) error {
 466 | 		tx.Set("1", "a", nil)
 467 | 		tx.Set("5", "5", nil)
 468 | 		tx.Delete("2")
 469 | 		tx.Set("6", "6", nil)
 470 | 		tx.DeleteAll()
 471 | 		tx.Set("7", "7", nil)
 472 | 		tx.Set("8", "8", nil)
 473 | 		tx.Set("6", "c", nil)
 474 | 		return errors.New("please rollback")
 475 | 	})
 476 | 	if err == nil || err.Error() != "please rollback" {
 477 | 		t.Fatal("expecteding 'please rollback' error")
 478 | 	}
 479 | 
 480 | 	res = ""
 481 | 	res2 = ""
 482 | 	db.View(func(tx *Tx) error {
 483 | 		tx.Ascend("", func(key, val string) bool {
 484 | 			res += key + ":" + val + "\n"
 485 | 			return true
 486 | 		})
 487 | 		tx.Ascend("all", func(key, val string) bool {
 488 | 			res2 += key + ":" + val + "\n"
 489 | 			return true
 490 | 		})
 491 | 		return nil
 492 | 	})
 493 | 	if res != "1:1\n2:2\n3:3\n4:4\naa:22\nbb:11\n" {
 494 | 		t.Fatal("fail")
 495 | 	}
 496 | 	if res2 != "1:1\nbb:11\n2:2\naa:22\n3:3\n4:4\n" {
 497 | 		t.Fatal("fail")
 498 | 	}
 499 | }
 500 | 
 501 | func TestAscendEqual(t *testing.T) {
 502 | 	db := testOpen(t)
 503 | 	defer testClose(db)
 504 | 	if err := db.Update(func(tx *Tx) error {
 505 | 		for i := 0; i < 300; i++ {
 506 | 			_, _, err := tx.Set(fmt.Sprintf("key:%05dA", i), fmt.Sprintf("%d", i+1000), nil)
 507 | 			if err != nil {
 508 | 				return err
 509 | 			}
 510 | 			_, _, err = tx.Set(fmt.Sprintf("key:%05dB", i), fmt.Sprintf("%d", i+1000), nil)
 511 | 			if err != nil {
 512 | 				return err
 513 | 			}
 514 | 		}
 515 | 		return tx.CreateIndex("num", "*", IndexInt)
 516 | 	}); err != nil {
 517 | 		t.Fatal(err)
 518 | 	}
 519 | 	var res []string
 520 | 	if err := db.View(func(tx *Tx) error {
 521 | 		return tx.AscendEqual("", "key:00055A", func(key, value string) bool {
 522 | 			res = append(res, key)
 523 | 			return true
 524 | 		})
 525 | 	}); err != nil {
 526 | 		t.Fatal(err)
 527 | 	}
 528 | 	if len(res) != 1 {
 529 | 		t.Fatalf("expected %v, got %v", 1, len(res))
 530 | 	}
 531 | 	if res[0] != "key:00055A" {
 532 | 		t.Fatalf("expected %v, got %v", "key:00055A", res[0])
 533 | 	}
 534 | 	res = nil
 535 | 	if err := db.View(func(tx *Tx) error {
 536 | 		return tx.AscendEqual("num", "1125", func(key, value string) bool {
 537 | 			res = append(res, key)
 538 | 			return true
 539 | 		})
 540 | 	}); err != nil {
 541 | 		t.Fatal(err)
 542 | 	}
 543 | 	if len(res) != 2 {
 544 | 		t.Fatalf("expected %v, got %v", 2, len(res))
 545 | 	}
 546 | 	if res[0] != "key:00125A" {
 547 | 		t.Fatalf("expected %v, got %v", "key:00125A", res[0])
 548 | 	}
 549 | 	if res[1] != "key:00125B" {
 550 | 		t.Fatalf("expected %v, got %v", "key:00125B", res[1])
 551 | 	}
 552 | }
 553 | func TestDescendEqual(t *testing.T) {
 554 | 	db := testOpen(t)
 555 | 	defer testClose(db)
 556 | 	if err := db.Update(func(tx *Tx) error {
 557 | 		for i := 0; i < 300; i++ {
 558 | 			_, _, err := tx.Set(fmt.Sprintf("key:%05dA", i), fmt.Sprintf("%d", i+1000), nil)
 559 | 			if err != nil {
 560 | 				return err
 561 | 			}
 562 | 			_, _, err = tx.Set(fmt.Sprintf("key:%05dB", i), fmt.Sprintf("%d", i+1000), nil)
 563 | 			if err != nil {
 564 | 				return err
 565 | 			}
 566 | 		}
 567 | 		return tx.CreateIndex("num", "*", IndexInt)
 568 | 	}); err != nil {
 569 | 		t.Fatal(err)
 570 | 	}
 571 | 	var res []string
 572 | 	if err := db.View(func(tx *Tx) error {
 573 | 		return tx.DescendEqual("", "key:00055A", func(key, value string) bool {
 574 | 			res = append(res, key)
 575 | 			return true
 576 | 		})
 577 | 	}); err != nil {
 578 | 		t.Fatal(err)
 579 | 	}
 580 | 	if len(res) != 1 {
 581 | 		t.Fatalf("expected %v, got %v", 1, len(res))
 582 | 	}
 583 | 	if res[0] != "key:00055A" {
 584 | 		t.Fatalf("expected %v, got %v", "key:00055A", res[0])
 585 | 	}
 586 | 	res = nil
 587 | 	if err := db.View(func(tx *Tx) error {
 588 | 		return tx.DescendEqual("num", "1125", func(key, value string) bool {
 589 | 			res = append(res, key)
 590 | 			return true
 591 | 		})
 592 | 	}); err != nil {
 593 | 		t.Fatal(err)
 594 | 	}
 595 | 	if len(res) != 2 {
 596 | 		t.Fatalf("expected %v, got %v", 2, len(res))
 597 | 	}
 598 | 	if res[0] != "key:00125B" {
 599 | 		t.Fatalf("expected %v, got %v", "key:00125B", res[0])
 600 | 	}
 601 | 	if res[1] != "key:00125A" {
 602 | 		t.Fatalf("expected %v, got %v", "key:00125A", res[1])
 603 | 	}
 604 | }
 605 | func TestVariousTx(t *testing.T) {
 606 | 	db := testOpen(t)
 607 | 	defer testClose(db)
 608 | 	if err := db.Update(func(tx *Tx) error {
 609 | 		_, _, err := tx.Set("hello", "planet", nil)
 610 | 		return err
 611 | 	}); err != nil {
 612 | 		t.Fatal(err)
 613 | 	}
 614 | 	errBroken := errors.New("broken")
 615 | 	if err := db.Update(func(tx *Tx) error {
 616 | 		_, _, _ = tx.Set("hello", "world", nil)
 617 | 		return errBroken
 618 | 	}); err != errBroken {
 619 | 		t.Fatalf("did not correctly receive the user-defined transaction error.")
 620 | 	}
 621 | 	var val string
 622 | 	err := db.View(func(tx *Tx) error {
 623 | 		var err error
 624 | 		val, err = tx.Get("hello")
 625 | 		return err
 626 | 	})
 627 | 	if err != nil {
 628 | 		t.Fatal(err)
 629 | 	}
 630 | 	if val == "world" {
 631 | 		t.Fatal("a rollbacked transaction got through")
 632 | 	}
 633 | 	if val != "planet" {
 634 | 		t.Fatalf("expecting '%v', got '%v'", "planet", val)
 635 | 	}
 636 | 	if err := db.Update(func(tx *Tx) error {
 637 | 		tx.db = nil
 638 | 		if _, _, err := tx.Set("hello", "planet", nil); err != ErrTxClosed {
 639 | 			t.Fatal("expecting a tx closed error")
 640 | 		}
 641 | 		if _, err := tx.Delete("hello"); err != ErrTxClosed {
 642 | 			t.Fatal("expecting a tx closed error")
 643 | 		}
 644 | 		if _, err := tx.Get("hello"); err != ErrTxClosed {
 645 | 			t.Fatal("expecting a tx closed error")
 646 | 		}
 647 | 		tx.db = db
 648 | 		tx.writable = false
 649 | 		if _, _, err := tx.Set("hello", "planet", nil); err != ErrTxNotWritable {
 650 | 			t.Fatal("expecting a tx not writable error")
 651 | 		}
 652 | 		if _, err := tx.Delete("hello"); err != ErrTxNotWritable {
 653 | 			t.Fatal("expecting a tx not writable error")
 654 | 		}
 655 | 		tx.writable = true
 656 | 		if _, err := tx.Get("something"); err != ErrNotFound {
 657 | 			t.Fatalf("expecting not found error")
 658 | 		}
 659 | 		if _, err := tx.Delete("something"); err != ErrNotFound {
 660 | 			t.Fatalf("expecting not found error")
 661 | 		}
 662 | 		if _, _, err := tx.Set("var", "val", &SetOptions{Expires: true, TTL: 0}); err != nil {
 663 | 			t.Fatal(err)
 664 | 		}
 665 | 		if _, err := tx.Get("var"); err != ErrNotFound {
 666 | 			t.Fatalf("expecting not found error")
 667 | 		}
 668 | 		if _, err := tx.Delete("var"); err != ErrNotFound {
 669 | 			tx.unlock()
 670 | 			t.Fatalf("expecting not found error")
 671 | 		}
 672 | 		return nil
 673 | 	}); err != nil {
 674 | 		t.Fatal(err)
 675 | 	}
 676 | 
 677 | 	// test non-managed transactions
 678 | 	tx, err := db.Begin(true)
 679 | 	if err != nil {
 680 | 		t.Fatal(err)
 681 | 	}
 682 | 	defer tx.Rollback()
 683 | 	_, _, err = tx.Set("howdy", "world", nil)
 684 | 	if err != nil {
 685 | 		t.Fatal(err)
 686 | 	}
 687 | 	if err := tx.Commit(); err != nil {
 688 | 		t.Fatal(err)
 689 | 	}
 690 | 	tx, err = db.Begin(false)
 691 | 	if err != nil {
 692 | 		t.Fatal(err)
 693 | 	}
 694 | 	defer tx.Rollback()
 695 | 	v, err := tx.Get("howdy")
 696 | 	if err != nil {
 697 | 		t.Fatal(err)
 698 | 	}
 699 | 	if v != "world" {
 700 | 		t.Fatalf("expecting '%v', got '%v'", "world", v)
 701 | 	}
 702 | 	if err := tx.Rollback(); err != nil {
 703 | 		t.Fatal(err)
 704 | 	}
 705 | 	tx, err = db.Begin(true)
 706 | 	if err != nil {
 707 | 		t.Fatal(err)
 708 | 	}
 709 | 	defer tx.Rollback()
 710 | 	v, err = tx.Get("howdy")
 711 | 	if err != nil {
 712 | 		t.Fatal(err)
 713 | 	}
 714 | 	if v != "world" {
 715 | 		t.Fatalf("expecting '%v', got '%v'", "world", v)
 716 | 	}
 717 | 	_, err = tx.Delete("howdy")
 718 | 	if err != nil {
 719 | 		t.Fatal(err)
 720 | 	}
 721 | 	if err := tx.Commit(); err != nil {
 722 | 		t.Fatal(err)
 723 | 	}
 724 | 
 725 | 	// test for invalid commits
 726 | 	if err := db.Update(func(tx *Tx) error {
 727 | 		// we are going to do some hackery
 728 | 		defer func() {
 729 | 			if v := recover(); v != nil {
 730 | 				if v.(string) != "managed tx commit not allowed" {
 731 | 					t.Fatal(v.(string))
 732 | 				}
 733 | 			}
 734 | 		}()
 735 | 		return tx.Commit()
 736 | 	}); err != nil {
 737 | 		t.Fatal(err)
 738 | 	}
 739 | 
 740 | 	// test for invalid commits
 741 | 	if err := db.Update(func(tx *Tx) error {
 742 | 		// we are going to do some hackery
 743 | 		defer func() {
 744 | 			if v := recover(); v != nil {
 745 | 				if v.(string) != "managed tx rollback not allowed" {
 746 | 					t.Fatal(v.(string))
 747 | 				}
 748 | 			}
 749 | 		}()
 750 | 		return tx.Rollback()
 751 | 	}); err != nil {
 752 | 		t.Fatal(err)
 753 | 	}
 754 | 
 755 | 	// test for closed transactions
 756 | 	if err := db.Update(func(tx *Tx) error {
 757 | 		tx.db = nil
 758 | 		return nil
 759 | 	}); err != ErrTxClosed {
 760 | 		t.Fatal("expecting tx closed error")
 761 | 	}
 762 | 	db.mu.Unlock()
 763 | 
 764 | 	// test for invalid writes
 765 | 	if err := db.Update(func(tx *Tx) error {
 766 | 		tx.writable = false
 767 | 		return nil
 768 | 	}); err != ErrTxNotWritable {
 769 | 		t.Fatal("expecting tx not writable error")
 770 | 	}
 771 | 	db.mu.Unlock()
 772 | 	// test for closed transactions
 773 | 	if err := db.View(func(tx *Tx) error {
 774 | 		tx.db = nil
 775 | 		return nil
 776 | 	}); err != ErrTxClosed {
 777 | 		t.Fatal("expecting tx closed error")
 778 | 	}
 779 | 	db.mu.RUnlock()
 780 | 	// flush to unwritable file
 781 | 	if err := db.Update(func(tx *Tx) error {
 782 | 		_, _, err := tx.Set("var1", "val1", nil)
 783 | 		if err != nil {
 784 | 			t.Fatal(err)
 785 | 		}
 786 | 		return tx.db.file.Close()
 787 | 	}); err == nil {
 788 | 		t.Fatal("should not be able to commit when the file is closed")
 789 | 	}
 790 | 	db.file, err = os.OpenFile("data.db", os.O_CREATE|os.O_RDWR, 0666)
 791 | 	if err != nil {
 792 | 		t.Fatal(err)
 793 | 	}
 794 | 	if _, err := db.file.Seek(0, 2); err != nil {
 795 | 		t.Fatal(err)
 796 | 	}
 797 | 	db.buf = nil
 798 | 	if err := db.CreateIndex("blank", "*", nil); err != nil {
 799 | 		t.Fatal(err)
 800 | 	}
 801 | 	if err := db.CreateIndex("real", "*", IndexInt); err != nil {
 802 | 		t.Fatal(err)
 803 | 	}
 804 | 	// test scanning
 805 | 	if err := db.Update(func(tx *Tx) error {
 806 | 		less, err := tx.GetLess("junk")
 807 | 		if err != ErrNotFound {
 808 | 			t.Fatalf("expecting a not found, got %v", err)
 809 | 		}
 810 | 		if less != nil {
 811 | 			t.Fatal("expecting nil, got a less function")
 812 | 		}
 813 | 		less, err = tx.GetLess("blank")
 814 | 		if err != ErrNotFound {
 815 | 			t.Fatalf("expecting a not found, got %v", err)
 816 | 		}
 817 | 		if less != nil {
 818 | 			t.Fatal("expecting nil, got a less function")
 819 | 		}
 820 | 		less, err = tx.GetLess("real")
 821 | 		if err != nil {
 822 | 			return err
 823 | 		}
 824 | 		if less == nil {
 825 | 			t.Fatal("expecting a less function, got nil")
 826 | 		}
 827 | 		_, _, err = tx.Set("nothing", "here", nil)
 828 | 		return err
 829 | 	}); err != nil {
 830 | 		t.Fatal(err)
 831 | 	}
 832 | 	if err := db.View(func(tx *Tx) error {
 833 | 		s := ""
 834 | 		err := tx.Ascend("", func(key, val string) bool {
 835 | 			s += key + ":" + val + "\n"
 836 | 			return true
 837 | 		})
 838 | 		if err != nil {
 839 | 			return err
 840 | 		}
 841 | 		if s != "hello:planet\nnothing:here\n" {
 842 | 			t.Fatal("invalid scan")
 843 | 		}
 844 | 		tx.db = nil
 845 | 		err = tx.Ascend("", func(key, val string) bool { return true })
 846 | 		if err != ErrTxClosed {
 847 | 			tx.unlock()
 848 | 			t.Fatal("expecting tx closed error")
 849 | 		}
 850 | 		tx.db = db
 851 | 		err = tx.Ascend("na", func(key, val string) bool { return true })
 852 | 		if err != ErrNotFound {
 853 | 			t.Fatal("expecting not found error")
 854 | 		}
 855 | 		err = tx.Ascend("blank", func(key, val string) bool { return true })
 856 | 		if err != nil {
 857 | 			t.Fatal(err)
 858 | 		}
 859 | 		s = ""
 860 | 		err = tx.AscendLessThan("", "liger", func(key, val string) bool {
 861 | 			s += key + ":" + val + "\n"
 862 | 			return true
 863 | 		})
 864 | 		if err != nil {
 865 | 			return err
 866 | 		}
 867 | 		if s != "hello:planet\n" {
 868 | 			t.Fatal("invalid scan")
 869 | 		}
 870 | 
 871 | 		s = ""
 872 | 		err = tx.Descend("", func(key, val string) bool {
 873 | 			s += key + ":" + val + "\n"
 874 | 			return true
 875 | 		})
 876 | 		if err != nil {
 877 | 			return err
 878 | 		}
 879 | 		if s != "nothing:here\nhello:planet\n" {
 880 | 			t.Fatal("invalid scan")
 881 | 		}
 882 | 
 883 | 		s = ""
 884 | 		err = tx.DescendLessOrEqual("", "liger", func(key, val string) bool {
 885 | 			s += key + ":" + val + "\n"
 886 | 			return true
 887 | 		})
 888 | 		if err != nil {
 889 | 			return err
 890 | 		}
 891 | 
 892 | 		if s != "hello:planet\n" {
 893 | 			t.Fatal("invalid scan")
 894 | 		}
 895 | 
 896 | 		s = ""
 897 | 		err = tx.DescendGreaterThan("", "liger", func(key, val string) bool {
 898 | 			s += key + ":" + val + "\n"
 899 | 			return true
 900 | 		})
 901 | 		if err != nil {
 902 | 			return err
 903 | 		}
 904 | 
 905 | 		if s != "nothing:here\n" {
 906 | 			t.Fatal("invalid scan")
 907 | 		}
 908 | 		s = ""
 909 | 		err = tx.DescendRange("", "liger", "apple", func(key, val string) bool {
 910 | 			s += key + ":" + val + "\n"
 911 | 			return true
 912 | 		})
 913 | 		if err != nil {
 914 | 			return err
 915 | 		}
 916 | 		if s != "hello:planet\n" {
 917 | 			t.Fatal("invalid scan")
 918 | 		}
 919 | 		return nil
 920 | 	}); err != nil {
 921 | 		t.Fatal(err)
 922 | 	}
 923 | 
 924 | 	// test some spatial stuff
 925 | 	if err := db.CreateSpatialIndex("spat", "rect:*", IndexRect); err != nil {
 926 | 		t.Fatal(err)
 927 | 	}
 928 | 	if err := db.CreateSpatialIndex("junk", "rect:*", nil); err != nil {
 929 | 		t.Fatal(err)
 930 | 	}
 931 | 	err = db.Update(func(tx *Tx) error {
 932 | 		rect, err := tx.GetRect("spat")
 933 | 		if err != nil {
 934 | 			return err
 935 | 		}
 936 | 		if rect == nil {
 937 | 			t.Fatal("expecting a rect function, got nil")
 938 | 		}
 939 | 		rect, err = tx.GetRect("junk")
 940 | 		if err != ErrNotFound {
 941 | 			t.Fatalf("expecting a not found, got %v", err)
 942 | 		}
 943 | 		if rect != nil {
 944 | 			t.Fatal("expecting nil, got a rect function")
 945 | 		}
 946 | 		rect, err = tx.GetRect("na")
 947 | 		if err != ErrNotFound {
 948 | 			t.Fatalf("expecting a not found, got %v", err)
 949 | 		}
 950 | 		if rect != nil {
 951 | 			t.Fatal("expecting nil, got a rect function")
 952 | 		}
 953 | 		if _, _, err := tx.Set("rect:1", "[10 10],[20 20]", nil); err != nil {
 954 | 			return err
 955 | 		}
 956 | 		if _, _, err := tx.Set("rect:2", "[15 15],[25 25]", nil); err != nil {
 957 | 			return err
 958 | 		}
 959 | 		if _, _, err := tx.Set("shape:1", "[12 12],[25 25]", nil); err != nil {
 960 | 			return err
 961 | 		}
 962 | 		s := ""
 963 | 		err = tx.Intersects("spat", "[5 5],[13 13]", func(key, val string) bool {
 964 | 			s += key + ":" + val + "\n"
 965 | 			return true
 966 | 		})
 967 | 		if err != nil {
 968 | 			return err
 969 | 		}
 970 | 		if s != "rect:1:[10 10],[20 20]\n" {
 971 | 			t.Fatal("invalid scan")
 972 | 		}
 973 | 		tx.db = nil
 974 | 		err = tx.Intersects("spat", "[5 5],[13 13]", func(key, val string) bool {
 975 | 			return true
 976 | 		})
 977 | 		if err != ErrTxClosed {
 978 | 			t.Fatal("expecting tx closed error")
 979 | 		}
 980 | 		tx.db = db
 981 | 		err = tx.Intersects("", "[5 5],[13 13]", func(key, val string) bool {
 982 | 			return true
 983 | 		})
 984 | 		if err != nil {
 985 | 			t.Fatal(err)
 986 | 		}
 987 | 		err = tx.Intersects("na", "[5 5],[13 13]", func(key, val string) bool {
 988 | 			return true
 989 | 		})
 990 | 		if err != ErrNotFound {
 991 | 			t.Fatal("expecting not found error")
 992 | 		}
 993 | 		err = tx.Intersects("junk", "[5 5],[13 13]", func(key, val string) bool {
 994 | 			return true
 995 | 		})
 996 | 		if err != nil {
 997 | 			t.Fatal(err)
 998 | 		}
 999 | 		n, err := tx.Len()
1000 | 		if err != nil {
1001 | 			t.Fatal(err)
1002 | 		}
1003 | 		if n != 5 {
1004 | 			t.Fatalf("expecting %v, got %v", 5, n)
1005 | 		}
1006 | 		tx.db = nil
1007 | 		_, err = tx.Len()
1008 | 		if err != ErrTxClosed {
1009 | 			t.Fatal("expecting tx closed error")
1010 | 		}
1011 | 		tx.db = db
1012 | 		return nil
1013 | 	})
1014 | 	if err != nil {
1015 | 		t.Fatal(err)
1016 | 	}
1017 | 	// test after closing
1018 | 	if err := db.Close(); err != nil {
1019 | 		t.Fatal(err)
1020 | 	}
1021 | 
1022 | 	if err := db.Update(func(tx *Tx) error { return nil }); err != ErrDatabaseClosed {
1023 | 		t.Fatalf("should not be able to perform transactionso on a closed database.")
1024 | 	}
1025 | }
1026 | 
1027 | func TestNearby(t *testing.T) {
1028 | 	rand.Seed(time.Now().UnixNano())
1029 | 	N := 100000
1030 | 	db, _ := Open(":memory:")
1031 | 	db.CreateSpatialIndex("points", "*", IndexRect)
1032 | 	db.Update(func(tx *Tx) error {
1033 | 		for i := 0; i < N; i++ {
1034 | 			p := Point(
1035 | 				rand.Float64()*100,
1036 | 				rand.Float64()*100,
1037 | 				rand.Float64()*100,
1038 | 				rand.Float64()*100,
1039 | 			)
1040 | 			tx.Set(fmt.Sprintf("p:%d", i), p, nil)
1041 | 		}
1042 | 		return nil
1043 | 	})
1044 | 	var keys, values []string
1045 | 	var dists []float64
1046 | 	var pdist float64
1047 | 	var i int
1048 | 	db.View(func(tx *Tx) error {
1049 | 		tx.Nearby("points", Point(0, 0, 0, 0), func(key, value string, dist float64) bool {
1050 | 			if i != 0 && dist < pdist {
1051 | 				t.Fatal("out of order")
1052 | 			}
1053 | 			keys = append(keys, key)
1054 | 			values = append(values, value)
1055 | 			dists = append(dists, dist)
1056 | 			pdist = dist
1057 | 			i++
1058 | 			return true
1059 | 		})
1060 | 		return nil
1061 | 	})
1062 | 	if len(keys) != N {
1063 | 		t.Fatalf("expected '%v', got '%v'", N, len(keys))
1064 | 	}
1065 | }
1066 | 
1067 | func Example_descKeys() {
1068 | 	db, _ := Open(":memory:")
1069 | 	db.CreateIndex("name", "*", IndexString)
1070 | 	db.Update(func(tx *Tx) error {
1071 | 		tx.Set("user:100:first", "Tom", nil)
1072 | 		tx.Set("user:100:last", "Johnson", nil)
1073 | 		tx.Set("user:101:first", "Janet", nil)
1074 | 		tx.Set("user:101:last", "Prichard", nil)
1075 | 		tx.Set("user:102:first", "Alan", nil)
1076 | 		tx.Set("user:102:last", "Cooper", nil)
1077 | 		return nil
1078 | 	})
1079 | 	db.View(func(tx *Tx) error {
1080 | 		tx.AscendKeys("user:101:*",
1081 | 			func(key, value string) bool {
1082 | 				fmt.Printf("%s: %s\n", key, value)
1083 | 				return true
1084 | 			})
1085 | 		tx.AscendKeys("user:10?:*",
1086 | 			func(key, value string) bool {
1087 | 				fmt.Printf("%s: %s\n", key, value)
1088 | 				return true
1089 | 			})
1090 | 		tx.AscendKeys("*2*",
1091 | 			func(key, value string) bool {
1092 | 				fmt.Printf("%s: %s\n", key, value)
1093 | 				return true
1094 | 			})
1095 | 		tx.DescendKeys("user:101:*",
1096 | 			func(key, value string) bool {
1097 | 				fmt.Printf("%s: %s\n", key, value)
1098 | 				return true
1099 | 			})
1100 | 		tx.DescendKeys("*",
1101 | 			func(key, value string) bool {
1102 | 				fmt.Printf("%s: %s\n", key, value)
1103 | 				return true
1104 | 			})
1105 | 		return nil
1106 | 	})
1107 | 
1108 | 	// Output:
1109 | 	// user:101:first: Janet
1110 | 	// user:101:last: Prichard
1111 | 	// user:100:first: Tom
1112 | 	// user:100:last: Johnson
1113 | 	// user:101:first: Janet
1114 | 	// user:101:last: Prichard
1115 | 	// user:102:first: Alan
1116 | 	// user:102:last: Cooper
1117 | 	// user:102:first: Alan
1118 | 	// user:102:last: Cooper
1119 | 	// user:101:last: Prichard
1120 | 	// user:101:first: Janet
1121 | 	// user:102:last: Cooper
1122 | 	// user:102:first: Alan
1123 | 	// user:101:last: Prichard
1124 | 	// user:101:first: Janet
1125 | 	// user:100:last: Johnson
1126 | 	// user:100:first: Tom
1127 | }
1128 | 
1129 | func ExampleDesc() {
1130 | 	db, _ := Open(":memory:")
1131 | 	db.CreateIndex("last_name_age", "*", IndexJSON("name.last"), Desc(IndexJSON("age")))
1132 | 	db.Update(func(tx *Tx) error {
1133 | 		tx.Set("1", `{"name":{"first":"Tom","last":"Johnson"},"age":38}`, nil)
1134 | 		tx.Set("2", `{"name":{"first":"Janet","last":"Prichard"},"age":47}`, nil)
1135 | 		tx.Set("3", `{"name":{"first":"Carol","last":"Anderson"},"age":52}`, nil)
1136 | 		tx.Set("4", `{"name":{"first":"Alan","last":"Cooper"},"age":28}`, nil)
1137 | 		tx.Set("5", `{"name":{"first":"Sam","last":"Anderson"},"age":51}`, nil)
1138 | 		tx.Set("6", `{"name":{"first":"Melinda","last":"Prichard"},"age":44}`, nil)
1139 | 		return nil
1140 | 	})
1141 | 	db.View(func(tx *Tx) error {
1142 | 		tx.Ascend("last_name_age", func(key, value string) bool {
1143 | 			fmt.Printf("%s: %s\n", key, value)
1144 | 			return true
1145 | 		})
1146 | 		return nil
1147 | 	})
1148 | 
1149 | 	// Output:
1150 | 	//3: {"name":{"first":"Carol","last":"Anderson"},"age":52}
1151 | 	//5: {"name":{"first":"Sam","last":"Anderson"},"age":51}
1152 | 	//4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
1153 | 	//1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
1154 | 	//2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
1155 | 	//6: {"name":{"first":"Melinda","last":"Prichard"},"age":44}
1156 | }
1157 | 
1158 | func ExampleDB_CreateIndex_jSON() {
1159 | 	db, _ := Open(":memory:")
1160 | 	db.CreateIndex("last_name", "*", IndexJSON("name.last"))
1161 | 	db.CreateIndex("age", "*", IndexJSON("age"))
1162 | 	db.Update(func(tx *Tx) error {
1163 | 		tx.Set("1", `{"name":{"first":"Tom","last":"Johnson"},"age":38}`, nil)
1164 | 		tx.Set("2", `{"name":{"first":"Janet","last":"Prichard"},"age":47}`, nil)
1165 | 		tx.Set("3", `{"name":{"first":"Carol","last":"Anderson"},"age":52}`, nil)
1166 | 		tx.Set("4", `{"name":{"first":"Alan","last":"Cooper"},"age":28}`, nil)
1167 | 		return nil
1168 | 	})
1169 | 	db.View(func(tx *Tx) error {
1170 | 		fmt.Println("Order by last name")
1171 | 		tx.Ascend("last_name", func(key, value string) bool {
1172 | 			fmt.Printf("%s: %s\n", key, value)
1173 | 			return true
1174 | 		})
1175 | 		fmt.Println("Order by age")
1176 | 		tx.Ascend("age", func(key, value string) bool {
1177 | 			fmt.Printf("%s: %s\n", key, value)
1178 | 			return true
1179 | 		})
1180 | 		fmt.Println("Order by age range 30-50")
1181 | 		tx.AscendRange("age", `{"age":30}`, `{"age":50}`, func(key, value string) bool {
1182 | 			fmt.Printf("%s: %s\n", key, value)
1183 | 			return true
1184 | 		})
1185 | 		return nil
1186 | 	})
1187 | 
1188 | 	// Output:
1189 | 	// Order by last name
1190 | 	// 3: {"name":{"first":"Carol","last":"Anderson"},"age":52}
1191 | 	// 4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
1192 | 	// 1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
1193 | 	// 2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
1194 | 	// Order by age
1195 | 	// 4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
1196 | 	// 1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
1197 | 	// 2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
1198 | 	// 3: {"name":{"first":"Carol","last":"Anderson"},"age":52}
1199 | 	// Order by age range 30-50
1200 | 	// 1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
1201 | 	// 2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
1202 | }
1203 | 
1204 | func ExampleDB_CreateIndex_strings() {
1205 | 	db, _ := Open(":memory:")
1206 | 	db.CreateIndex("name", "*", IndexString)
1207 | 	db.Update(func(tx *Tx) error {
1208 | 		tx.Set("1", "Tom", nil)
1209 | 		tx.Set("2", "Janet", nil)
1210 | 		tx.Set("3", "Carol", nil)
1211 | 		tx.Set("4", "Alan", nil)
1212 | 		tx.Set("5", "Sam", nil)
1213 | 		tx.Set("6", "Melinda", nil)
1214 | 		return nil
1215 | 	})
1216 | 	db.View(func(tx *Tx) error {
1217 | 		tx.Ascend("name", func(key, value string) bool {
1218 | 			fmt.Printf("%s: %s\n", key, value)
1219 | 			return true
1220 | 		})
1221 | 		return nil
1222 | 	})
1223 | 
1224 | 	// Output:
1225 | 	//4: Alan
1226 | 	//3: Carol
1227 | 	//2: Janet
1228 | 	//6: Melinda
1229 | 	//5: Sam
1230 | 	//1: Tom
1231 | }
1232 | 
1233 | func ExampleDB_CreateIndex_ints() {
1234 | 	db, _ := Open(":memory:")
1235 | 	db.CreateIndex("age", "*", IndexInt)
1236 | 	db.Update(func(tx *Tx) error {
1237 | 		tx.Set("1", "30", nil)
1238 | 		tx.Set("2", "51", nil)
1239 | 		tx.Set("3", "16", nil)
1240 | 		tx.Set("4", "76", nil)
1241 | 		tx.Set("5", "23", nil)
1242 | 		tx.Set("6", "43", nil)
1243 | 		return nil
1244 | 	})
1245 | 	db.View(func(tx *Tx) error {
1246 | 		tx.Ascend("age", func(key, value string) bool {
1247 | 			fmt.Printf("%s: %s\n", key, value)
1248 | 			return true
1249 | 		})
1250 | 		return nil
1251 | 	})
1252 | 
1253 | 	// Output:
1254 | 	//3: 16
1255 | 	//5: 23
1256 | 	//1: 30
1257 | 	//6: 43
1258 | 	//2: 51
1259 | 	//4: 76
1260 | }
1261 | func ExampleDB_CreateIndex_multipleFields() {
1262 | 	db, _ := Open(":memory:")
1263 | 	db.CreateIndex("last_name_age", "*", IndexJSON("name.last"), IndexJSON("age"))
1264 | 	db.Update(func(tx *Tx) error {
1265 | 		tx.Set("1", `{"name":{"first":"Tom","last":"Johnson"},"age":38}`, nil)
1266 | 		tx.Set("2", `{"name":{"first":"Janet","last":"Prichard"},"age":47}`, nil)
1267 | 		tx.Set("3", `{"name":{"first":"Carol","last":"Anderson"},"age":52}`, nil)
1268 | 		tx.Set("4", `{"name":{"first":"Alan","last":"Cooper"},"age":28}`, nil)
1269 | 		tx.Set("5", `{"name":{"first":"Sam","last":"Anderson"},"age":51}`, nil)
1270 | 		tx.Set("6", `{"name":{"first":"Melinda","last":"Prichard"},"age":44}`, nil)
1271 | 		return nil
1272 | 	})
1273 | 	db.View(func(tx *Tx) error {
1274 | 		tx.Ascend("last_name_age", func(key, value string) bool {
1275 | 			fmt.Printf("%s: %s\n", key, value)
1276 | 			return true
1277 | 		})
1278 | 		return nil
1279 | 	})
1280 | 
1281 | 	// Output:
1282 | 	//5: {"name":{"first":"Sam","last":"Anderson"},"age":51}
1283 | 	//3: {"name":{"first":"Carol","last":"Anderson"},"age":52}
1284 | 	//4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
1285 | 	//1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
1286 | 	//6: {"name":{"first":"Melinda","last":"Prichard"},"age":44}
1287 | 	//2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
1288 | }
1289 | 
1290 | func TestNoExpiringItem(t *testing.T) {
1291 | 	item := &dbItem{key: "key", val: "val"}
1292 | 	if !item.expiresAt().Equal(maxTime) {
1293 | 		t.Fatal("item.expiresAt() != maxTime")
1294 | 	}
1295 | 	if min, max := item.Rect(nil); min != nil || max != nil {
1296 | 		t.Fatal("item min,max should both be nil")
1297 | 	}
1298 | }
1299 | func TestAutoShrink(t *testing.T) {
1300 | 	db := testOpen(t)
1301 | 	defer testClose(db)
1302 | 	for i := 0; i < 1000; i++ {
1303 | 		err := db.Update(func(tx *Tx) error {
1304 | 			for i := 0; i < 20; i++ {
1305 | 				if _, _, err := tx.Set(fmt.Sprintf("HELLO:%d", i), "WORLD", nil); err != nil {
1306 | 					return err
1307 | 				}
1308 | 			}
1309 | 			return nil
1310 | 		})
1311 | 		if err != nil {
1312 | 			t.Fatal(err)
1313 | 		}
1314 | 	}
1315 | 	db = testReOpen(t, db)
1316 | 	defer testClose(db)
1317 | 	db.config.AutoShrinkMinSize = 64 * 1024 // 64K
1318 | 	for i := 0; i < 2000; i++ {
1319 | 		err := db.Update(func(tx *Tx) error {
1320 | 			for i := 0; i < 20; i++ {
1321 | 				if _, _, err := tx.Set(fmt.Sprintf("HELLO:%d", i), "WORLD", nil); err != nil {
1322 | 					return err
1323 | 				}
1324 | 			}
1325 | 			return nil
1326 | 		})
1327 | 		if err != nil {
1328 | 			t.Fatal(err)
1329 | 		}
1330 | 	}
1331 | 	time.Sleep(time.Second * 3)
1332 | 	db = testReOpen(t, db)
1333 | 	defer testClose(db)
1334 | 	err := db.View(func(tx *Tx) error {
1335 | 		n, err := tx.Len()
1336 | 		if err != nil {
1337 | 			return err
1338 | 		}
1339 | 		if n != 20 {
1340 | 			t.Fatalf("expecting 20, got %v", n)
1341 | 		}
1342 | 		return nil
1343 | 	})
1344 | 	if err != nil {
1345 | 		t.Fatal(err)
1346 | 	}
1347 | }
1348 | 
1349 | // test database format loading
1350 | func TestDatabaseFormat(t *testing.T) {
1351 | 	// should succeed
1352 | 	func() {
1353 | 		resp := strings.Join([]string{
1354 | 			"*3\r\n$3\r\nset\r\n$4\r\nvar1\r\n$4\r\n1234\r\n",
1355 | 			"*3\r\n$3\r\nset\r\n$4\r\nvar2\r\n$4\r\n1234\r\n",
1356 | 			"*2\r\n$3\r\ndel\r\n$4\r\nvar1\r\n",
1357 | 			"*5\r\n$3\r\nset\r\n$3\r\nvar\r\n$3\r\nval\r\n$2\r\nex\r\n$2\r\n10\r\n",
1358 | 		}, "")
1359 | 		if err := os.RemoveAll("data.db"); err != nil {
1360 | 			t.Fatal(err)
1361 | 		}
1362 | 		if err := ioutil.WriteFile("data.db", []byte(resp), 0666); err != nil {
1363 | 			t.Fatal(err)
1364 | 		}
1365 | 		db := testOpen(t)
1366 | 		defer testClose(db)
1367 | 	}()
1368 | 	testFormat := func(t *testing.T, expectValid bool, resp string, do func(db *DB) error) {
1369 | 		t.Helper()
1370 | 		os.RemoveAll("data.db")
1371 | 		if err := ioutil.WriteFile("data.db", []byte(resp), 0666); err != nil {
1372 | 			t.Fatal(err)
1373 | 		}
1374 | 		defer os.RemoveAll("data.db")
1375 | 
1376 | 		db, err := Open("data.db")
1377 | 		if err == nil {
1378 | 			if do != nil {
1379 | 				if err := do(db); err != nil {
1380 | 					t.Fatal(err)
1381 | 				}
1382 | 			}
1383 | 			if err := db.Close(); err != nil {
1384 | 				t.Fatal(err)
1385 | 			}
1386 | 		}
1387 | 		if err == nil && !expectValid {
1388 | 			t.Fatalf("expected invalid database")
1389 | 		} else if err != nil && expectValid {
1390 | 			t.Fatalf("expected valid database, got '%s'", err)
1391 | 		}
1392 | 	}
1393 | 
1394 | 	// basic valid commands
1395 | 	testFormat(t, true, "*2\r\n$3\r\nDEL\r\n$5\r\nHELLO\r\n", nil)
1396 | 	testFormat(t, true, "*3\r\n$3\r\nSET\r\n$5\r\nHELLO\r\n$5\r\nWORLD\r\n", nil)
1397 | 	testFormat(t, true, "*1\r\n$7\r\nFLUSHDB\r\n", nil)
1398 | 
1399 | 	// commands with invalid names or arguments
1400 | 	testFormat(t, false, "*3\r\n$3\r\nDEL\r\n$5\r\nHELLO\r\n$5\r\nWORLD\r\n", nil)
1401 | 	testFormat(t, false, "*2\r\n$3\r\nSET\r\n$5\r\nHELLO\r\n", nil)
1402 | 	testFormat(t, false, "*1\r\n$6\r\nSET123\r\n", nil)
1403 | 
1404 | 	// partial tail commands should be ignored but allowed
1405 | 	pcmd := "*3\r\n$3\r\nSET\r\n$5\r\nHELLO\r\n$5\r\nWORLD\r\n"
1406 | 	for i := 1; i < len(pcmd); i++ {
1407 | 		cmd := "*3\r\n$3\r\nSET\r\n$5\r\nHELLO\r\n$5\r\nJELLO\r\n"
1408 | 		testFormat(t, true, cmd+pcmd[:len(pcmd)-i],
1409 | 			func(db *DB) error {
1410 | 				return db.View(func(tx *Tx) error {
1411 | 					val, err := tx.Get("HELLO")
1412 | 					if err != nil {
1413 | 						return err
1414 | 					}
1415 | 					if val != "JELLO" {
1416 | 						return fmt.Errorf("expected '%s', got '%s'", "JELLO", val)
1417 | 					}
1418 | 					return nil
1419 | 				})
1420 | 			})
1421 | 	}
1422 | 
1423 | 	// commands with invalid formatting
1424 | 	testFormat(t, false, "^3\r\n$3\r\nSET\r\n$5\r\nHELLO\r\n$5\r\nWORLD\r\n", nil)
1425 | 	testFormat(t, false, "*3\n$3\r\nSET\r\n$5\r\nHELLO\r\n$5\r\nWORLD\r\n", nil)
1426 | 	testFormat(t, false, "*\n$3\r\nSET\r\n$5\r\nHELLO\r\n$5\r\nWORLD\r\n", nil)
1427 | 	testFormat(t, false, "*3\r\n^3\r\nSET\r\n$5\r\nHELLO\r\n$5\r\nWORLD\r\n", nil)
1428 | 	testFormat(t, false, "*3\r\n$\r\nSET\r\n$5\r\nHELLO\r\n$5\r\nWORLD\r\n", nil)
1429 | 	testFormat(t, false, "*3\r\n$3\nSET\r\n$5\r\nHELLO\r\n$5\r\nWORLD\r\n", nil)
1430 | 	testFormat(t, false, "*3\r\n$3SET\r\n$5\r\nHELLO\r\n$5\r\nWORLD\r\n", nil)
1431 | 	testFormat(t, false, "*3\r\n$3\r\nSET\r\n$5\r\nHELLO\r\n$5\r\nWORLD\r\n123\n", nil)
1432 | 
1433 | 	// commands with nuls
1434 | 	testFormat(t, true, "\u0000*3\r\n$3\r\nSET\r\n$5\r\nHELLO\r\n$5\r\nWORLD\r\n"+
1435 | 		"\u0000\u0000*3\r\n$3\r\nSET\r\n$5\r\nHELLO\r\n$5\r\nJELLO\r\n\u0000", func(db *DB) error {
1436 | 		return db.View(func(tx *Tx) error {
1437 | 			val, err := tx.Get("HELLO")
1438 | 			if err != nil {
1439 | 				return err
1440 | 			}
1441 | 			if val != "JELLO" {
1442 | 				return fmt.Errorf("expected '%s', got '%s'", "JELLO", val)
1443 | 			}
1444 | 			return nil
1445 | 		})
1446 | 	})
1447 | 
1448 | }
1449 | 
1450 | func TestInsertsAndDeleted(t *testing.T) {
1451 | 	db := testOpen(t)
1452 | 	defer testClose(db)
1453 | 	if err := db.CreateIndex("any", "*", IndexString); err != nil {
1454 | 		t.Fatal(err)
1455 | 	}
1456 | 	if err := db.CreateSpatialIndex("rect", "*", IndexRect); err != nil {
1457 | 		t.Fatal(err)
1458 | 	}
1459 | 	if err := db.Update(func(tx *Tx) error {
1460 | 		if _, _, err := tx.Set("item1", "value1", &SetOptions{Expires: true, TTL: time.Second}); err != nil {
1461 | 			return err
1462 | 		}
1463 | 		if _, _, err := tx.Set("item2", "value2", nil); err != nil {
1464 | 			return err
1465 | 		}
1466 | 		if _, _, err := tx.Set("item3", "value3", &SetOptions{Expires: true, TTL: time.Second}); err != nil {
1467 | 			return err
1468 | 		}
1469 | 		return nil
1470 | 	}); err != nil {
1471 | 		t.Fatal(err)
1472 | 	}
1473 | 
1474 | 	// test replacing items in the database
1475 | 	if err := db.Update(func(tx *Tx) error {
1476 | 		if _, _, err := tx.Set("item1", "nvalue1", nil); err != nil {
1477 | 			return err
1478 | 		}
1479 | 		if _, _, err := tx.Set("item2", "nvalue2", nil); err != nil {
1480 | 			return err
1481 | 		}
1482 | 		if _, err := tx.Delete("item3"); err != nil {
1483 | 			return err
1484 | 		}
1485 | 		return nil
1486 | 	}); err != nil {
1487 | 		t.Fatal(err)
1488 | 	}
1489 | }
1490 | 
1491 | func TestInsertDoesNotMisuseIndex(t *testing.T) {
1492 | 	db := testOpen(t)
1493 | 	defer testClose(db)
1494 | 	fail := func(a, b string) bool { t.Fatal("Misused index"); return false }
1495 | 	if err := db.CreateIndex("some", "a*", fail); err != nil {
1496 | 		// Only one item is eligible for the index, so no comparison is necessary.
1497 | 		t.Fatal(err)
1498 | 	}
1499 | 	if err := db.Update(func(tx *Tx) error {
1500 | 		if _, _, err := tx.Set("a", "1", nil); err != nil {
1501 | 			return err
1502 | 		}
1503 | 		if _, _, err := tx.Set("b", "1", nil); err != nil {
1504 | 			return err
1505 | 		}
1506 | 		return nil
1507 | 	}); err != nil {
1508 | 		t.Fatal(err)
1509 | 	}
1510 | 
1511 | 	if err := db.Update(func(tx *Tx) error {
1512 | 		_, _, err := tx.Set("b", "2", nil)
1513 | 		return err
1514 | 	}); err != nil {
1515 | 		t.Fatal(err)
1516 | 	}
1517 | }
1518 | 
1519 | func TestDeleteDoesNotMisuseIndex(t *testing.T) {
1520 | 	db := testOpen(t)
1521 | 	defer testClose(db)
1522 | 	fail := func(a, b string) bool { t.Fatal("Misused index"); return false }
1523 | 	if err := db.CreateIndex("some", "a*", fail); err != nil {
1524 | 		// Only one item is eligible for the index, so no comparison is necessary.
1525 | 		t.Fatal(err)
1526 | 	}
1527 | 	if err := db.Update(func(tx *Tx) error {
1528 | 		if _, _, err := tx.Set("a", "1", nil); err != nil {
1529 | 			return err
1530 | 		}
1531 | 		if _, _, err := tx.Set("b", "1", nil); err != nil {
1532 | 			return err
1533 | 		}
1534 | 		return nil
1535 | 	}); err != nil {
1536 | 		t.Fatal(err)
1537 | 	}
1538 | 
1539 | 	if err := db.Update(func(tx *Tx) error {
1540 | 		_, err := tx.Delete("b")
1541 | 		return err
1542 | 	}); err != nil {
1543 | 		t.Fatal(err)
1544 | 	}
1545 | }
1546 | 
1547 | // test index compare functions
1548 | func TestIndexCompare(t *testing.T) {
1549 | 	if !IndexFloat("1.5", "1.6") {
1550 | 		t.Fatalf("expected true, got false")
1551 | 	}
1552 | 	if !IndexInt("-1", "2") {
1553 | 		t.Fatalf("expected true, got false")
1554 | 	}
1555 | 	if !IndexUint("10", "25") {
1556 | 		t.Fatalf("expected true, got false")
1557 | 	}
1558 | 	if !IndexBinary("Hello", "hello") {
1559 | 		t.Fatalf("expected true, got false")
1560 | 	}
1561 | 	if IndexString("hello", "hello") {
1562 | 		t.Fatalf("expected false, got true")
1563 | 	}
1564 | 	if IndexString("Hello", "hello") {
1565 | 		t.Fatalf("expected false, got true")
1566 | 	}
1567 | 	if IndexString("hello", "Hello") {
1568 | 		t.Fatalf("expected false, got true")
1569 | 	}
1570 | 	if !IndexString("gello", "Hello") {
1571 | 		t.Fatalf("expected true, got false")
1572 | 	}
1573 | 	if IndexString("Hello", "gello") {
1574 | 		t.Fatalf("expected false, got true")
1575 | 	}
1576 | 	if Rect(IndexRect("[1 2 3 4],[5 6 7 8]")) != "[1 2 3 4],[5 6 7 8]" {
1577 | 		t.Fatalf("expected '%v', got '%v'", "[1 2 3 4],[5 6 7 8]", Rect(IndexRect("[1 2 3 4],[5 6 7 8]")))
1578 | 	}
1579 | 	if Rect(IndexRect("[1 2 3 4]")) != "[1 2 3 4]" {
1580 | 		t.Fatalf("expected '%v', got '%v'", "[1 2 3 4]", Rect(IndexRect("[1 2 3 4]")))
1581 | 	}
1582 | 	if Rect(nil, nil) != "[]" {
1583 | 		t.Fatalf("expected '%v', got '%v'", "", Rect(nil, nil))
1584 | 	}
1585 | 	if Point(1, 2, 3) != "[1 2 3]" {
1586 | 		t.Fatalf("expected '%v', got '%v'", "[1 2 3]", Point(1, 2, 3))
1587 | 	}
1588 | }
1589 | 
1590 | // test opening a folder.
1591 | func TestOpeningAFolder(t *testing.T) {
1592 | 	if err := os.RemoveAll("dir.tmp"); err != nil {
1593 | 		t.Fatal(err)
1594 | 	}
1595 | 	if err := os.Mkdir("dir.tmp", 0700); err != nil {
1596 | 		t.Fatal(err)
1597 | 	}
1598 | 	defer func() { _ = os.RemoveAll("dir.tmp") }()
1599 | 	db, err := Open("dir.tmp")
1600 | 	if err == nil {
1601 | 		if err := db.Close(); err != nil {
1602 | 			t.Fatal(err)
1603 | 		}
1604 | 		t.Fatalf("opening a directory should not be allowed")
1605 | 	}
1606 | }
1607 | 
1608 | // test opening an invalid resp file.
1609 | func TestOpeningInvalidDatabaseFile(t *testing.T) {
1610 | 	if err := os.RemoveAll("data.db"); err != nil {
1611 | 		t.Fatal(err)
1612 | 	}
1613 | 	if err := ioutil.WriteFile("data.db", []byte("invalid\r\nfile"), 0666); err != nil {
1614 | 		t.Fatal(err)
1615 | 	}
1616 | 	defer func() { _ = os.RemoveAll("data.db") }()
1617 | 	db, err := Open("data.db")
1618 | 	if err == nil {
1619 | 		if err := db.Close(); err != nil {
1620 | 			t.Fatal(err)
1621 | 		}
1622 | 		t.Fatalf("invalid database should not be allowed")
1623 | 	}
1624 | }
1625 | 
1626 | // test closing a closed database.
1627 | func TestOpeningClosedDatabase(t *testing.T) {
1628 | 	if err := os.RemoveAll("data.db"); err != nil {
1629 | 		t.Fatal(err)
1630 | 	}
1631 | 	db, err := Open("data.db")
1632 | 	if err != nil {
1633 | 		t.Fatal(err)
1634 | 	}
1635 | 	defer func() { _ = os.RemoveAll("data.db") }()
1636 | 	if err := db.Close(); err != nil {
1637 | 		t.Fatal(err)
1638 | 	}
1639 | 	if err := db.Close(); err != ErrDatabaseClosed {
1640 | 		t.Fatal("should not be able to close a closed database")
1641 | 	}
1642 | 	db, err = Open(":memory:")
1643 | 	if err != nil {
1644 | 		t.Fatal(err)
1645 | 	}
1646 | 	if err := db.Close(); err != nil {
1647 | 		t.Fatal(err)
1648 | 	}
1649 | 	if err := db.Close(); err != ErrDatabaseClosed {
1650 | 		t.Fatal("should not be able to close a closed database")
1651 | 	}
1652 | }
1653 | 
1654 | // test shrinking a database.
1655 | func TestShrink(t *testing.T) {
1656 | 	db := testOpen(t)
1657 | 	defer testClose(db)
1658 | 	if err := db.Shrink(); err != nil {
1659 | 		t.Fatal(err)
1660 | 	}
1661 | 	fi, err := os.Stat("data.db")
1662 | 	if err != nil {
1663 | 		t.Fatal(err)
1664 | 	}
1665 | 	if fi.Size() != 0 {
1666 | 		t.Fatalf("expected %v, got %v", 0, fi.Size())
1667 | 	}
1668 | 	// add 10 items
1669 | 	err = db.Update(func(tx *Tx) error {
1670 | 		for i := 0; i < 10; i++ {
1671 | 			if _, _, err := tx.Set(fmt.Sprintf("key%d", i), fmt.Sprintf("val%d", i), nil); err != nil {
1672 | 				return err
1673 | 			}
1674 | 		}
1675 | 		return nil
1676 | 	})
1677 | 	if err != nil {
1678 | 		t.Fatal(err)
1679 | 	}
1680 | 	// add the same 10 items
1681 | 	// this will create 10 duplicate log entries
1682 | 	err = db.Update(func(tx *Tx) error {
1683 | 		for i := 0; i < 10; i++ {
1684 | 			if _, _, err := tx.Set(fmt.Sprintf("key%d", i), fmt.Sprintf("val%d", i), nil); err != nil {
1685 | 				return err
1686 | 			}
1687 | 		}
1688 | 		return nil
1689 | 	})
1690 | 	if err != nil {
1691 | 		t.Fatal(err)
1692 | 	}
1693 | 	fi, err = os.Stat("data.db")
1694 | 	if err != nil {
1695 | 		t.Fatal(err)
1696 | 	}
1697 | 	sz1 := fi.Size()
1698 | 	if sz1 == 0 {
1699 | 		t.Fatalf("expected > 0, got %v", sz1)
1700 | 	}
1701 | 	if err := db.Shrink(); err != nil {
1702 | 		t.Fatal(err)
1703 | 	}
1704 | 	fi, err = os.Stat("data.db")
1705 | 	if err != nil {
1706 | 		t.Fatal(err)
1707 | 	}
1708 | 	sz2 := fi.Size()
1709 | 	if sz2 >= sz1 {
1710 | 		t.Fatalf("expected < %v, got %v", sz1, sz2)
1711 | 	}
1712 | 	if err := db.Close(); err != nil {
1713 | 		t.Fatal(err)
1714 | 	}
1715 | 	if err := db.Shrink(); err != ErrDatabaseClosed {
1716 | 		t.Fatal("shrink on a closed databse should not be allowed")
1717 | 	}
1718 | 	// Now we will open a db that does not persist
1719 | 	db, err = Open(":memory:")
1720 | 	if err != nil {
1721 | 		t.Fatal(err)
1722 | 	}
1723 | 	defer func() { _ = db.Close() }()
1724 | 	// add 10 items
1725 | 	err = db.Update(func(tx *Tx) error {
1726 | 		for i := 0; i < 10; i++ {
1727 | 			if _, _, err := tx.Set(fmt.Sprintf("key%d", i), fmt.Sprintf("val%d", i), nil); err != nil {
1728 | 				return err
1729 | 			}
1730 | 		}
1731 | 		return nil
1732 | 	})
1733 | 	if err != nil {
1734 | 		t.Fatal(err)
1735 | 	}
1736 | 	// add the same 10 items
1737 | 	// this will create 10 duplicate log entries
1738 | 	err = db.Update(func(tx *Tx) error {
1739 | 		for i := 0; i < 10; i++ {
1740 | 			if _, _, err := tx.Set(fmt.Sprintf("key%d", i), fmt.Sprintf("val%d", i), nil); err != nil {
1741 | 				return err
1742 | 			}
1743 | 		}
1744 | 		return nil
1745 | 	})
1746 | 	if err != nil {
1747 | 		t.Fatal(err)
1748 | 	}
1749 | 	err = db.View(func(tx *Tx) error {
1750 | 		n, err := tx.Len()
1751 | 		if err != nil {
1752 | 			t.Fatal(err)
1753 | 		}
1754 | 		if n != 10 {
1755 | 			t.Fatalf("expecting %v, got %v", 10, n)
1756 | 		}
1757 | 		return nil
1758 | 	})
1759 | 	if err != nil {
1760 | 		t.Fatal(err)
1761 | 	}
1762 | 	// this should succeed even though it's basically a noop.
1763 | 	if err := db.Shrink(); err != nil {
1764 | 		t.Fatal(err)
1765 | 	}
1766 | }
1767 | 
1768 | func TestVariousIndexOperations(t *testing.T) {
1769 | 	db := testOpen(t)
1770 | 	defer testClose(db)
1771 | 	// test creating an index with no index name.
1772 | 	err := db.CreateIndex("", "", nil)
1773 | 	if err == nil {
1774 | 		t.Fatal("should not be able to create an index with no name")
1775 | 	}
1776 | 	// test creating an index with a name that has already been used.
1777 | 	err = db.CreateIndex("hello", "", nil)
1778 | 	if err != nil {
1779 | 		t.Fatal(err)
1780 | 	}
1781 | 	err = db.CreateIndex("hello", "", nil)
1782 | 	if err == nil {
1783 | 		t.Fatal("should not be able to create a duplicate index")
1784 | 	}
1785 | 	err = db.Update(func(tx *Tx) error {
1786 | 
1787 | 		if _, _, err := tx.Set("user:1", "tom", nil); err != nil {
1788 | 			return err
1789 | 		}
1790 | 		if _, _, err := tx.Set("user:2", "janet", nil); err != nil {
1791 | 			return err
1792 | 		}
1793 | 		if _, _, err := tx.Set("alt:1", "from", nil); err != nil {
1794 | 			return err
1795 | 		}
1796 | 		if _, _, err := tx.Set("alt:2", "there", nil); err != nil {
1797 | 			return err
1798 | 		}
1799 | 		if _, _, err := tx.Set("rect:1", "[1 2],[3 4]", nil); err != nil {
1800 | 			return err
1801 | 		}
1802 | 		if _, _, err := tx.Set("rect:2", "[5 6],[7 8]", nil); err != nil {
1803 | 			return err
1804 | 		}
1805 | 		return nil
1806 | 	})
1807 | 	if err != nil {
1808 | 		t.Fatal(err)
1809 | 	}
1810 | 	// test creating an index after adding items. use pattern matching. have some items in the match and some not.
1811 | 	if err := db.CreateIndex("string", "user:*", IndexString); err != nil {
1812 | 		t.Fatal(err)
1813 | 	}
1814 | 	// test creating a spatial index after adding items. use pattern matching. have some items in the match and some not.
1815 | 	if err := db.CreateSpatialIndex("rect", "rect:*", IndexRect); err != nil {
1816 | 		t.Fatal(err)
1817 | 	}
1818 | 	// test dropping an index
1819 | 	if err := db.DropIndex("hello"); err != nil {
1820 | 		t.Fatal(err)
1821 | 	}
1822 | 	// test dropping an index with no name
1823 | 	if err := db.DropIndex(""); err == nil {
1824 | 		t.Fatal("should not be allowed to drop an index with no name")
1825 | 	}
1826 | 	// test dropping an index with no name
1827 | 	if err := db.DropIndex("na"); err == nil {
1828 | 		t.Fatal("should not be allowed to drop an index that does not exist")
1829 | 	}
1830 | 	// test retrieving index names
1831 | 	names, err := db.Indexes()
1832 | 	if err != nil {
1833 | 		t.Fatal(err)
1834 | 	}
1835 | 	if strings.Join(names, ",") != "rect,string" {
1836 | 		t.Fatalf("expecting '%v', got '%v'", "rect,string", strings.Join(names, ","))
1837 | 	}
1838 | 	// test creating an index after closing database
1839 | 	if err := db.Close(); err != nil {
1840 | 		t.Fatal(err)
1841 | 	}
1842 | 	if err := db.CreateIndex("new-index", "", nil); err != ErrDatabaseClosed {
1843 | 		t.Fatal("should not be able to create an index on a closed database")
1844 | 	}
1845 | 	// test getting index names after closing database
1846 | 	if _, err := db.Indexes(); err != ErrDatabaseClosed {
1847 | 		t.Fatal("should not be able to get index names on a closed database")
1848 | 	}
1849 | 	// test dropping an index after closing database
1850 | 	if err := db.DropIndex("rect"); err != ErrDatabaseClosed {
1851 | 		t.Fatal("should not be able to drop an index on a closed database")
1852 | 	}
1853 | }
1854 | 
1855 | func test(t *testing.T, a, b bool) {
1856 | 	if a != b {
1857 | 		t.Fatal("failed, bummer...")
1858 | 	}
1859 | }
1860 | 
1861 | func TestBasic(t *testing.T) {
1862 | 	rand.Seed(time.Now().UnixNano())
1863 | 	db := testOpen(t)
1864 | 	defer testClose(db)
1865 | 
1866 | 	// create a simple index
1867 | 	if err := db.CreateIndex("users", "fun:user:*", IndexString); err != nil {
1868 | 		t.Fatal(err)
1869 | 	}
1870 | 
1871 | 	// create a spatial index
1872 | 	if err := db.CreateSpatialIndex("rects", "rect:*", IndexRect); err != nil {
1873 | 		t.Fatal(err)
1874 | 	}
1875 | 	if true {
1876 | 		err := db.Update(func(tx *Tx) error {
1877 | 			if _, _, err := tx.Set("fun:user:0", "tom", nil); err != nil {
1878 | 				return err
1879 | 			}
1880 | 			if _, _, err := tx.Set("fun:user:1", "Randi", nil); err != nil {
1881 | 				return err
1882 | 			}
1883 | 			if _, _, err := tx.Set("fun:user:2", "jane", nil); err != nil {
1884 | 				return err
1885 | 			}
1886 | 			if _, _, err := tx.Set("fun:user:4", "Janet", nil); err != nil {
1887 | 				return err
1888 | 			}
1889 | 			if _, _, err := tx.Set("fun:user:5", "Paula", nil); err != nil {
1890 | 				return err
1891 | 			}
1892 | 			if _, _, err := tx.Set("fun:user:6", "peter", nil); err != nil {
1893 | 				return err
1894 | 			}
1895 | 			if _, _, err := tx.Set("fun:user:7", "Terri", nil); err != nil {
1896 | 				return err
1897 | 			}
1898 | 			return nil
1899 | 		})
1900 | 		if err != nil {
1901 | 			t.Fatal(err)
1902 | 		}
1903 | 		// add some random items
1904 | 		start := time.Now()
1905 | 		if err := db.Update(func(tx *Tx) error {
1906 | 			for _, i := range rand.Perm(100) {
1907 | 				if _, _, err := tx.Set(fmt.Sprintf("tag:%d", i+100), fmt.Sprintf("val:%d", rand.Int()%100+100), nil); err != nil {
1908 | 					return err
1909 | 				}
1910 | 			}
1911 | 			return nil
1912 | 		}); err != nil {
1913 | 			t.Fatal(err)
1914 | 		}
1915 | 		if false {
1916 | 			println(time.Since(start).String(), db.keys.Len())
1917 | 		}
1918 | 		// add some random rects
1919 | 		if err := db.Update(func(tx *Tx) error {
1920 | 			if _, _, err := tx.Set("rect:1", Rect([]float64{10, 10}, []float64{20, 20}), nil); err != nil {
1921 | 				return err
1922 | 			}
1923 | 			if _, _, err := tx.Set("rect:2", Rect([]float64{15, 15}, []float64{24, 24}), nil); err != nil {
1924 | 				return err
1925 | 			}
1926 | 			if _, _, err := tx.Set("rect:3", Rect([]float64{17, 17}, []float64{27, 27}), nil); err != nil {
1927 | 				return err
1928 | 			}
1929 | 			return nil
1930 | 		}); err != nil {
1931 | 			t.Fatal(err)
1932 | 		}
1933 | 	}
1934 | 	// verify the data has been created
1935 | 	buf := &bytes.Buffer{}
1936 | 	err := db.View(func(tx *Tx) error {
1937 | 		err := tx.Ascend("users", func(key, val string) bool {
1938 | 			fmt.Fprintf(buf, "%s %s\n", key, val)
1939 | 			return true
1940 | 		})
1941 | 		if err != nil {
1942 | 			t.Fatal(err)
1943 | 		}
1944 | 		err = tx.AscendRange("", "tag:170", "tag:172", func(key, val string) bool {
1945 | 			fmt.Fprintf(buf, "%s\n", key)
1946 | 			return true
1947 | 		})
1948 | 		if err != nil {
1949 | 			t.Fatal(err)
1950 | 		}
1951 | 		err = tx.AscendGreaterOrEqual("", "tag:195", func(key, val string) bool {
1952 | 			fmt.Fprintf(buf, "%s\n", key)
1953 | 			return true
1954 | 		})
1955 | 		if err != nil {
1956 | 			t.Fatal(err)
1957 | 		}
1958 | 		err = tx.AscendGreaterOrEqual("", "rect:", func(key, val string) bool {
1959 | 			if !strings.HasPrefix(key, "rect:") {
1960 | 				return false
1961 | 			}
1962 | 			min, max := IndexRect(val)
1963 | 			fmt.Fprintf(buf, "%s: %v,%v\n", key, min, max)
1964 | 			return true
1965 | 		})
1966 | 		if err != nil {
1967 | 			t.Fatal(err)
1968 | 		}
1969 | 		expect := make([]string, 2)
1970 | 		n := 0
1971 | 		err = tx.Intersects("rects", "[0 0],[15 15]", func(key, val string) bool {
1972 | 			if n == 2 {
1973 | 				t.Fatalf("too many rects where received, expecting only two")
1974 | 			}
1975 | 			min, max := IndexRect(val)
1976 | 			s := fmt.Sprintf("%s: %v,%v\n", key, min, max)
1977 | 			if key == "rect:1" {
1978 | 				expect[0] = s
1979 | 			} else if key == "rect:2" {
1980 | 				expect[1] = s
1981 | 			}
1982 | 			n++
1983 | 			return true
1984 | 		})
1985 | 		if err != nil {
1986 | 			t.Fatal(err)
1987 | 		}
1988 | 		for _, s := range expect {
1989 | 			if _, err := buf.WriteString(s); err != nil {
1990 | 				return err
1991 | 			}
1992 | 		}
1993 | 		return nil
1994 | 	})
1995 | 	if err != nil {
1996 | 		t.Fatal(err)
1997 | 	}
1998 | 	res := `
1999 | fun:user:2 jane
2000 | fun:user:4 Janet
2001 | fun:user:5 Paula
2002 | fun:user:6 peter
2003 | fun:user:1 Randi
2004 | fun:user:7 Terri
2005 | fun:user:0 tom
2006 | tag:170
2007 | tag:171
2008 | tag:195
2009 | tag:196
2010 | tag:197
2011 | tag:198
2012 | tag:199
2013 | rect:1: [10 10],[20 20]
2014 | rect:2: [15 15],[24 24]
2015 | rect:3: [17 17],[27 27]
2016 | rect:1: [10 10],[20 20]
2017 | rect:2: [15 15],[24 24]
2018 | `
2019 | 	res = strings.Replace(res, "\r", "", -1)
2020 | 	if strings.TrimSpace(buf.String()) != strings.TrimSpace(res) {
2021 | 		t.Fatalf("expected [%v], got [%v]", strings.TrimSpace(res), strings.TrimSpace(buf.String()))
2022 | 	}
2023 | }
2024 | 
2025 | func TestIndexAscend(t *testing.T) {
2026 | 	rand.Seed(time.Now().UnixNano())
2027 | 	db := testOpen(t)
2028 | 	defer testClose(db)
2029 | 
2030 | 	// create a simple index
2031 | 	if err := db.CreateIndex("usr", "usr:*", IndexInt); err != nil {
2032 | 		t.Fatal(err)
2033 | 	}
2034 | 	if err := db.Update(func(tx *Tx) error {
2035 | 		for i := 10; i > 0; i-- {
2036 | 			tx.Set(fmt.Sprintf("usr:%d", i), fmt.Sprintf("%d", 10-i), nil)
2037 | 		}
2038 | 		return nil
2039 | 	}); err != nil {
2040 | 		t.Fatal(err)
2041 | 	}
2042 | 
2043 | 	buf := &bytes.Buffer{}
2044 | 	err := db.View(func(tx *Tx) error {
2045 | 		tx.Ascend("usr", func(key, value string) bool {
2046 | 			fmt.Fprintf(buf, "%s %s\n", key, value)
2047 | 			return true
2048 | 		})
2049 | 		fmt.Fprintln(buf)
2050 | 
2051 | 		tx.AscendGreaterOrEqual("usr", "8", func(key, value string) bool {
2052 | 			fmt.Fprintf(buf, "%s %s\n", key, value)
2053 | 			return true
2054 | 		})
2055 | 		fmt.Fprintln(buf)
2056 | 
2057 | 		tx.AscendLessThan("usr", "3", func(key, value string) bool {
2058 | 			fmt.Fprintf(buf, "%s %s\n", key, value)
2059 | 			return true
2060 | 		})
2061 | 		fmt.Fprintln(buf)
2062 | 
2063 | 		tx.AscendRange("usr", "4", "8", func(key, value string) bool {
2064 | 			fmt.Fprintf(buf, "%s %s\n", key, value)
2065 | 			return true
2066 | 		})
2067 | 		return nil
2068 | 	})
2069 | 
2070 | 	if err != nil {
2071 | 		t.Fatal(err)
2072 | 	}
2073 | 
2074 | 	res := `
2075 | usr:10 0
2076 | usr:9 1
2077 | usr:8 2
2078 | usr:7 3
2079 | usr:6 4
2080 | usr:5 5
2081 | usr:4 6
2082 | usr:3 7
2083 | usr:2 8
2084 | usr:1 9
2085 | 
2086 | usr:2 8
2087 | usr:1 9
2088 | 
2089 | usr:10 0
2090 | usr:9 1
2091 | usr:8 2
2092 | 
2093 | usr:6 4
2094 | usr:5 5
2095 | usr:4 6
2096 | usr:3 7
2097 | `
2098 | 	res = strings.Replace(res, "\r", "", -1)
2099 | 	s1 := strings.TrimSpace(buf.String())
2100 | 	s2 := strings.TrimSpace(res)
2101 | 	if s1 != s2 {
2102 | 		t.Fatalf("expected [%v], got [%v]", s1, s2)
2103 | 	}
2104 | }
2105 | 
2106 | func testRectStringer(min, max []float64) error {
2107 | 	nmin, nmax := IndexRect(Rect(min, max))
2108 | 	if len(nmin) != len(min) {
2109 | 		return fmt.Errorf("rect=%v,%v, expect=%v,%v", nmin, nmax, min, max)
2110 | 	}
2111 | 	for i := 0; i < len(min); i++ {
2112 | 		if min[i] != nmin[i] || max[i] != nmax[i] {
2113 | 			return fmt.Errorf("rect=%v,%v, expect=%v,%v", nmin, nmax, min, max)
2114 | 		}
2115 | 	}
2116 | 	return nil
2117 | }
2118 | func TestRectStrings(t *testing.T) {
2119 | 	test(t, Rect(IndexRect(Point(1))) == "[1]", true)
2120 | 	test(t, Rect(IndexRect(Point(1, 2, 3, 4))) == "[1 2 3 4]", true)
2121 | 	test(t, Rect(IndexRect(Rect(IndexRect("[1 2],[1 2]")))) == "[1 2]", true)
2122 | 	test(t, Rect(IndexRect(Rect(IndexRect("[1 2],[2 2]")))) == "[1 2],[2 2]", true)
2123 | 	test(t, Rect(IndexRect(Rect(IndexRect("[1 2],[2 2],[3]")))) == "[1 2],[2 2]", true)
2124 | 	test(t, Rect(IndexRect(Rect(IndexRect("[1 2]")))) == "[1 2]", true)
2125 | 	test(t, Rect(IndexRect(Rect(IndexRect("[1.5 2 4.5 5.6]")))) == "[1.5 2 4.5 5.6]", true)
2126 | 	test(t, Rect(IndexRect(Rect(IndexRect("[1.5 2 4.5 5.6 -1],[]")))) == "[1.5 2 4.5 5.6 -1]", true)
2127 | 	test(t, Rect(IndexRect(Rect(IndexRect("[]")))) == "[]", true)
2128 | 	test(t, Rect(IndexRect(Rect(IndexRect("")))) == "[]", true)
2129 | 	if err := testRectStringer(nil, nil); err != nil {
2130 | 		t.Fatal(err)
2131 | 	}
2132 | 	if err := testRectStringer([]float64{}, []float64{}); err != nil {
2133 | 		t.Fatal(err)
2134 | 	}
2135 | 	if err := testRectStringer([]float64{1}, []float64{2}); err != nil {
2136 | 		t.Fatal(err)
2137 | 	}
2138 | 	if err := testRectStringer([]float64{1, 2}, []float64{3, 4}); err != nil {
2139 | 		t.Fatal(err)
2140 | 	}
2141 | 	if err := testRectStringer([]float64{1, 2, 3}, []float64{4, 5, 6}); err != nil {
2142 | 		t.Fatal(err)
2143 | 	}
2144 | 	if err := testRectStringer([]float64{1, 2, 3, 4}, []float64{5, 6, 7, 8}); err != nil {
2145 | 		t.Fatal(err)
2146 | 	}
2147 | 	if err := testRectStringer([]float64{1, 2, 3, 4, 5}, []float64{6, 7, 8, 9, 10}); err != nil {
2148 | 		t.Fatal(err)
2149 | 	}
2150 | }
2151 | 
2152 | // TestTTLReOpen test setting a TTL and then immediately closing the database and
2153 | // then waiting the TTL before reopening. The key should not be accessible.
2154 | func TestTTLReOpen(t *testing.T) {
2155 | 	ttl := time.Second * 3
2156 | 	db := testOpen(t)
2157 | 	defer testClose(db)
2158 | 	err := db.Update(func(tx *Tx) error {
2159 | 		if _, _, err := tx.Set("key1", "val1", &SetOptions{Expires: true, TTL: ttl}); err != nil {
2160 | 			return err
2161 | 		}
2162 | 		return nil
2163 | 	})
2164 | 	if err != nil {
2165 | 		t.Fatal(err)
2166 | 	}
2167 | 	db = testReOpenDelay(t, db, ttl/4)
2168 | 	err = db.View(func(tx *Tx) error {
2169 | 		val, err := tx.Get("key1")
2170 | 		if err != nil {
2171 | 			return err
2172 | 		}
2173 | 		if val != "val1" {
2174 | 			t.Fatalf("expecting '%v', got '%v'", "val1", val)
2175 | 		}
2176 | 		return nil
2177 | 	})
2178 | 	if err != nil {
2179 | 		t.Fatal(err)
2180 | 	}
2181 | 	db = testReOpenDelay(t, db, ttl-ttl/4)
2182 | 	defer testClose(db)
2183 | 	err = db.View(func(tx *Tx) error {
2184 | 		val, err := tx.Get("key1")
2185 | 		if err == nil || err != ErrNotFound || val != "" {
2186 | 			t.Fatal("expecting not found")
2187 | 		}
2188 | 
2189 | 		return nil
2190 | 	})
2191 | 	if err != nil {
2192 | 		t.Fatal(err)
2193 | 	}
2194 | }
2195 | 
2196 | func TestTTL(t *testing.T) {
2197 | 	db := testOpen(t)
2198 | 	defer testClose(db)
2199 | 	err := db.Update(func(tx *Tx) error {
2200 | 		if _, _, err := tx.Set("key1", "val1", &SetOptions{Expires: true, TTL: time.Second}); err != nil {
2201 | 			return err
2202 | 		}
2203 | 		if _, _, err := tx.Set("key2", "val2", nil); err != nil {
2204 | 			return err
2205 | 		}
2206 | 		return nil
2207 | 	})
2208 | 	if err != nil {
2209 | 		t.Fatal(err)
2210 | 	}
2211 | 	err = db.View(func(tx *Tx) error {
2212 | 		dur1, err := tx.TTL("key1")
2213 | 		if err != nil {
2214 | 			t.Fatal(err)
2215 | 		}
2216 | 		if dur1 > time.Second || dur1 <= 0 {
2217 | 			t.Fatalf("expecting between zero and one second, got '%v'", dur1)
2218 | 		}
2219 | 		dur1, err = tx.TTL("key2")
2220 | 		if err != nil {
2221 | 			t.Fatal(err)
2222 | 		}
2223 | 		if dur1 >= 0 {
2224 | 			t.Fatalf("expecting a negative value, got '%v'", dur1)
2225 | 		}
2226 | 		return nil
2227 | 	})
2228 | 	if err != nil {
2229 | 		t.Fatal(err)
2230 | 	}
2231 | }
2232 | 
2233 | func TestConfig(t *testing.T) {
2234 | 	db := testOpen(t)
2235 | 	defer testClose(db)
2236 | 
2237 | 	err := db.SetConfig(Config{SyncPolicy: SyncPolicy(-1)})
2238 | 	if err == nil {
2239 | 		t.Fatal("expecting a config syncpolicy error")
2240 | 	}
2241 | 	err = db.SetConfig(Config{SyncPolicy: SyncPolicy(3)})
2242 | 	if err == nil {
2243 | 		t.Fatal("expecting a config syncpolicy error")
2244 | 	}
2245 | 	err = db.SetConfig(Config{SyncPolicy: Never})
2246 | 	if err != nil {
2247 | 		t.Fatal(err)
2248 | 	}
2249 | 	err = db.SetConfig(Config{SyncPolicy: EverySecond})
2250 | 	if err != nil {
2251 | 		t.Fatal(err)
2252 | 	}
2253 | 	err = db.SetConfig(Config{AutoShrinkMinSize: 100, AutoShrinkPercentage: 200, SyncPolicy: Always})
2254 | 	if err != nil {
2255 | 		t.Fatal(err)
2256 | 	}
2257 | 
2258 | 	var c Config
2259 | 	if err := db.ReadConfig(&c); err != nil {
2260 | 		t.Fatal(err)
2261 | 	}
2262 | 	if c.AutoShrinkMinSize != 100 || c.AutoShrinkPercentage != 200 && c.SyncPolicy != Always {
2263 | 		t.Fatalf("expecting %v, %v, and %v, got %v, %v, and %v", 100, 200, Always, c.AutoShrinkMinSize, c.AutoShrinkPercentage, c.SyncPolicy)
2264 | 	}
2265 | }
2266 | func testUint64Hex(n uint64) string {
2267 | 	s := strconv.FormatUint(n, 16)
2268 | 	s = "0000000000000000" + s
2269 | 	return s[len(s)-16:]
2270 | }
2271 | func textHexUint64(s string) uint64 {
2272 | 	n, _ := strconv.ParseUint(s, 16, 64)
2273 | 	return n
2274 | }
2275 | func benchClose(t *testing.B, persist bool, db *DB) {
2276 | 	if persist {
2277 | 		if err := os.RemoveAll("data.db"); err != nil {
2278 | 			t.Fatal(err)
2279 | 		}
2280 | 	}
2281 | 	if err := db.Close(); err != nil {
2282 | 		t.Fatal(err)
2283 | 	}
2284 | }
2285 | 
2286 | func benchOpenFillData(t *testing.B, N int,
2287 | 	set, persist, random bool,
2288 | 	geo bool,
2289 | 	batch int) (db *DB, keys, vals []string) {
2290 | 	///
2291 | 	t.StopTimer()
2292 | 	rand.Seed(time.Now().UnixNano())
2293 | 	var err error
2294 | 	if persist {
2295 | 		if err := os.RemoveAll("data.db"); err != nil {
2296 | 			t.Fatal(err)
2297 | 		}
2298 | 		db, err = Open("data.db")
2299 | 	} else {
2300 | 		db, err = Open(":memory:")
2301 | 	}
2302 | 	if err != nil {
2303 | 		t.Fatal(err)
2304 | 	}
2305 | 	keys = make([]string, N)
2306 | 	vals = make([]string, N)
2307 | 	perm := rand.Perm(N)
2308 | 	for i := 0; i < N; i++ {
2309 | 		if random && set {
2310 | 			keys[perm[i]] = testUint64Hex(uint64(i))
2311 | 			vals[perm[i]] = strconv.FormatInt(rand.Int63()%1000+1000, 10)
2312 | 		} else {
2313 | 			keys[i] = testUint64Hex(uint64(i))
2314 | 			vals[i] = strconv.FormatInt(rand.Int63()%1000+1000, 10)
2315 | 		}
2316 | 	}
2317 | 	if set {
2318 | 		t.StartTimer()
2319 | 	}
2320 | 	for i := 0; i < N; {
2321 | 		err := db.Update(func(tx *Tx) error {
2322 | 			var err error
2323 | 			for j := 0; j < batch && i < N; j++ {
2324 | 				_, _, err = tx.Set(keys[i], vals[i], nil)
2325 | 				i++
2326 | 			}
2327 | 			return err
2328 | 		})
2329 | 		if err != nil {
2330 | 			t.Fatal(err)
2331 | 		}
2332 | 	}
2333 | 	if set {
2334 | 		t.StopTimer()
2335 | 	}
2336 | 	var n uint64
2337 | 	err = db.View(func(tx *Tx) error {
2338 | 		err := tx.Ascend("", func(key, value string) bool {
2339 | 			n2 := textHexUint64(key)
2340 | 			if n2 != n {
2341 | 				t.Fatalf("expecting '%v', got '%v'", n2, n)
2342 | 			}
2343 | 			n++
2344 | 			return true
2345 | 		})
2346 | 		return err
2347 | 	})
2348 | 	if err != nil {
2349 | 		t.Fatal(err)
2350 | 	}
2351 | 	if n != uint64(N) {
2352 | 		t.Fatalf("expecting '%v', got '%v'", N, n)
2353 | 	}
2354 | 	t.StartTimer()
2355 | 	return db, keys, vals
2356 | }
2357 | 
2358 | func benchSetGet(t *testing.B, set, persist, random bool, batch int) {
2359 | 	N := t.N
2360 | 	for N > 0 {
2361 | 		n := 0
2362 | 		if N >= 100000 {
2363 | 			n = 100000
2364 | 		} else {
2365 | 			n = N
2366 | 		}
2367 | 		func() {
2368 | 			db, keys, _ := benchOpenFillData(t, n, set, persist, random, false, batch)
2369 | 			defer benchClose(t, persist, db)
2370 | 			if !set {
2371 | 				for i := 0; i < n; {
2372 | 					err := db.View(func(tx *Tx) error {
2373 | 						var err error
2374 | 						for j := 0; j < batch && i < n; j++ {
2375 | 							_, err = tx.Get(keys[i])
2376 | 							i++
2377 | 						}
2378 | 						return err
2379 | 					})
2380 | 					if err != nil {
2381 | 						t.Fatal(err)
2382 | 					}
2383 | 				}
2384 | 			}
2385 | 		}()
2386 | 		N -= n
2387 | 	}
2388 | }
2389 | 
2390 | // Set Persist
2391 | func Benchmark_Set_Persist_Random_1(t *testing.B) {
2392 | 	benchSetGet(t, true, true, true, 1)
2393 | }
2394 | func Benchmark_Set_Persist_Random_10(t *testing.B) {
2395 | 	benchSetGet(t, true, true, true, 10)
2396 | }
2397 | func Benchmark_Set_Persist_Random_100(t *testing.B) {
2398 | 	benchSetGet(t, true, true, true, 100)
2399 | }
2400 | func Benchmark_Set_Persist_Sequential_1(t *testing.B) {
2401 | 	benchSetGet(t, true, true, false, 1)
2402 | }
2403 | func Benchmark_Set_Persist_Sequential_10(t *testing.B) {
2404 | 	benchSetGet(t, true, true, false, 10)
2405 | }
2406 | func Benchmark_Set_Persist_Sequential_100(t *testing.B) {
2407 | 	benchSetGet(t, true, true, false, 100)
2408 | }
2409 | 
2410 | // Set NoPersist
2411 | func Benchmark_Set_NoPersist_Random_1(t *testing.B) {
2412 | 	benchSetGet(t, true, false, true, 1)
2413 | }
2414 | func Benchmark_Set_NoPersist_Random_10(t *testing.B) {
2415 | 	benchSetGet(t, true, false, true, 10)
2416 | }
2417 | func Benchmark_Set_NoPersist_Random_100(t *testing.B) {
2418 | 	benchSetGet(t, true, false, true, 100)
2419 | }
2420 | func Benchmark_Set_NoPersist_Sequential_1(t *testing.B) {
2421 | 	benchSetGet(t, true, false, false, 1)
2422 | }
2423 | func Benchmark_Set_NoPersist_Sequential_10(t *testing.B) {
2424 | 	benchSetGet(t, true, false, false, 10)
2425 | }
2426 | func Benchmark_Set_NoPersist_Sequential_100(t *testing.B) {
2427 | 	benchSetGet(t, true, false, false, 100)
2428 | }
2429 | 
2430 | // Get
2431 | func Benchmark_Get_1(t *testing.B) {
2432 | 	benchSetGet(t, false, false, false, 1)
2433 | }
2434 | func Benchmark_Get_10(t *testing.B) {
2435 | 	benchSetGet(t, false, false, false, 10)
2436 | }
2437 | func Benchmark_Get_100(t *testing.B) {
2438 | 	benchSetGet(t, false, false, false, 100)
2439 | }
2440 | 
2441 | func benchScan(t *testing.B, asc bool, count int) {
2442 | 	N := count
2443 | 	db, _, _ := benchOpenFillData(t, N, false, false, false, false, 100)
2444 | 	defer benchClose(t, false, db)
2445 | 	for i := 0; i < t.N; i++ {
2446 | 		count := 0
2447 | 		err := db.View(func(tx *Tx) error {
2448 | 			if asc {
2449 | 				return tx.Ascend("", func(key, val string) bool {
2450 | 					count++
2451 | 					return true
2452 | 				})
2453 | 			}
2454 | 			return tx.Descend("", func(key, val string) bool {
2455 | 				count++
2456 | 				return true
2457 | 			})
2458 | 
2459 | 		})
2460 | 		if err != nil {
2461 | 			t.Fatal(err)
2462 | 		}
2463 | 		if count != N {
2464 | 			t.Fatalf("expecting '%v', got '%v'", N, count)
2465 | 		}
2466 | 	}
2467 | }
2468 | 
2469 | func Benchmark_Ascend_1(t *testing.B) {
2470 | 	benchScan(t, true, 1)
2471 | }
2472 | func Benchmark_Ascend_10(t *testing.B) {
2473 | 	benchScan(t, true, 10)
2474 | }
2475 | func Benchmark_Ascend_100(t *testing.B) {
2476 | 	benchScan(t, true, 100)
2477 | }
2478 | func Benchmark_Ascend_1000(t *testing.B) {
2479 | 	benchScan(t, true, 1000)
2480 | }
2481 | func Benchmark_Ascend_10000(t *testing.B) {
2482 | 	benchScan(t, true, 10000)
2483 | }
2484 | 
2485 | func Benchmark_Descend_1(t *testing.B) {
2486 | 	benchScan(t, false, 1)
2487 | }
2488 | func Benchmark_Descend_10(t *testing.B) {
2489 | 	benchScan(t, false, 10)
2490 | }
2491 | func Benchmark_Descend_100(t *testing.B) {
2492 | 	benchScan(t, false, 100)
2493 | }
2494 | func Benchmark_Descend_1000(t *testing.B) {
2495 | 	benchScan(t, false, 1000)
2496 | }
2497 | func Benchmark_Descend_10000(t *testing.B) {
2498 | 	benchScan(t, false, 10000)
2499 | }
2500 | 
2501 | /*
2502 | 	func Benchmark_Spatial_2D(t *testing.B) {
2503 | 		N := 100000
2504 | 		db, _, _ := benchOpenFillData(t, N, true, true, false, true, 100)
2505 | 		defer benchClose(t, false, db)
2506 | 
2507 | }
2508 | */
2509 | func TestCoverCloseAlreadyClosed(t *testing.T) {
2510 | 	db := testOpen(t)
2511 | 	defer testClose(db)
2512 | 	_ = db.file.Close()
2513 | 	if err := db.Close(); err == nil {
2514 | 		t.Fatal("expecting an error")
2515 | 	}
2516 | }
2517 | 
2518 | func TestCoverConfigClosed(t *testing.T) {
2519 | 	db := testOpen(t)
2520 | 	defer testClose(db)
2521 | 	_ = db.Close()
2522 | 	var config Config
2523 | 	if err := db.ReadConfig(&config); err != ErrDatabaseClosed {
2524 | 		t.Fatal("expecting database closed error")
2525 | 	}
2526 | 	if err := db.SetConfig(config); err != ErrDatabaseClosed {
2527 | 		t.Fatal("expecting database closed error")
2528 | 	}
2529 | }
2530 | func TestCoverShrinkShrink(t *testing.T) {
2531 | 	db := testOpen(t)
2532 | 	defer testClose(db)
2533 | 	if err := db.Update(func(tx *Tx) error {
2534 | 		for i := 0; i < 10000; i++ {
2535 | 			_, _, err := tx.Set(fmt.Sprintf("%d", i), fmt.Sprintf("%d", i), nil)
2536 | 			if err != nil {
2537 | 				return err
2538 | 			}
2539 | 		}
2540 | 		return nil
2541 | 	}); err != nil {
2542 | 		t.Fatal(err)
2543 | 	}
2544 | 	if err := db.Update(func(tx *Tx) error {
2545 | 		for i := 250; i < 250+100; i++ {
2546 | 			_, err := tx.Delete(fmt.Sprintf("%d", i))
2547 | 			if err != nil {
2548 | 				return err
2549 | 			}
2550 | 		}
2551 | 		return nil
2552 | 	}); err != nil {
2553 | 		t.Fatal(err)
2554 | 	}
2555 | 	var err1, err2 error
2556 | 	var wg sync.WaitGroup
2557 | 	wg.Add(2)
2558 | 	go func() {
2559 | 		defer wg.Done()
2560 | 		err1 = db.Shrink()
2561 | 	}()
2562 | 	go func() {
2563 | 		defer wg.Done()
2564 | 		err2 = db.Shrink()
2565 | 	}()
2566 | 	wg.Wait()
2567 | 	//println(123)
2568 | 	//fmt.Printf("%v\n%v\n", err1, err2)
2569 | 	if err1 != ErrShrinkInProcess && err2 != ErrShrinkInProcess {
2570 | 		t.Fatal("expecting a shrink in process error")
2571 | 	}
2572 | 	db = testReOpen(t, db)
2573 | 	defer testClose(db)
2574 | 	if err := db.View(func(tx *Tx) error {
2575 | 		n, err := tx.Len()
2576 | 		if err != nil {
2577 | 			return err
2578 | 		}
2579 | 		if n != 9900 {
2580 | 			t.Fatal("expecting 9900 items")
2581 | 		}
2582 | 		return nil
2583 | 	}); err != nil {
2584 | 		t.Fatal(err)
2585 | 	}
2586 | }
2587 | 
2588 | func TestPreviousItem(t *testing.T) {
2589 | 	db := testOpen(t)
2590 | 	defer testClose(db)
2591 | 	err := db.Update(func(tx *Tx) error {
2592 | 		_, _, err := tx.Set("hello", "world", nil)
2593 | 		if err != nil {
2594 | 			return err
2595 | 		}
2596 | 		prev, replaced, err := tx.Set("hello", "planet", nil)
2597 | 		if err != nil {
2598 | 			return err
2599 | 		}
2600 | 		if !replaced {
2601 | 			t.Fatal("should be replaced")
2602 | 		}
2603 | 		if prev != "world" {
2604 | 			t.Fatalf("expecting '%v', got '%v'", "world", prev)
2605 | 		}
2606 | 		return nil
2607 | 	})
2608 | 	if err != nil {
2609 | 		t.Fatal(err)
2610 | 	}
2611 | }
2612 | 
2613 | func TestJSONIndex(t *testing.T) {
2614 | 	db := testOpen(t)
2615 | 	defer testClose(db)
2616 | 
2617 | 	_ = db.CreateIndex("last_name", "*", IndexJSON("name.last"))
2618 | 	_ = db.CreateIndex("last_name_cs", "*", IndexJSONCaseSensitive("name.last"))
2619 | 	_ = db.CreateIndex("age", "*", IndexJSON("age"))
2620 | 	_ = db.CreateIndex("student", "*", IndexJSON("student"))
2621 | 	_ = db.Update(func(tx *Tx) error {
2622 | 		_, _, _ = tx.Set("1", `{"name":{"first":"Tom","last":"Johnson"},"age":38,"student":false}`, nil)
2623 | 		_, _, _ = tx.Set("2", `{"name":{"first":"Janet","last":"Prichard"},"age":47,"student":true}`, nil)
2624 | 		_, _, _ = tx.Set("3", `{"name":{"first":"Carol","last":"Anderson"},"age":52,"student":true}`, nil)
2625 | 		_, _, _ = tx.Set("4", `{"name":{"first":"Alan","last":"Cooper"},"age":28,"student":false}`, nil)
2626 | 		_, _, _ = tx.Set("5", `{"name":{"first":"bill","last":"frank"},"age":21,"student":true}`, nil)
2627 | 		_, _, _ = tx.Set("6", `{"name":{"first":"sally","last":"randall"},"age":68,"student":false}`, nil)
2628 | 		return nil
2629 | 	})
2630 | 	var keys []string
2631 | 	_ = db.View(func(tx *Tx) error {
2632 | 		_ = tx.Ascend("last_name_cs", func(key, value string) bool {
2633 | 			//fmt.Printf("%s: %s\n", key, value)
2634 | 			keys = append(keys, key)
2635 | 			return true
2636 | 		})
2637 | 		_ = tx.Ascend("last_name", func(key, value string) bool {
2638 | 			//fmt.Printf("%s: %s\n", key, value)
2639 | 			keys = append(keys, key)
2640 | 			return true
2641 | 		})
2642 | 		_ = tx.Ascend("age", func(key, value string) bool {
2643 | 			//fmt.Printf("%s: %s\n", key, value)
2644 | 			keys = append(keys, key)
2645 | 			return true
2646 | 		})
2647 | 		_ = tx.Ascend("student", func(key, value string) bool {
2648 | 			//fmt.Printf("%s: %s\n", key, value)
2649 | 			keys = append(keys, key)
2650 | 			return true
2651 | 		})
2652 | 		return nil
2653 | 	})
2654 | 	expect := "3,4,1,2,5,6,3,4,5,1,2,6,5,4,1,2,3,6,1,4,6,2,3,5"
2655 | 	if strings.Join(keys, ",") != expect {
2656 | 		t.Fatalf("expected %v, got %v", expect, strings.Join(keys, ","))
2657 | 	}
2658 | }
2659 | 
2660 | func TestOnExpiredSync(t *testing.T) {
2661 | 	db := testOpen(t)
2662 | 	defer testClose(db)
2663 | 
2664 | 	var config Config
2665 | 	if err := db.ReadConfig(&config); err != nil {
2666 | 		t.Fatal(err)
2667 | 	}
2668 | 	hits := make(chan int, 3)
2669 | 	config.OnExpiredSync = func(key, value string, tx *Tx) error {
2670 | 		n, err := strconv.Atoi(value)
2671 | 		if err != nil {
2672 | 			return err
2673 | 		}
2674 | 		defer func() { hits <- n }()
2675 | 		if n >= 2 {
2676 | 			_, err = tx.Delete(key)
2677 | 			if err != ErrNotFound {
2678 | 				return err
2679 | 			}
2680 | 			return nil
2681 | 		}
2682 | 		n++
2683 | 		_, _, err = tx.Set(key, strconv.Itoa(n), &SetOptions{Expires: true, TTL: time.Millisecond * 100})
2684 | 		return err
2685 | 	}
2686 | 	if err := db.SetConfig(config); err != nil {
2687 | 		t.Fatal(err)
2688 | 	}
2689 | 	err := db.Update(func(tx *Tx) error {
2690 | 		_, _, err := tx.Set("K", "0", &SetOptions{Expires: true, TTL: time.Millisecond * 100})
2691 | 		return err
2692 | 	})
2693 | 	if err != nil {
2694 | 		t.Fail()
2695 | 	}
2696 | 
2697 | 	done := make(chan struct{})
2698 | 	go func() {
2699 | 		ticks := time.NewTicker(time.Millisecond * 50)
2700 | 		defer ticks.Stop()
2701 | 		for {
2702 | 			select {
2703 | 			case <-done:
2704 | 				return
2705 | 			case <-ticks.C:
2706 | 				err := db.View(func(tx *Tx) error {
2707 | 					v, err := tx.Get("K", true)
2708 | 					if err != nil {
2709 | 						return err
2710 | 					}
2711 | 					n, err := strconv.Atoi(v)
2712 | 					if err != nil {
2713 | 						return err
2714 | 					}
2715 | 					if n < 0 || n > 2 {
2716 | 						t.Fail()
2717 | 					}
2718 | 					return nil
2719 | 				})
2720 | 				if err != nil {
2721 | 					t.Fail()
2722 | 				}
2723 | 			}
2724 | 		}
2725 | 	}()
2726 | 
2727 | OUTER1:
2728 | 	for {
2729 | 		select {
2730 | 		case <-time.After(time.Second * 2):
2731 | 			t.Fail()
2732 | 		case v := <-hits:
2733 | 			if v >= 2 {
2734 | 				break OUTER1
2735 | 			}
2736 | 		}
2737 | 	}
2738 | 	err = db.View(func(tx *Tx) error {
2739 | 		defer close(done)
2740 | 		v, err := tx.Get("K")
2741 | 		if err != nil {
2742 | 			t.Fail()
2743 | 			return err
2744 | 		}
2745 | 		if v != "2" {
2746 | 			t.Fail()
2747 | 		}
2748 | 		return nil
2749 | 	})
2750 | 	if err != nil {
2751 | 		t.Fail()
2752 | 	}
2753 | }
2754 | 
2755 | func TestTransactionLeak(t *testing.T) {
2756 | 	// This tests an bug identified in Issue #69. When inside a Update
2757 | 	// transaction, a Set after a Delete for a key that previously exists will
2758 | 	// remove the key when the transaction was rolledback.
2759 | 	buntDB, err := Open(":memory:")
2760 | 	if err != nil {
2761 | 		t.Fatal(err)
2762 | 	}
2763 | 	bd := buntDB
2764 | 	err = bd.Update(func(tx *Tx) error {
2765 | 		_, _, err := tx.Set("a", "a", nil)
2766 | 		return err
2767 | 	})
2768 | 	if err != nil {
2769 | 		t.Fatal(err)
2770 | 	}
2771 | 	bd.View(func(tx *Tx) error {
2772 | 		val, err := tx.Get("a")
2773 | 		if err != nil {
2774 | 			return err
2775 | 		}
2776 | 		if val != "a" {
2777 | 			return errors.New("mismatch")
2778 | 		}
2779 | 		return nil
2780 | 	})
2781 | 	if err != nil {
2782 | 		t.Fatal(err)
2783 | 	}
2784 | 	bd.Update(func(tx *Tx) error {
2785 | 		val, err := tx.Delete("a")
2786 | 		if err != nil {
2787 | 			return err
2788 | 		}
2789 | 		if val != "a" {
2790 | 			return errors.New("mismatch")
2791 | 		}
2792 | 		val, err = tx.Get("a")
2793 | 		if err != ErrNotFound {
2794 | 			return fmt.Errorf("expected NotFound, got %v", err)
2795 | 		}
2796 | 		if val != "" {
2797 | 			return errors.New("mismatch")
2798 | 		}
2799 | 		val, rep, err := tx.Set("a", "b", nil)
2800 | 		if err != nil {
2801 | 			return err
2802 | 		}
2803 | 		if rep {
2804 | 			return errors.New("replaced")
2805 | 		}
2806 | 		if val != "" {
2807 | 			return errors.New("mismatch")
2808 | 		}
2809 | 		val, err = tx.Get("a")
2810 | 		if err != nil {
2811 | 			return err
2812 | 		}
2813 | 		if val != "b" {
2814 | 			return errors.New("mismatch")
2815 | 		}
2816 | 		return errors.New("rollback")
2817 | 	})
2818 | 	if err != nil {
2819 | 		t.Fatal(err)
2820 | 	}
2821 | 	bd.View(func(tx *Tx) error {
2822 | 		val, err := tx.Get("a")
2823 | 		if err != nil {
2824 | 			return err
2825 | 		}
2826 | 		if val != "a" {
2827 | 			return errors.New("mismatch")
2828 | 		}
2829 | 		return nil
2830 | 	})
2831 | 	if err != nil {
2832 | 		t.Fatal(err)
2833 | 	}
2834 | }
2835 | 
2836 | func TestReloadNotInvalid(t *testing.T) {
2837 | 	rand.Seed(time.Now().UnixNano())
2838 | 	os.RemoveAll("data.db")
2839 | 	defer os.RemoveAll("data.db")
2840 | 	start := time.Now()
2841 | 	ii := 0
2842 | 	for time.Since(start) < time.Second*5 {
2843 | 		func() {
2844 | 			db, err := Open("data.db")
2845 | 			if err != nil {
2846 | 				t.Fatal(err)
2847 | 			}
2848 | 			defer func() {
2849 | 				if err := db.Close(); err != nil {
2850 | 					panic(err)
2851 | 				}
2852 | 				// truncate at a random point in the file
2853 | 				f, err := os.OpenFile("data.db", os.O_RDWR, 0666)
2854 | 				if err != nil {
2855 | 					panic(err)
2856 | 				}
2857 | 				defer f.Close()
2858 | 				sz, err := f.Seek(0, 2)
2859 | 				if err != nil {
2860 | 					panic(err)
2861 | 				}
2862 | 				n := sz/2 + int64(rand.Intn(int(sz/2)))
2863 | 				err = f.Truncate(n)
2864 | 				if err != nil {
2865 | 					panic(err)
2866 | 				}
2867 | 			}()
2868 | 			N := 500
2869 | 			lotsa.Ops(N, 16, func(i, t int) {
2870 | 				if i == N/2 && ii&7 == 0 {
2871 | 					if err := db.Shrink(); err != nil {
2872 | 						panic(err)
2873 | 					}
2874 | 				}
2875 | 				err := db.Update(func(tx *Tx) error {
2876 | 					_, _, err := tx.Set(fmt.Sprintf("key:%d", i), fmt.Sprintf("val:%d", i), &SetOptions{
2877 | 						Expires: true,
2878 | 						TTL:     30,
2879 | 					})
2880 | 					return err
2881 | 				})
2882 | 				if err != nil {
2883 | 					panic(err)
2884 | 				}
2885 | 			})
2886 | 		}()
2887 | 		ii++
2888 | 	}
2889 | }
2890 | 
2891 | func TestEstSize(t *testing.T) {
2892 | 	t.Run("estIntSize", func(t *testing.T) {
2893 | 		assert.Assert(estIntSize(0) == 1)
2894 | 		assert.Assert(estIntSize(1) == 1)
2895 | 		assert.Assert(estIntSize(9) == 1)
2896 | 		assert.Assert(estIntSize(10) == 2)
2897 | 		assert.Assert(estIntSize(11) == 2)
2898 | 		assert.Assert(estIntSize(19) == 2)
2899 | 		assert.Assert(estIntSize(20) == 2)
2900 | 		assert.Assert(estIntSize(113) == 3)
2901 | 		assert.Assert(estIntSize(3822) == 4)
2902 | 		assert.Assert(estIntSize(-1) == 2)
2903 | 		assert.Assert(estIntSize(-12) == 3)
2904 | 		assert.Assert(estIntSize(-124) == 4)
2905 | 	})
2906 | }
2907 | 
2908 | func TestWrappedError(t *testing.T) {
2909 | 	defer func() {
2910 | 		if err, ok := recover().(error); ok {
2911 | 			if strings.HasPrefix(err.Error(), "buntdb: ") {
2912 | 				err := errors.Unwrap(err)
2913 | 				if err.Error() != "my fake error" {
2914 | 					t.Fatal("!")
2915 | 				}
2916 | 			}
2917 | 		}
2918 | 	}()
2919 | 	panicErr(errors.New("my fake error"))
2920 | }
2921 | 
2922 | func TestIssue112(t *testing.T) {
2923 | 	defer os.RemoveAll("data.db")
2924 | 	db, err := Open("data.db")
2925 | 	if err != nil {
2926 | 		panicErr(err)
2927 | 	}
2928 | 	err = db.Update(func(tx *Tx) error {
2929 | 		_, _, err := tx.Set("key:112", "value:112-old", nil)
2930 | 		return err
2931 | 	})
2932 | 	if err != nil {
2933 | 		panicErr(err)
2934 | 	}
2935 | 	err = db.Update(func(tx *Tx) error {
2936 | 		_, _, err := tx.Set("key:112", "value:112", &SetOptions{Expires: true, TTL: time.Second})
2937 | 		return err
2938 | 	})
2939 | 	if err != nil {
2940 | 		panicErr(err)
2941 | 	}
2942 | 	db.Close()
2943 | 	time.Sleep(time.Second)
2944 | 	db, err = Open("data.db")
2945 | 	if err != nil {
2946 | 		panicErr(err)
2947 | 	}
2948 | 	err = db.View(func(tx *Tx) error {
2949 | 		_, err := tx.Get("key:112")
2950 | 		return err
2951 | 	})
2952 | 	assert.Assert(err == ErrNotFound)
2953 | 	db.View(func(tx *Tx) error {
2954 | 		err := tx.Ascend("", func(key, value string) bool {
2955 | 			t.Fatalf("key: %s, value: %s\n", key, value)
2956 | 			return true
2957 | 		})
2958 | 		return err
2959 | 	})
2960 | }
2961 | 


--------------------------------------------------------------------------------
/go.mod:
--------------------------------------------------------------------------------
 1 | module github.com/tidwall/buntdb
 2 | 
 3 | go 1.18
 4 | 
 5 | require (
 6 | 	github.com/tidwall/assert v0.1.0
 7 | 	github.com/tidwall/btree v1.4.2
 8 | 	github.com/tidwall/gjson v1.14.3
 9 | 	github.com/tidwall/grect v0.1.4
10 | 	github.com/tidwall/lotsa v1.0.2
11 | 	github.com/tidwall/match v1.1.1
12 | 	github.com/tidwall/rtred v0.1.2
13 | )
14 | 
15 | require (
16 | 	github.com/tidwall/pretty v1.2.0 // indirect
17 | 	github.com/tidwall/tinyqueue v0.1.1 // indirect
18 | )
19 | 


--------------------------------------------------------------------------------
/go.sum:
--------------------------------------------------------------------------------
 1 | github.com/tidwall/assert v0.1.0 h1:aWcKyRBUAdLoVebxo95N7+YZVTFF/ASTr7BN4sLP6XI=
 2 | github.com/tidwall/assert v0.1.0/go.mod h1:QLYtGyeqse53vuELQheYl9dngGCJQ+mTtlxcktb+Kj8=
 3 | github.com/tidwall/btree v1.4.2 h1:PpkaieETJMUxYNADsjgtNRcERX7mGc/GP2zp/r5FM3g=
 4 | github.com/tidwall/btree v1.4.2/go.mod h1:LGm8L/DZjPLmeWGjv5kFrY8dL4uVhMmzmmLYmsObdKE=
 5 | github.com/tidwall/gjson v1.12.1/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
 6 | github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
 7 | github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
 8 | github.com/tidwall/grect v0.1.4 h1:dA3oIgNgWdSspFzn1kS4S/RDpZFLrIxAZOdJKjYapOg=
 9 | github.com/tidwall/grect v0.1.4/go.mod h1:9FBsaYRaR0Tcy4UwefBX/UDcDcDy9V5jUcxHzv2jd5Q=
10 | github.com/tidwall/lotsa v1.0.2 h1:dNVBH5MErdaQ/xd9s769R31/n2dXavsQ0Yf4TMEHHw8=
11 | github.com/tidwall/lotsa v1.0.2/go.mod h1:X6NiU+4yHA3fE3Puvpnn1XMDrFZrE9JO2/w+UMuqgR8=
12 | github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
13 | github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
14 | github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs=
15 | github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
16 | github.com/tidwall/rtred v0.1.2 h1:exmoQtOLvDoO8ud++6LwVsAMTu0KPzLTUrMln8u1yu8=
17 | github.com/tidwall/rtred v0.1.2/go.mod h1:hd69WNXQ5RP9vHd7dqekAz+RIdtfBogmglkZSRxCHFQ=
18 | github.com/tidwall/tinyqueue v0.1.1 h1:SpNEvEggbpyN5DIReaJ2/1ndroY8iyEGxPYxoSaymYE=
19 | github.com/tidwall/tinyqueue v0.1.1/go.mod h1:O/QNHwrnjqr6IHItYrzoHAKYhBkLI67Q096fQP5zMYw=
20 | 


--------------------------------------------------------------------------------
/logo.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/tidwall/buntdb/3daff4e1233584685027938bde39971cc239f2b2/logo.png


--------------------------------------------------------------------------------