├── .github ├── ISSUE_TEMPLATE │ ├── config.yml │ └── open_an_issue.md ├── config.yml └── workflows │ ├── generated-pr.yml │ ├── go-check.yml │ ├── go-test.yml │ ├── release-check.yml │ ├── releaser.yml │ ├── stale.yml │ └── tagpush.yml ├── .gitignore ├── .gx └── lastpubver ├── LICENSE ├── README.md ├── datastore.go ├── ds_test.go ├── go.mod ├── go.sum └── version.json /.github/ISSUE_TEMPLATE/config.yml: -------------------------------------------------------------------------------- 1 | blank_issues_enabled: false 2 | contact_links: 3 | - name: Getting Help on IPFS 4 | url: https://ipfs.io/help 5 | about: All information about how and where to get help on IPFS. 6 | - name: IPFS Official Forum 7 | url: https://discuss.ipfs.io 8 | about: Please post general questions, support requests, and discussions here. 9 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/open_an_issue.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Open an issue 3 | about: Only for actionable issues relevant to this repository. 4 | title: '' 5 | labels: need/triage 6 | assignees: '' 7 | 8 | --- 9 | 20 | -------------------------------------------------------------------------------- /.github/config.yml: -------------------------------------------------------------------------------- 1 | # Configuration for welcome - https://github.com/behaviorbot/welcome 2 | 3 | # Configuration for new-issue-welcome - https://github.com/behaviorbot/new-issue-welcome 4 | # Comment to be posted to on first time issues 5 | newIssueWelcomeComment: > 6 | Thank you for submitting your first issue to this repository! A maintainer 7 | will be here shortly to triage and review. 8 | 9 | In the meantime, please double-check that you have provided all the 10 | necessary information to make this process easy! Any information that can 11 | help save additional round trips is useful! We currently aim to give 12 | initial feedback within **two business days**. If this does not happen, feel 13 | free to leave a comment. 14 | 15 | Please keep an eye on how this issue will be labeled, as labels give an 16 | overview of priorities, assignments and additional actions requested by the 17 | maintainers: 18 | 19 | - "Priority" labels will show how urgent this is for the team. 20 | - "Status" labels will show if this is ready to be worked on, blocked, or in progress. 21 | - "Need" labels will indicate if additional input or analysis is required. 22 | 23 | Finally, remember to use https://discuss.ipfs.io if you just need general 24 | support. 25 | 26 | # Configuration for new-pr-welcome - https://github.com/behaviorbot/new-pr-welcome 27 | # Comment to be posted to on PRs from first time contributors in your repository 28 | newPRWelcomeComment: > 29 | Thank you for submitting this PR! 30 | 31 | A maintainer will be here shortly to review it. 32 | 33 | We are super grateful, but we are also overloaded! Help us by making sure 34 | that: 35 | 36 | * The context for this PR is clear, with relevant discussion, decisions 37 | and stakeholders linked/mentioned. 38 | 39 | * Your contribution itself is clear (code comments, self-review for the 40 | rest) and in its best form. Follow the [code contribution 41 | guidelines](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md#code-contribution-guidelines) 42 | if they apply. 43 | 44 | Getting other community members to do a review would be great help too on 45 | complex PRs (you can ask in the chats/forums). If you are unsure about 46 | something, just leave us a comment. 47 | 48 | Next steps: 49 | 50 | * A maintainer will triage and assign priority to this PR, commenting on 51 | any missing things and potentially assigning a reviewer for high 52 | priority items. 53 | 54 | * The PR gets reviews, discussed and approvals as needed. 55 | 56 | * The PR is merged by maintainers when it has been approved and comments addressed. 57 | 58 | We currently aim to provide initial feedback/triaging within **two business 59 | days**. Please keep an eye on any labelling actions, as these will indicate 60 | priorities and status of your contribution. 61 | 62 | We are very grateful for your contribution! 63 | 64 | 65 | # Configuration for first-pr-merge - https://github.com/behaviorbot/first-pr-merge 66 | # Comment to be posted to on pull requests merged by a first time user 67 | # Currently disabled 68 | #firstPRMergeComment: "" 69 | -------------------------------------------------------------------------------- /.github/workflows/generated-pr.yml: -------------------------------------------------------------------------------- 1 | name: Close Generated PRs 2 | 3 | on: 4 | schedule: 5 | - cron: '0 0 * * *' 6 | workflow_dispatch: 7 | 8 | permissions: 9 | issues: write 10 | pull-requests: write 11 | 12 | jobs: 13 | stale: 14 | uses: ipdxco/unified-github-workflows/.github/workflows/reusable-generated-pr.yml@v1 15 | -------------------------------------------------------------------------------- /.github/workflows/go-check.yml: -------------------------------------------------------------------------------- 1 | name: Go Checks 2 | 3 | on: 4 | pull_request: 5 | push: 6 | branches: ["master"] 7 | workflow_dispatch: 8 | 9 | permissions: 10 | contents: read 11 | 12 | concurrency: 13 | group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} 14 | cancel-in-progress: true 15 | 16 | jobs: 17 | go-check: 18 | uses: ipdxco/unified-github-workflows/.github/workflows/go-check.yml@v1.0 19 | -------------------------------------------------------------------------------- /.github/workflows/go-test.yml: -------------------------------------------------------------------------------- 1 | name: Go Test 2 | 3 | on: 4 | pull_request: 5 | push: 6 | branches: ["master"] 7 | workflow_dispatch: 8 | 9 | permissions: 10 | contents: read 11 | 12 | concurrency: 13 | group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }} 14 | cancel-in-progress: true 15 | 16 | jobs: 17 | go-test: 18 | uses: ipdxco/unified-github-workflows/.github/workflows/go-test.yml@v1.0 19 | secrets: 20 | CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} 21 | -------------------------------------------------------------------------------- /.github/workflows/release-check.yml: -------------------------------------------------------------------------------- 1 | name: Release Checker 2 | 3 | on: 4 | pull_request_target: 5 | paths: [ 'version.json' ] 6 | types: [ opened, synchronize, reopened, labeled, unlabeled ] 7 | workflow_dispatch: 8 | 9 | permissions: 10 | contents: write 11 | pull-requests: write 12 | 13 | concurrency: 14 | group: ${{ github.workflow }}-${{ github.ref }} 15 | cancel-in-progress: true 16 | 17 | jobs: 18 | release-check: 19 | uses: ipdxco/unified-github-workflows/.github/workflows/release-check.yml@v1.0 20 | -------------------------------------------------------------------------------- /.github/workflows/releaser.yml: -------------------------------------------------------------------------------- 1 | name: Releaser 2 | 3 | on: 4 | push: 5 | paths: [ 'version.json' ] 6 | workflow_dispatch: 7 | 8 | permissions: 9 | contents: write 10 | 11 | concurrency: 12 | group: ${{ github.workflow }}-${{ github.sha }} 13 | cancel-in-progress: true 14 | 15 | jobs: 16 | releaser: 17 | uses: ipdxco/unified-github-workflows/.github/workflows/releaser.yml@v1.0 18 | -------------------------------------------------------------------------------- /.github/workflows/stale.yml: -------------------------------------------------------------------------------- 1 | name: Close Stale Issues 2 | 3 | on: 4 | schedule: 5 | - cron: '0 0 * * *' 6 | workflow_dispatch: 7 | 8 | permissions: 9 | issues: write 10 | pull-requests: write 11 | 12 | jobs: 13 | stale: 14 | uses: ipdxco/unified-github-workflows/.github/workflows/reusable-stale-issue.yml@v1 15 | -------------------------------------------------------------------------------- /.github/workflows/tagpush.yml: -------------------------------------------------------------------------------- 1 | name: Tag Push Checker 2 | 3 | on: 4 | push: 5 | tags: 6 | - v* 7 | 8 | permissions: 9 | contents: read 10 | issues: write 11 | 12 | concurrency: 13 | group: ${{ github.workflow }}-${{ github.ref }} 14 | cancel-in-progress: true 15 | 16 | jobs: 17 | releaser: 18 | uses: ipdxco/unified-github-workflows/.github/workflows/tagpush.yml@v1.0 19 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.swp 2 | -------------------------------------------------------------------------------- /.gx/lastpubver: -------------------------------------------------------------------------------- 1 | 1.3.0: QmbgYmpUkuCDnXi4hci3Jt797iVXbpuBKRTCqGz57h48Sk 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License 2 | 3 | Copyright (c) 2016 Jeromy Johnson 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # go-ds-leveldb 2 | 3 | [![](https://img.shields.io/badge/made%20by-Protocol%20Labs-blue.svg?style=flat-square)](http://ipn.io) 4 | [![](https://img.shields.io/badge/project-IPFS-blue.svg?style=flat-square)](http://ipfs.io/) 5 | [![standard-readme compliant](https://img.shields.io/badge/standard--readme-OK-green.svg?style=flat-square)](https://github.com/RichardLitt/standard-readme) 6 | [![GoDoc](https://godoc.org/github.com/ipfs/go-ds-leveldb?status.svg)](https://godoc.org/github.com/ipfs/go-ds-leveldb) 7 | [![Build Status](https://travis-ci.org/ipfs/go-ds-leveldb.svg?branch=master)](https://travis-ci.org/ipfs/go-ds-leveldb) 8 | 9 | > A go-datastore implementation using LevelDB 10 | 11 | `go-ds-leveldb` implements the [go-datastore](https://github.com/ipfs/go-datastore) interface using a LevelDB backend. 12 | 13 | ## Lead Maintainer 14 | 15 | [Steven Allen](https://github.com/Stebalien) 16 | 17 | ## Table of Contents 18 | 19 | - [Install](#install) 20 | - [Usage](#usage) 21 | - [Contribute](#contribute) 22 | - [License](#license) 23 | 24 | ## Install 25 | 26 | This module can be installed like a regular go module: 27 | 28 | ``` 29 | go get github.com/ipfs/go-ds-leveldb 30 | ``` 31 | 32 | It uses [Gx](https://github.com/whyrusleeping/gx) to manage dependencies. You can use `make deps` to rewrite imports to the gx-specified versions. 33 | 34 | ## Usage 35 | 36 | ``` 37 | import "github.com/ipfs/go-ds-leveldb" 38 | ``` 39 | 40 | Check the [GoDoc documentation](https://godoc.org/github.com/ipfs/go-ds-leveldb) 41 | 42 | 43 | ## Contribute 44 | 45 | PRs accepted. 46 | 47 | Small note: If editing the README, please conform to the [standard-readme](https://github.com/RichardLitt/standard-readme) specification. 48 | 49 | ## License 50 | 51 | MIT © Protocol Labs, Inc. 52 | -------------------------------------------------------------------------------- /datastore.go: -------------------------------------------------------------------------------- 1 | package leveldb 2 | 3 | import ( 4 | "context" 5 | "os" 6 | "path/filepath" 7 | "sync" 8 | 9 | ds "github.com/ipfs/go-datastore" 10 | dsq "github.com/ipfs/go-datastore/query" 11 | "github.com/syndtr/goleveldb/leveldb" 12 | "github.com/syndtr/goleveldb/leveldb/errors" 13 | "github.com/syndtr/goleveldb/leveldb/iterator" 14 | "github.com/syndtr/goleveldb/leveldb/opt" 15 | "github.com/syndtr/goleveldb/leveldb/storage" 16 | "github.com/syndtr/goleveldb/leveldb/util" 17 | ) 18 | 19 | type Datastore struct { 20 | *accessor 21 | DB *leveldb.DB 22 | path string 23 | } 24 | 25 | var _ ds.Datastore = (*Datastore)(nil) 26 | var _ ds.TxnDatastore = (*Datastore)(nil) 27 | var _ ds.Txn = (*transaction)(nil) 28 | var _ ds.PersistentDatastore = (*Datastore)(nil) 29 | var _ ds.Batching = (*Datastore)(nil) 30 | var _ ds.Batch = (*leveldbBatch)(nil) 31 | 32 | // Options is an alias of syndtr/goleveldb/opt.Options which might be extended 33 | // in the future. 34 | type Options opt.Options 35 | 36 | // NewDatastore returns a new datastore backed by leveldb 37 | // 38 | // for path == "", an in memory backend will be chosen 39 | func NewDatastore(path string, opts *Options) (*Datastore, error) { 40 | var nopts opt.Options 41 | if opts != nil { 42 | nopts = opt.Options(*opts) 43 | } 44 | 45 | var err error 46 | var db *leveldb.DB 47 | 48 | if path == "" { 49 | db, err = leveldb.Open(storage.NewMemStorage(), &nopts) 50 | } else { 51 | db, err = leveldb.OpenFile(path, &nopts) 52 | if errors.IsCorrupted(err) && !nopts.GetReadOnly() { 53 | db, err = leveldb.RecoverFile(path, &nopts) 54 | } 55 | } 56 | 57 | if err != nil { 58 | return nil, err 59 | } 60 | 61 | ds := Datastore{ 62 | accessor: &accessor{ldb: db, syncWrites: true, closeLk: new(sync.RWMutex)}, 63 | DB: db, 64 | path: path, 65 | } 66 | return &ds, nil 67 | } 68 | 69 | // An extraction of the common interface between LevelDB Transactions and the DB itself. 70 | // 71 | // It allows to plug in either inside the `accessor`. 72 | type levelDbOps interface { 73 | Put(key, value []byte, wo *opt.WriteOptions) error 74 | Get(key []byte, ro *opt.ReadOptions) (value []byte, err error) 75 | Has(key []byte, ro *opt.ReadOptions) (ret bool, err error) 76 | Delete(key []byte, wo *opt.WriteOptions) error 77 | NewIterator(slice *util.Range, ro *opt.ReadOptions) iterator.Iterator 78 | } 79 | 80 | // Datastore operations using either the DB or a transaction as the backend. 81 | type accessor struct { 82 | ldb levelDbOps 83 | syncWrites bool 84 | closeLk *sync.RWMutex 85 | } 86 | 87 | func (a *accessor) Put(ctx context.Context, key ds.Key, value []byte) (err error) { 88 | a.closeLk.RLock() 89 | defer a.closeLk.RUnlock() 90 | return a.ldb.Put(key.Bytes(), value, &opt.WriteOptions{Sync: a.syncWrites}) 91 | } 92 | 93 | func (a *accessor) Sync(ctx context.Context, prefix ds.Key) error { 94 | return nil 95 | } 96 | 97 | func (a *accessor) Get(ctx context.Context, key ds.Key) (value []byte, err error) { 98 | a.closeLk.RLock() 99 | defer a.closeLk.RUnlock() 100 | val, err := a.ldb.Get(key.Bytes(), nil) 101 | if err != nil { 102 | if err == leveldb.ErrNotFound { 103 | return nil, ds.ErrNotFound 104 | } 105 | return nil, err 106 | } 107 | return val, nil 108 | } 109 | 110 | func (a *accessor) Has(ctx context.Context, key ds.Key) (exists bool, err error) { 111 | a.closeLk.RLock() 112 | defer a.closeLk.RUnlock() 113 | return a.ldb.Has(key.Bytes(), nil) 114 | } 115 | 116 | func (a *accessor) GetSize(ctx context.Context, key ds.Key) (size int, err error) { 117 | return ds.GetBackedSize(ctx, a, key) 118 | } 119 | 120 | func (a *accessor) Delete(ctx context.Context, key ds.Key) (err error) { 121 | a.closeLk.RLock() 122 | defer a.closeLk.RUnlock() 123 | return a.ldb.Delete(key.Bytes(), &opt.WriteOptions{Sync: a.syncWrites}) 124 | } 125 | 126 | func (a *accessor) Query(ctx context.Context, q dsq.Query) (dsq.Results, error) { 127 | a.closeLk.RLock() 128 | defer a.closeLk.RUnlock() 129 | var rnge *util.Range 130 | 131 | // make a copy of the query for the fallback naive query implementation. 132 | // don't modify the original so res.Query() returns the correct results. 133 | qNaive := q 134 | prefix := ds.NewKey(q.Prefix).String() 135 | if prefix != "/" { 136 | rnge = util.BytesPrefix([]byte(prefix + "/")) 137 | qNaive.Prefix = "" 138 | } 139 | i := a.ldb.NewIterator(rnge, nil) 140 | next := i.Next 141 | if len(q.Orders) > 0 { 142 | switch q.Orders[0].(type) { 143 | case dsq.OrderByKey, *dsq.OrderByKey: 144 | qNaive.Orders = nil 145 | case dsq.OrderByKeyDescending, *dsq.OrderByKeyDescending: 146 | next = func() bool { 147 | next = i.Prev 148 | return i.Last() 149 | } 150 | qNaive.Orders = nil 151 | default: 152 | } 153 | } 154 | r := dsq.ResultsFromIterator(q, dsq.Iterator{ 155 | Next: func() (dsq.Result, bool) { 156 | a.closeLk.RLock() 157 | defer a.closeLk.RUnlock() 158 | if !next() { 159 | return dsq.Result{}, false 160 | } 161 | k := string(i.Key()) 162 | e := dsq.Entry{Key: k, Size: len(i.Value())} 163 | 164 | if !q.KeysOnly { 165 | buf := make([]byte, len(i.Value())) 166 | copy(buf, i.Value()) 167 | e.Value = buf 168 | } 169 | return dsq.Result{Entry: e}, true 170 | }, 171 | Close: func() error { 172 | a.closeLk.RLock() 173 | defer a.closeLk.RUnlock() 174 | i.Release() 175 | return nil 176 | }, 177 | }) 178 | return dsq.NaiveQueryApply(qNaive, r), nil 179 | } 180 | 181 | // DiskUsage returns the current disk size used by this levelDB. 182 | // For in-mem datastores, it will return 0. 183 | func (d *Datastore) DiskUsage(ctx context.Context) (uint64, error) { 184 | d.closeLk.RLock() 185 | defer d.closeLk.RUnlock() 186 | if d.path == "" { // in-mem 187 | return 0, nil 188 | } 189 | 190 | var du uint64 191 | 192 | err := filepath.Walk(d.path, func(path string, info os.FileInfo, err error) error { 193 | if err != nil { 194 | return err 195 | } 196 | du += uint64(info.Size()) 197 | return nil 198 | }) 199 | 200 | if err != nil { 201 | return 0, err 202 | } 203 | 204 | return du, nil 205 | } 206 | 207 | // LevelDB needs to be closed. 208 | func (d *Datastore) Close() error { 209 | d.closeLk.Lock() 210 | defer d.closeLk.Unlock() 211 | return d.DB.Close() 212 | } 213 | 214 | type leveldbBatch struct { 215 | b *leveldb.Batch 216 | db *leveldb.DB 217 | closeLk *sync.RWMutex 218 | syncWrites bool 219 | } 220 | 221 | func (d *Datastore) Batch(ctx context.Context) (ds.Batch, error) { 222 | return &leveldbBatch{ 223 | b: new(leveldb.Batch), 224 | db: d.DB, 225 | closeLk: d.closeLk, 226 | syncWrites: d.syncWrites, 227 | }, nil 228 | } 229 | 230 | func (b *leveldbBatch) Put(ctx context.Context, key ds.Key, value []byte) error { 231 | b.b.Put(key.Bytes(), value) 232 | return nil 233 | } 234 | 235 | func (b *leveldbBatch) Commit(ctx context.Context) error { 236 | b.closeLk.RLock() 237 | defer b.closeLk.RUnlock() 238 | return b.db.Write(b.b, &opt.WriteOptions{Sync: b.syncWrites}) 239 | } 240 | 241 | func (b *leveldbBatch) Delete(ctx context.Context, key ds.Key) error { 242 | b.b.Delete(key.Bytes()) 243 | return nil 244 | } 245 | 246 | // A leveldb transaction embedding the accessor backed by the transaction. 247 | type transaction struct { 248 | *accessor 249 | tx *leveldb.Transaction 250 | } 251 | 252 | func (t *transaction) Commit(ctx context.Context) error { 253 | t.closeLk.RLock() 254 | defer t.closeLk.RUnlock() 255 | return t.tx.Commit() 256 | } 257 | 258 | func (t *transaction) Discard(ctx context.Context) { 259 | t.closeLk.RLock() 260 | defer t.closeLk.RUnlock() 261 | t.tx.Discard() 262 | } 263 | 264 | func (d *Datastore) NewTransaction(ctx context.Context, readOnly bool) (ds.Txn, error) { 265 | d.closeLk.RLock() 266 | defer d.closeLk.RUnlock() 267 | tx, err := d.DB.OpenTransaction() 268 | if err != nil { 269 | return nil, err 270 | } 271 | accessor := &accessor{ldb: tx, syncWrites: false, closeLk: d.closeLk} 272 | return &transaction{accessor, tx}, nil 273 | } 274 | -------------------------------------------------------------------------------- /ds_test.go: -------------------------------------------------------------------------------- 1 | package leveldb 2 | 3 | import ( 4 | "bytes" 5 | "context" 6 | "fmt" 7 | "os" 8 | "sort" 9 | "testing" 10 | 11 | ds "github.com/ipfs/go-datastore" 12 | dsq "github.com/ipfs/go-datastore/query" 13 | dstest "github.com/ipfs/go-datastore/test" 14 | ) 15 | 16 | var testcases = map[string]string{ 17 | "/a": "a", 18 | "/a/b": "ab", 19 | "/a/b/c": "abc", 20 | "/a/b/d": "a/b/d", 21 | "/a/c": "ac", 22 | "/a/d": "ad", 23 | "/e": "e", 24 | "/f": "f", 25 | } 26 | 27 | var bg = context.Background() 28 | 29 | // returns datastore, and a function to call on exit. 30 | // (this garbage collects). So: 31 | // 32 | // d, close := newDS(t) 33 | // defer close() 34 | func newDS(t *testing.T) (*Datastore, func()) { 35 | path, err := os.MkdirTemp("", "testing_leveldb_") 36 | if err != nil { 37 | t.Fatal(err) 38 | } 39 | 40 | d, err := NewDatastore(path, nil) 41 | if err != nil { 42 | t.Fatal(err) 43 | } 44 | return d, func() { 45 | if err := d.Close(); err != nil { 46 | t.Fatal(err) 47 | } 48 | if err := os.RemoveAll(path); err != nil { 49 | t.Fatal(err) 50 | } 51 | } 52 | } 53 | 54 | // newDSMem returns an in-memory datastore. 55 | func newDSMem(t *testing.T) *Datastore { 56 | d, err := NewDatastore("", nil) 57 | if err != nil { 58 | t.Fatal(err) 59 | } 60 | return d 61 | } 62 | 63 | func addTestCases(t *testing.T, d *Datastore, testcases map[string]string) { 64 | for k, v := range testcases { 65 | dsk := ds.NewKey(k) 66 | if err := d.Put(bg, dsk, []byte(v)); err != nil { 67 | t.Fatal(err) 68 | } 69 | } 70 | 71 | for k, v := range testcases { 72 | dsk := ds.NewKey(k) 73 | v2, err := d.Get(bg, dsk) 74 | if err != nil { 75 | t.Fatal(err) 76 | } 77 | if string(v2) != v { 78 | t.Errorf("%s values differ: %s != %s", k, v, v2) 79 | } 80 | } 81 | 82 | } 83 | 84 | func testQuery(t *testing.T, d *Datastore) { 85 | addTestCases(t, d, testcases) 86 | 87 | rs, err := d.Query(bg, dsq.Query{Prefix: "/a/"}) 88 | if err != nil { 89 | t.Fatal(err) 90 | } 91 | 92 | expectMatches(t, []string{ 93 | "/a/b", 94 | "/a/b/c", 95 | "/a/b/d", 96 | "/a/c", 97 | "/a/d", 98 | }, rs) 99 | 100 | // test offset and limit 101 | 102 | rs, err = d.Query(bg, dsq.Query{Prefix: "/a/", Offset: 2, Limit: 2}) 103 | if err != nil { 104 | t.Fatal(err) 105 | } 106 | 107 | expectMatches(t, []string{ 108 | "/a/b/d", 109 | "/a/c", 110 | }, rs) 111 | 112 | // test order 113 | 114 | rs, err = d.Query(bg, dsq.Query{Orders: []dsq.Order{dsq.OrderByKey{}}}) 115 | if err != nil { 116 | t.Fatal(err) 117 | } 118 | 119 | keys := make([]string, 0, len(testcases)) 120 | for k := range testcases { 121 | keys = append(keys, k) 122 | } 123 | sort.Strings(keys) 124 | 125 | expectOrderedMatches(t, keys, rs) 126 | 127 | rs, err = d.Query(bg, dsq.Query{Orders: []dsq.Order{dsq.OrderByKeyDescending{}}}) 128 | if err != nil { 129 | t.Fatal(err) 130 | } 131 | 132 | // reverse 133 | for i, j := 0, len(keys)-1; i < j; i, j = i+1, j-1 { 134 | keys[i], keys[j] = keys[j], keys[i] 135 | } 136 | 137 | expectOrderedMatches(t, keys, rs) 138 | } 139 | 140 | func TestQuery(t *testing.T) { 141 | d, close := newDS(t) 142 | defer close() 143 | testQuery(t, d) 144 | } 145 | func TestQueryMem(t *testing.T) { 146 | d := newDSMem(t) 147 | testQuery(t, d) 148 | } 149 | 150 | func TestQueryRespectsProcess(t *testing.T) { 151 | d, close := newDS(t) 152 | defer close() 153 | addTestCases(t, d, testcases) 154 | } 155 | 156 | func TestCloseRace(t *testing.T) { 157 | d, close := newDS(t) 158 | for n := 0; n < 100; n++ { 159 | d.Put(bg, ds.NewKey(fmt.Sprintf("%d", n)), []byte(fmt.Sprintf("test%d", n))) 160 | } 161 | 162 | tx, _ := d.NewTransaction(bg, false) 163 | tx.Put(bg, ds.NewKey("txnversion"), []byte("bump")) 164 | 165 | closeCh := make(chan interface{}) 166 | 167 | go func() { 168 | close() 169 | closeCh <- nil 170 | }() 171 | for k := range testcases { 172 | tx.Get(bg, ds.NewKey(k)) 173 | } 174 | tx.Commit(bg) 175 | <-closeCh 176 | } 177 | 178 | func TestCloseSafety(t *testing.T) { 179 | d, close := newDS(t) 180 | addTestCases(t, d, testcases) 181 | 182 | tx, _ := d.NewTransaction(bg, false) 183 | err := tx.Put(bg, ds.NewKey("test"), []byte("test")) 184 | if err != nil { 185 | t.Error("Failed to put in a txn.") 186 | } 187 | close() 188 | err = tx.Commit(bg) 189 | if err == nil { 190 | t.Error("committing after close should fail.") 191 | } 192 | } 193 | 194 | func TestQueryRespectsProcessMem(t *testing.T) { 195 | d := newDSMem(t) 196 | addTestCases(t, d, testcases) 197 | } 198 | 199 | func expectMatches(t *testing.T, expect []string, actualR dsq.Results) { 200 | t.Helper() 201 | actual, err := actualR.Rest() 202 | if err != nil { 203 | t.Error(err) 204 | } 205 | 206 | if len(actual) != len(expect) { 207 | t.Error("not enough", expect, actual) 208 | } 209 | for _, k := range expect { 210 | found := false 211 | for _, e := range actual { 212 | if e.Key == k { 213 | found = true 214 | } 215 | } 216 | if !found { 217 | t.Error(k, "not found") 218 | } 219 | } 220 | } 221 | 222 | func expectOrderedMatches(t *testing.T, expect []string, actualR dsq.Results) { 223 | t.Helper() 224 | actual, err := actualR.Rest() 225 | if err != nil { 226 | t.Error(err) 227 | } 228 | 229 | if len(actual) != len(expect) { 230 | t.Error("not enough", expect, actual) 231 | } 232 | for i := range expect { 233 | if expect[i] != actual[i].Key { 234 | t.Errorf("expected %q, got %q", expect[i], actual[i].Key) 235 | } 236 | } 237 | } 238 | 239 | func testBatching(t *testing.T, d *Datastore) { 240 | b, err := d.Batch(bg) 241 | if err != nil { 242 | t.Fatal(err) 243 | } 244 | 245 | for k, v := range testcases { 246 | err := b.Put(bg, ds.NewKey(k), []byte(v)) 247 | if err != nil { 248 | t.Fatal(err) 249 | } 250 | } 251 | 252 | err = b.Commit(bg) 253 | if err != nil { 254 | t.Fatal(err) 255 | } 256 | 257 | for k, v := range testcases { 258 | val, err := d.Get(bg, ds.NewKey(k)) 259 | if err != nil { 260 | t.Fatal(err) 261 | } 262 | 263 | if v != string(val) { 264 | t.Fatal("got wrong data!") 265 | } 266 | } 267 | } 268 | 269 | func TestBatching(t *testing.T) { 270 | d, done := newDS(t) 271 | defer done() 272 | testBatching(t, d) 273 | } 274 | 275 | func TestBatchingMem(t *testing.T) { 276 | d := newDSMem(t) 277 | testBatching(t, d) 278 | } 279 | 280 | func TestDiskUsage(t *testing.T) { 281 | d, done := newDS(t) 282 | addTestCases(t, d, testcases) 283 | du, err := d.DiskUsage(bg) 284 | if err != nil { 285 | t.Fatal(err) 286 | } 287 | 288 | if du == 0 { 289 | t.Fatal("expected some disk usage") 290 | } 291 | 292 | k := ds.NewKey("more") 293 | err = d.Put(bg, k, []byte("value")) 294 | if err != nil { 295 | t.Fatal(err) 296 | } 297 | 298 | du2, err := d.DiskUsage(bg) 299 | if err != nil { 300 | t.Fatal(err) 301 | } 302 | if du2 <= du { 303 | t.Fatal("size should have increased") 304 | } 305 | 306 | done() 307 | 308 | // This should fail 309 | _, err = d.DiskUsage(bg) 310 | if err == nil { 311 | t.Fatal("DiskUsage should fail when we cannot walk path") 312 | } 313 | } 314 | 315 | func TestDiskUsageInMem(t *testing.T) { 316 | d := newDSMem(t) 317 | du, _ := d.DiskUsage(bg) 318 | if du != 0 { 319 | t.Fatal("inmem dbs have 0 disk usage") 320 | } 321 | } 322 | 323 | func TestTransactionCommit(t *testing.T) { 324 | key := ds.NewKey("/test/key1") 325 | 326 | d, done := newDS(t) 327 | defer done() 328 | 329 | txn, err := d.NewTransaction(bg, false) 330 | if err != nil { 331 | t.Fatal(err) 332 | } 333 | defer txn.Discard(bg) 334 | 335 | if err := txn.Put(bg, key, []byte("hello")); err != nil { 336 | t.Fatal(err) 337 | } 338 | if val, err := d.Get(bg, key); err != ds.ErrNotFound { 339 | t.Fatalf("expected ErrNotFound, got err: %v, value: %v", err, val) 340 | } 341 | if err := txn.Commit(bg); err != nil { 342 | t.Fatal(err) 343 | } 344 | if val, err := d.Get(bg, key); err != nil || !bytes.Equal(val, []byte("hello")) { 345 | t.Fatalf("expected entry present after commit, got err: %v, value: %v", err, val) 346 | } 347 | } 348 | 349 | func TestTransactionDiscard(t *testing.T) { 350 | key := ds.NewKey("/test/key1") 351 | 352 | d, done := newDS(t) 353 | defer done() 354 | 355 | txn, err := d.NewTransaction(bg, false) 356 | if err != nil { 357 | t.Fatal(err) 358 | } 359 | defer txn.Discard(bg) 360 | 361 | if err := txn.Put(bg, key, []byte("hello")); err != nil { 362 | t.Fatal(err) 363 | } 364 | if val, err := d.Get(bg, key); err != ds.ErrNotFound { 365 | t.Fatalf("expected ErrNotFound, got err: %v, value: %v", err, val) 366 | } 367 | if txn.Discard(bg); err != nil { 368 | t.Fatal(err) 369 | } 370 | if val, err := d.Get(bg, key); err != ds.ErrNotFound { 371 | t.Fatalf("expected ErrNotFound, got err: %v, value: %v", err, val) 372 | } 373 | } 374 | 375 | func TestTransactionManyOperations(t *testing.T) { 376 | keys := []ds.Key{ds.NewKey("/test/key1"), ds.NewKey("/test/key2"), ds.NewKey("/test/key3"), ds.NewKey("/test/key4"), ds.NewKey("/test/key5")} 377 | 378 | d, done := newDS(t) 379 | defer done() 380 | 381 | txn, err := d.NewTransaction(bg, false) 382 | if err != nil { 383 | t.Fatal(err) 384 | } 385 | defer txn.Discard(bg) 386 | 387 | // Insert all entries. 388 | for i := 0; i < 5; i++ { 389 | if err := txn.Put(bg, keys[i], []byte(fmt.Sprintf("hello%d", i))); err != nil { 390 | t.Fatal(err) 391 | } 392 | } 393 | 394 | // Remove the third entry. 395 | if err := txn.Delete(bg, keys[2]); err != nil { 396 | t.Fatal(err) 397 | } 398 | 399 | // Check existences. 400 | if has, err := txn.Has(bg, keys[1]); err != nil || !has { 401 | t.Fatalf("expected key[1] to be present, err: %v, has: %v", err, has) 402 | } 403 | if has, err := txn.Has(bg, keys[2]); err != nil || has { 404 | t.Fatalf("expected key[2] to be absent, err: %v, has: %v", err, has) 405 | } 406 | 407 | var res dsq.Results 408 | if res, err = txn.Query(bg, dsq.Query{Prefix: "/test"}); err != nil { 409 | t.Fatalf("query failed, err: %v", err) 410 | } 411 | if entries, err := res.Rest(); err != nil || len(entries) != 4 { 412 | t.Fatalf("query failed or contained unexpected number of entries, err: %v, results: %v", err, entries) 413 | } 414 | 415 | txn.Discard(bg) 416 | } 417 | 418 | func TestSuite(t *testing.T) { 419 | d := newDSMem(t) 420 | defer d.Close() 421 | dstest.SubtestAll(t, d) 422 | } 423 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/ipfs/go-ds-leveldb 2 | 3 | go 1.23 4 | 5 | require ( 6 | github.com/ipfs/go-datastore v0.8.2 7 | github.com/syndtr/goleveldb v1.0.0 8 | ) 9 | 10 | require ( 11 | github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db // indirect 12 | github.com/google/uuid v1.6.0 // indirect 13 | github.com/ipfs/go-detect-race v0.0.1 // indirect 14 | ) 15 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= 2 | github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 3 | github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db h1:woRePGFeVFfLKN/pOkfl+p/TAqKOfFu+7KPlMVpok/w= 4 | github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= 5 | github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= 6 | github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= 7 | github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= 8 | github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= 9 | github.com/ipfs/go-datastore v0.8.2 h1:Jy3wjqQR6sg/LhyY0NIePZC3Vux19nLtg7dx0TVqr6U= 10 | github.com/ipfs/go-datastore v0.8.2/go.mod h1:W+pI1NsUsz3tcsAACMtfC+IZdnQTnC/7VfPoJBQuts0= 11 | github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ6cOk= 12 | github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps= 13 | github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= 14 | github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= 15 | github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= 16 | github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= 17 | github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= 18 | github.com/onsi/ginkgo v1.7.0 h1:WSHQ+IS43OoUrWtD1/bbclrwK8TTH5hzp+umCiuxHgs= 19 | github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= 20 | github.com/onsi/gomega v1.4.3 h1:RE1xgDvH7imwFD45h+u2SgIfERHlS2yNG4DObb5BSKU= 21 | github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= 22 | github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII= 23 | github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o= 24 | github.com/syndtr/goleveldb v1.0.0 h1:fBdIW9lB4Iz0n9khmH8w27SJ3QEJ7+IgjPEwGSZiFdE= 25 | github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ= 26 | golang.org/x/net v0.0.0-20180906233101-161cd47e91fd h1:nTDtHvHSdCn1m6ITfMRqtOd/9+7a3s8RBNOZ3eYZzJA= 27 | golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 28 | golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 29 | golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e h1:o3PsSEY8E4eXWkXrIP9YJALUkVZqzHJT5DOasTyn8Vs= 30 | golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 31 | golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= 32 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 33 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 34 | gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= 35 | gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= 36 | gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4= 37 | gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= 38 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= 39 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= 40 | gopkg.in/yaml.v2 v2.2.1 h1:mUhvW9EsL+naU5Q3cakzfE91YhliOondGd6ZrsDBHQE= 41 | gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 42 | -------------------------------------------------------------------------------- /version.json: -------------------------------------------------------------------------------- 1 | { 2 | "version": "v0.5.2" 3 | } 4 | --------------------------------------------------------------------------------