├── .github └── workflows │ └── tests.yaml ├── .gitignore ├── .readthedocs.yaml ├── MANIFEST.in ├── README.md ├── docs ├── Makefile ├── api.rst ├── conf.py ├── index.rst ├── installation.rst ├── quickstart.rst └── requirements.txt ├── lsm.pyx ├── pyproject.toml ├── setup.py ├── src ├── lsm.h ├── lsmInt.h ├── lsm_ckpt.c ├── lsm_file.c ├── lsm_log.c ├── lsm_main.c ├── lsm_mem.c ├── lsm_mutex.c ├── lsm_shared.c ├── lsm_sorted.c ├── lsm_str.c ├── lsm_tree.c ├── lsm_unix.c ├── lsm_varint.c └── lsm_win32.c └── tests.py /.github/workflows/tests.yaml: -------------------------------------------------------------------------------- 1 | name: Tests 2 | on: [push] 3 | jobs: 4 | tests: 5 | name: ${{ matrix.python-version }} 6 | runs-on: ubuntu-latest 7 | strategy: 8 | fail-fast: false 9 | matrix: 10 | python-version: [3.8, "3.10", "3.11", "3.13"] 11 | steps: 12 | - uses: actions/checkout@v2 13 | - uses: actions/setup-python@v2 14 | with: 15 | python-version: ${{ matrix.python-version }} 16 | - name: pip deps 17 | run: | 18 | pip install cython setuptools 19 | python setup.py build_ext -i 20 | - name: runtests 21 | run: python tests.py 22 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | lsm.c 2 | lsm*.so 3 | MANIFEST 4 | build/* 5 | -------------------------------------------------------------------------------- /.readthedocs.yaml: -------------------------------------------------------------------------------- 1 | version: 2 2 | build: 3 | os: ubuntu-22.04 4 | tools: 5 | python: "3.11" 6 | sphinx: 7 | configuration: docs/conf.py 8 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.md 2 | include LICENSE 3 | include SQLITE_LICENSE 4 | include lsm.c 5 | include lsm.pyx 6 | include pyproject.toml 7 | include tests.py 8 | recursive-include docs * 9 | recursive-include src * 10 | 11 | global-exclude *.pyc 12 | global-exclude *.o 13 | global-exclude *.so.0.1 14 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![Python LSM-DB](http://media.charlesleifer.com/blog/photos/lsm.png) 2 | 3 | Fast Python bindings for [SQLite's LSM key/value store](http://www.sqlite.org/src4/doc/trunk/www/lsmusr.wiki). 4 | The LSM storage engine was initially written as part of the experimental 5 | SQLite4 rewrite (now abandoned). More recently, the LSM source code was moved 6 | into the SQLite3 [source tree](https://www.sqlite.org/cgi/src/dir?ci=e148cdad35520e66&name=ext/lsm1) 7 | and has seen some improvements and fixes. This project uses the LSM code from 8 | the SQLite3 source tree. 9 | 10 | Features: 11 | 12 | * Embedded zero-conf database. 13 | * Keys support in-order traversal using cursors. 14 | * Transactional (including nested transactions). 15 | * Single writer/multiple reader MVCC based transactional concurrency model. 16 | * On-disk database stored in a single file. 17 | * Data is durable in the face of application or power failure. 18 | * Thread-safe. 19 | * Python 2.x and 3.x. 20 | 21 | Limitations: 22 | 23 | * Not tested on Windoze. 24 | 25 | The source for Python lsm-db is [hosted on GitHub](https://github.com/coleifer/python-lsm-db). 26 | 27 | If you encounter any bugs in the library, please [open an issue](https://github.com/coleifer/python-lsm-db/issues/new), 28 | including a description of the bug and any related traceback. 29 | 30 | ## Quick-start 31 | 32 | Below is a sample interactive console session designed to show some of the 33 | basic features and functionality of the ``lsm-db`` Python library. Also check 34 | out the [API documentation](https://lsm-db.readthedocs.io/en/latest/api.html). 35 | 36 | To begin, instantiate a `LSM` object, specifying a path to a database file. 37 | 38 | ```python 39 | 40 | >>> from lsm import LSM 41 | >>> db = LSM('test.ldb') 42 | ``` 43 | 44 | ### Key/Value Features 45 | 46 | `lsm-db` is a key/value store, and has a dictionary-like API: 47 | 48 | ```python 49 | 50 | >>> db['foo'] = 'bar' 51 | >>> print db['foo'] 52 | bar 53 | 54 | >>> for i in range(4): 55 | ... db['k%s' % i] = str(i) 56 | ... 57 | 58 | >>> 'k3' in db 59 | True 60 | >>> 'k4' in db 61 | False 62 | 63 | >>> del db['k3'] 64 | >>> db['k3'] 65 | Traceback (most recent call last): 66 | File "", line 1, in 67 | File "lsm.pyx", line 973, in lsm.LSM.__getitem__ (lsm.c:7142) 68 | File "lsm.pyx", line 777, in lsm.LSM.fetch (lsm.c:5756) 69 | File "lsm.pyx", line 778, in lsm.LSM.fetch (lsm.c:5679) 70 | File "lsm.pyx", line 1289, in lsm.Cursor.seek (lsm.c:12122) 71 | File "lsm.pyx", line 1311, in lsm.Cursor.seek (lsm.c:12008) 72 | KeyError: 'k3' 73 | ``` 74 | 75 | By default when you attempt to look up a key, ``lsm-db`` will search for an 76 | exact match. You can also search for the closest key, if the specific key you 77 | are searching for does not exist: 78 | 79 | ```python 80 | 81 | >>> from lsm import SEEK_LE, SEEK_GE 82 | >>> db['k1xx', SEEK_LE] # Here we will match "k1". 83 | '1' 84 | >>> db['k1xx', SEEK_GE] # Here we will match "k2". 85 | '2' 86 | ``` 87 | 88 | `LSM` supports other common dictionary methods such as: 89 | 90 | * `keys()` 91 | * `values()` 92 | * `update()` 93 | 94 | ### Slices and Iteration 95 | 96 | The database can be iterated through directly, or sliced. When you are slicing 97 | the database the start and end keys need not exist -- ``lsm-db`` will find the 98 | closest key (details can be found in the [LSM.fetch_range()](https://lsm-db.readthedocs.io/en/latest/api.html#lsm.LSM.fetch_range) 99 | documentation). 100 | 101 | ```python 102 | 103 | >>> [item for item in db] 104 | [('foo', 'bar'), ('k0', '0'), ('k1', '1'), ('k2', '2')] 105 | 106 | >>> db['k0':'k99'] 107 | 108 | 109 | >>> list(db['k0':'k99']) 110 | [('k0', '0'), ('k1', '1'), ('k2', '2')] 111 | ``` 112 | 113 | You can use open-ended slices. If the lower- or upper-bound is outside the 114 | range of keys an empty list is returned. 115 | 116 | ```python 117 | 118 | >>> list(db['k0':]) 119 | [('k0', '0'), ('k1', '1'), ('k2', '2')] 120 | 121 | >>> list(db[:'k1']) 122 | [('foo', 'bar'), ('k0', '0'), ('k1', '1')] 123 | 124 | >>> list(db[:'aaa']) 125 | [] 126 | ``` 127 | 128 | To retrieve keys in reverse order, simply use a higher key as the first 129 | parameter of your slice. If you are retrieving an open-ended slice, you can 130 | specify ``True`` as the ``step`` parameter of the slice. 131 | 132 | ```python 133 | 134 | >>> list(db['k1':'aaa']) # Since 'k1' > 'aaa', keys are retrieved in reverse: 135 | [('k1', '1'), ('k0', '0'), ('foo', 'bar')] 136 | 137 | >>> list(db['k1'::True]) # Open-ended slices specify True for step: 138 | [('k1', '1'), ('k0', '0'), ('foo', 'bar')] 139 | ``` 140 | 141 | You can also **delete** slices of keys, but note that the delete **will not** 142 | include the keys themselves: 143 | 144 | ```python 145 | 146 | >>> del db['k0':'k99'] 147 | 148 | >>> list(db) # Note that 'k0' still exists. 149 | [('foo', 'bar'), ('k0', '0')] 150 | ``` 151 | 152 | ### Cursors 153 | 154 | While slicing may cover most use-cases, for finer-grained control you can use 155 | cursors for traversing records. 156 | 157 | ```python 158 | 159 | >>> with db.cursor() as cursor: 160 | ... for key, value in cursor: 161 | ... print key, '=>', value 162 | ... 163 | foo => bar 164 | k0 => 0 165 | 166 | >>> db.update({'k1': '1', 'k2': '2', 'k3': '3'}) 167 | 168 | >>> with db.cursor() as cursor: 169 | ... cursor.first() 170 | ... print cursor.key() 171 | ... cursor.last() 172 | ... print cursor.key() 173 | ... cursor.previous() 174 | ... print cursor.key() 175 | ... 176 | foo 177 | k3 178 | k2 179 | 180 | >>> with db.cursor() as cursor: 181 | ... cursor.seek('k0', SEEK_GE) 182 | ... print list(cursor.fetch_until('k99')) 183 | ... 184 | [('k0', '0'), ('k1', '1'), ('k2', '2'), ('k3', '3')] 185 | ``` 186 | 187 | It is very important to close a cursor when you are through using it. For this 188 | reason, it is recommended you use the `LSM.cursor()` context-manager, which 189 | ensures the cursor is closed properly. 190 | 191 | ### Transactions 192 | 193 | ``lsm-db`` supports nested transactions. The simplest way to use transactions 194 | is with the `LSM.transaction()` method, which doubles as a context-manager or 195 | decorator. 196 | 197 | ```python 198 | 199 | >>> with db.transaction() as txn: 200 | ... db['k1'] = '1-mod' 201 | ... with db.transaction() as txn2: 202 | ... db['k2'] = '2-mod' 203 | ... txn2.rollback() 204 | ... 205 | True 206 | >>> print db['k1'], db['k2'] 207 | 1-mod 2 208 | ``` 209 | 210 | You can commit or roll-back transactions part-way through a wrapped block: 211 | 212 | ```python 213 | 214 | >>> with db.transaction() as txn: 215 | ... db['k1'] = 'outer txn' 216 | ... txn.commit() # The write is preserved. 217 | ... 218 | ... db['k1'] = 'outer txn-2' 219 | ... with db.transaction() as txn2: 220 | ... db['k1'] = 'inner-txn' # This is commited after the block ends. 221 | ... print db['k1'] # Prints "inner-txn". 222 | ... txn.rollback() # Rolls back both the changes from txn2 and the preceding write. 223 | ... print db['k1'] 224 | ... 225 | 1 <- Return value from call to commit(). 226 | inner-txn <- Printed after end of txn2. 227 | True <- Return value of call to rollback(). 228 | outer txn <- Printed after rollback. 229 | ``` 230 | 231 | If you like, you can also explicitly call `LSM.begin()`, `LSM.commit()`, and 232 | `LSM.rollback()`. 233 | 234 | ```python 235 | 236 | >>> db.begin() 237 | >>> db['foo'] = 'baze' 238 | >>> print db['foo'] 239 | baze 240 | >>> db.rollback() 241 | True 242 | >>> print db['foo'] 243 | bar 244 | ``` 245 | 246 | ### Reading more 247 | 248 | For more information, check out the project's documentation, hosted at 249 | readthedocs: 250 | 251 | https://lsm-db.readthedocs.io/en/latest/ 252 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | PAPER = 8 | BUILDDIR = _build 9 | 10 | # User-friendly check for sphinx-build 11 | ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) 12 | $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) 13 | endif 14 | 15 | # Internal variables. 16 | PAPEROPT_a4 = -D latex_paper_size=a4 17 | PAPEROPT_letter = -D latex_paper_size=letter 18 | ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 19 | # the i18n builder cannot share the environment and doctrees with the others 20 | I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 21 | 22 | .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext 23 | 24 | help: 25 | @echo "Please use \`make ' where is one of" 26 | @echo " html to make standalone HTML files" 27 | @echo " dirhtml to make HTML files named index.html in directories" 28 | @echo " singlehtml to make a single large HTML file" 29 | @echo " pickle to make pickle files" 30 | @echo " json to make JSON files" 31 | @echo " htmlhelp to make HTML files and a HTML help project" 32 | @echo " qthelp to make HTML files and a qthelp project" 33 | @echo " applehelp to make an Apple Help Book" 34 | @echo " devhelp to make HTML files and a Devhelp project" 35 | @echo " epub to make an epub" 36 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 37 | @echo " latexpdf to make LaTeX files and run them through pdflatex" 38 | @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" 39 | @echo " text to make text files" 40 | @echo " man to make manual pages" 41 | @echo " texinfo to make Texinfo files" 42 | @echo " info to make Texinfo files and run them through makeinfo" 43 | @echo " gettext to make PO message catalogs" 44 | @echo " changes to make an overview of all changed/added/deprecated items" 45 | @echo " xml to make Docutils-native XML files" 46 | @echo " pseudoxml to make pseudoxml-XML files for display purposes" 47 | @echo " linkcheck to check all external links for integrity" 48 | @echo " doctest to run all doctests embedded in the documentation (if enabled)" 49 | @echo " coverage to run coverage check of the documentation (if enabled)" 50 | 51 | clean: 52 | rm -rf $(BUILDDIR)/* 53 | 54 | html: 55 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html 56 | @echo 57 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." 58 | 59 | dirhtml: 60 | $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml 61 | @echo 62 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." 63 | 64 | singlehtml: 65 | $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml 66 | @echo 67 | @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." 68 | 69 | pickle: 70 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle 71 | @echo 72 | @echo "Build finished; now you can process the pickle files." 73 | 74 | json: 75 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json 76 | @echo 77 | @echo "Build finished; now you can process the JSON files." 78 | 79 | htmlhelp: 80 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp 81 | @echo 82 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 83 | ".hhp project file in $(BUILDDIR)/htmlhelp." 84 | 85 | qthelp: 86 | $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp 87 | @echo 88 | @echo "Build finished; now you can run "qcollectiongenerator" with the" \ 89 | ".qhcp project file in $(BUILDDIR)/qthelp, like this:" 90 | @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/lsm-db.qhcp" 91 | @echo "To view the help file:" 92 | @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/lsm-db.qhc" 93 | 94 | applehelp: 95 | $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp 96 | @echo 97 | @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." 98 | @echo "N.B. You won't be able to view it unless you put it in" \ 99 | "~/Library/Documentation/Help or install it in your application" \ 100 | "bundle." 101 | 102 | devhelp: 103 | $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp 104 | @echo 105 | @echo "Build finished." 106 | @echo "To view the help file:" 107 | @echo "# mkdir -p $$HOME/.local/share/devhelp/lsm-db" 108 | @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/lsm-db" 109 | @echo "# devhelp" 110 | 111 | epub: 112 | $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub 113 | @echo 114 | @echo "Build finished. The epub file is in $(BUILDDIR)/epub." 115 | 116 | latex: 117 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 118 | @echo 119 | @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." 120 | @echo "Run \`make' in that directory to run these through (pdf)latex" \ 121 | "(use \`make latexpdf' here to do that automatically)." 122 | 123 | latexpdf: 124 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 125 | @echo "Running LaTeX files through pdflatex..." 126 | $(MAKE) -C $(BUILDDIR)/latex all-pdf 127 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 128 | 129 | latexpdfja: 130 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 131 | @echo "Running LaTeX files through platex and dvipdfmx..." 132 | $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja 133 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 134 | 135 | text: 136 | $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text 137 | @echo 138 | @echo "Build finished. The text files are in $(BUILDDIR)/text." 139 | 140 | man: 141 | $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man 142 | @echo 143 | @echo "Build finished. The manual pages are in $(BUILDDIR)/man." 144 | 145 | texinfo: 146 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 147 | @echo 148 | @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." 149 | @echo "Run \`make' in that directory to run these through makeinfo" \ 150 | "(use \`make info' here to do that automatically)." 151 | 152 | info: 153 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 154 | @echo "Running Texinfo files through makeinfo..." 155 | make -C $(BUILDDIR)/texinfo info 156 | @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." 157 | 158 | gettext: 159 | $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale 160 | @echo 161 | @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." 162 | 163 | changes: 164 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes 165 | @echo 166 | @echo "The overview file is in $(BUILDDIR)/changes." 167 | 168 | linkcheck: 169 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck 170 | @echo 171 | @echo "Link check complete; look for any errors in the above output " \ 172 | "or in $(BUILDDIR)/linkcheck/output.txt." 173 | 174 | doctest: 175 | $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest 176 | @echo "Testing of doctests in the sources finished, look at the " \ 177 | "results in $(BUILDDIR)/doctest/output.txt." 178 | 179 | coverage: 180 | $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage 181 | @echo "Testing of coverage in the sources finished, look at the " \ 182 | "results in $(BUILDDIR)/coverage/python.txt." 183 | 184 | xml: 185 | $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml 186 | @echo 187 | @echo "Build finished. The XML files are in $(BUILDDIR)/xml." 188 | 189 | pseudoxml: 190 | $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml 191 | @echo 192 | @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." 193 | -------------------------------------------------------------------------------- /docs/api.rst: -------------------------------------------------------------------------------- 1 | .. _api: 2 | 3 | API Documentation 4 | ================= 5 | 6 | .. py:module:: lsm 7 | 8 | .. autoclass:: LSM 9 | :members: 10 | __init__, 11 | open, 12 | close, 13 | page_size, 14 | block_size, 15 | multiple_processes, 16 | readonly, 17 | write_safety, 18 | autoflush, 19 | autowork, 20 | automerge, 21 | autocheckpoint, 22 | mmap, 23 | transaction_log, 24 | pages_written, 25 | pages_read, 26 | checkpoint_size, 27 | tree_size, 28 | __enter__, 29 | insert, 30 | update, 31 | fetch, 32 | fetch_bulk, 33 | fetch_range, 34 | delete, 35 | delete_range, 36 | __getitem__, 37 | __setitem__, 38 | __delitem__, 39 | __contains__, 40 | __iter__, 41 | __reversed__, 42 | keys, 43 | values, 44 | flush, 45 | work, 46 | checkpoint, 47 | begin, 48 | commit, 49 | rollback, 50 | transaction, 51 | cursor 52 | 53 | 54 | .. autoclass:: Cursor 55 | :members: 56 | open, 57 | close, 58 | __enter__, 59 | __iter__, 60 | compare, 61 | seek, 62 | is_valid, 63 | first, 64 | last, 65 | next, 66 | previous, 67 | fetch_until, 68 | fetch_range, 69 | key, 70 | value, 71 | keys, 72 | values 73 | 74 | 75 | .. autoclass:: Transaction 76 | :members: 77 | commit, 78 | rollback 79 | 80 | Constants 81 | --------- 82 | 83 | Seek methods, can be used when fetching records or slices. 84 | 85 | ``SEEK_EQ`` 86 | The cursor is left at EOF (invalidated). A call to lsm_csr_valid() 87 | returns non-zero. 88 | 89 | ``SEEK_LE`` 90 | The cursor is left pointing to the largest key in the database that 91 | is smaller than (pKey/nKey). If the database contains no keys smaller 92 | than (pKey/nKey), the cursor is left at EOF. 93 | 94 | ``SEEK_GE`` 95 | The cursor is left pointing to the smallest key in the database that 96 | is larger than (pKey/nKey). If the database contains no keys larger 97 | than (pKey/nKey), the cursor is left at EOF. 98 | 99 | If the fourth parameter is ``SEEK_LEFAST``, this function searches the 100 | database in a similar manner to ``SEEK_LE``, with two differences: 101 | 102 | Even if a key can be found (the cursor is not left at EOF), the 103 | lsm_csr_value() function may not be used (attempts to do so return 104 | LSM_MISUSE). 105 | 106 | The key that the cursor is left pointing to may be one that has 107 | been recently deleted from the database. In this case it is 108 | guaranteed that the returned key is larger than any key currently 109 | in the database that is less than or equal to (pKey/nKey). 110 | 111 | ``SEEK_LEFAST`` requests are intended to be used to allocate database 112 | keys. 113 | 114 | Used in calls to :py:meth:`LSM.set_safety`. 115 | 116 | * ``SAFETY_OFF`` 117 | * ``SAFETY_NORMAL`` 118 | * ``SAFETY_FULL`` 119 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # lsm-db documentation build configuration file, created by 4 | # sphinx-quickstart on Mon Aug 3 01:29:51 2015. 5 | # 6 | # This file is execfile()d with the current directory set to its 7 | # containing dir. 8 | # 9 | # Note that not all possible configuration values are present in this 10 | # autogenerated file. 11 | # 12 | # All configuration values have a default; values that are commented out 13 | # serve to show the default. 14 | 15 | import sys 16 | import os 17 | import shlex 18 | 19 | # If extensions (or modules to document with autodoc) are in another directory, 20 | # add these directories to sys.path here. If the directory is relative to the 21 | # documentation root, use os.path.abspath to make it absolute, like shown here. 22 | #sys.path.insert(0, os.path.abspath('.')) 23 | 24 | # -- General configuration ------------------------------------------------ 25 | 26 | # If your documentation needs a minimal Sphinx version, state it here. 27 | #needs_sphinx = '1.0' 28 | 29 | # Add any Sphinx extension module names here, as strings. They can be 30 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 31 | # ones. 32 | extensions = ['sphinx.ext.autodoc'] 33 | 34 | # Add any paths that contain templates here, relative to this directory. 35 | templates_path = ['_templates'] 36 | 37 | # The suffix(es) of source filenames. 38 | # You can specify multiple suffix as a list of string: 39 | # source_suffix = ['.rst', '.md'] 40 | source_suffix = '.rst' 41 | 42 | # The encoding of source files. 43 | #source_encoding = 'utf-8-sig' 44 | 45 | # The master toctree document. 46 | master_doc = 'index' 47 | 48 | # General information about the project. 49 | project = u'lsm-db' 50 | copyright = u'2015, Charles Leifer' 51 | author = u'Charles Leifer' 52 | 53 | # The version info for the project you're documenting, acts as replacement for 54 | # |version| and |release|, also used in various other places throughout the 55 | # built documents. 56 | # 57 | # The short X.Y version. 58 | version = '0.1.0' 59 | # The full version, including alpha/beta/rc tags. 60 | release = '0.1.0' 61 | 62 | # The language for content autogenerated by Sphinx. Refer to documentation 63 | # for a list of supported languages. 64 | # 65 | # This is also used if you do content translation via gettext catalogs. 66 | # Usually you set "language" from the command line for these cases. 67 | language = None 68 | 69 | # There are two options for replacing |today|: either, you set today to some 70 | # non-false value, then it is used: 71 | #today = '' 72 | # Else, today_fmt is used as the format for a strftime call. 73 | #today_fmt = '%B %d, %Y' 74 | 75 | # List of patterns, relative to source directory, that match files and 76 | # directories to ignore when looking for source files. 77 | exclude_patterns = ['_build'] 78 | 79 | # The reST default role (used for this markup: `text`) to use for all 80 | # documents. 81 | #default_role = None 82 | 83 | # If true, '()' will be appended to :func: etc. cross-reference text. 84 | #add_function_parentheses = True 85 | 86 | # If true, the current module name will be prepended to all description 87 | # unit titles (such as .. function::). 88 | #add_module_names = True 89 | 90 | # If true, sectionauthor and moduleauthor directives will be shown in the 91 | # output. They are ignored by default. 92 | #show_authors = False 93 | 94 | # The name of the Pygments (syntax highlighting) style to use. 95 | pygments_style = 'sphinx' 96 | 97 | # A list of ignored prefixes for module index sorting. 98 | #modindex_common_prefix = [] 99 | 100 | # If true, keep warnings as "system message" paragraphs in the built documents. 101 | #keep_warnings = False 102 | 103 | # If true, `todo` and `todoList` produce output, else they produce nothing. 104 | todo_include_todos = False 105 | 106 | 107 | # -- Options for HTML output ---------------------------------------------- 108 | 109 | # The theme to use for HTML and HTML Help pages. See the documentation for 110 | # a list of builtin themes. 111 | html_theme = 'default' 112 | 113 | # Theme options are theme-specific and customize the look and feel of a theme 114 | # further. For a list of options available for each theme, see the 115 | # documentation. 116 | #html_theme_options = {} 117 | 118 | # Add any paths that contain custom themes here, relative to this directory. 119 | #html_theme_path = [] 120 | 121 | # The name for this set of Sphinx documents. If None, it defaults to 122 | # " v documentation". 123 | #html_title = None 124 | 125 | # A shorter title for the navigation bar. Default is the same as html_title. 126 | #html_short_title = None 127 | 128 | # The name of an image file (relative to this directory) to place at the top 129 | # of the sidebar. 130 | #html_logo = None 131 | 132 | # The name of an image file (within the static path) to use as favicon of the 133 | # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 134 | # pixels large. 135 | #html_favicon = None 136 | 137 | # Add any paths that contain custom static files (such as style sheets) here, 138 | # relative to this directory. They are copied after the builtin static files, 139 | # so a file named "default.css" will overwrite the builtin "default.css". 140 | html_static_path = ['_static'] 141 | 142 | # Add any extra paths that contain custom files (such as robots.txt or 143 | # .htaccess) here, relative to this directory. These files are copied 144 | # directly to the root of the documentation. 145 | #html_extra_path = [] 146 | 147 | # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, 148 | # using the given strftime format. 149 | #html_last_updated_fmt = '%b %d, %Y' 150 | 151 | # If true, SmartyPants will be used to convert quotes and dashes to 152 | # typographically correct entities. 153 | #html_use_smartypants = True 154 | 155 | # Custom sidebar templates, maps document names to template names. 156 | #html_sidebars = {} 157 | 158 | # Additional templates that should be rendered to pages, maps page names to 159 | # template names. 160 | #html_additional_pages = {} 161 | 162 | # If false, no module index is generated. 163 | #html_domain_indices = True 164 | 165 | # If false, no index is generated. 166 | #html_use_index = True 167 | 168 | # If true, the index is split into individual pages for each letter. 169 | #html_split_index = False 170 | 171 | # If true, links to the reST sources are added to the pages. 172 | #html_show_sourcelink = True 173 | 174 | # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. 175 | #html_show_sphinx = True 176 | 177 | # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. 178 | #html_show_copyright = True 179 | 180 | # If true, an OpenSearch description file will be output, and all pages will 181 | # contain a tag referring to it. The value of this option must be the 182 | # base URL from which the finished HTML is served. 183 | #html_use_opensearch = '' 184 | 185 | # This is the file name suffix for HTML files (e.g. ".xhtml"). 186 | #html_file_suffix = None 187 | 188 | # Language to be used for generating the HTML full-text search index. 189 | # Sphinx supports the following languages: 190 | # 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' 191 | # 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr' 192 | #html_search_language = 'en' 193 | 194 | # A dictionary with options for the search language support, empty by default. 195 | # Now only 'ja' uses this config value 196 | #html_search_options = {'type': 'default'} 197 | 198 | # The name of a javascript file (relative to the configuration directory) that 199 | # implements a search results scorer. If empty, the default will be used. 200 | #html_search_scorer = 'scorer.js' 201 | 202 | # Output file base name for HTML help builder. 203 | htmlhelp_basename = 'lsm-dbdoc' 204 | 205 | # -- Options for LaTeX output --------------------------------------------- 206 | 207 | latex_elements = { 208 | # The paper size ('letterpaper' or 'a4paper'). 209 | #'papersize': 'letterpaper', 210 | 211 | # The font size ('10pt', '11pt' or '12pt'). 212 | #'pointsize': '10pt', 213 | 214 | # Additional stuff for the LaTeX preamble. 215 | #'preamble': '', 216 | 217 | # Latex figure (float) alignment 218 | #'figure_align': 'htbp', 219 | } 220 | 221 | # Grouping the document tree into LaTeX files. List of tuples 222 | # (source start file, target name, title, 223 | # author, documentclass [howto, manual, or own class]). 224 | latex_documents = [ 225 | (master_doc, 'lsm-db.tex', u'lsm-db Documentation', 226 | u'Charles Leifer', 'manual'), 227 | ] 228 | 229 | # The name of an image file (relative to this directory) to place at the top of 230 | # the title page. 231 | #latex_logo = None 232 | 233 | # For "manual" documents, if this is true, then toplevel headings are parts, 234 | # not chapters. 235 | #latex_use_parts = False 236 | 237 | # If true, show page references after internal links. 238 | #latex_show_pagerefs = False 239 | 240 | # If true, show URL addresses after external links. 241 | #latex_show_urls = False 242 | 243 | # Documents to append as an appendix to all manuals. 244 | #latex_appendices = [] 245 | 246 | # If false, no module index is generated. 247 | #latex_domain_indices = True 248 | 249 | 250 | # -- Options for manual page output --------------------------------------- 251 | 252 | # One entry per manual page. List of tuples 253 | # (source start file, name, description, authors, manual section). 254 | man_pages = [ 255 | (master_doc, 'lsm-db', u'lsm-db Documentation', 256 | [author], 1) 257 | ] 258 | 259 | # If true, show URL addresses after external links. 260 | #man_show_urls = False 261 | 262 | 263 | # -- Options for Texinfo output ------------------------------------------- 264 | 265 | # Grouping the document tree into Texinfo files. List of tuples 266 | # (source start file, target name, title, author, 267 | # dir menu entry, description, category) 268 | texinfo_documents = [ 269 | (master_doc, 'lsm-db', u'lsm-db Documentation', 270 | author, 'lsm-db', 'One line description of project.', 271 | 'Miscellaneous'), 272 | ] 273 | 274 | # Documents to append as an appendix to all manuals. 275 | #texinfo_appendices = [] 276 | 277 | # If false, no module index is generated. 278 | #texinfo_domain_indices = True 279 | 280 | # How to display URL addresses: 'footnote', 'no', or 'inline'. 281 | #texinfo_show_urls = 'footnote' 282 | 283 | # If true, do not generate a @detailmenu in the "Top" node's menu. 284 | #texinfo_no_detailmenu = False 285 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | .. lsm-db documentation master file, created by 2 | sphinx-quickstart on Mon Aug 3 01:29:51 2015. 3 | You can adapt this file completely to your liking, but it should at least 4 | contain the root `toctree` directive. 5 | 6 | python lsm-db 7 | ============= 8 | 9 | .. image:: http://media.charlesleifer.com/blog/photos/lsm.png 10 | 11 | Fast Python bindings for `SQLite's LSM key/value store `_. 12 | The LSM storage engine was initially written as part of the experimental 13 | SQLite4 rewrite (now abandoned). More recently, the LSM source code was moved 14 | into the SQLite3 `source tree `_ 15 | and has seen some improvements and fixes. This project uses the LSM code from 16 | the SQLite3 source tree. 17 | 18 | Features: 19 | 20 | * Embedded zero-conf database. 21 | * Keys support in-order traversal using cursors. 22 | * Transactional (including nested transactions). 23 | * Single writer/multiple reader MVCC based transactional concurrency model. 24 | * On-disk database stored in a single file. 25 | * Data is durable in the face of application or power failure. 26 | * Thread-safe. 27 | * Python 2.x and 3.x. 28 | 29 | Limitations: 30 | 31 | * Not tested on Windoze. 32 | 33 | The source for Python lsm-db is `hosted on GitHub `_. 34 | 35 | .. note:: 36 | If you encounter any bugs in the library, please `open an issue `_, including a description of the bug and any related traceback. 37 | 38 | Contents: 39 | 40 | .. toctree:: 41 | :maxdepth: 2 42 | :glob: 43 | 44 | installation 45 | quickstart 46 | api 47 | 48 | 49 | Indices and tables 50 | ================== 51 | 52 | * :ref:`genindex` 53 | * :ref:`modindex` 54 | * :ref:`search` 55 | 56 | -------------------------------------------------------------------------------- /docs/installation.rst: -------------------------------------------------------------------------------- 1 | .. _installation: 2 | 3 | Installation 4 | ============ 5 | 6 | You can use ``pip`` to install ``lsm-db``: 7 | 8 | .. code-block:: console 9 | 10 | pip install lsm-db 11 | 12 | The project is hosted at https://github.com/coleifer/python-lsm-db and can be installed from source: 13 | 14 | .. code-block:: console 15 | 16 | git clone https://github.com/coleifer/python-lsm-db 17 | cd lsm-db 18 | python setup.py build 19 | python setup.py install 20 | 21 | .. note:: 22 | ``lsm-db`` depends on `Cython `_ to generate the Python extension. By default, lsm-db ships with a pre-generated C source file, so it is not strictly necessary to install Cython in order to compile ``lsm-db``, but you may wish to install Cython to ensure the generated source is compatible with your setup. 23 | 24 | After installing lsm-db, you can run the unit tests by executing the ``tests`` module: 25 | 26 | .. code-block:: console 27 | 28 | python tests.py 29 | -------------------------------------------------------------------------------- /docs/quickstart.rst: -------------------------------------------------------------------------------- 1 | .. _quickstart: 2 | 3 | Quick-start 4 | =========== 5 | 6 | Below is a sample interactive console session designed to show some of the basic features and functionality of the ``lsm-db`` Python library. Also check out the :ref:`API documentation `. 7 | 8 | To begin, instantiate a :py:class:`lsm.LSM` object, specifying a path to a database file. 9 | 10 | .. code-block:: pycon 11 | 12 | >>> from lsm import LSM 13 | >>> db = LSM('test.ldb') 14 | 15 | Key/Value Features 16 | ------------------ 17 | 18 | ``lsm-db`` is a key/value store, and has a dictionary-like API: 19 | 20 | .. code-block:: pycon 21 | 22 | >>> db['foo'] = 'bar' 23 | >>> print db['foo'] 24 | bar 25 | 26 | >>> for i in range(4): 27 | ... db['k%s' % i] = str(i) 28 | ... 29 | 30 | >>> 'k3' in db 31 | True 32 | >>> 'k4' in db 33 | False 34 | 35 | >>> del db['k3'] 36 | >>> db['k3'] 37 | Traceback (most recent call last): 38 | File "", line 1, in 39 | File "lsm.pyx", line 973, in lsm.LSM.__getitem__ (lsm.c:7142) 40 | File "lsm.pyx", line 777, in lsm.LSM.fetch (lsm.c:5756) 41 | File "lsm.pyx", line 778, in lsm.LSM.fetch (lsm.c:5679) 42 | File "lsm.pyx", line 1289, in lsm.Cursor.seek (lsm.c:12122) 43 | File "lsm.pyx", line 1311, in lsm.Cursor.seek (lsm.c:12008) 44 | KeyError: 'k3' 45 | 46 | By default when you attempt to look up a key, ``lsm-db`` will search for an exact match. You can also search for the closest key, if the specific key you are searching for does not exist: 47 | 48 | .. code-block:: pycon 49 | 50 | >>> from lsm import SEEK_LE, SEEK_GE 51 | >>> db['k1xx', SEEK_LE] # Here we will match "k1". 52 | '1' 53 | >>> db['k1xx', SEEK_GE] # Here we will match "k2". 54 | '2' 55 | 56 | :py:class:`LSM` supports other common dictionary methods such as: 57 | 58 | * :py:meth:`~lsm.LSM.keys` 59 | * :py:meth:`~lsm.LSM.values` 60 | * :py:meth:`~lsm.LSM.update` 61 | 62 | Slices and Iteration 63 | -------------------- 64 | 65 | The database can be iterated through directly, or sliced. When you are slicing the database the start and end keys need not exist -- ``lsm-db`` will find the closest key (details can be found in the :py:meth:`~lsm.LSM.fetch` documentation). 66 | 67 | .. code-block:: pycon 68 | 69 | >>> [item for item in db] 70 | [('foo', 'bar'), ('k0', '0'), ('k1', '1'), ('k2', '2')] 71 | 72 | >>> db['k0':'k99'] 73 | 74 | 75 | >>> list(db['k0':'k99']) 76 | [('k0', '0'), ('k1', '1'), ('k2', '2')] 77 | 78 | You can use open-ended slices. If the lower- or upper-bound is outside the range of keys an empty list is returned. 79 | 80 | .. code-block:: pycon 81 | 82 | >>> list(db['k0':]) 83 | [('k0', '0'), ('k1', '1'), ('k2', '2')] 84 | 85 | >>> list(db[:'k1']) 86 | [('foo', 'bar'), ('k0', '0'), ('k1', '1')] 87 | 88 | >>> list(db[:'aaa']) 89 | [] 90 | 91 | To retrieve keys in reverse order, simply use a higher key as the first parameter of your slice. If you are retrieving an open-ended slice, you can specify ``True`` as the ``step`` parameter of the slice. 92 | 93 | .. code-block:: pycon 94 | 95 | >>> list(db['k1':'aaa']) # Since 'k1' > 'aaa', keys are retrieved in reverse: 96 | [('k1', '1'), ('k0', '0'), ('foo', 'bar')] 97 | 98 | >>> list(db['k1'::True]) # Open-ended slices specify True for step: 99 | [('k1', '1'), ('k0', '0'), ('foo', 'bar')] 100 | 101 | You can also **delete** slices of keys, but note that the delete **will not** include the keys themselves: 102 | 103 | .. code-block:: pycon 104 | 105 | >>> del db['k0':'k99'] 106 | 107 | >>> list(db) # Note that 'k0' still exists. 108 | [('foo', 'bar'), ('k0', '0')] 109 | 110 | Cursors 111 | ------- 112 | 113 | While slicing may cover most use-cases, for finer-grained control you can use cursors for traversing records. 114 | 115 | .. code-block:: pycon 116 | 117 | >>> with db.cursor() as cursor: 118 | ... for key, value in cursor: 119 | ... print key, '=>', value 120 | ... 121 | foo => bar 122 | k0 => 0 123 | 124 | >>> db.update({'k1': '1', 'k2': '2', 'k3': '3'}) 125 | 126 | >>> with db.cursor() as cursor: 127 | ... cursor.first() 128 | ... print cursor.key() 129 | ... cursor.last() 130 | ... print cursor.key() 131 | ... cursor.previous() 132 | ... print cursor.key() 133 | ... 134 | foo 135 | k3 136 | k2 137 | 138 | >>> with db.cursor() as cursor: 139 | ... cursor.seek('k0', SEEK_GE) 140 | ... print list(cursor.fetch_until('k99')) 141 | ... 142 | [('k0', '0'), ('k1', '1'), ('k2', '2'), ('k3', '3')] 143 | 144 | .. note:: 145 | It is very important to close a cursor when you are through using it. For this reason, it is recommended you use the :py:meth:`~lsm.LSM.cursor` context-manager, which ensures the cursor is closed properly. 146 | 147 | Transactions 148 | ------------ 149 | 150 | ``lsm-db`` supports nested transactions. The simplest way to use transactions is with the :py:meth:`~lsm.LSM.transaction` method, which doubles as a context-manager or decorator. 151 | 152 | .. code-block:: pycon 153 | 154 | >>> with db.transaction() as txn: 155 | ... db['k1'] = '1-mod' 156 | ... with db.transaction() as txn2: 157 | ... db['k2'] = '2-mod' 158 | ... txn2.rollback() 159 | ... 160 | True 161 | >>> print db['k1'], db['k2'] 162 | 1-mod 2 163 | 164 | You can commit or roll-back transactions part-way through a wrapped block: 165 | 166 | .. code-block:: pycon 167 | 168 | >>> with db.transaction() as txn: 169 | ... db['k1'] = 'outer txn' 170 | ... txn.commit() # The write is preserved. 171 | ... 172 | ... db['k1'] = 'outer txn-2' 173 | ... with db.transaction() as txn2: 174 | ... db['k1'] = 'inner-txn' # This is commited after the block ends. 175 | ... print db['k1'] # Prints "inner-txn". 176 | ... txn.rollback() # Rolls back both the changes from txn2 and the preceding write. 177 | ... print db['k1'] 178 | ... 179 | 1 <- Return value from call to commit(). 180 | inner-txn <- Printed after end of txn2. 181 | True <- Return value of call to rollback(). 182 | outer txn <- Printed after rollback. 183 | 184 | If you like, you can also explicitly call :py:meth:`~lsm.LSM.begin`, :py:meth:`~lsm.LSM.commit`, and :py:meth:`~lsm.LSM.rollback`: 185 | 186 | .. code-block:: pycon 187 | 188 | >>> db.begin() 189 | >>> db['foo'] = 'baze' 190 | >>> print db['foo'] 191 | baze 192 | >>> db.rollback() 193 | True 194 | >>> print db['foo'] 195 | bar 196 | -------------------------------------------------------------------------------- /docs/requirements.txt: -------------------------------------------------------------------------------- 1 | Cython 2 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = ["setuptools", "wheel", "cython"] 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import glob 2 | import os 3 | 4 | from setuptools import setup 5 | from setuptools.extension import Extension 6 | try: 7 | from Cython.Build import cythonize 8 | except ImportError: 9 | import warnings 10 | cython_installed = False 11 | warnings.warn('Cython not installed, using pre-generated C source file.') 12 | else: 13 | cython_installed = True 14 | 15 | 16 | if cython_installed: 17 | python_source = 'lsm.pyx' 18 | else: 19 | python_source = 'lsm.c' 20 | cythonize = lambda obj: obj 21 | 22 | library_source = glob.glob('src/*.c') 23 | lsm_extension = Extension( 24 | 'lsm', 25 | sources=[python_source] + library_source) 26 | 27 | setup( 28 | name='lsm-db', 29 | version='0.7.2', 30 | description='Python bindings for the SQLite4 LSM database.', 31 | author='Charles Leifer', 32 | author_email='', 33 | ext_modules=cythonize([lsm_extension]), 34 | ) 35 | -------------------------------------------------------------------------------- /src/lsm.h: -------------------------------------------------------------------------------- 1 | /* 2 | ** 2011-08-10 3 | ** 4 | ** The author disclaims copyright to this source code. In place of 5 | ** a legal notice, here is a blessing: 6 | ** 7 | ** May you do good and not evil. 8 | ** May you find forgiveness for yourself and forgive others. 9 | ** May you share freely, never taking more than you give. 10 | ** 11 | ************************************************************************* 12 | ** 13 | ** This file defines the LSM API. 14 | */ 15 | #ifndef _LSM_H 16 | #define _LSM_H 17 | #include 18 | #ifdef __cplusplus 19 | extern "C" { 20 | #endif 21 | 22 | /* 23 | ** Opaque handle types. 24 | */ 25 | typedef struct lsm_compress lsm_compress; /* Compression library functions */ 26 | typedef struct lsm_compress_factory lsm_compress_factory; 27 | typedef struct lsm_cursor lsm_cursor; /* Database cursor handle */ 28 | typedef struct lsm_db lsm_db; /* Database connection handle */ 29 | typedef struct lsm_env lsm_env; /* Runtime environment */ 30 | typedef struct lsm_file lsm_file; /* OS file handle */ 31 | typedef struct lsm_mutex lsm_mutex; /* Mutex handle */ 32 | 33 | /* 64-bit integer type used for file offsets. */ 34 | typedef long long int lsm_i64; /* 64-bit signed integer type */ 35 | 36 | /* Candidate values for the 3rd argument to lsm_env.xLock() */ 37 | #define LSM_LOCK_UNLOCK 0 38 | #define LSM_LOCK_SHARED 1 39 | #define LSM_LOCK_EXCL 2 40 | 41 | /* Flags for lsm_env.xOpen() */ 42 | #define LSM_OPEN_READONLY 0x0001 43 | 44 | /* 45 | ** CAPI: Database Runtime Environment 46 | ** 47 | ** Run-time environment used by LSM 48 | */ 49 | struct lsm_env { 50 | int nByte; /* Size of this structure in bytes */ 51 | int iVersion; /* Version number of this structure (1) */ 52 | /****** file i/o ***********************************************/ 53 | void *pVfsCtx; 54 | int (*xFullpath)(lsm_env*, const char *, char *, int *); 55 | int (*xOpen)(lsm_env*, const char *, int flags, lsm_file **); 56 | int (*xRead)(lsm_file *, lsm_i64, void *, int); 57 | int (*xWrite)(lsm_file *, lsm_i64, void *, int); 58 | int (*xTruncate)(lsm_file *, lsm_i64); 59 | int (*xSync)(lsm_file *); 60 | int (*xSectorSize)(lsm_file *); 61 | int (*xRemap)(lsm_file *, lsm_i64, void **, lsm_i64*); 62 | int (*xFileid)(lsm_file *, void *pBuf, int *pnBuf); 63 | int (*xClose)(lsm_file *); 64 | int (*xUnlink)(lsm_env*, const char *); 65 | int (*xLock)(lsm_file*, int, int); 66 | int (*xTestLock)(lsm_file*, int, int, int); 67 | int (*xShmMap)(lsm_file*, int, int, void **); 68 | void (*xShmBarrier)(void); 69 | int (*xShmUnmap)(lsm_file*, int); 70 | /****** memory allocation ****************************************/ 71 | void *pMemCtx; 72 | void *(*xMalloc)(lsm_env*, size_t); /* malloc(3) function */ 73 | void *(*xRealloc)(lsm_env*, void *, size_t); /* realloc(3) function */ 74 | void (*xFree)(lsm_env*, void *); /* free(3) function */ 75 | size_t (*xSize)(lsm_env*, void *); /* xSize function */ 76 | /****** mutexes ****************************************************/ 77 | void *pMutexCtx; 78 | int (*xMutexStatic)(lsm_env*,int,lsm_mutex**); /* Obtain a static mutex */ 79 | int (*xMutexNew)(lsm_env*, lsm_mutex**); /* Get a new dynamic mutex */ 80 | void (*xMutexDel)(lsm_mutex *); /* Delete an allocated mutex */ 81 | void (*xMutexEnter)(lsm_mutex *); /* Grab a mutex */ 82 | int (*xMutexTry)(lsm_mutex *); /* Attempt to obtain a mutex */ 83 | void (*xMutexLeave)(lsm_mutex *); /* Leave a mutex */ 84 | int (*xMutexHeld)(lsm_mutex *); /* Return true if mutex is held */ 85 | int (*xMutexNotHeld)(lsm_mutex *); /* Return true if mutex not held */ 86 | /****** other ****************************************************/ 87 | int (*xSleep)(lsm_env*, int microseconds); 88 | 89 | /* New fields may be added in future releases, in which case the 90 | ** iVersion value will increase. */ 91 | }; 92 | 93 | /* 94 | ** Values that may be passed as the second argument to xMutexStatic. 95 | */ 96 | #define LSM_MUTEX_GLOBAL 1 97 | #define LSM_MUTEX_HEAP 2 98 | 99 | /* 100 | ** CAPI: LSM Error Codes 101 | */ 102 | #define LSM_OK 0 103 | #define LSM_ERROR 1 104 | #define LSM_BUSY 5 105 | #define LSM_NOMEM 7 106 | #define LSM_READONLY 8 107 | #define LSM_IOERR 10 108 | #define LSM_CORRUPT 11 109 | #define LSM_FULL 13 110 | #define LSM_CANTOPEN 14 111 | #define LSM_PROTOCOL 15 112 | #define LSM_MISUSE 21 113 | 114 | #define LSM_MISMATCH 50 115 | 116 | 117 | #define LSM_IOERR_NOENT (LSM_IOERR | (1<<8)) 118 | 119 | /* 120 | ** CAPI: Creating and Destroying Database Connection Handles 121 | ** 122 | ** Open and close a database connection handle. 123 | */ 124 | int lsm_new(lsm_env*, lsm_db **ppDb); 125 | int lsm_close(lsm_db *pDb); 126 | 127 | /* 128 | ** CAPI: Connecting to a Database 129 | */ 130 | int lsm_open(lsm_db *pDb, const char *zFilename); 131 | 132 | /* 133 | ** CAPI: Obtaining pointers to database environments 134 | ** 135 | ** Return a pointer to the environment used by the database connection 136 | ** passed as the first argument. Assuming the argument is valid, this 137 | ** function always returns a valid environment pointer - it cannot fail. 138 | */ 139 | lsm_env *lsm_get_env(lsm_db *pDb); 140 | 141 | /* 142 | ** The lsm_default_env() function returns a pointer to the default LSM 143 | ** environment for the current platform. 144 | */ 145 | lsm_env *lsm_default_env(void); 146 | 147 | 148 | /* 149 | ** CAPI: Configuring a database connection. 150 | ** 151 | ** The lsm_config() function is used to configure a database connection. 152 | */ 153 | int lsm_config(lsm_db *, int, ...); 154 | 155 | /* 156 | ** The following values may be passed as the second argument to lsm_config(). 157 | ** 158 | ** LSM_CONFIG_AUTOFLUSH: 159 | ** A read/write integer parameter. 160 | ** 161 | ** This value determines the amount of data allowed to accumulate in a 162 | ** live in-memory tree before it is marked as old. After committing a 163 | ** transaction, a connection checks if the size of the live in-memory tree, 164 | ** including data structure overhead, is greater than the value of this 165 | ** option in KB. If it is, and there is not already an old in-memory tree, 166 | ** the live in-memory tree is marked as old. 167 | ** 168 | ** The maximum allowable value is 1048576 (1GB). There is no minimum 169 | ** value. If this parameter is set to zero, then an attempt is made to 170 | ** mark the live in-memory tree as old after each transaction is committed. 171 | ** 172 | ** The default value is 1024 (1MB). 173 | ** 174 | ** LSM_CONFIG_PAGE_SIZE: 175 | ** A read/write integer parameter. This parameter may only be set before 176 | ** lsm_open() has been called. 177 | ** 178 | ** LSM_CONFIG_BLOCK_SIZE: 179 | ** A read/write integer parameter. 180 | ** 181 | ** This parameter may only be set before lsm_open() has been called. It 182 | ** must be set to a power of two between 64 and 65536, inclusive (block 183 | ** sizes between 64KB and 64MB). 184 | ** 185 | ** If the connection creates a new database, the block size of the new 186 | ** database is set to the value of this option in KB. After lsm_open() 187 | ** has been called, querying this parameter returns the actual block 188 | ** size of the opened database. 189 | ** 190 | ** The default value is 1024 (1MB blocks). 191 | ** 192 | ** LSM_CONFIG_SAFETY: 193 | ** A read/write integer parameter. Valid values are 0, 1 (the default) 194 | ** and 2. This parameter determines how robust the database is in the 195 | ** face of a system crash (e.g. a power failure or operating system 196 | ** crash). As follows: 197 | ** 198 | ** 0 (off): No robustness. A system crash may corrupt the database. 199 | ** 200 | ** 1 (normal): Some robustness. A system crash may not corrupt the 201 | ** database file, but recently committed transactions may 202 | ** be lost following recovery. 203 | ** 204 | ** 2 (full): Full robustness. A system crash may not corrupt the 205 | ** database file. Following recovery the database file 206 | ** contains all successfully committed transactions. 207 | ** 208 | ** LSM_CONFIG_AUTOWORK: 209 | ** A read/write integer parameter. 210 | ** 211 | ** LSM_CONFIG_AUTOCHECKPOINT: 212 | ** A read/write integer parameter. 213 | ** 214 | ** If this option is set to non-zero value N, then a checkpoint is 215 | ** automatically attempted after each N KB of data have been written to 216 | ** the database file. 217 | ** 218 | ** The amount of uncheckpointed data already written to the database file 219 | ** is a global parameter. After performing database work (writing to the 220 | ** database file), the process checks if the total amount of uncheckpointed 221 | ** data exceeds the value of this paramter. If so, a checkpoint is performed. 222 | ** This means that this option may cause the connection to perform a 223 | ** checkpoint even if the current connection has itself written very little 224 | ** data into the database file. 225 | ** 226 | ** The default value is 2048 (checkpoint every 2MB). 227 | ** 228 | ** LSM_CONFIG_MMAP: 229 | ** A read/write integer parameter. If this value is set to 0, then the 230 | ** database file is accessed using ordinary read/write IO functions. Or, 231 | ** if it is set to 1, then the database file is memory mapped and accessed 232 | ** that way. If this parameter is set to any value N greater than 1, then 233 | ** up to the first N KB of the file are memory mapped, and any remainder 234 | ** accessed using read/write IO. 235 | ** 236 | ** The default value is 1 on 64-bit platforms and 32768 on 32-bit platforms. 237 | ** 238 | ** 239 | ** LSM_CONFIG_USE_LOG: 240 | ** A read/write boolean parameter. True (the default) to use the log 241 | ** file normally. False otherwise. 242 | ** 243 | ** LSM_CONFIG_AUTOMERGE: 244 | ** A read/write integer parameter. The minimum number of segments to 245 | ** merge together at a time. Default value 4. 246 | ** 247 | ** LSM_CONFIG_MAX_FREELIST: 248 | ** A read/write integer parameter. The maximum number of free-list 249 | ** entries that are stored in a database checkpoint (the others are 250 | ** stored elsewhere in the database). 251 | ** 252 | ** There is no reason for an application to configure or query this 253 | ** parameter. It is only present because configuring a small value 254 | ** makes certain parts of the lsm code easier to test. 255 | ** 256 | ** LSM_CONFIG_MULTIPLE_PROCESSES: 257 | ** A read/write boolean parameter. This parameter may only be set before 258 | ** lsm_open() has been called. If true, the library uses shared-memory 259 | ** and posix advisory locks to co-ordinate access by clients from within 260 | ** multiple processes. Otherwise, if false, all database clients must be 261 | ** located in the same process. The default value is true. 262 | ** 263 | ** LSM_CONFIG_SET_COMPRESSION: 264 | ** Set the compression methods used to compress and decompress database 265 | ** content. The argument to this option should be a pointer to a structure 266 | ** of type lsm_compress. The lsm_config() method takes a copy of the 267 | ** structures contents. 268 | ** 269 | ** This option may only be used before lsm_open() is called. Invoking it 270 | ** after lsm_open() has been called results in an LSM_MISUSE error. 271 | ** 272 | ** LSM_CONFIG_GET_COMPRESSION: 273 | ** Query the compression methods used to compress and decompress database 274 | ** content. 275 | ** 276 | ** LSM_CONFIG_SET_COMPRESSION_FACTORY: 277 | ** Configure a factory method to be invoked in case of an LSM_MISMATCH 278 | ** error. 279 | ** 280 | ** LSM_CONFIG_READONLY: 281 | ** A read/write boolean parameter. This parameter may only be set before 282 | ** lsm_open() is called. 283 | */ 284 | #define LSM_CONFIG_AUTOFLUSH 1 285 | #define LSM_CONFIG_PAGE_SIZE 2 286 | #define LSM_CONFIG_SAFETY 3 287 | #define LSM_CONFIG_BLOCK_SIZE 4 288 | #define LSM_CONFIG_AUTOWORK 5 289 | #define LSM_CONFIG_MMAP 7 290 | #define LSM_CONFIG_USE_LOG 8 291 | #define LSM_CONFIG_AUTOMERGE 9 292 | #define LSM_CONFIG_MAX_FREELIST 10 293 | #define LSM_CONFIG_MULTIPLE_PROCESSES 11 294 | #define LSM_CONFIG_AUTOCHECKPOINT 12 295 | #define LSM_CONFIG_SET_COMPRESSION 13 296 | #define LSM_CONFIG_GET_COMPRESSION 14 297 | #define LSM_CONFIG_SET_COMPRESSION_FACTORY 15 298 | #define LSM_CONFIG_READONLY 16 299 | 300 | #define LSM_SAFETY_OFF 0 301 | #define LSM_SAFETY_NORMAL 1 302 | #define LSM_SAFETY_FULL 2 303 | 304 | /* 305 | ** CAPI: Compression and/or Encryption Hooks 306 | */ 307 | struct lsm_compress { 308 | void *pCtx; 309 | unsigned int iId; 310 | int (*xBound)(void *, int nSrc); 311 | int (*xCompress)(void *, char *, int *, const char *, int); 312 | int (*xUncompress)(void *, char *, int *, const char *, int); 313 | void (*xFree)(void *pCtx); 314 | }; 315 | 316 | struct lsm_compress_factory { 317 | void *pCtx; 318 | int (*xFactory)(void *, lsm_db *, unsigned int); 319 | void (*xFree)(void *pCtx); 320 | }; 321 | 322 | #define LSM_COMPRESSION_EMPTY 0 323 | #define LSM_COMPRESSION_NONE 1 324 | 325 | /* 326 | ** CAPI: Allocating and Freeing Memory 327 | ** 328 | ** Invoke the memory allocation functions that belong to environment 329 | ** pEnv. Or the system defaults if no memory allocation functions have 330 | ** been registered. 331 | */ 332 | void *lsm_malloc(lsm_env*, size_t); 333 | void *lsm_realloc(lsm_env*, void *, size_t); 334 | void lsm_free(lsm_env*, void *); 335 | 336 | /* 337 | ** CAPI: Querying a Connection For Operational Data 338 | ** 339 | ** Query a database connection for operational statistics or data. 340 | */ 341 | int lsm_info(lsm_db *, int, ...); 342 | 343 | int lsm_get_user_version(lsm_db *, unsigned int *); 344 | int lsm_set_user_version(lsm_db *, unsigned int); 345 | 346 | /* 347 | ** The following values may be passed as the second argument to lsm_info(). 348 | ** 349 | ** LSM_INFO_NWRITE: 350 | ** The third parameter should be of type (int *). The location pointed 351 | ** to by the third parameter is set to the number of 4KB pages written to 352 | ** the database file during the lifetime of this connection. 353 | ** 354 | ** LSM_INFO_NREAD: 355 | ** The third parameter should be of type (int *). The location pointed 356 | ** to by the third parameter is set to the number of 4KB pages read from 357 | ** the database file during the lifetime of this connection. 358 | ** 359 | ** LSM_INFO_DB_STRUCTURE: 360 | ** The third argument should be of type (char **). The location pointed 361 | ** to is populated with a pointer to a nul-terminated string containing 362 | ** the string representation of a Tcl data-structure reflecting the 363 | ** current structure of the database file. Specifically, the current state 364 | ** of the worker snapshot. The returned string should be eventually freed 365 | ** by the caller using lsm_free(). 366 | ** 367 | ** The returned list contains one element for each level in the database, 368 | ** in order from most to least recent. Each element contains a 369 | ** single element for each segment comprising the corresponding level, 370 | ** starting with the lhs segment, then each of the rhs segments (if any) 371 | ** in order from most to least recent. 372 | ** 373 | ** Each segment element is itself a list of 4 integer values, as follows: 374 | ** 375 | **
  1. First page of segment 376 | **
  2. Last page of segment 377 | **
  3. Root page of segment (if applicable) 378 | **
  4. Total number of pages in segment 379 | **
380 | ** 381 | ** LSM_INFO_ARRAY_STRUCTURE: 382 | ** There should be two arguments passed following this option (i.e. a 383 | ** total of four arguments passed to lsm_info()). The first argument 384 | ** should be the page number of the first page in a database array 385 | ** (perhaps obtained from an earlier INFO_DB_STRUCTURE call). The second 386 | ** trailing argument should be of type (char **). The location pointed 387 | ** to is populated with a pointer to a nul-terminated string that must 388 | ** be eventually freed using lsm_free() by the caller. 389 | ** 390 | ** The output string contains the text representation of a Tcl list of 391 | ** integers. Each pair of integers represent a range of pages used by 392 | ** the identified array. For example, if the array occupies database 393 | ** pages 993 to 1024, then pages 2048 to 2777, then the returned string 394 | ** will be "993 1024 2048 2777". 395 | ** 396 | ** If the specified integer argument does not correspond to the first 397 | ** page of any database array, LSM_ERROR is returned and the output 398 | ** pointer is set to a NULL value. 399 | ** 400 | ** LSM_INFO_LOG_STRUCTURE: 401 | ** The third argument should be of type (char **). The location pointed 402 | ** to is populated with a pointer to a nul-terminated string containing 403 | ** the string representation of a Tcl data-structure. The returned 404 | ** string should be eventually freed by the caller using lsm_free(). 405 | ** 406 | ** The Tcl structure returned is a list of six integers that describe 407 | ** the current structure of the log file. 408 | ** 409 | ** LSM_INFO_ARRAY_PAGES: 410 | ** 411 | ** LSM_INFO_PAGE_ASCII_DUMP: 412 | ** As with LSM_INFO_ARRAY_STRUCTURE, there should be two arguments passed 413 | ** with calls that specify this option - an integer page number and a 414 | ** (char **) used to return a nul-terminated string that must be later 415 | ** freed using lsm_free(). In this case the output string is populated 416 | ** with a human-readable description of the page content. 417 | ** 418 | ** If the page cannot be decoded, it is not an error. In this case the 419 | ** human-readable output message will report the systems failure to 420 | ** interpret the page data. 421 | ** 422 | ** LSM_INFO_PAGE_HEX_DUMP: 423 | ** This argument is similar to PAGE_ASCII_DUMP, except that keys and 424 | ** values are represented using hexadecimal notation instead of ascii. 425 | ** 426 | ** LSM_INFO_FREELIST: 427 | ** The third argument should be of type (char **). The location pointed 428 | ** to is populated with a pointer to a nul-terminated string containing 429 | ** the string representation of a Tcl data-structure. The returned 430 | ** string should be eventually freed by the caller using lsm_free(). 431 | ** 432 | ** The Tcl structure returned is a list containing one element for each 433 | ** free block in the database. The element itself consists of two 434 | ** integers - the block number and the id of the snapshot that freed it. 435 | ** 436 | ** LSM_INFO_CHECKPOINT_SIZE: 437 | ** The third argument should be of type (int *). The location pointed to 438 | ** by this argument is populated with the number of KB written to the 439 | ** database file since the most recent checkpoint. 440 | ** 441 | ** LSM_INFO_TREE_SIZE: 442 | ** If this value is passed as the second argument to an lsm_info() call, it 443 | ** should be followed by two arguments of type (int *) (for a total of four 444 | ** arguments). 445 | ** 446 | ** At any time, there are either one or two tree structures held in shared 447 | ** memory that new database clients will access (there may also be additional 448 | ** tree structures being used by older clients - this API does not provide 449 | ** information on them). One tree structure - the current tree - is used to 450 | ** accumulate new data written to the database. The other tree structure - 451 | ** the old tree - is a read-only tree holding older data and may be flushed 452 | ** to disk at any time. 453 | ** 454 | ** Assuming no error occurs, the location pointed to by the first of the two 455 | ** (int *) arguments is set to the size of the old in-memory tree in KB. 456 | ** The second is set to the size of the current, or live in-memory tree. 457 | ** 458 | ** LSM_INFO_COMPRESSION_ID: 459 | ** This value should be followed by a single argument of type 460 | ** (unsigned int *). If successful, the location pointed to is populated 461 | ** with the database compression id before returning. 462 | */ 463 | #define LSM_INFO_NWRITE 1 464 | #define LSM_INFO_NREAD 2 465 | #define LSM_INFO_DB_STRUCTURE 3 466 | #define LSM_INFO_LOG_STRUCTURE 4 467 | #define LSM_INFO_ARRAY_STRUCTURE 5 468 | #define LSM_INFO_PAGE_ASCII_DUMP 6 469 | #define LSM_INFO_PAGE_HEX_DUMP 7 470 | #define LSM_INFO_FREELIST 8 471 | #define LSM_INFO_ARRAY_PAGES 9 472 | #define LSM_INFO_CHECKPOINT_SIZE 10 473 | #define LSM_INFO_TREE_SIZE 11 474 | #define LSM_INFO_FREELIST_SIZE 12 475 | #define LSM_INFO_COMPRESSION_ID 13 476 | 477 | 478 | /* 479 | ** CAPI: Opening and Closing Write Transactions 480 | ** 481 | ** These functions are used to open and close transactions and nested 482 | ** sub-transactions. 483 | ** 484 | ** The lsm_begin() function is used to open transactions and sub-transactions. 485 | ** A successful call to lsm_begin() ensures that there are at least iLevel 486 | ** nested transactions open. To open a top-level transaction, pass iLevel=1. 487 | ** To open a sub-transaction within the top-level transaction, iLevel=2. 488 | ** Passing iLevel=0 is a no-op. 489 | ** 490 | ** lsm_commit() is used to commit transactions and sub-transactions. A 491 | ** successful call to lsm_commit() ensures that there are at most iLevel 492 | ** nested transactions open. To commit a top-level transaction, pass iLevel=0. 493 | ** To commit all sub-transactions inside the main transaction, pass iLevel=1. 494 | ** 495 | ** Function lsm_rollback() is used to roll back transactions and 496 | ** sub-transactions. A successful call to lsm_rollback() restores the database 497 | ** to the state it was in when the iLevel'th nested sub-transaction (if any) 498 | ** was first opened. And then closes transactions to ensure that there are 499 | ** at most iLevel nested transactions open. Passing iLevel=0 rolls back and 500 | ** closes the top-level transaction. iLevel=1 also rolls back the top-level 501 | ** transaction, but leaves it open. iLevel=2 rolls back the sub-transaction 502 | ** nested directly inside the top-level transaction (and leaves it open). 503 | */ 504 | int lsm_begin(lsm_db *pDb, int iLevel); 505 | int lsm_commit(lsm_db *pDb, int iLevel); 506 | int lsm_rollback(lsm_db *pDb, int iLevel); 507 | 508 | /* 509 | ** CAPI: Writing to a Database 510 | ** 511 | ** Write a new value into the database. If a value with a duplicate key 512 | ** already exists it is replaced. 513 | */ 514 | int lsm_insert(lsm_db*, const void *pKey, int nKey, const void *pVal, int nVal); 515 | 516 | /* 517 | ** Delete a value from the database. No error is returned if the specified 518 | ** key value does not exist in the database. 519 | */ 520 | int lsm_delete(lsm_db *, const void *pKey, int nKey); 521 | 522 | /* 523 | ** Delete all database entries with keys that are greater than (pKey1/nKey1) 524 | ** and smaller than (pKey2/nKey2). Note that keys (pKey1/nKey1) and 525 | ** (pKey2/nKey2) themselves, if they exist in the database, are not deleted. 526 | ** 527 | ** Return LSM_OK if successful, or an LSM error code otherwise. 528 | */ 529 | int lsm_delete_range(lsm_db *, 530 | const void *pKey1, int nKey1, const void *pKey2, int nKey2 531 | ); 532 | 533 | /* 534 | ** CAPI: Explicit Database Work and Checkpointing 535 | ** 536 | ** This function is called by a thread to work on the database structure. 537 | */ 538 | int lsm_work(lsm_db *pDb, int nMerge, int nKB, int *pnWrite); 539 | 540 | int lsm_flush(lsm_db *pDb); 541 | 542 | /* 543 | ** Attempt to checkpoint the current database snapshot. Return an LSM 544 | ** error code if an error occurs or LSM_OK otherwise. 545 | ** 546 | ** If the current snapshot has already been checkpointed, calling this 547 | ** function is a no-op. In this case if pnKB is not NULL, *pnKB is 548 | ** set to 0. Or, if the current snapshot is successfully checkpointed 549 | ** by this function and pbKB is not NULL, *pnKB is set to the number 550 | ** of bytes written to the database file since the previous checkpoint 551 | ** (the same measure as returned by the LSM_INFO_CHECKPOINT_SIZE query). 552 | */ 553 | int lsm_checkpoint(lsm_db *pDb, int *pnKB); 554 | 555 | /* 556 | ** CAPI: Opening and Closing Database Cursors 557 | ** 558 | ** Open and close a database cursor. 559 | */ 560 | int lsm_csr_open(lsm_db *pDb, lsm_cursor **ppCsr); 561 | int lsm_csr_close(lsm_cursor *pCsr); 562 | 563 | /* 564 | ** CAPI: Positioning Database Cursors 565 | ** 566 | ** If the fourth parameter is LSM_SEEK_EQ, LSM_SEEK_GE or LSM_SEEK_LE, 567 | ** this function searches the database for an entry with key (pKey/nKey). 568 | ** If an error occurs, an LSM error code is returned. Otherwise, LSM_OK. 569 | ** 570 | ** If no error occurs and the requested key is present in the database, the 571 | ** cursor is left pointing to the entry with the specified key. Or, if the 572 | ** specified key is not present in the database the state of the cursor 573 | ** depends on the value passed as the final parameter, as follows: 574 | ** 575 | ** LSM_SEEK_EQ: 576 | ** The cursor is left at EOF (invalidated). A call to lsm_csr_valid() 577 | ** returns non-zero. 578 | ** 579 | ** LSM_SEEK_LE: 580 | ** The cursor is left pointing to the largest key in the database that 581 | ** is smaller than (pKey/nKey). If the database contains no keys smaller 582 | ** than (pKey/nKey), the cursor is left at EOF. 583 | ** 584 | ** LSM_SEEK_GE: 585 | ** The cursor is left pointing to the smallest key in the database that 586 | ** is larger than (pKey/nKey). If the database contains no keys larger 587 | ** than (pKey/nKey), the cursor is left at EOF. 588 | ** 589 | ** If the fourth parameter is LSM_SEEK_LEFAST, this function searches the 590 | ** database in a similar manner to LSM_SEEK_LE, with two differences: 591 | ** 592 | **
  1. Even if a key can be found (the cursor is not left at EOF), the 593 | ** lsm_csr_value() function may not be used (attempts to do so return 594 | ** LSM_MISUSE). 595 | ** 596 | **
  2. The key that the cursor is left pointing to may be one that has 597 | ** been recently deleted from the database. In this case it is 598 | ** guaranteed that the returned key is larger than any key currently 599 | ** in the database that is less than or equal to (pKey/nKey). 600 | **
601 | ** 602 | ** LSM_SEEK_LEFAST requests are intended to be used to allocate database 603 | ** keys. 604 | */ 605 | int lsm_csr_seek(lsm_cursor *pCsr, const void *pKey, int nKey, int eSeek); 606 | 607 | int lsm_csr_first(lsm_cursor *pCsr); 608 | int lsm_csr_last(lsm_cursor *pCsr); 609 | 610 | /* 611 | ** Advance the specified cursor to the next or previous key in the database. 612 | ** Return LSM_OK if successful, or an LSM error code otherwise. 613 | ** 614 | ** Functions lsm_csr_seek(), lsm_csr_first() and lsm_csr_last() are "seek" 615 | ** functions. Whether or not lsm_csr_next and lsm_csr_prev may be called 616 | ** successfully also depends on the most recent seek function called on 617 | ** the cursor. Specifically: 618 | ** 619 | **
    620 | **
  • At least one seek function must have been called on the cursor. 621 | **
  • To call lsm_csr_next(), the most recent call to a seek function must 622 | ** have been either lsm_csr_first() or a call to lsm_csr_seek() specifying 623 | ** LSM_SEEK_GE. 624 | **
  • To call lsm_csr_prev(), the most recent call to a seek function must 625 | ** have been either lsm_csr_last() or a call to lsm_csr_seek() specifying 626 | ** LSM_SEEK_LE. 627 | **
628 | ** 629 | ** Otherwise, if the above conditions are not met when lsm_csr_next or 630 | ** lsm_csr_prev is called, LSM_MISUSE is returned and the cursor position 631 | ** remains unchanged. 632 | */ 633 | int lsm_csr_next(lsm_cursor *pCsr); 634 | int lsm_csr_prev(lsm_cursor *pCsr); 635 | 636 | /* 637 | ** Values that may be passed as the fourth argument to lsm_csr_seek(). 638 | */ 639 | #define LSM_SEEK_LEFAST -2 640 | #define LSM_SEEK_LE -1 641 | #define LSM_SEEK_EQ 0 642 | #define LSM_SEEK_GE 1 643 | 644 | /* 645 | ** CAPI: Extracting Data From Database Cursors 646 | ** 647 | ** Retrieve data from a database cursor. 648 | */ 649 | int lsm_csr_valid(lsm_cursor *pCsr); 650 | int lsm_csr_key(lsm_cursor *pCsr, const void **ppKey, int *pnKey); 651 | int lsm_csr_value(lsm_cursor *pCsr, const void **ppVal, int *pnVal); 652 | 653 | /* 654 | ** If no error occurs, this function compares the database key passed via 655 | ** the pKey/nKey arguments with the key that the cursor passed as the first 656 | ** argument currently points to. If the cursors key is less than, equal to 657 | ** or greater than pKey/nKey, *piRes is set to less than, equal to or greater 658 | ** than zero before returning. LSM_OK is returned in this case. 659 | ** 660 | ** Or, if an error occurs, an LSM error code is returned and the final 661 | ** value of *piRes is undefined. If the cursor does not point to a valid 662 | ** key when this function is called, LSM_MISUSE is returned. 663 | */ 664 | int lsm_csr_cmp(lsm_cursor *pCsr, const void *pKey, int nKey, int *piRes); 665 | 666 | /* 667 | ** CAPI: Change these!! 668 | ** 669 | ** Configure a callback to which debugging and other messages should 670 | ** be directed. Only useful for debugging lsm. 671 | */ 672 | void lsm_config_log(lsm_db *, void (*)(void *, int, const char *), void *); 673 | 674 | /* 675 | ** Configure a callback that is invoked if the database connection ever 676 | ** writes to the database file. 677 | */ 678 | void lsm_config_work_hook(lsm_db *, void (*)(lsm_db *, void *), void *); 679 | 680 | /* ENDOFAPI */ 681 | #ifdef __cplusplus 682 | } /* End of the 'extern "C"' block */ 683 | #endif 684 | #endif /* ifndef _LSM_H */ 685 | -------------------------------------------------------------------------------- /src/lsm_main.c: -------------------------------------------------------------------------------- 1 | /* 2 | ** 2011-08-18 3 | ** 4 | ** The author disclaims copyright to this source code. In place of 5 | ** a legal notice, here is a blessing: 6 | ** 7 | ** May you do good and not evil. 8 | ** May you find forgiveness for yourself and forgive others. 9 | ** May you share freely, never taking more than you give. 10 | ** 11 | ************************************************************************* 12 | ** 13 | ** The main interface to the LSM module. 14 | */ 15 | #include "lsmInt.h" 16 | 17 | 18 | #ifdef LSM_DEBUG 19 | /* 20 | ** This function returns a copy of its only argument. 21 | ** 22 | ** When the library is built with LSM_DEBUG defined, this function is called 23 | ** whenever an error code is generated (not propagated - generated). So 24 | ** if the library is mysteriously returning (say) LSM_IOERR, a breakpoint 25 | ** may be set in this function to determine why. 26 | */ 27 | int lsmErrorBkpt(int rc){ 28 | /* Set breakpoint here! */ 29 | return rc; 30 | } 31 | 32 | /* 33 | ** This function contains various assert() statements that test that the 34 | ** lsm_db structure passed as an argument is internally consistent. 35 | */ 36 | static void assert_db_state(lsm_db *pDb){ 37 | 38 | /* If there is at least one cursor or a write transaction open, the database 39 | ** handle must be holding a pointer to a client snapshot. And the reverse 40 | ** - if there are no open cursors and no write transactions then there must 41 | ** not be a client snapshot. */ 42 | 43 | assert( (pDb->pCsr!=0||pDb->nTransOpen>0)==(pDb->iReader>=0||pDb->bRoTrans) ); 44 | 45 | assert( (pDb->iReader<0 && pDb->bRoTrans==0) || pDb->pClient!=0 ); 46 | 47 | assert( pDb->nTransOpen>=0 ); 48 | } 49 | #else 50 | # define assert_db_state(x) 51 | #endif 52 | 53 | /* 54 | ** The default key-compare function. 55 | */ 56 | static int xCmp(void *p1, int n1, void *p2, int n2){ 57 | int res; 58 | res = memcmp(p1, p2, LSM_MIN(n1, n2)); 59 | if( res==0 ) res = (n1-n2); 60 | return res; 61 | } 62 | 63 | static void xLog(void *pCtx, int rc, const char *z){ 64 | (void)(rc); 65 | (void)(pCtx); 66 | fprintf(stderr, "%s\n", z); 67 | fflush(stderr); 68 | } 69 | 70 | /* 71 | ** Allocate a new db handle. 72 | */ 73 | int lsm_new(lsm_env *pEnv, lsm_db **ppDb){ 74 | lsm_db *pDb; 75 | 76 | /* If the user did not provide an environment, use the default. */ 77 | if( pEnv==0 ) pEnv = lsm_default_env(); 78 | assert( pEnv ); 79 | 80 | /* Allocate the new database handle */ 81 | *ppDb = pDb = (lsm_db *)lsmMallocZero(pEnv, sizeof(lsm_db)); 82 | if( pDb==0 ) return LSM_NOMEM_BKPT; 83 | 84 | /* Initialize the new object */ 85 | pDb->pEnv = pEnv; 86 | pDb->nTreeLimit = LSM_DFLT_AUTOFLUSH; 87 | pDb->nAutockpt = LSM_DFLT_AUTOCHECKPOINT; 88 | pDb->bAutowork = LSM_DFLT_AUTOWORK; 89 | pDb->eSafety = LSM_DFLT_SAFETY; 90 | pDb->xCmp = xCmp; 91 | pDb->nDfltPgsz = LSM_DFLT_PAGE_SIZE; 92 | pDb->nDfltBlksz = LSM_DFLT_BLOCK_SIZE; 93 | pDb->nMerge = LSM_DFLT_AUTOMERGE; 94 | pDb->nMaxFreelist = LSM_MAX_FREELIST_ENTRIES; 95 | pDb->bUseLog = LSM_DFLT_USE_LOG; 96 | pDb->iReader = -1; 97 | pDb->iRwclient = -1; 98 | pDb->bMultiProc = LSM_DFLT_MULTIPLE_PROCESSES; 99 | pDb->iMmap = LSM_DFLT_MMAP; 100 | pDb->xLog = xLog; 101 | pDb->compress.iId = LSM_COMPRESSION_NONE; 102 | return LSM_OK; 103 | } 104 | 105 | lsm_env *lsm_get_env(lsm_db *pDb){ 106 | assert( pDb->pEnv ); 107 | return pDb->pEnv; 108 | } 109 | 110 | /* 111 | ** If database handle pDb is currently holding a client snapshot, but does 112 | ** not have any open cursors or write transactions, release it. 113 | */ 114 | static void dbReleaseClientSnapshot(lsm_db *pDb){ 115 | if( pDb->nTransOpen==0 && pDb->pCsr==0 ){ 116 | lsmFinishReadTrans(pDb); 117 | } 118 | } 119 | 120 | static int getFullpathname( 121 | lsm_env *pEnv, 122 | const char *zRel, 123 | char **pzAbs 124 | ){ 125 | int nAlloc = 0; 126 | char *zAlloc = 0; 127 | int nReq = 0; 128 | int rc; 129 | 130 | do{ 131 | nAlloc = nReq; 132 | rc = pEnv->xFullpath(pEnv, zRel, zAlloc, &nReq); 133 | if( nReq>nAlloc ){ 134 | zAlloc = lsmReallocOrFreeRc(pEnv, zAlloc, nReq, &rc); 135 | } 136 | }while( nReq>nAlloc && rc==LSM_OK ); 137 | 138 | if( rc!=LSM_OK ){ 139 | lsmFree(pEnv, zAlloc); 140 | zAlloc = 0; 141 | } 142 | *pzAbs = zAlloc; 143 | return rc; 144 | } 145 | 146 | /* 147 | ** Check that the bits in the db->mLock mask are consistent with the 148 | ** value stored in db->iRwclient. An assert shall fail otherwise. 149 | */ 150 | static void assertRwclientLockValue(lsm_db *db){ 151 | #ifndef NDEBUG 152 | u64 msk; /* Mask of mLock bits for RWCLIENT locks */ 153 | u64 rwclient = 0; /* Bit corresponding to db->iRwclient */ 154 | 155 | if( db->iRwclient>=0 ){ 156 | rwclient = ((u64)1 << (LSM_LOCK_RWCLIENT(db->iRwclient)-1)); 157 | } 158 | msk = ((u64)1 << (LSM_LOCK_RWCLIENT(LSM_LOCK_NRWCLIENT)-1)) - 1; 159 | msk -= (((u64)1 << (LSM_LOCK_RWCLIENT(0)-1)) - 1); 160 | 161 | assert( (db->mLock & msk)==rwclient ); 162 | #endif 163 | } 164 | 165 | /* 166 | ** Open a new connection to database zFilename. 167 | */ 168 | int lsm_open(lsm_db *pDb, const char *zFilename){ 169 | int rc; 170 | 171 | if( pDb->pDatabase ){ 172 | rc = LSM_MISUSE; 173 | }else{ 174 | char *zFull; 175 | 176 | /* Translate the possibly relative pathname supplied by the user into 177 | ** an absolute pathname. This is required because the supplied path 178 | ** is used (either directly or with "-log" appended to it) for more 179 | ** than one purpose - to open both the database and log files, and 180 | ** perhaps to unlink the log file during disconnection. An absolute 181 | ** path is required to ensure that the correct files are operated 182 | ** on even if the application changes the cwd. */ 183 | rc = getFullpathname(pDb->pEnv, zFilename, &zFull); 184 | assert( rc==LSM_OK || zFull==0 ); 185 | 186 | /* Connect to the database. */ 187 | if( rc==LSM_OK ){ 188 | rc = lsmDbDatabaseConnect(pDb, zFull); 189 | } 190 | 191 | if( pDb->bReadonly==0 ){ 192 | /* Configure the file-system connection with the page-size and block-size 193 | ** of this database. Even if the database file is zero bytes in size 194 | ** on disk, these values have been set in shared-memory by now, and so 195 | ** are guaranteed not to change during the lifetime of this connection. 196 | */ 197 | if( rc==LSM_OK && LSM_OK==(rc = lsmCheckpointLoad(pDb, 0)) ){ 198 | lsmFsSetPageSize(pDb->pFS, lsmCheckpointPgsz(pDb->aSnapshot)); 199 | lsmFsSetBlockSize(pDb->pFS, lsmCheckpointBlksz(pDb->aSnapshot)); 200 | } 201 | } 202 | 203 | lsmFree(pDb->pEnv, zFull); 204 | assertRwclientLockValue(pDb); 205 | } 206 | 207 | assert( pDb->bReadonly==0 || pDb->bReadonly==1 ); 208 | assert( rc!=LSM_OK || (pDb->pShmhdr==0)==(pDb->bReadonly==1) ); 209 | 210 | return rc; 211 | } 212 | 213 | int lsm_close(lsm_db *pDb){ 214 | int rc = LSM_OK; 215 | if( pDb ){ 216 | assert_db_state(pDb); 217 | if( pDb->pCsr || pDb->nTransOpen ){ 218 | rc = LSM_MISUSE_BKPT; 219 | }else{ 220 | lsmMCursorFreeCache(pDb); 221 | lsmFreeSnapshot(pDb->pEnv, pDb->pClient); 222 | pDb->pClient = 0; 223 | 224 | assertRwclientLockValue(pDb); 225 | 226 | lsmDbDatabaseRelease(pDb); 227 | lsmLogClose(pDb); 228 | lsmFsClose(pDb->pFS); 229 | /* assert( pDb->mLock==0 ); */ 230 | 231 | /* Invoke any destructors registered for the compression or 232 | ** compression factory callbacks. */ 233 | if( pDb->factory.xFree ) pDb->factory.xFree(pDb->factory.pCtx); 234 | if( pDb->compress.xFree ) pDb->compress.xFree(pDb->compress.pCtx); 235 | 236 | lsmFree(pDb->pEnv, pDb->rollback.aArray); 237 | lsmFree(pDb->pEnv, pDb->aTrans); 238 | lsmFree(pDb->pEnv, pDb->apShm); 239 | lsmFree(pDb->pEnv, pDb); 240 | } 241 | } 242 | return rc; 243 | } 244 | 245 | int lsm_config(lsm_db *pDb, int eParam, ...){ 246 | int rc = LSM_OK; 247 | va_list ap; 248 | va_start(ap, eParam); 249 | 250 | switch( eParam ){ 251 | case LSM_CONFIG_AUTOFLUSH: { 252 | /* This parameter is read and written in KB. But all internal 253 | ** processing is done in bytes. */ 254 | int *piVal = va_arg(ap, int *); 255 | int iVal = *piVal; 256 | if( iVal>=0 && iVal<=(1024*1024) ){ 257 | pDb->nTreeLimit = iVal*1024; 258 | } 259 | *piVal = (pDb->nTreeLimit / 1024); 260 | break; 261 | } 262 | 263 | case LSM_CONFIG_AUTOWORK: { 264 | int *piVal = va_arg(ap, int *); 265 | if( *piVal>=0 ){ 266 | pDb->bAutowork = *piVal; 267 | } 268 | *piVal = pDb->bAutowork; 269 | break; 270 | } 271 | 272 | case LSM_CONFIG_AUTOCHECKPOINT: { 273 | /* This parameter is read and written in KB. But all internal processing 274 | ** (including the lsm_db.nAutockpt variable) is done in bytes. */ 275 | int *piVal = va_arg(ap, int *); 276 | if( *piVal>=0 ){ 277 | int iVal = *piVal; 278 | pDb->nAutockpt = (i64)iVal * 1024; 279 | } 280 | *piVal = (int)(pDb->nAutockpt / 1024); 281 | break; 282 | } 283 | 284 | case LSM_CONFIG_PAGE_SIZE: { 285 | int *piVal = va_arg(ap, int *); 286 | if( pDb->pDatabase ){ 287 | /* If lsm_open() has been called, this is a read-only parameter. 288 | ** Set the output variable to the page-size according to the 289 | ** FileSystem object. */ 290 | *piVal = lsmFsPageSize(pDb->pFS); 291 | }else{ 292 | if( *piVal>=256 && *piVal<=65536 && ((*piVal-1) & *piVal)==0 ){ 293 | pDb->nDfltPgsz = *piVal; 294 | }else{ 295 | *piVal = pDb->nDfltPgsz; 296 | } 297 | } 298 | break; 299 | } 300 | 301 | case LSM_CONFIG_BLOCK_SIZE: { 302 | /* This parameter is read and written in KB. But all internal 303 | ** processing is done in bytes. */ 304 | int *piVal = va_arg(ap, int *); 305 | if( pDb->pDatabase ){ 306 | /* If lsm_open() has been called, this is a read-only parameter. 307 | ** Set the output variable to the block-size in KB according to the 308 | ** FileSystem object. */ 309 | *piVal = lsmFsBlockSize(pDb->pFS) / 1024; 310 | }else{ 311 | int iVal = *piVal; 312 | if( iVal>=64 && iVal<=65536 && ((iVal-1) & iVal)==0 ){ 313 | pDb->nDfltBlksz = iVal * 1024; 314 | }else{ 315 | *piVal = pDb->nDfltBlksz / 1024; 316 | } 317 | } 318 | break; 319 | } 320 | 321 | case LSM_CONFIG_SAFETY: { 322 | int *piVal = va_arg(ap, int *); 323 | if( *piVal>=0 && *piVal<=2 ){ 324 | pDb->eSafety = *piVal; 325 | } 326 | *piVal = pDb->eSafety; 327 | break; 328 | } 329 | 330 | case LSM_CONFIG_MMAP: { 331 | int *piVal = va_arg(ap, int *); 332 | if( pDb->iReader<0 && *piVal>=0 ){ 333 | pDb->iMmap = *piVal; 334 | rc = lsmFsConfigure(pDb); 335 | } 336 | *piVal = pDb->iMmap; 337 | break; 338 | } 339 | 340 | case LSM_CONFIG_USE_LOG: { 341 | int *piVal = va_arg(ap, int *); 342 | if( pDb->nTransOpen==0 && (*piVal==0 || *piVal==1) ){ 343 | pDb->bUseLog = *piVal; 344 | } 345 | *piVal = pDb->bUseLog; 346 | break; 347 | } 348 | 349 | case LSM_CONFIG_AUTOMERGE: { 350 | int *piVal = va_arg(ap, int *); 351 | if( *piVal>1 ) pDb->nMerge = *piVal; 352 | *piVal = pDb->nMerge; 353 | break; 354 | } 355 | 356 | case LSM_CONFIG_MAX_FREELIST: { 357 | int *piVal = va_arg(ap, int *); 358 | if( *piVal>=2 && *piVal<=LSM_MAX_FREELIST_ENTRIES ){ 359 | pDb->nMaxFreelist = *piVal; 360 | } 361 | *piVal = pDb->nMaxFreelist; 362 | break; 363 | } 364 | 365 | case LSM_CONFIG_MULTIPLE_PROCESSES: { 366 | int *piVal = va_arg(ap, int *); 367 | if( pDb->pDatabase ){ 368 | /* If lsm_open() has been called, this is a read-only parameter. 369 | ** Set the output variable to true if this connection is currently 370 | ** in multi-process mode. */ 371 | *piVal = lsmDbMultiProc(pDb); 372 | }else{ 373 | pDb->bMultiProc = *piVal = (*piVal!=0); 374 | } 375 | break; 376 | } 377 | 378 | case LSM_CONFIG_READONLY: { 379 | int *piVal = va_arg(ap, int *); 380 | /* If lsm_open() has been called, this is a read-only parameter. */ 381 | if( pDb->pDatabase==0 && *piVal>=0 ){ 382 | pDb->bReadonly = *piVal = (*piVal!=0); 383 | } 384 | *piVal = pDb->bReadonly; 385 | break; 386 | } 387 | 388 | case LSM_CONFIG_SET_COMPRESSION: { 389 | lsm_compress *p = va_arg(ap, lsm_compress *); 390 | if( pDb->iReader>=0 && pDb->bInFactory==0 ){ 391 | /* May not change compression schemes with an open transaction */ 392 | rc = LSM_MISUSE_BKPT; 393 | }else{ 394 | if( pDb->compress.xFree ){ 395 | /* Invoke any destructor belonging to the current compression. */ 396 | pDb->compress.xFree(pDb->compress.pCtx); 397 | } 398 | if( p->xBound==0 ){ 399 | memset(&pDb->compress, 0, sizeof(lsm_compress)); 400 | pDb->compress.iId = LSM_COMPRESSION_NONE; 401 | }else{ 402 | memcpy(&pDb->compress, p, sizeof(lsm_compress)); 403 | } 404 | rc = lsmFsConfigure(pDb); 405 | } 406 | break; 407 | } 408 | 409 | case LSM_CONFIG_SET_COMPRESSION_FACTORY: { 410 | lsm_compress_factory *p = va_arg(ap, lsm_compress_factory *); 411 | if( pDb->factory.xFree ){ 412 | /* Invoke any destructor belonging to the current factory. */ 413 | pDb->factory.xFree(pDb->factory.pCtx); 414 | } 415 | memcpy(&pDb->factory, p, sizeof(lsm_compress_factory)); 416 | break; 417 | } 418 | 419 | case LSM_CONFIG_GET_COMPRESSION: { 420 | lsm_compress *p = va_arg(ap, lsm_compress *); 421 | memcpy(p, &pDb->compress, sizeof(lsm_compress)); 422 | break; 423 | } 424 | 425 | default: 426 | rc = LSM_MISUSE; 427 | break; 428 | } 429 | 430 | va_end(ap); 431 | return rc; 432 | } 433 | 434 | void lsmAppendSegmentList(LsmString *pStr, char *zPre, Segment *pSeg){ 435 | lsmStringAppendf(pStr, "%s{%lld %lld %lld %lld}", zPre, 436 | pSeg->iFirst, pSeg->iLastPg, pSeg->iRoot, pSeg->nSize 437 | ); 438 | } 439 | 440 | static int infoGetWorker(lsm_db *pDb, Snapshot **pp, int *pbUnlock){ 441 | int rc = LSM_OK; 442 | 443 | assert( *pbUnlock==0 ); 444 | if( !pDb->pWorker ){ 445 | rc = lsmBeginWork(pDb); 446 | if( rc!=LSM_OK ) return rc; 447 | *pbUnlock = 1; 448 | } 449 | if( pp ) *pp = pDb->pWorker; 450 | return rc; 451 | } 452 | 453 | static void infoFreeWorker(lsm_db *pDb, int bUnlock){ 454 | if( bUnlock ){ 455 | int rcdummy = LSM_BUSY; 456 | lsmFinishWork(pDb, 0, &rcdummy); 457 | } 458 | } 459 | 460 | int lsmStructList( 461 | lsm_db *pDb, /* Database handle */ 462 | char **pzOut /* OUT: Nul-terminated string (tcl list) */ 463 | ){ 464 | Level *pTopLevel = 0; /* Top level of snapshot to report on */ 465 | int rc = LSM_OK; 466 | Level *p; 467 | LsmString s; 468 | Snapshot *pWorker; /* Worker snapshot */ 469 | int bUnlock = 0; 470 | 471 | /* Obtain the worker snapshot */ 472 | rc = infoGetWorker(pDb, &pWorker, &bUnlock); 473 | if( rc!=LSM_OK ) return rc; 474 | 475 | /* Format the contents of the snapshot as text */ 476 | pTopLevel = lsmDbSnapshotLevel(pWorker); 477 | lsmStringInit(&s, pDb->pEnv); 478 | for(p=pTopLevel; rc==LSM_OK && p; p=p->pNext){ 479 | int i; 480 | lsmStringAppendf(&s, "%s{%d", (s.n ? " " : ""), (int)p->iAge); 481 | lsmAppendSegmentList(&s, " ", &p->lhs); 482 | for(i=0; rc==LSM_OK && inRight; i++){ 483 | lsmAppendSegmentList(&s, " ", &p->aRhs[i]); 484 | } 485 | lsmStringAppend(&s, "}", 1); 486 | } 487 | rc = s.n>=0 ? LSM_OK : LSM_NOMEM; 488 | 489 | /* Release the snapshot and return */ 490 | infoFreeWorker(pDb, bUnlock); 491 | *pzOut = s.z; 492 | return rc; 493 | } 494 | 495 | static int infoFreelistCb(void *pCtx, int iBlk, i64 iSnapshot){ 496 | LsmString *pStr = (LsmString *)pCtx; 497 | lsmStringAppendf(pStr, "%s{%d %lld}", (pStr->n?" ":""), iBlk, iSnapshot); 498 | return 0; 499 | } 500 | 501 | int lsmInfoFreelist(lsm_db *pDb, char **pzOut){ 502 | Snapshot *pWorker; /* Worker snapshot */ 503 | int bUnlock = 0; 504 | LsmString s; 505 | int rc; 506 | 507 | /* Obtain the worker snapshot */ 508 | rc = infoGetWorker(pDb, &pWorker, &bUnlock); 509 | if( rc!=LSM_OK ) return rc; 510 | 511 | lsmStringInit(&s, pDb->pEnv); 512 | rc = lsmWalkFreelist(pDb, 0, infoFreelistCb, &s); 513 | if( rc!=LSM_OK ){ 514 | lsmFree(pDb->pEnv, s.z); 515 | }else{ 516 | *pzOut = s.z; 517 | } 518 | 519 | /* Release the snapshot and return */ 520 | infoFreeWorker(pDb, bUnlock); 521 | return rc; 522 | } 523 | 524 | static int infoTreeSize(lsm_db *db, int *pnOldKB, int *pnNewKB){ 525 | ShmHeader *pShm = db->pShmhdr; 526 | TreeHeader *p = &pShm->hdr1; 527 | 528 | /* The following code suffers from two race conditions, as it accesses and 529 | ** trusts the contents of shared memory without verifying checksums: 530 | ** 531 | ** * The two values read - TreeHeader.root.nByte and oldroot.nByte - are 532 | ** 32-bit fields. It is assumed that reading from one of these 533 | ** is atomic - that it is not possible to read a partially written 534 | ** garbage value. However the two values may be mutually inconsistent. 535 | ** 536 | ** * TreeHeader.iLogOff is a 64-bit value. And lsmCheckpointLogOffset() 537 | ** reads a 64-bit value from a snapshot stored in shared memory. It 538 | ** is assumed that in each case it is possible to read a partially 539 | ** written garbage value. If this occurs, then the value returned 540 | ** for the size of the "old" tree may reflect the size of an "old" 541 | ** tree that was recently flushed to disk. 542 | ** 543 | ** Given the context in which this function is called (as a result of an 544 | ** lsm_info(LSM_INFO_TREE_SIZE) request), neither of these are considered to 545 | ** be problems. 546 | */ 547 | *pnNewKB = ((int)p->root.nByte + 1023) / 1024; 548 | if( p->iOldShmid ){ 549 | if( p->iOldLog==lsmCheckpointLogOffset(pShm->aSnap1) ){ 550 | *pnOldKB = 0; 551 | }else{ 552 | *pnOldKB = ((int)p->oldroot.nByte + 1023) / 1024; 553 | } 554 | }else{ 555 | *pnOldKB = 0; 556 | } 557 | 558 | return LSM_OK; 559 | } 560 | 561 | int lsm_info(lsm_db *pDb, int eParam, ...){ 562 | int rc = LSM_OK; 563 | va_list ap; 564 | va_start(ap, eParam); 565 | 566 | switch( eParam ){ 567 | case LSM_INFO_NWRITE: { 568 | int *piVal = va_arg(ap, int *); 569 | *piVal = lsmFsNWrite(pDb->pFS); 570 | break; 571 | } 572 | 573 | case LSM_INFO_NREAD: { 574 | int *piVal = va_arg(ap, int *); 575 | *piVal = lsmFsNRead(pDb->pFS); 576 | break; 577 | } 578 | 579 | case LSM_INFO_DB_STRUCTURE: { 580 | char **pzVal = va_arg(ap, char **); 581 | rc = lsmStructList(pDb, pzVal); 582 | break; 583 | } 584 | 585 | case LSM_INFO_ARRAY_STRUCTURE: { 586 | LsmPgno pgno = va_arg(ap, LsmPgno); 587 | char **pzVal = va_arg(ap, char **); 588 | rc = lsmInfoArrayStructure(pDb, 0, pgno, pzVal); 589 | break; 590 | } 591 | 592 | case LSM_INFO_ARRAY_PAGES: { 593 | LsmPgno pgno = va_arg(ap, LsmPgno); 594 | char **pzVal = va_arg(ap, char **); 595 | rc = lsmInfoArrayPages(pDb, pgno, pzVal); 596 | break; 597 | } 598 | 599 | case LSM_INFO_PAGE_HEX_DUMP: 600 | case LSM_INFO_PAGE_ASCII_DUMP: { 601 | LsmPgno pgno = va_arg(ap, LsmPgno); 602 | char **pzVal = va_arg(ap, char **); 603 | int bUnlock = 0; 604 | rc = infoGetWorker(pDb, 0, &bUnlock); 605 | if( rc==LSM_OK ){ 606 | int bHex = (eParam==LSM_INFO_PAGE_HEX_DUMP); 607 | rc = lsmInfoPageDump(pDb, pgno, bHex, pzVal); 608 | } 609 | infoFreeWorker(pDb, bUnlock); 610 | break; 611 | } 612 | 613 | case LSM_INFO_LOG_STRUCTURE: { 614 | char **pzVal = va_arg(ap, char **); 615 | rc = lsmInfoLogStructure(pDb, pzVal); 616 | break; 617 | } 618 | 619 | case LSM_INFO_FREELIST: { 620 | char **pzVal = va_arg(ap, char **); 621 | rc = lsmInfoFreelist(pDb, pzVal); 622 | break; 623 | } 624 | 625 | case LSM_INFO_CHECKPOINT_SIZE: { 626 | int *pnKB = va_arg(ap, int *); 627 | rc = lsmCheckpointSize(pDb, pnKB); 628 | break; 629 | } 630 | 631 | case LSM_INFO_TREE_SIZE: { 632 | int *pnOld = va_arg(ap, int *); 633 | int *pnNew = va_arg(ap, int *); 634 | rc = infoTreeSize(pDb, pnOld, pnNew); 635 | break; 636 | } 637 | 638 | case LSM_INFO_COMPRESSION_ID: { 639 | unsigned int *piOut = va_arg(ap, unsigned int *); 640 | if( pDb->pClient ){ 641 | *piOut = pDb->pClient->iCmpId; 642 | }else{ 643 | rc = lsmInfoCompressionId(pDb, piOut); 644 | } 645 | break; 646 | } 647 | 648 | default: 649 | rc = LSM_MISUSE; 650 | break; 651 | } 652 | 653 | va_end(ap); 654 | return rc; 655 | } 656 | 657 | static int doWriteOp( 658 | lsm_db *pDb, 659 | int bDeleteRange, 660 | const void *pKey, int nKey, /* Key to write or delete */ 661 | const void *pVal, int nVal /* Value to write. Or nVal==-1 for a delete */ 662 | ){ 663 | int rc = LSM_OK; /* Return code */ 664 | int bCommit = 0; /* True to commit before returning */ 665 | 666 | if( pDb->nTransOpen==0 ){ 667 | bCommit = 1; 668 | rc = lsm_begin(pDb, 1); 669 | } 670 | 671 | if( rc==LSM_OK ){ 672 | int eType = (bDeleteRange ? LSM_DRANGE : (nVal>=0?LSM_WRITE:LSM_DELETE)); 673 | rc = lsmLogWrite(pDb, eType, (void *)pKey, nKey, (void *)pVal, nVal); 674 | } 675 | 676 | lsmSortedSaveTreeCursors(pDb); 677 | 678 | if( rc==LSM_OK ){ 679 | int pgsz = lsmFsPageSize(pDb->pFS); 680 | int nQuant = LSM_AUTOWORK_QUANT * pgsz; 681 | int nBefore; 682 | int nAfter; 683 | int nDiff; 684 | 685 | if( nQuant>pDb->nTreeLimit ){ 686 | nQuant = LSM_MAX(pDb->nTreeLimit, pgsz); 687 | } 688 | 689 | nBefore = lsmTreeSize(pDb); 690 | if( bDeleteRange ){ 691 | rc = lsmTreeDelete(pDb, (void *)pKey, nKey, (void *)pVal, nVal); 692 | }else{ 693 | rc = lsmTreeInsert(pDb, (void *)pKey, nKey, (void *)pVal, nVal); 694 | } 695 | 696 | nAfter = lsmTreeSize(pDb); 697 | nDiff = (nAfter/nQuant) - (nBefore/nQuant); 698 | if( rc==LSM_OK && pDb->bAutowork && nDiff!=0 ){ 699 | rc = lsmSortedAutoWork(pDb, nDiff * LSM_AUTOWORK_QUANT); 700 | } 701 | } 702 | 703 | /* If a transaction was opened at the start of this function, commit it. 704 | ** Or, if an error has occurred, roll it back. */ 705 | if( bCommit ){ 706 | if( rc==LSM_OK ){ 707 | rc = lsm_commit(pDb, 0); 708 | }else{ 709 | lsm_rollback(pDb, 0); 710 | } 711 | } 712 | 713 | return rc; 714 | } 715 | 716 | /* 717 | ** Write a new value into the database. 718 | */ 719 | int lsm_insert( 720 | lsm_db *db, /* Database connection */ 721 | const void *pKey, int nKey, /* Key to write or delete */ 722 | const void *pVal, int nVal /* Value to write. Or nVal==-1 for a delete */ 723 | ){ 724 | return doWriteOp(db, 0, pKey, nKey, pVal, nVal); 725 | } 726 | 727 | /* 728 | ** Delete a value from the database. 729 | */ 730 | int lsm_delete(lsm_db *db, const void *pKey, int nKey){ 731 | return doWriteOp(db, 0, pKey, nKey, 0, -1); 732 | } 733 | 734 | /* 735 | ** Delete a range of database keys. 736 | */ 737 | int lsm_delete_range( 738 | lsm_db *db, /* Database handle */ 739 | const void *pKey1, int nKey1, /* Lower bound of range to delete */ 740 | const void *pKey2, int nKey2 /* Upper bound of range to delete */ 741 | ){ 742 | int rc = LSM_OK; 743 | if( db->xCmp((void *)pKey1, nKey1, (void *)pKey2, nKey2)<0 ){ 744 | rc = doWriteOp(db, 1, pKey1, nKey1, pKey2, nKey2); 745 | } 746 | return rc; 747 | } 748 | 749 | /* 750 | ** Open a new cursor handle. 751 | ** 752 | ** If there are currently no other open cursor handles, and no open write 753 | ** transaction, open a read transaction here. 754 | */ 755 | int lsm_csr_open(lsm_db *pDb, lsm_cursor **ppCsr){ 756 | int rc = LSM_OK; /* Return code */ 757 | MultiCursor *pCsr = 0; /* New cursor object */ 758 | 759 | /* Open a read transaction if one is not already open. */ 760 | assert_db_state(pDb); 761 | 762 | if( pDb->pShmhdr==0 ){ 763 | assert( pDb->bReadonly ); 764 | rc = lsmBeginRoTrans(pDb); 765 | }else if( pDb->iReader<0 ){ 766 | rc = lsmBeginReadTrans(pDb); 767 | } 768 | 769 | /* Allocate the multi-cursor. */ 770 | if( rc==LSM_OK ){ 771 | rc = lsmMCursorNew(pDb, &pCsr); 772 | } 773 | 774 | /* If an error has occured, set the output to NULL and delete any partially 775 | ** allocated cursor. If this means there are no open cursors, release the 776 | ** client snapshot. */ 777 | if( rc!=LSM_OK ){ 778 | lsmMCursorClose(pCsr, 0); 779 | dbReleaseClientSnapshot(pDb); 780 | } 781 | 782 | assert_db_state(pDb); 783 | *ppCsr = (lsm_cursor *)pCsr; 784 | return rc; 785 | } 786 | 787 | /* 788 | ** Close a cursor opened using lsm_csr_open(). 789 | */ 790 | int lsm_csr_close(lsm_cursor *p){ 791 | if( p ){ 792 | lsm_db *pDb = lsmMCursorDb((MultiCursor *)p); 793 | assert_db_state(pDb); 794 | lsmMCursorClose((MultiCursor *)p, 1); 795 | dbReleaseClientSnapshot(pDb); 796 | assert_db_state(pDb); 797 | } 798 | return LSM_OK; 799 | } 800 | 801 | /* 802 | ** Attempt to seek the cursor to the database entry specified by pKey/nKey. 803 | ** If an error occurs (e.g. an OOM or IO error), return an LSM error code. 804 | ** Otherwise, return LSM_OK. 805 | */ 806 | int lsm_csr_seek(lsm_cursor *pCsr, const void *pKey, int nKey, int eSeek){ 807 | return lsmMCursorSeek((MultiCursor *)pCsr, 0, (void *)pKey, nKey, eSeek); 808 | } 809 | 810 | int lsm_csr_next(lsm_cursor *pCsr){ 811 | return lsmMCursorNext((MultiCursor *)pCsr); 812 | } 813 | 814 | int lsm_csr_prev(lsm_cursor *pCsr){ 815 | return lsmMCursorPrev((MultiCursor *)pCsr); 816 | } 817 | 818 | int lsm_csr_first(lsm_cursor *pCsr){ 819 | return lsmMCursorFirst((MultiCursor *)pCsr); 820 | } 821 | 822 | int lsm_csr_last(lsm_cursor *pCsr){ 823 | return lsmMCursorLast((MultiCursor *)pCsr); 824 | } 825 | 826 | int lsm_csr_valid(lsm_cursor *pCsr){ 827 | return lsmMCursorValid((MultiCursor *)pCsr); 828 | } 829 | 830 | int lsm_csr_key(lsm_cursor *pCsr, const void **ppKey, int *pnKey){ 831 | return lsmMCursorKey((MultiCursor *)pCsr, (void **)ppKey, pnKey); 832 | } 833 | 834 | int lsm_csr_value(lsm_cursor *pCsr, const void **ppVal, int *pnVal){ 835 | return lsmMCursorValue((MultiCursor *)pCsr, (void **)ppVal, pnVal); 836 | } 837 | 838 | void lsm_config_log( 839 | lsm_db *pDb, 840 | void (*xLog)(void *, int, const char *), 841 | void *pCtx 842 | ){ 843 | pDb->xLog = xLog; 844 | pDb->pLogCtx = pCtx; 845 | } 846 | 847 | void lsm_config_work_hook( 848 | lsm_db *pDb, 849 | void (*xWork)(lsm_db *, void *), 850 | void *pCtx 851 | ){ 852 | pDb->xWork = xWork; 853 | pDb->pWorkCtx = pCtx; 854 | } 855 | 856 | void lsmLogMessage(lsm_db *pDb, int rc, const char *zFormat, ...){ 857 | if( pDb->xLog ){ 858 | LsmString s; 859 | va_list ap, ap2; 860 | lsmStringInit(&s, pDb->pEnv); 861 | va_start(ap, zFormat); 862 | va_start(ap2, zFormat); 863 | lsmStringVAppendf(&s, zFormat, ap, ap2); 864 | va_end(ap); 865 | va_end(ap2); 866 | pDb->xLog(pDb->pLogCtx, rc, s.z); 867 | lsmStringClear(&s); 868 | } 869 | } 870 | 871 | int lsm_begin(lsm_db *pDb, int iLevel){ 872 | int rc; 873 | 874 | assert_db_state( pDb ); 875 | rc = (pDb->bReadonly ? LSM_READONLY : LSM_OK); 876 | 877 | /* A value less than zero means open one more transaction. */ 878 | if( iLevel<0 ) iLevel = pDb->nTransOpen + 1; 879 | if( iLevel>pDb->nTransOpen ){ 880 | int i; 881 | 882 | /* Extend the pDb->aTrans[] array if required. */ 883 | if( rc==LSM_OK && pDb->nTransAllocpEnv, pDb->aTrans, nByte); 887 | if( !aNew ){ 888 | rc = LSM_NOMEM; 889 | }else{ 890 | nByte = sizeof(TransMark) * (iLevel+1 - pDb->nTransAlloc); 891 | memset(&aNew[pDb->nTransAlloc], 0, nByte); 892 | pDb->nTransAlloc = iLevel+1; 893 | pDb->aTrans = aNew; 894 | } 895 | } 896 | 897 | if( rc==LSM_OK && pDb->nTransOpen==0 ){ 898 | rc = lsmBeginWriteTrans(pDb); 899 | } 900 | 901 | if( rc==LSM_OK ){ 902 | for(i=pDb->nTransOpen; iaTrans[i].tree); 904 | lsmLogTell(pDb, &pDb->aTrans[i].log); 905 | } 906 | pDb->nTransOpen = iLevel; 907 | } 908 | } 909 | 910 | return rc; 911 | } 912 | 913 | int lsm_commit(lsm_db *pDb, int iLevel){ 914 | int rc = LSM_OK; 915 | 916 | assert_db_state( pDb ); 917 | 918 | /* A value less than zero means close the innermost nested transaction. */ 919 | if( iLevel<0 ) iLevel = LSM_MAX(0, pDb->nTransOpen - 1); 920 | 921 | if( iLevelnTransOpen ){ 922 | if( iLevel==0 ){ 923 | int rc2; 924 | /* Commit the transaction to disk. */ 925 | if( rc==LSM_OK ) rc = lsmLogCommit(pDb); 926 | if( rc==LSM_OK && pDb->eSafety==LSM_SAFETY_FULL ){ 927 | rc = lsmFsSyncLog(pDb->pFS); 928 | } 929 | rc2 = lsmFinishWriteTrans(pDb, (rc==LSM_OK)); 930 | if( rc==LSM_OK ) rc = rc2; 931 | } 932 | pDb->nTransOpen = iLevel; 933 | } 934 | dbReleaseClientSnapshot(pDb); 935 | return rc; 936 | } 937 | 938 | int lsm_rollback(lsm_db *pDb, int iLevel){ 939 | int rc = LSM_OK; 940 | assert_db_state( pDb ); 941 | 942 | if( pDb->nTransOpen ){ 943 | /* A value less than zero means close the innermost nested transaction. */ 944 | if( iLevel<0 ) iLevel = LSM_MAX(0, pDb->nTransOpen - 1); 945 | 946 | if( iLevel<=pDb->nTransOpen ){ 947 | TransMark *pMark = &pDb->aTrans[(iLevel==0 ? 0 : iLevel-1)]; 948 | lsmTreeRollback(pDb, &pMark->tree); 949 | if( iLevel ) lsmLogSeek(pDb, &pMark->log); 950 | pDb->nTransOpen = iLevel; 951 | } 952 | 953 | if( pDb->nTransOpen==0 ){ 954 | lsmFinishWriteTrans(pDb, 0); 955 | } 956 | dbReleaseClientSnapshot(pDb); 957 | } 958 | 959 | return rc; 960 | } 961 | 962 | int lsm_get_user_version(lsm_db *pDb, unsigned int *piUsr){ 963 | int rc = LSM_OK; /* Return code */ 964 | 965 | /* Open a read transaction if one is not already open. */ 966 | assert_db_state(pDb); 967 | if( pDb->pShmhdr==0 ){ 968 | assert( pDb->bReadonly ); 969 | rc = lsmBeginRoTrans(pDb); 970 | }else if( pDb->iReader<0 ){ 971 | rc = lsmBeginReadTrans(pDb); 972 | } 973 | 974 | /* Allocate the multi-cursor. */ 975 | if( rc==LSM_OK ){ 976 | *piUsr = pDb->treehdr.iUsrVersion; 977 | } 978 | 979 | dbReleaseClientSnapshot(pDb); 980 | assert_db_state(pDb); 981 | return rc; 982 | } 983 | 984 | int lsm_set_user_version(lsm_db *pDb, unsigned int iUsr){ 985 | int rc = LSM_OK; /* Return code */ 986 | int bCommit = 0; /* True to commit before returning */ 987 | 988 | if( pDb->nTransOpen==0 ){ 989 | bCommit = 1; 990 | rc = lsm_begin(pDb, 1); 991 | } 992 | 993 | if( rc==LSM_OK ){ 994 | pDb->treehdr.iUsrVersion = iUsr; 995 | } 996 | 997 | /* If a transaction was opened at the start of this function, commit it. 998 | ** Or, if an error has occurred, roll it back. */ 999 | if( bCommit ){ 1000 | if( rc==LSM_OK ){ 1001 | rc = lsm_commit(pDb, 0); 1002 | }else{ 1003 | lsm_rollback(pDb, 0); 1004 | } 1005 | } 1006 | 1007 | return rc; 1008 | } 1009 | -------------------------------------------------------------------------------- /src/lsm_mem.c: -------------------------------------------------------------------------------- 1 | /* 2 | ** 2011-08-18 3 | ** 4 | ** The author disclaims copyright to this source code. In place of 5 | ** a legal notice, here is a blessing: 6 | ** 7 | ** May you do good and not evil. 8 | ** May you find forgiveness for yourself and forgive others. 9 | ** May you share freely, never taking more than you give. 10 | ** 11 | ************************************************************************* 12 | ** 13 | ** Helper routines for memory allocation. 14 | */ 15 | #include "lsmInt.h" 16 | 17 | /* 18 | ** The following routines are called internally by LSM sub-routines. In 19 | ** this case a valid environment pointer must be supplied. 20 | */ 21 | void *lsmMalloc(lsm_env *pEnv, size_t N){ 22 | assert( pEnv ); 23 | return pEnv->xMalloc(pEnv, N); 24 | } 25 | void lsmFree(lsm_env *pEnv, void *p){ 26 | assert( pEnv ); 27 | pEnv->xFree(pEnv, p); 28 | } 29 | void *lsmRealloc(lsm_env *pEnv, void *p, size_t N){ 30 | assert( pEnv ); 31 | return pEnv->xRealloc(pEnv, p, N); 32 | } 33 | 34 | /* 35 | ** Core memory allocation routines for LSM. 36 | */ 37 | void *lsm_malloc(lsm_env *pEnv, size_t N){ 38 | return lsmMalloc(pEnv ? pEnv : lsm_default_env(), N); 39 | } 40 | void lsm_free(lsm_env *pEnv, void *p){ 41 | lsmFree(pEnv ? pEnv : lsm_default_env(), p); 42 | } 43 | void *lsm_realloc(lsm_env *pEnv, void *p, size_t N){ 44 | return lsmRealloc(pEnv ? pEnv : lsm_default_env(), p, N); 45 | } 46 | 47 | void *lsmMallocZero(lsm_env *pEnv, size_t N){ 48 | void *pRet; 49 | assert( pEnv ); 50 | pRet = lsmMalloc(pEnv, N); 51 | if( pRet ) memset(pRet, 0, N); 52 | return pRet; 53 | } 54 | 55 | void *lsmMallocRc(lsm_env *pEnv, size_t N, int *pRc){ 56 | void *pRet = 0; 57 | if( *pRc==LSM_OK ){ 58 | pRet = lsmMalloc(pEnv, N); 59 | if( pRet==0 ){ 60 | *pRc = LSM_NOMEM_BKPT; 61 | } 62 | } 63 | return pRet; 64 | } 65 | 66 | void *lsmMallocZeroRc(lsm_env *pEnv, size_t N, int *pRc){ 67 | void *pRet = 0; 68 | if( *pRc==LSM_OK ){ 69 | pRet = lsmMallocZero(pEnv, N); 70 | if( pRet==0 ){ 71 | *pRc = LSM_NOMEM_BKPT; 72 | } 73 | } 74 | return pRet; 75 | } 76 | 77 | void *lsmReallocOrFree(lsm_env *pEnv, void *p, size_t N){ 78 | void *pNew; 79 | pNew = lsm_realloc(pEnv, p, N); 80 | if( !pNew ) lsm_free(pEnv, p); 81 | return pNew; 82 | } 83 | 84 | void *lsmReallocOrFreeRc(lsm_env *pEnv, void *p, size_t N, int *pRc){ 85 | void *pRet = 0; 86 | if( *pRc ){ 87 | lsmFree(pEnv, p); 88 | }else{ 89 | pRet = lsmReallocOrFree(pEnv, p, N); 90 | if( !pRet ) *pRc = LSM_NOMEM_BKPT; 91 | } 92 | return pRet; 93 | } 94 | 95 | char *lsmMallocStrdup(lsm_env *pEnv, const char *zIn){ 96 | int nByte; 97 | char *zRet; 98 | nByte = strlen(zIn); 99 | zRet = lsmMalloc(pEnv, nByte+1); 100 | if( zRet ){ 101 | memcpy(zRet, zIn, nByte+1); 102 | } 103 | return zRet; 104 | } 105 | -------------------------------------------------------------------------------- /src/lsm_mutex.c: -------------------------------------------------------------------------------- 1 | /* 2 | ** 2012-01-30 3 | ** 4 | ** The author disclaims copyright to this source code. In place of 5 | ** a legal notice, here is a blessing: 6 | ** 7 | ** May you do good and not evil. 8 | ** May you find forgiveness for yourself and forgive others. 9 | ** May you share freely, never taking more than you give. 10 | ** 11 | ************************************************************************* 12 | ** 13 | ** Mutex functions for LSM. 14 | */ 15 | #include "lsmInt.h" 16 | 17 | /* 18 | ** Allocate a new mutex. 19 | */ 20 | int lsmMutexNew(lsm_env *pEnv, lsm_mutex **ppNew){ 21 | return pEnv->xMutexNew(pEnv, ppNew); 22 | } 23 | 24 | /* 25 | ** Return a handle for one of the static mutexes. 26 | */ 27 | int lsmMutexStatic(lsm_env *pEnv, int iMutex, lsm_mutex **ppStatic){ 28 | return pEnv->xMutexStatic(pEnv, iMutex, ppStatic); 29 | } 30 | 31 | /* 32 | ** Free a mutex allocated by lsmMutexNew(). 33 | */ 34 | void lsmMutexDel(lsm_env *pEnv, lsm_mutex *pMutex){ 35 | if( pMutex ) pEnv->xMutexDel(pMutex); 36 | } 37 | 38 | /* 39 | ** Enter a mutex. 40 | */ 41 | void lsmMutexEnter(lsm_env *pEnv, lsm_mutex *pMutex){ 42 | pEnv->xMutexEnter(pMutex); 43 | } 44 | 45 | /* 46 | ** Attempt to enter a mutex, but do not block. If successful, return zero. 47 | ** Otherwise, if the mutex is already held by some other thread and is not 48 | ** entered, return non zero. 49 | ** 50 | ** Each successful call to this function must be matched by a call to 51 | ** lsmMutexLeave(). 52 | */ 53 | int lsmMutexTry(lsm_env *pEnv, lsm_mutex *pMutex){ 54 | return pEnv->xMutexTry(pMutex); 55 | } 56 | 57 | /* 58 | ** Leave a mutex. 59 | */ 60 | void lsmMutexLeave(lsm_env *pEnv, lsm_mutex *pMutex){ 61 | pEnv->xMutexLeave(pMutex); 62 | } 63 | 64 | #ifndef NDEBUG 65 | /* 66 | ** Return non-zero if the mutex passed as the second argument is held 67 | ** by the calling thread, or zero otherwise. If the implementation is not 68 | ** able to tell if the mutex is held by the caller, it should return 69 | ** non-zero. 70 | ** 71 | ** This function is only used as part of assert() statements. 72 | */ 73 | int lsmMutexHeld(lsm_env *pEnv, lsm_mutex *pMutex){ 74 | return pEnv->xMutexHeld ? pEnv->xMutexHeld(pMutex) : 1; 75 | } 76 | 77 | /* 78 | ** Return non-zero if the mutex passed as the second argument is not 79 | ** held by the calling thread, or zero otherwise. If the implementation 80 | ** is not able to tell if the mutex is held by the caller, it should 81 | ** return non-zero. 82 | ** 83 | ** This function is only used as part of assert() statements. 84 | */ 85 | int lsmMutexNotHeld(lsm_env *pEnv, lsm_mutex *pMutex){ 86 | return pEnv->xMutexNotHeld ? pEnv->xMutexNotHeld(pMutex) : 1; 87 | } 88 | #endif 89 | -------------------------------------------------------------------------------- /src/lsm_str.c: -------------------------------------------------------------------------------- 1 | /* 2 | ** 2012-04-27 3 | ** 4 | ** The author disclaims copyright to this source code. In place of 5 | ** a legal notice, here is a blessing: 6 | ** 7 | ** May you do good and not evil. 8 | ** May you find forgiveness for yourself and forgive others. 9 | ** May you share freely, never taking more than you give. 10 | ** 11 | ************************************************************************* 12 | ** 13 | ** Dynamic string functions. 14 | */ 15 | #include "lsmInt.h" 16 | 17 | /* 18 | ** Turn bulk and uninitialized memory into an LsmString object 19 | */ 20 | void lsmStringInit(LsmString *pStr, lsm_env *pEnv){ 21 | memset(pStr, 0, sizeof(pStr[0])); 22 | pStr->pEnv = pEnv; 23 | } 24 | 25 | /* 26 | ** Increase the memory allocated for holding the string. Realloc as needed. 27 | ** 28 | ** If a memory allocation error occurs, set pStr->n to -1 and free the existing 29 | ** allocation. If a prior memory allocation has occurred, this routine is a 30 | ** no-op. 31 | */ 32 | int lsmStringExtend(LsmString *pStr, int nNew){ 33 | assert( nNew>0 ); 34 | if( pStr->n<0 ) return LSM_NOMEM; 35 | if( pStr->n + nNew >= pStr->nAlloc ){ 36 | int nAlloc = pStr->n + nNew + 100; 37 | char *zNew = lsmRealloc(pStr->pEnv, pStr->z, nAlloc); 38 | if( zNew==0 ){ 39 | lsmFree(pStr->pEnv, pStr->z); 40 | nAlloc = 0; 41 | pStr->n = -1; 42 | } 43 | pStr->nAlloc = nAlloc; 44 | pStr->z = zNew; 45 | } 46 | return (pStr->z ? LSM_OK : LSM_NOMEM_BKPT); 47 | } 48 | 49 | /* 50 | ** Clear an LsmString object, releasing any allocated memory that it holds. 51 | ** This also clears the error indication (if any). 52 | */ 53 | void lsmStringClear(LsmString *pStr){ 54 | lsmFree(pStr->pEnv, pStr->z); 55 | lsmStringInit(pStr, pStr->pEnv); 56 | } 57 | 58 | /* 59 | ** Append N bytes of text to the end of an LsmString object. If 60 | ** N is negative, append the entire string. 61 | ** 62 | ** If the string is in an error state, this routine is a no-op. 63 | */ 64 | int lsmStringAppend(LsmString *pStr, const char *z, int N){ 65 | int rc; 66 | if( N<0 ) N = (int)strlen(z); 67 | rc = lsmStringExtend(pStr, N+1); 68 | if( pStr->nAlloc ){ 69 | memcpy(pStr->z+pStr->n, z, N+1); 70 | pStr->n += N; 71 | } 72 | return rc; 73 | } 74 | 75 | int lsmStringBinAppend(LsmString *pStr, const u8 *a, int n){ 76 | int rc; 77 | rc = lsmStringExtend(pStr, n); 78 | if( pStr->nAlloc ){ 79 | memcpy(pStr->z+pStr->n, a, n); 80 | pStr->n += n; 81 | } 82 | return rc; 83 | } 84 | 85 | /* 86 | ** Append printf-formatted content to an LsmString. 87 | */ 88 | void lsmStringVAppendf( 89 | LsmString *pStr, 90 | const char *zFormat, 91 | va_list ap1, 92 | va_list ap2 93 | ){ 94 | #if (!defined(__STDC_VERSION__) || (__STDC_VERSION__<199901L)) && \ 95 | !defined(__APPLE__) 96 | extern int vsnprintf(char *str, size_t size, const char *format, va_list ap) 97 | /* Compatibility crutch for C89 compilation mode. sqlite3_vsnprintf() 98 | does not work identically and causes test failures if used here. 99 | For the time being we are assuming that the target has vsnprintf(), 100 | but that is not guaranteed to be the case for pure C89 platforms. 101 | */; 102 | #endif 103 | int nWrite; 104 | int nAvail; 105 | 106 | nAvail = pStr->nAlloc - pStr->n; 107 | nWrite = vsnprintf(pStr->z + pStr->n, nAvail, zFormat, ap1); 108 | 109 | if( nWrite>=nAvail ){ 110 | lsmStringExtend(pStr, nWrite+1); 111 | if( pStr->nAlloc==0 ) return; 112 | nWrite = vsnprintf(pStr->z + pStr->n, nWrite+1, zFormat, ap2); 113 | } 114 | 115 | pStr->n += nWrite; 116 | pStr->z[pStr->n] = 0; 117 | } 118 | 119 | void lsmStringAppendf(LsmString *pStr, const char *zFormat, ...){ 120 | va_list ap, ap2; 121 | va_start(ap, zFormat); 122 | va_start(ap2, zFormat); 123 | lsmStringVAppendf(pStr, zFormat, ap, ap2); 124 | va_end(ap); 125 | va_end(ap2); 126 | } 127 | 128 | int lsmStrlen(const char *zName){ 129 | int nRet = 0; 130 | while( zName[nRet] ) nRet++; 131 | return nRet; 132 | } 133 | 134 | /* 135 | ** Write into memory obtained from lsm_malloc(). 136 | */ 137 | char *lsmMallocPrintf(lsm_env *pEnv, const char *zFormat, ...){ 138 | LsmString s; 139 | va_list ap, ap2; 140 | lsmStringInit(&s, pEnv); 141 | va_start(ap, zFormat); 142 | va_start(ap2, zFormat); 143 | lsmStringVAppendf(&s, zFormat, ap, ap2); 144 | va_end(ap); 145 | va_end(ap2); 146 | if( s.n<0 ) return 0; 147 | return (char *)lsmReallocOrFree(pEnv, s.z, s.n+1); 148 | } 149 | -------------------------------------------------------------------------------- /src/lsm_unix.c: -------------------------------------------------------------------------------- 1 | /* 2 | ** 2011-12-03 3 | ** 4 | ** The author disclaims copyright to this source code. In place of 5 | ** a legal notice, here is a blessing: 6 | ** 7 | ** May you do good and not evil. 8 | ** May you find forgiveness for yourself and forgive others. 9 | ** May you share freely, never taking more than you give. 10 | ** 11 | ************************************************************************* 12 | ** 13 | ** Unix-specific run-time environment implementation for LSM. 14 | */ 15 | 16 | #ifndef _WIN32 17 | 18 | #if defined(__GNUC__) || defined(__TINYC__) 19 | /* workaround for ftruncate() visibility on gcc. */ 20 | # ifndef _XOPEN_SOURCE 21 | # define _XOPEN_SOURCE 500 22 | # endif 23 | #endif 24 | 25 | #include 26 | #include 27 | 28 | #include 29 | #include 30 | #include 31 | #include 32 | 33 | #include 34 | #include 35 | #include 36 | #include 37 | 38 | #include 39 | #include 40 | 41 | #include 42 | #include "lsmInt.h" 43 | 44 | /* There is no fdatasync() call on Android */ 45 | #ifdef __ANDROID__ 46 | # define fdatasync(x) fsync(x) 47 | #endif 48 | 49 | /* 50 | ** An open file is an instance of the following object 51 | */ 52 | typedef struct PosixFile PosixFile; 53 | struct PosixFile { 54 | lsm_env *pEnv; /* The run-time environment */ 55 | const char *zName; /* Full path to file */ 56 | int fd; /* The open file descriptor */ 57 | int shmfd; /* Shared memory file-descriptor */ 58 | void *pMap; /* Pointer to mapping of file fd */ 59 | off_t nMap; /* Size of mapping at pMap in bytes */ 60 | int nShm; /* Number of entries in array apShm[] */ 61 | void **apShm; /* Array of 32K shared memory segments */ 62 | }; 63 | 64 | static char *posixShmFile(PosixFile *p){ 65 | char *zShm; 66 | int nName = strlen(p->zName); 67 | zShm = (char *)lsmMalloc(p->pEnv, nName+4+1); 68 | if( zShm ){ 69 | memcpy(zShm, p->zName, nName); 70 | memcpy(&zShm[nName], "-shm", 5); 71 | } 72 | return zShm; 73 | } 74 | 75 | static int lsmPosixOsOpen( 76 | lsm_env *pEnv, 77 | const char *zFile, 78 | int flags, 79 | lsm_file **ppFile 80 | ){ 81 | int rc = LSM_OK; 82 | PosixFile *p; 83 | 84 | p = lsm_malloc(pEnv, sizeof(PosixFile)); 85 | if( p==0 ){ 86 | rc = LSM_NOMEM; 87 | }else{ 88 | int bReadonly = (flags & LSM_OPEN_READONLY); 89 | int oflags = (bReadonly ? O_RDONLY : (O_RDWR|O_CREAT)); 90 | memset(p, 0, sizeof(PosixFile)); 91 | p->zName = zFile; 92 | p->pEnv = pEnv; 93 | p->fd = open(zFile, oflags, 0644); 94 | if( p->fd<0 ){ 95 | lsm_free(pEnv, p); 96 | p = 0; 97 | if( errno==ENOENT ){ 98 | rc = lsmErrorBkpt(LSM_IOERR_NOENT); 99 | }else{ 100 | rc = LSM_IOERR_BKPT; 101 | } 102 | } 103 | } 104 | 105 | *ppFile = (lsm_file *)p; 106 | return rc; 107 | } 108 | 109 | static int lsmPosixOsWrite( 110 | lsm_file *pFile, /* File to write to */ 111 | lsm_i64 iOff, /* Offset to write to */ 112 | void *pData, /* Write data from this buffer */ 113 | int nData /* Bytes of data to write */ 114 | ){ 115 | int rc = LSM_OK; 116 | PosixFile *p = (PosixFile *)pFile; 117 | off_t offset; 118 | 119 | offset = lseek(p->fd, (off_t)iOff, SEEK_SET); 120 | if( offset!=iOff ){ 121 | rc = LSM_IOERR_BKPT; 122 | }else{ 123 | ssize_t prc = write(p->fd, pData, (size_t)nData); 124 | if( prc<0 ) rc = LSM_IOERR_BKPT; 125 | } 126 | 127 | return rc; 128 | } 129 | 130 | static int lsmPosixOsTruncate( 131 | lsm_file *pFile, /* File to write to */ 132 | lsm_i64 nSize /* Size to truncate file to */ 133 | ){ 134 | PosixFile *p = (PosixFile *)pFile; 135 | int rc = LSM_OK; /* Return code */ 136 | int prc; /* Posix Return Code */ 137 | struct stat sStat; /* Result of fstat() invocation */ 138 | 139 | prc = fstat(p->fd, &sStat); 140 | if( prc==0 && sStat.st_size>nSize ){ 141 | prc = ftruncate(p->fd, (off_t)nSize); 142 | } 143 | if( prc<0 ) rc = LSM_IOERR_BKPT; 144 | 145 | return rc; 146 | } 147 | 148 | static int lsmPosixOsRead( 149 | lsm_file *pFile, /* File to read from */ 150 | lsm_i64 iOff, /* Offset to read from */ 151 | void *pData, /* Read data into this buffer */ 152 | int nData /* Bytes of data to read */ 153 | ){ 154 | int rc = LSM_OK; 155 | PosixFile *p = (PosixFile *)pFile; 156 | off_t offset; 157 | 158 | offset = lseek(p->fd, (off_t)iOff, SEEK_SET); 159 | if( offset!=iOff ){ 160 | rc = LSM_IOERR_BKPT; 161 | }else{ 162 | ssize_t prc = read(p->fd, pData, (size_t)nData); 163 | if( prc<0 ){ 164 | rc = LSM_IOERR_BKPT; 165 | }else if( prcpMap ){ 182 | prc = msync(p->pMap, p->nMap, MS_SYNC); 183 | } 184 | if( prc==0 ) prc = fdatasync(p->fd); 185 | if( prc<0 ) rc = LSM_IOERR_BKPT; 186 | #else 187 | (void)pFile; 188 | #endif 189 | 190 | return rc; 191 | } 192 | 193 | static int lsmPosixOsSectorSize(lsm_file *pFile){ 194 | return 512; 195 | } 196 | 197 | static int lsmPosixOsRemap( 198 | lsm_file *pFile, 199 | lsm_i64 iMin, 200 | void **ppOut, 201 | lsm_i64 *pnOut 202 | ){ 203 | off_t iSz; 204 | int prc; 205 | PosixFile *p = (PosixFile *)pFile; 206 | struct stat buf; 207 | 208 | /* If the file is between 0 and 2MB in size, extend it in chunks of 256K. 209 | ** Thereafter, in chunks of 1MB at a time. */ 210 | const int aIncrSz[] = {256*1024, 1024*1024}; 211 | int nIncrSz = aIncrSz[iMin>(2*1024*1024)]; 212 | 213 | if( p->pMap ){ 214 | munmap(p->pMap, p->nMap); 215 | *ppOut = p->pMap = 0; 216 | *pnOut = p->nMap = 0; 217 | } 218 | 219 | if( iMin>=0 ){ 220 | memset(&buf, 0, sizeof(buf)); 221 | prc = fstat(p->fd, &buf); 222 | if( prc!=0 ) return LSM_IOERR_BKPT; 223 | iSz = buf.st_size; 224 | if( iSzfd, iSz); 227 | if( prc!=0 ) return LSM_IOERR_BKPT; 228 | } 229 | 230 | p->pMap = mmap(0, iSz, PROT_READ|PROT_WRITE, MAP_SHARED, p->fd, 0); 231 | if( p->pMap==MAP_FAILED ){ 232 | p->pMap = 0; 233 | return LSM_IOERR_BKPT; 234 | } 235 | p->nMap = iSz; 236 | } 237 | 238 | *ppOut = p->pMap; 239 | *pnOut = p->nMap; 240 | return LSM_OK; 241 | } 242 | 243 | static int lsmPosixOsFullpath( 244 | lsm_env *pEnv, 245 | const char *zName, 246 | char *zOut, 247 | int *pnOut 248 | ){ 249 | int nBuf = *pnOut; 250 | int nReq; 251 | 252 | if( zName[0]!='/' ){ 253 | char *z; 254 | char *zTmp; 255 | int nTmp = 512; 256 | zTmp = lsmMalloc(pEnv, nTmp); 257 | while( zTmp ){ 258 | z = getcwd(zTmp, nTmp); 259 | if( z || errno!=ERANGE ) break; 260 | nTmp = nTmp*2; 261 | zTmp = lsmReallocOrFree(pEnv, zTmp, nTmp); 262 | } 263 | if( zTmp==0 ) return LSM_NOMEM_BKPT; 264 | if( z==0 ) return LSM_IOERR_BKPT; 265 | assert( z==zTmp ); 266 | 267 | nTmp = strlen(zTmp); 268 | nReq = nTmp + 1 + strlen(zName) + 1; 269 | if( nReq<=nBuf ){ 270 | memcpy(zOut, zTmp, nTmp); 271 | zOut[nTmp] = '/'; 272 | memcpy(&zOut[nTmp+1], zName, strlen(zName)+1); 273 | } 274 | lsmFree(pEnv, zTmp); 275 | }else{ 276 | nReq = strlen(zName)+1; 277 | if( nReq<=nBuf ){ 278 | memcpy(zOut, zName, strlen(zName)+1); 279 | } 280 | } 281 | 282 | *pnOut = nReq; 283 | return LSM_OK; 284 | } 285 | 286 | static int lsmPosixOsFileid( 287 | lsm_file *pFile, 288 | void *pBuf, 289 | int *pnBuf 290 | ){ 291 | int prc; 292 | int nBuf; 293 | int nReq; 294 | PosixFile *p = (PosixFile *)pFile; 295 | struct stat buf; 296 | 297 | nBuf = *pnBuf; 298 | nReq = (sizeof(buf.st_dev) + sizeof(buf.st_ino)); 299 | *pnBuf = nReq; 300 | if( nReq>nBuf ) return LSM_OK; 301 | 302 | memset(&buf, 0, sizeof(buf)); 303 | prc = fstat(p->fd, &buf); 304 | if( prc!=0 ) return LSM_IOERR_BKPT; 305 | 306 | memcpy(pBuf, &buf.st_dev, sizeof(buf.st_dev)); 307 | memcpy(&(((u8 *)pBuf)[sizeof(buf.st_dev)]), &buf.st_ino, sizeof(buf.st_ino)); 308 | return LSM_OK; 309 | } 310 | 311 | static int lsmPosixOsUnlink(lsm_env *pEnv, const char *zFile){ 312 | int prc = unlink(zFile); 313 | return prc ? LSM_IOERR_BKPT : LSM_OK; 314 | } 315 | 316 | static int lsmPosixOsLock(lsm_file *pFile, int iLock, int eType){ 317 | int rc = LSM_OK; 318 | PosixFile *p = (PosixFile *)pFile; 319 | static const short aType[3] = { F_UNLCK, F_RDLCK, F_WRLCK }; 320 | struct flock lock; 321 | 322 | assert( aType[LSM_LOCK_UNLOCK]==F_UNLCK ); 323 | assert( aType[LSM_LOCK_SHARED]==F_RDLCK ); 324 | assert( aType[LSM_LOCK_EXCL]==F_WRLCK ); 325 | assert( eType>=0 && eType0 && iLock<=32 ); 327 | 328 | memset(&lock, 0, sizeof(lock)); 329 | lock.l_whence = SEEK_SET; 330 | lock.l_len = 1; 331 | lock.l_type = aType[eType]; 332 | lock.l_start = (4096-iLock); 333 | 334 | if( fcntl(p->fd, F_SETLK, &lock) ){ 335 | int e = errno; 336 | if( e==EACCES || e==EAGAIN ){ 337 | rc = LSM_BUSY; 338 | }else{ 339 | rc = LSM_IOERR_BKPT; 340 | } 341 | } 342 | 343 | return rc; 344 | } 345 | 346 | static int lsmPosixOsTestLock(lsm_file *pFile, int iLock, int nLock, int eType){ 347 | int rc = LSM_OK; 348 | PosixFile *p = (PosixFile *)pFile; 349 | static const short aType[3] = { 0, F_RDLCK, F_WRLCK }; 350 | struct flock lock; 351 | 352 | assert( eType==LSM_LOCK_SHARED || eType==LSM_LOCK_EXCL ); 353 | assert( aType[LSM_LOCK_SHARED]==F_RDLCK ); 354 | assert( aType[LSM_LOCK_EXCL]==F_WRLCK ); 355 | assert( eType>=0 && eType0 && iLock<=32 ); 357 | 358 | memset(&lock, 0, sizeof(lock)); 359 | lock.l_whence = SEEK_SET; 360 | lock.l_len = nLock; 361 | lock.l_type = aType[eType]; 362 | lock.l_start = (4096-iLock-nLock+1); 363 | 364 | if( fcntl(p->fd, F_GETLK, &lock) ){ 365 | rc = LSM_IOERR_BKPT; 366 | }else if( lock.l_type!=F_UNLCK ){ 367 | rc = LSM_BUSY; 368 | } 369 | 370 | return rc; 371 | } 372 | 373 | static int lsmPosixOsShmMap(lsm_file *pFile, int iChunk, int sz, void **ppShm){ 374 | PosixFile *p = (PosixFile *)pFile; 375 | 376 | *ppShm = 0; 377 | assert( sz==LSM_SHM_CHUNK_SIZE ); 378 | if( iChunk>=p->nShm ){ 379 | int i; 380 | void **apNew; 381 | int nNew = iChunk+1; 382 | off_t nReq = nNew * LSM_SHM_CHUNK_SIZE; 383 | struct stat sStat; 384 | 385 | /* If the shared-memory file has not been opened, open it now. */ 386 | if( p->shmfd<=0 ){ 387 | char *zShm = posixShmFile(p); 388 | if( !zShm ) return LSM_NOMEM_BKPT; 389 | p->shmfd = open(zShm, O_RDWR|O_CREAT, 0644); 390 | lsmFree(p->pEnv, zShm); 391 | if( p->shmfd<0 ){ 392 | return LSM_IOERR_BKPT; 393 | } 394 | } 395 | 396 | /* If the shared-memory file is not large enough to contain the 397 | ** requested chunk, cause it to grow. */ 398 | if( fstat(p->shmfd, &sStat) ){ 399 | return LSM_IOERR_BKPT; 400 | } 401 | if( sStat.st_sizeshmfd, nReq) ){ 403 | return LSM_IOERR_BKPT; 404 | } 405 | } 406 | 407 | apNew = (void **)lsmRealloc(p->pEnv, p->apShm, sizeof(void *) * nNew); 408 | if( !apNew ) return LSM_NOMEM_BKPT; 409 | for(i=p->nShm; iapShm = apNew; 413 | p->nShm = nNew; 414 | } 415 | 416 | if( p->apShm[iChunk]==0 ){ 417 | p->apShm[iChunk] = mmap(0, LSM_SHM_CHUNK_SIZE, 418 | PROT_READ|PROT_WRITE, MAP_SHARED, p->shmfd, iChunk*LSM_SHM_CHUNK_SIZE 419 | ); 420 | if( p->apShm[iChunk]==MAP_FAILED ){ 421 | p->apShm[iChunk] = 0; 422 | return LSM_IOERR_BKPT; 423 | } 424 | } 425 | 426 | *ppShm = p->apShm[iChunk]; 427 | return LSM_OK; 428 | } 429 | 430 | static void lsmPosixOsShmBarrier(void){ 431 | } 432 | 433 | static int lsmPosixOsShmUnmap(lsm_file *pFile, int bDelete){ 434 | PosixFile *p = (PosixFile *)pFile; 435 | if( p->shmfd>0 ){ 436 | int i; 437 | for(i=0; inShm; i++){ 438 | if( p->apShm[i] ){ 439 | munmap(p->apShm[i], LSM_SHM_CHUNK_SIZE); 440 | p->apShm[i] = 0; 441 | } 442 | } 443 | close(p->shmfd); 444 | p->shmfd = 0; 445 | if( bDelete ){ 446 | char *zShm = posixShmFile(p); 447 | if( zShm ) unlink(zShm); 448 | lsmFree(p->pEnv, zShm); 449 | } 450 | } 451 | return LSM_OK; 452 | } 453 | 454 | 455 | static int lsmPosixOsClose(lsm_file *pFile){ 456 | PosixFile *p = (PosixFile *)pFile; 457 | lsmPosixOsShmUnmap(pFile, 0); 458 | if( p->pMap ) munmap(p->pMap, p->nMap); 459 | close(p->fd); 460 | lsm_free(p->pEnv, p->apShm); 461 | lsm_free(p->pEnv, p); 462 | return LSM_OK; 463 | } 464 | 465 | static int lsmPosixOsSleep(lsm_env *pEnv, int us){ 466 | #if 0 467 | /* Apparently on Android usleep() returns void */ 468 | if( usleep(us) ) return LSM_IOERR; 469 | #endif 470 | usleep(us); 471 | return LSM_OK; 472 | } 473 | 474 | /**************************************************************************** 475 | ** Memory allocation routines. 476 | */ 477 | #define BLOCK_HDR_SIZE ROUND8( sizeof(size_t) ) 478 | 479 | static void *lsmPosixOsMalloc(lsm_env *pEnv, size_t N){ 480 | unsigned char * m; 481 | N += BLOCK_HDR_SIZE; 482 | m = (unsigned char *)malloc(N); 483 | *((size_t*)m) = N; 484 | return m + BLOCK_HDR_SIZE; 485 | } 486 | 487 | static void lsmPosixOsFree(lsm_env *pEnv, void *p){ 488 | if(p){ 489 | free( ((unsigned char *)p) - BLOCK_HDR_SIZE ); 490 | } 491 | } 492 | 493 | static void *lsmPosixOsRealloc(lsm_env *pEnv, void *p, size_t N){ 494 | unsigned char * m = (unsigned char *)p; 495 | if(1>N){ 496 | lsmPosixOsFree( pEnv, p ); 497 | return NULL; 498 | }else if(NULL==p){ 499 | return lsmPosixOsMalloc(pEnv, N); 500 | }else{ 501 | void * re = NULL; 502 | m -= BLOCK_HDR_SIZE; 503 | #if 0 /* arguable: don't shrink */ 504 | size_t * sz = (size_t*)m; 505 | if(*sz >= (size_t)N){ 506 | return p; 507 | } 508 | #endif 509 | re = realloc( m, N + BLOCK_HDR_SIZE ); 510 | if(re){ 511 | m = (unsigned char *)re; 512 | *((size_t*)m) = N; 513 | return m + BLOCK_HDR_SIZE; 514 | }else{ 515 | return NULL; 516 | } 517 | } 518 | } 519 | 520 | static size_t lsmPosixOsMSize(lsm_env *pEnv, void *p){ 521 | unsigned char * m = (unsigned char *)p; 522 | return *((size_t*)(m-BLOCK_HDR_SIZE)); 523 | } 524 | #undef BLOCK_HDR_SIZE 525 | 526 | 527 | #ifdef LSM_MUTEX_PTHREADS 528 | /************************************************************************* 529 | ** Mutex methods for pthreads based systems. If LSM_MUTEX_PTHREADS is 530 | ** missing then a no-op implementation of mutexes found in lsm_mutex.c 531 | ** will be used instead. 532 | */ 533 | #include 534 | 535 | typedef struct PthreadMutex PthreadMutex; 536 | struct PthreadMutex { 537 | lsm_env *pEnv; 538 | pthread_mutex_t mutex; 539 | #ifdef LSM_DEBUG 540 | pthread_t owner; 541 | #endif 542 | }; 543 | 544 | #ifdef LSM_DEBUG 545 | # define LSM_PTHREAD_STATIC_MUTEX { 0, PTHREAD_MUTEX_INITIALIZER, 0 } 546 | #else 547 | # define LSM_PTHREAD_STATIC_MUTEX { 0, PTHREAD_MUTEX_INITIALIZER } 548 | #endif 549 | 550 | static int lsmPosixOsMutexStatic( 551 | lsm_env *pEnv, 552 | int iMutex, 553 | lsm_mutex **ppStatic 554 | ){ 555 | static PthreadMutex sMutex[2] = { 556 | LSM_PTHREAD_STATIC_MUTEX, 557 | LSM_PTHREAD_STATIC_MUTEX 558 | }; 559 | 560 | assert( iMutex==LSM_MUTEX_GLOBAL || iMutex==LSM_MUTEX_HEAP ); 561 | assert( LSM_MUTEX_GLOBAL==1 && LSM_MUTEX_HEAP==2 ); 562 | 563 | *ppStatic = (lsm_mutex *)&sMutex[iMutex-1]; 564 | return LSM_OK; 565 | } 566 | 567 | static int lsmPosixOsMutexNew(lsm_env *pEnv, lsm_mutex **ppNew){ 568 | PthreadMutex *pMutex; /* Pointer to new mutex */ 569 | pthread_mutexattr_t attr; /* Attributes object */ 570 | 571 | pMutex = (PthreadMutex *)lsmMallocZero(pEnv, sizeof(PthreadMutex)); 572 | if( !pMutex ) return LSM_NOMEM_BKPT; 573 | 574 | pMutex->pEnv = pEnv; 575 | pthread_mutexattr_init(&attr); 576 | pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE); 577 | pthread_mutex_init(&pMutex->mutex, &attr); 578 | pthread_mutexattr_destroy(&attr); 579 | 580 | *ppNew = (lsm_mutex *)pMutex; 581 | return LSM_OK; 582 | } 583 | 584 | static void lsmPosixOsMutexDel(lsm_mutex *p){ 585 | PthreadMutex *pMutex = (PthreadMutex *)p; 586 | pthread_mutex_destroy(&pMutex->mutex); 587 | lsmFree(pMutex->pEnv, pMutex); 588 | } 589 | 590 | static void lsmPosixOsMutexEnter(lsm_mutex *p){ 591 | PthreadMutex *pMutex = (PthreadMutex *)p; 592 | pthread_mutex_lock(&pMutex->mutex); 593 | 594 | #ifdef LSM_DEBUG 595 | assert( !pthread_equal(pMutex->owner, pthread_self()) ); 596 | pMutex->owner = pthread_self(); 597 | assert( pthread_equal(pMutex->owner, pthread_self()) ); 598 | #endif 599 | } 600 | 601 | static int lsmPosixOsMutexTry(lsm_mutex *p){ 602 | int ret; 603 | PthreadMutex *pMutex = (PthreadMutex *)p; 604 | ret = pthread_mutex_trylock(&pMutex->mutex); 605 | #ifdef LSM_DEBUG 606 | if( ret==0 ){ 607 | assert( !pthread_equal(pMutex->owner, pthread_self()) ); 608 | pMutex->owner = pthread_self(); 609 | assert( pthread_equal(pMutex->owner, pthread_self()) ); 610 | } 611 | #endif 612 | return ret; 613 | } 614 | 615 | static void lsmPosixOsMutexLeave(lsm_mutex *p){ 616 | PthreadMutex *pMutex = (PthreadMutex *)p; 617 | #ifdef LSM_DEBUG 618 | assert( pthread_equal(pMutex->owner, pthread_self()) ); 619 | pMutex->owner = 0; 620 | assert( !pthread_equal(pMutex->owner, pthread_self()) ); 621 | #endif 622 | pthread_mutex_unlock(&pMutex->mutex); 623 | } 624 | 625 | #ifdef LSM_DEBUG 626 | static int lsmPosixOsMutexHeld(lsm_mutex *p){ 627 | PthreadMutex *pMutex = (PthreadMutex *)p; 628 | return pMutex ? pthread_equal(pMutex->owner, pthread_self()) : 1; 629 | } 630 | static int lsmPosixOsMutexNotHeld(lsm_mutex *p){ 631 | PthreadMutex *pMutex = (PthreadMutex *)p; 632 | return pMutex ? !pthread_equal(pMutex->owner, pthread_self()) : 1; 633 | } 634 | #endif 635 | /* 636 | ** End of pthreads mutex implementation. 637 | *************************************************************************/ 638 | #else 639 | /************************************************************************* 640 | ** Noop mutex implementation 641 | */ 642 | typedef struct NoopMutex NoopMutex; 643 | struct NoopMutex { 644 | lsm_env *pEnv; /* Environment handle (for xFree()) */ 645 | int bHeld; /* True if mutex is held */ 646 | int bStatic; /* True for a static mutex */ 647 | }; 648 | static NoopMutex aStaticNoopMutex[2] = { 649 | {0, 0, 1}, 650 | {0, 0, 1}, 651 | }; 652 | 653 | static int lsmPosixOsMutexStatic( 654 | lsm_env *pEnv, 655 | int iMutex, 656 | lsm_mutex **ppStatic 657 | ){ 658 | assert( iMutex>=1 && iMutex<=(int)array_size(aStaticNoopMutex) ); 659 | *ppStatic = (lsm_mutex *)&aStaticNoopMutex[iMutex-1]; 660 | return LSM_OK; 661 | } 662 | static int lsmPosixOsMutexNew(lsm_env *pEnv, lsm_mutex **ppNew){ 663 | NoopMutex *p; 664 | p = (NoopMutex *)lsmMallocZero(pEnv, sizeof(NoopMutex)); 665 | if( p ) p->pEnv = pEnv; 666 | *ppNew = (lsm_mutex *)p; 667 | return (p ? LSM_OK : LSM_NOMEM_BKPT); 668 | } 669 | static void lsmPosixOsMutexDel(lsm_mutex *pMutex) { 670 | NoopMutex *p = (NoopMutex *)pMutex; 671 | assert( p->bStatic==0 && p->pEnv ); 672 | lsmFree(p->pEnv, p); 673 | } 674 | static void lsmPosixOsMutexEnter(lsm_mutex *pMutex){ 675 | NoopMutex *p = (NoopMutex *)pMutex; 676 | assert( p->bHeld==0 ); 677 | p->bHeld = 1; 678 | } 679 | static int lsmPosixOsMutexTry(lsm_mutex *pMutex){ 680 | NoopMutex *p = (NoopMutex *)pMutex; 681 | assert( p->bHeld==0 ); 682 | p->bHeld = 1; 683 | return 0; 684 | } 685 | static void lsmPosixOsMutexLeave(lsm_mutex *pMutex){ 686 | NoopMutex *p = (NoopMutex *)pMutex; 687 | assert( p->bHeld==1 ); 688 | p->bHeld = 0; 689 | } 690 | #ifdef LSM_DEBUG 691 | static int lsmPosixOsMutexHeld(lsm_mutex *pMutex){ 692 | NoopMutex *p = (NoopMutex *)pMutex; 693 | return p ? p->bHeld : 1; 694 | } 695 | static int lsmPosixOsMutexNotHeld(lsm_mutex *pMutex){ 696 | NoopMutex *p = (NoopMutex *)pMutex; 697 | return p ? !p->bHeld : 1; 698 | } 699 | #endif 700 | /***************************************************************************/ 701 | #endif /* else LSM_MUTEX_NONE */ 702 | 703 | /* Without LSM_DEBUG, the MutexHeld tests are never called */ 704 | #ifndef LSM_DEBUG 705 | # define lsmPosixOsMutexHeld 0 706 | # define lsmPosixOsMutexNotHeld 0 707 | #endif 708 | 709 | lsm_env *lsm_default_env(void){ 710 | static lsm_env posix_env = { 711 | sizeof(lsm_env), /* nByte */ 712 | 1, /* iVersion */ 713 | /***** file i/o ******************/ 714 | 0, /* pVfsCtx */ 715 | lsmPosixOsFullpath, /* xFullpath */ 716 | lsmPosixOsOpen, /* xOpen */ 717 | lsmPosixOsRead, /* xRead */ 718 | lsmPosixOsWrite, /* xWrite */ 719 | lsmPosixOsTruncate, /* xTruncate */ 720 | lsmPosixOsSync, /* xSync */ 721 | lsmPosixOsSectorSize, /* xSectorSize */ 722 | lsmPosixOsRemap, /* xRemap */ 723 | lsmPosixOsFileid, /* xFileid */ 724 | lsmPosixOsClose, /* xClose */ 725 | lsmPosixOsUnlink, /* xUnlink */ 726 | lsmPosixOsLock, /* xLock */ 727 | lsmPosixOsTestLock, /* xTestLock */ 728 | lsmPosixOsShmMap, /* xShmMap */ 729 | lsmPosixOsShmBarrier, /* xShmBarrier */ 730 | lsmPosixOsShmUnmap, /* xShmUnmap */ 731 | /***** memory allocation *********/ 732 | 0, /* pMemCtx */ 733 | lsmPosixOsMalloc, /* xMalloc */ 734 | lsmPosixOsRealloc, /* xRealloc */ 735 | lsmPosixOsFree, /* xFree */ 736 | lsmPosixOsMSize, /* xSize */ 737 | /***** mutexes *********************/ 738 | 0, /* pMutexCtx */ 739 | lsmPosixOsMutexStatic, /* xMutexStatic */ 740 | lsmPosixOsMutexNew, /* xMutexNew */ 741 | lsmPosixOsMutexDel, /* xMutexDel */ 742 | lsmPosixOsMutexEnter, /* xMutexEnter */ 743 | lsmPosixOsMutexTry, /* xMutexTry */ 744 | lsmPosixOsMutexLeave, /* xMutexLeave */ 745 | lsmPosixOsMutexHeld, /* xMutexHeld */ 746 | lsmPosixOsMutexNotHeld, /* xMutexNotHeld */ 747 | /***** other *********************/ 748 | lsmPosixOsSleep, /* xSleep */ 749 | }; 750 | return &posix_env; 751 | } 752 | 753 | #endif 754 | -------------------------------------------------------------------------------- /src/lsm_varint.c: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | ** 2012-02-08 4 | ** 5 | ** The author disclaims copyright to this source code. In place of 6 | ** a legal notice, here is a blessing: 7 | ** 8 | ** May you do good and not evil. 9 | ** May you find forgiveness for yourself and forgive others. 10 | ** May you share freely, never taking more than you give. 11 | ** 12 | ************************************************************************* 13 | ** 14 | ** SQLite4-compatible varint implementation. 15 | */ 16 | #include "lsmInt.h" 17 | 18 | /************************************************************************* 19 | ** The following is a copy of the varint.c module from SQLite 4. 20 | */ 21 | 22 | /* 23 | ** Decode the varint in z[]. Write the integer value into *pResult and 24 | ** return the number of bytes in the varint. 25 | */ 26 | static int lsmSqlite4GetVarint64(const unsigned char *z, u64 *pResult){ 27 | unsigned int x; 28 | if( z[0]<=240 ){ 29 | *pResult = z[0]; 30 | return 1; 31 | } 32 | if( z[0]<=248 ){ 33 | *pResult = (z[0]-241)*256 + z[1] + 240; 34 | return 2; 35 | } 36 | if( z[0]==249 ){ 37 | *pResult = 2288 + 256*z[1] + z[2]; 38 | return 3; 39 | } 40 | if( z[0]==250 ){ 41 | *pResult = (z[1]<<16) + (z[2]<<8) + z[3]; 42 | return 4; 43 | } 44 | x = (z[1]<<24) + (z[2]<<16) + (z[3]<<8) + z[4]; 45 | if( z[0]==251 ){ 46 | *pResult = x; 47 | return 5; 48 | } 49 | if( z[0]==252 ){ 50 | *pResult = (((u64)x)<<8) + z[5]; 51 | return 6; 52 | } 53 | if( z[0]==253 ){ 54 | *pResult = (((u64)x)<<16) + (z[5]<<8) + z[6]; 55 | return 7; 56 | } 57 | if( z[0]==254 ){ 58 | *pResult = (((u64)x)<<24) + (z[5]<<16) + (z[6]<<8) + z[7]; 59 | return 8; 60 | } 61 | *pResult = (((u64)x)<<32) + 62 | (0xffffffff & ((z[5]<<24) + (z[6]<<16) + (z[7]<<8) + z[8])); 63 | return 9; 64 | } 65 | 66 | /* 67 | ** Write a 32-bit unsigned integer as 4 big-endian bytes. 68 | */ 69 | static void lsmVarintWrite32(unsigned char *z, unsigned int y){ 70 | z[0] = (unsigned char)(y>>24); 71 | z[1] = (unsigned char)(y>>16); 72 | z[2] = (unsigned char)(y>>8); 73 | z[3] = (unsigned char)(y); 74 | } 75 | 76 | /* 77 | ** Write a varint into z[]. The buffer z[] must be at least 9 characters 78 | ** long to accommodate the largest possible varint. Return the number of 79 | ** bytes of z[] used. 80 | */ 81 | static int lsmSqlite4PutVarint64(unsigned char *z, u64 x){ 82 | unsigned int w, y; 83 | if( x<=240 ){ 84 | z[0] = (unsigned char)x; 85 | return 1; 86 | } 87 | if( x<=2287 ){ 88 | y = (unsigned int)(x - 240); 89 | z[0] = (unsigned char)(y/256 + 241); 90 | z[1] = (unsigned char)(y%256); 91 | return 2; 92 | } 93 | if( x<=67823 ){ 94 | y = (unsigned int)(x - 2288); 95 | z[0] = 249; 96 | z[1] = (unsigned char)(y/256); 97 | z[2] = (unsigned char)(y%256); 98 | return 3; 99 | } 100 | y = (unsigned int)x; 101 | w = (unsigned int)(x>>32); 102 | if( w==0 ){ 103 | if( y<=16777215 ){ 104 | z[0] = 250; 105 | z[1] = (unsigned char)(y>>16); 106 | z[2] = (unsigned char)(y>>8); 107 | z[3] = (unsigned char)(y); 108 | return 4; 109 | } 110 | z[0] = 251; 111 | lsmVarintWrite32(z+1, y); 112 | return 5; 113 | } 114 | if( w<=255 ){ 115 | z[0] = 252; 116 | z[1] = (unsigned char)w; 117 | lsmVarintWrite32(z+2, y); 118 | return 6; 119 | } 120 | if( w<=32767 ){ 121 | z[0] = 253; 122 | z[1] = (unsigned char)(w>>8); 123 | z[2] = (unsigned char)w; 124 | lsmVarintWrite32(z+3, y); 125 | return 7; 126 | } 127 | if( w<=16777215 ){ 128 | z[0] = 254; 129 | z[1] = (unsigned char)(w>>16); 130 | z[2] = (unsigned char)(w>>8); 131 | z[3] = (unsigned char)w; 132 | lsmVarintWrite32(z+4, y); 133 | return 8; 134 | } 135 | z[0] = 255; 136 | lsmVarintWrite32(z+1, w); 137 | lsmVarintWrite32(z+5, y); 138 | return 9; 139 | } 140 | 141 | /* 142 | ** End of SQLite 4 code. 143 | *************************************************************************/ 144 | 145 | int lsmVarintPut64(u8 *aData, i64 iVal){ 146 | return lsmSqlite4PutVarint64(aData, (u64)iVal); 147 | } 148 | 149 | int lsmVarintGet64(const u8 *aData, i64 *piVal){ 150 | return lsmSqlite4GetVarint64(aData, (u64 *)piVal); 151 | } 152 | 153 | int lsmVarintPut32(u8 *aData, int iVal){ 154 | return lsmSqlite4PutVarint64(aData, (u64)iVal); 155 | } 156 | 157 | int lsmVarintGet32(u8 *z, int *piVal){ 158 | u64 i; 159 | int ret; 160 | 161 | if( z[0]<=240 ){ 162 | *piVal = z[0]; 163 | return 1; 164 | } 165 | if( z[0]<=248 ){ 166 | *piVal = (z[0]-241)*256 + z[1] + 240; 167 | return 2; 168 | } 169 | if( z[0]==249 ){ 170 | *piVal = 2288 + 256*z[1] + z[2]; 171 | return 3; 172 | } 173 | if( z[0]==250 ){ 174 | *piVal = (z[1]<<16) + (z[2]<<8) + z[3]; 175 | return 4; 176 | } 177 | 178 | ret = lsmSqlite4GetVarint64(z, &i); 179 | *piVal = (int)i; 180 | return ret; 181 | } 182 | 183 | int lsmVarintLen32(int n){ 184 | u8 aData[9]; 185 | return lsmVarintPut32(aData, n); 186 | } 187 | 188 | int lsmVarintLen64(i64 n){ 189 | u8 aData[9]; 190 | return lsmVarintPut64(aData, n); 191 | } 192 | 193 | /* 194 | ** The argument is the first byte of a varint. This function returns the 195 | ** total number of bytes in the entire varint (including the first byte). 196 | */ 197 | int lsmVarintSize(u8 c){ 198 | if( c<241 ) return 1; 199 | if( c<249 ) return 2; 200 | return (int)(c - 246); 201 | } 202 | -------------------------------------------------------------------------------- /src/lsm_win32.c: -------------------------------------------------------------------------------- 1 | /* 2 | ** 2011-12-03 3 | ** 4 | ** The author disclaims copyright to this source code. In place of 5 | ** a legal notice, here is a blessing: 6 | ** 7 | ** May you do good and not evil. 8 | ** May you find forgiveness for yourself and forgive others. 9 | ** May you share freely, never taking more than you give. 10 | ** 11 | ************************************************************************* 12 | ** 13 | ** Win32-specific run-time environment implementation for LSM. 14 | */ 15 | 16 | #ifdef _WIN32 17 | 18 | #include 19 | #include 20 | 21 | #include 22 | #include 23 | #include 24 | #include 25 | 26 | #include "windows.h" 27 | 28 | #include "lsmInt.h" 29 | 30 | /* 31 | ** An open file is an instance of the following object 32 | */ 33 | typedef struct Win32File Win32File; 34 | struct Win32File { 35 | lsm_env *pEnv; /* The run-time environment */ 36 | const char *zName; /* Full path to file */ 37 | 38 | HANDLE hFile; /* Open file handle */ 39 | HANDLE hShmFile; /* File handle for *-shm file */ 40 | 41 | SYSTEM_INFO sysInfo; /* Operating system information */ 42 | HANDLE hMap; /* File handle for mapping */ 43 | LPVOID pMap; /* Pointer to mapping of file fd */ 44 | size_t nMap; /* Size of mapping at pMap in bytes */ 45 | int nShm; /* Number of entries in ahShm[]/apShm[] */ 46 | LPHANDLE ahShm; /* Array of handles for shared mappings */ 47 | LPVOID *apShm; /* Array of 32K shared memory segments */ 48 | }; 49 | 50 | static char *win32ShmFile(Win32File *pWin32File){ 51 | char *zShm; 52 | int nName = strlen(pWin32File->zName); 53 | zShm = (char *)lsmMallocZero(pWin32File->pEnv, nName+4+1); 54 | if( zShm ){ 55 | memcpy(zShm, pWin32File->zName, nName); 56 | memcpy(&zShm[nName], "-shm", 5); 57 | } 58 | return zShm; 59 | } 60 | 61 | static int win32Sleep(int us){ 62 | Sleep((us + 999) / 1000); 63 | return LSM_OK; 64 | } 65 | 66 | /* 67 | ** The number of times that an I/O operation will be retried following a 68 | ** locking error - probably caused by antivirus software. Also the initial 69 | ** delay before the first retry. The delay increases linearly with each 70 | ** retry. 71 | */ 72 | #ifndef LSM_WIN32_IOERR_RETRY 73 | # define LSM_WIN32_IOERR_RETRY 10 74 | #endif 75 | #ifndef LSM_WIN32_IOERR_RETRY_DELAY 76 | # define LSM_WIN32_IOERR_RETRY_DELAY 25000 77 | #endif 78 | static int win32IoerrRetry = LSM_WIN32_IOERR_RETRY; 79 | static int win32IoerrRetryDelay = LSM_WIN32_IOERR_RETRY_DELAY; 80 | 81 | /* 82 | ** The "win32IoerrCanRetry1" macro is used to determine if a particular 83 | ** I/O error code obtained via GetLastError() is eligible to be retried. 84 | ** It must accept the error code DWORD as its only argument and should 85 | ** return non-zero if the error code is transient in nature and the 86 | ** operation responsible for generating the original error might succeed 87 | ** upon being retried. The argument to this macro should be a variable. 88 | ** 89 | ** Additionally, a macro named "win32IoerrCanRetry2" may be defined. If 90 | ** it is defined, it will be consulted only when the macro 91 | ** "win32IoerrCanRetry1" returns zero. The "win32IoerrCanRetry2" macro 92 | ** is completely optional and may be used to include additional error 93 | ** codes in the set that should result in the failing I/O operation being 94 | ** retried by the caller. If defined, the "win32IoerrCanRetry2" macro 95 | ** must exhibit external semantics identical to those of the 96 | ** "win32IoerrCanRetry1" macro. 97 | */ 98 | #if !defined(win32IoerrCanRetry1) 99 | #define win32IoerrCanRetry1(a) (((a)==ERROR_ACCESS_DENIED) || \ 100 | ((a)==ERROR_SHARING_VIOLATION) || \ 101 | ((a)==ERROR_LOCK_VIOLATION) || \ 102 | ((a)==ERROR_DEV_NOT_EXIST) || \ 103 | ((a)==ERROR_NETNAME_DELETED) || \ 104 | ((a)==ERROR_SEM_TIMEOUT) || \ 105 | ((a)==ERROR_NETWORK_UNREACHABLE)) 106 | #endif 107 | 108 | /* 109 | ** If an I/O error occurs, invoke this routine to see if it should be 110 | ** retried. Return TRUE to retry. Return FALSE to give up with an 111 | ** error. 112 | */ 113 | static int win32RetryIoerr( 114 | lsm_env *pEnv, 115 | int *pnRetry 116 | ){ 117 | DWORD lastErrno; 118 | if( *pnRetry>=win32IoerrRetry ){ 119 | return 0; 120 | } 121 | lastErrno = GetLastError(); 122 | if( win32IoerrCanRetry1(lastErrno) ){ 123 | win32Sleep(win32IoerrRetryDelay*(1+*pnRetry)); 124 | ++*pnRetry; 125 | return 1; 126 | } 127 | #if defined(win32IoerrCanRetry2) 128 | else if( win32IoerrCanRetry2(lastErrno) ){ 129 | win32Sleep(win32IoerrRetryDelay*(1+*pnRetry)); 130 | ++*pnRetry; 131 | return 1; 132 | } 133 | #endif 134 | return 0; 135 | } 136 | 137 | /* 138 | ** Convert a UTF-8 string to Microsoft Unicode. 139 | ** 140 | ** Space to hold the returned string is obtained from lsmMalloc(). 141 | */ 142 | static LPWSTR win32Utf8ToUnicode(lsm_env *pEnv, const char *zText){ 143 | int nChar; 144 | LPWSTR zWideText; 145 | 146 | nChar = MultiByteToWideChar(CP_UTF8, 0, zText, -1, NULL, 0); 147 | if( nChar==0 ){ 148 | return 0; 149 | } 150 | zWideText = lsmMallocZero(pEnv, nChar * sizeof(WCHAR)); 151 | if( zWideText==0 ){ 152 | return 0; 153 | } 154 | nChar = MultiByteToWideChar(CP_UTF8, 0, zText, -1, zWideText, nChar); 155 | if( nChar==0 ){ 156 | lsmFree(pEnv, zWideText); 157 | zWideText = 0; 158 | } 159 | return zWideText; 160 | } 161 | 162 | /* 163 | ** Convert a Microsoft Unicode string to UTF-8. 164 | ** 165 | ** Space to hold the returned string is obtained from lsmMalloc(). 166 | */ 167 | static char *win32UnicodeToUtf8(lsm_env *pEnv, LPCWSTR zWideText){ 168 | int nByte; 169 | char *zText; 170 | 171 | nByte = WideCharToMultiByte(CP_UTF8, 0, zWideText, -1, 0, 0, 0, 0); 172 | if( nByte == 0 ){ 173 | return 0; 174 | } 175 | zText = lsmMallocZero(pEnv, nByte); 176 | if( zText==0 ){ 177 | return 0; 178 | } 179 | nByte = WideCharToMultiByte(CP_UTF8, 0, zWideText, -1, zText, nByte, 0, 0); 180 | if( nByte == 0 ){ 181 | lsmFree(pEnv, zText); 182 | zText = 0; 183 | } 184 | return zText; 185 | } 186 | 187 | #if !defined(win32IsNotFound) 188 | #define win32IsNotFound(a) (((a)==ERROR_FILE_NOT_FOUND) || \ 189 | ((a)==ERROR_PATH_NOT_FOUND)) 190 | #endif 191 | 192 | static int win32Open( 193 | lsm_env *pEnv, 194 | const char *zFile, 195 | int flags, 196 | LPHANDLE phFile 197 | ){ 198 | int rc; 199 | LPWSTR zConverted; 200 | 201 | zConverted = win32Utf8ToUnicode(pEnv, zFile); 202 | if( zConverted==0 ){ 203 | rc = LSM_NOMEM_BKPT; 204 | }else{ 205 | int bReadonly = (flags & LSM_OPEN_READONLY); 206 | DWORD dwDesiredAccess; 207 | DWORD dwShareMode = FILE_SHARE_READ | FILE_SHARE_WRITE; 208 | DWORD dwCreationDisposition; 209 | DWORD dwFlagsAndAttributes = FILE_ATTRIBUTE_NORMAL; 210 | HANDLE hFile; 211 | int nRetry = 0; 212 | if( bReadonly ){ 213 | dwDesiredAccess = GENERIC_READ; 214 | dwCreationDisposition = OPEN_EXISTING; 215 | }else{ 216 | dwDesiredAccess = GENERIC_READ | GENERIC_WRITE; 217 | dwCreationDisposition = OPEN_ALWAYS; 218 | } 219 | while( (hFile = CreateFileW((LPCWSTR)zConverted, 220 | dwDesiredAccess, 221 | dwShareMode, NULL, 222 | dwCreationDisposition, 223 | dwFlagsAndAttributes, 224 | NULL))==INVALID_HANDLE_VALUE && 225 | win32RetryIoerr(pEnv, &nRetry) ){ 226 | /* Noop */ 227 | } 228 | lsmFree(pEnv, zConverted); 229 | if( hFile!=INVALID_HANDLE_VALUE ){ 230 | *phFile = hFile; 231 | rc = LSM_OK; 232 | }else{ 233 | if( win32IsNotFound(GetLastError()) ){ 234 | rc = lsmErrorBkpt(LSM_IOERR_NOENT); 235 | }else{ 236 | rc = LSM_IOERR_BKPT; 237 | } 238 | } 239 | } 240 | return rc; 241 | } 242 | 243 | static int lsmWin32OsOpen( 244 | lsm_env *pEnv, 245 | const char *zFile, 246 | int flags, 247 | lsm_file **ppFile 248 | ){ 249 | int rc = LSM_OK; 250 | Win32File *pWin32File; 251 | 252 | pWin32File = lsmMallocZero(pEnv, sizeof(Win32File)); 253 | if( pWin32File==0 ){ 254 | rc = LSM_NOMEM_BKPT; 255 | }else{ 256 | HANDLE hFile = NULL; 257 | 258 | rc = win32Open(pEnv, zFile, flags, &hFile); 259 | if( rc==LSM_OK ){ 260 | memset(&pWin32File->sysInfo, 0, sizeof(SYSTEM_INFO)); 261 | GetSystemInfo(&pWin32File->sysInfo); 262 | pWin32File->pEnv = pEnv; 263 | pWin32File->zName = zFile; 264 | pWin32File->hFile = hFile; 265 | }else{ 266 | lsmFree(pEnv, pWin32File); 267 | pWin32File = 0; 268 | } 269 | } 270 | *ppFile = (lsm_file *)pWin32File; 271 | return rc; 272 | } 273 | 274 | static int lsmWin32OsWrite( 275 | lsm_file *pFile, /* File to write to */ 276 | lsm_i64 iOff, /* Offset to write to */ 277 | void *pData, /* Write data from this buffer */ 278 | int nData /* Bytes of data to write */ 279 | ){ 280 | Win32File *pWin32File = (Win32File *)pFile; 281 | OVERLAPPED overlapped; /* The offset for WriteFile. */ 282 | u8 *aRem = (u8 *)pData; /* Data yet to be written */ 283 | int nRem = nData; /* Number of bytes yet to be written */ 284 | int nRetry = 0; /* Number of retrys */ 285 | 286 | memset(&overlapped, 0, sizeof(OVERLAPPED)); 287 | overlapped.Offset = (LONG)(iOff & 0XFFFFFFFF); 288 | overlapped.OffsetHigh = (LONG)((iOff>>32) & 0x7FFFFFFF); 289 | while( nRem>0 ){ 290 | DWORD nWrite = 0; /* Bytes written using WriteFile */ 291 | if( !WriteFile(pWin32File->hFile, aRem, nRem, &nWrite, &overlapped) ){ 292 | if( win32RetryIoerr(pWin32File->pEnv, &nRetry) ) continue; 293 | break; 294 | } 295 | assert( nWrite==0 || nWrite<=(DWORD)nRem ); 296 | if( nWrite==0 || nWrite>(DWORD)nRem ){ 297 | break; 298 | } 299 | iOff += nWrite; 300 | overlapped.Offset = (LONG)(iOff & 0xFFFFFFFF); 301 | overlapped.OffsetHigh = (LONG)((iOff>>32) & 0x7FFFFFFF); 302 | aRem += nWrite; 303 | nRem -= nWrite; 304 | } 305 | if( nRem!=0 ) return LSM_IOERR_BKPT; 306 | return LSM_OK; 307 | } 308 | 309 | static int win32Truncate( 310 | HANDLE hFile, 311 | lsm_i64 nSize 312 | ){ 313 | LARGE_INTEGER offset; 314 | offset.QuadPart = nSize; 315 | if( !SetFilePointerEx(hFile, offset, 0, FILE_BEGIN) ){ 316 | return LSM_IOERR_BKPT; 317 | } 318 | if (!SetEndOfFile(hFile) ){ 319 | return LSM_IOERR_BKPT; 320 | } 321 | return LSM_OK; 322 | } 323 | 324 | static int lsmWin32OsTruncate( 325 | lsm_file *pFile, /* File to write to */ 326 | lsm_i64 nSize /* Size to truncate file to */ 327 | ){ 328 | Win32File *pWin32File = (Win32File *)pFile; 329 | return win32Truncate(pWin32File->hFile, nSize); 330 | } 331 | 332 | static int lsmWin32OsRead( 333 | lsm_file *pFile, /* File to read from */ 334 | lsm_i64 iOff, /* Offset to read from */ 335 | void *pData, /* Read data into this buffer */ 336 | int nData /* Bytes of data to read */ 337 | ){ 338 | Win32File *pWin32File = (Win32File *)pFile; 339 | OVERLAPPED overlapped; /* The offset for ReadFile */ 340 | DWORD nRead = 0; /* Bytes read using ReadFile */ 341 | int nRetry = 0; /* Number of retrys */ 342 | 343 | memset(&overlapped, 0, sizeof(OVERLAPPED)); 344 | overlapped.Offset = (LONG)(iOff & 0XFFFFFFFF); 345 | overlapped.OffsetHigh = (LONG)((iOff>>32) & 0X7FFFFFFF); 346 | while( !ReadFile(pWin32File->hFile, pData, nData, &nRead, &overlapped) && 347 | GetLastError()!=ERROR_HANDLE_EOF ){ 348 | if( win32RetryIoerr(pWin32File->pEnv, &nRetry) ) continue; 349 | return LSM_IOERR_BKPT; 350 | } 351 | if( nRead<(DWORD)nData ){ 352 | /* Unread parts of the buffer must be zero-filled */ 353 | memset(&((char*)pData)[nRead], 0, nData - nRead); 354 | } 355 | return LSM_OK; 356 | } 357 | 358 | static int lsmWin32OsSync(lsm_file *pFile){ 359 | int rc = LSM_OK; 360 | 361 | #ifndef LSM_NO_SYNC 362 | Win32File *pWin32File = (Win32File *)pFile; 363 | 364 | if( pWin32File->pMap!=NULL ){ 365 | if( !FlushViewOfFile(pWin32File->pMap, 0) ){ 366 | rc = LSM_IOERR_BKPT; 367 | } 368 | } 369 | if( rc==LSM_OK && !FlushFileBuffers(pWin32File->hFile) ){ 370 | rc = LSM_IOERR_BKPT; 371 | } 372 | #else 373 | unused_parameter(pFile); 374 | #endif 375 | 376 | return rc; 377 | } 378 | 379 | static int lsmWin32OsSectorSize(lsm_file *pFile){ 380 | return 512; 381 | } 382 | 383 | static void win32Unmap(Win32File *pWin32File){ 384 | if( pWin32File->pMap!=NULL ){ 385 | UnmapViewOfFile(pWin32File->pMap); 386 | pWin32File->pMap = NULL; 387 | pWin32File->nMap = 0; 388 | } 389 | if( pWin32File->hMap!=NULL ){ 390 | CloseHandle(pWin32File->hMap); 391 | pWin32File->hMap = NULL; 392 | } 393 | } 394 | 395 | static int lsmWin32OsRemap( 396 | lsm_file *pFile, 397 | lsm_i64 iMin, 398 | void **ppOut, 399 | lsm_i64 *pnOut 400 | ){ 401 | Win32File *pWin32File = (Win32File *)pFile; 402 | 403 | /* If the file is between 0 and 2MB in size, extend it in chunks of 256K. 404 | ** Thereafter, in chunks of 1MB at a time. */ 405 | const int aIncrSz[] = {256*1024, 1024*1024}; 406 | int nIncrSz = aIncrSz[iMin>(2*1024*1024)]; 407 | 408 | *ppOut = NULL; 409 | *pnOut = 0; 410 | 411 | win32Unmap(pWin32File); 412 | if( iMin>=0 ){ 413 | LARGE_INTEGER fileSize; 414 | DWORD dwSizeHigh; 415 | DWORD dwSizeLow; 416 | HANDLE hMap; 417 | LPVOID pMap; 418 | memset(&fileSize, 0, sizeof(LARGE_INTEGER)); 419 | if( !GetFileSizeEx(pWin32File->hFile, &fileSize) ){ 420 | return LSM_IOERR_BKPT; 421 | } 422 | assert( fileSize.QuadPart>=0 ); 423 | if( fileSize.QuadPart> 32); 433 | hMap = CreateFileMappingW(pWin32File->hFile, NULL, PAGE_READWRITE, 434 | dwSizeHigh, dwSizeLow, NULL); 435 | if( hMap==NULL ){ 436 | return LSM_IOERR_BKPT; 437 | } 438 | pWin32File->hMap = hMap; 439 | assert( fileSize.QuadPart<=0xFFFFFFFF ); 440 | pMap = MapViewOfFile(hMap, FILE_MAP_WRITE | FILE_MAP_READ, 0, 0, 441 | (SIZE_T)fileSize.QuadPart); 442 | if( pMap==NULL ){ 443 | return LSM_IOERR_BKPT; 444 | } 445 | pWin32File->pMap = pMap; 446 | pWin32File->nMap = (SIZE_T)fileSize.QuadPart; 447 | } 448 | *ppOut = pWin32File->pMap; 449 | *pnOut = pWin32File->nMap; 450 | return LSM_OK; 451 | } 452 | 453 | static BOOL win32IsDriveLetterAndColon( 454 | const char *zPathname 455 | ){ 456 | return ( isalpha(zPathname[0]) && zPathname[1]==':' ); 457 | } 458 | 459 | static int lsmWin32OsFullpath( 460 | lsm_env *pEnv, 461 | const char *zName, 462 | char *zOut, 463 | int *pnOut 464 | ){ 465 | DWORD nByte; 466 | void *zConverted; 467 | LPWSTR zTempWide; 468 | char *zTempUtf8; 469 | 470 | if( zName[0]=='/' && win32IsDriveLetterAndColon(zName+1) ){ 471 | zName++; 472 | } 473 | zConverted = win32Utf8ToUnicode(pEnv, zName); 474 | if( zConverted==0 ){ 475 | return LSM_NOMEM_BKPT; 476 | } 477 | nByte = GetFullPathNameW((LPCWSTR)zConverted, 0, 0, 0); 478 | if( nByte==0 ){ 479 | lsmFree(pEnv, zConverted); 480 | return LSM_IOERR_BKPT; 481 | } 482 | nByte += 3; 483 | zTempWide = lsmMallocZero(pEnv, nByte * sizeof(zTempWide[0])); 484 | if( zTempWide==0 ){ 485 | lsmFree(pEnv, zConverted); 486 | return LSM_NOMEM_BKPT; 487 | } 488 | nByte = GetFullPathNameW((LPCWSTR)zConverted, nByte, zTempWide, 0); 489 | if( nByte==0 ){ 490 | lsmFree(pEnv, zConverted); 491 | lsmFree(pEnv, zTempWide); 492 | return LSM_IOERR_BKPT; 493 | } 494 | lsmFree(pEnv, zConverted); 495 | zTempUtf8 = win32UnicodeToUtf8(pEnv, zTempWide); 496 | lsmFree(pEnv, zTempWide); 497 | if( zTempUtf8 ){ 498 | int nOut = *pnOut; 499 | int nLen = strlen(zTempUtf8) + 1; 500 | if( nLen<=nOut ){ 501 | snprintf(zOut, nOut, "%s", zTempUtf8); 502 | } 503 | lsmFree(pEnv, zTempUtf8); 504 | *pnOut = nLen; 505 | return LSM_OK; 506 | }else{ 507 | return LSM_NOMEM_BKPT; 508 | } 509 | } 510 | 511 | static int lsmWin32OsFileid( 512 | lsm_file *pFile, 513 | void *pBuf, 514 | int *pnBuf 515 | ){ 516 | int nBuf; 517 | int nReq; 518 | u8 *pBuf2 = (u8 *)pBuf; 519 | Win32File *pWin32File = (Win32File *)pFile; 520 | BY_HANDLE_FILE_INFORMATION fileInfo; 521 | 522 | nBuf = *pnBuf; 523 | nReq = (sizeof(fileInfo.dwVolumeSerialNumber) + 524 | sizeof(fileInfo.nFileIndexHigh) + 525 | sizeof(fileInfo.nFileIndexLow)); 526 | *pnBuf = nReq; 527 | if( nReq>nBuf ) return LSM_OK; 528 | memset(&fileInfo, 0, sizeof(BY_HANDLE_FILE_INFORMATION)); 529 | if( !GetFileInformationByHandle(pWin32File->hFile, &fileInfo) ){ 530 | return LSM_IOERR_BKPT; 531 | } 532 | nReq = sizeof(fileInfo.dwVolumeSerialNumber); 533 | memcpy(pBuf2, &fileInfo.dwVolumeSerialNumber, nReq); 534 | pBuf2 += nReq; 535 | nReq = sizeof(fileInfo.nFileIndexHigh); 536 | memcpy(pBuf, &fileInfo.nFileIndexHigh, nReq); 537 | pBuf2 += nReq; 538 | nReq = sizeof(fileInfo.nFileIndexLow); 539 | memcpy(pBuf2, &fileInfo.nFileIndexLow, nReq); 540 | return LSM_OK; 541 | } 542 | 543 | static int win32Delete( 544 | lsm_env *pEnv, 545 | const char *zFile 546 | ){ 547 | int rc; 548 | LPWSTR zConverted; 549 | 550 | zConverted = win32Utf8ToUnicode(pEnv, zFile); 551 | if( zConverted==0 ){ 552 | rc = LSM_NOMEM_BKPT; 553 | }else{ 554 | int nRetry = 0; 555 | DWORD attr; 556 | 557 | do { 558 | attr = GetFileAttributesW(zConverted); 559 | if ( attr==INVALID_FILE_ATTRIBUTES ){ 560 | rc = LSM_IOERR_BKPT; 561 | break; 562 | } 563 | if ( attr&FILE_ATTRIBUTE_DIRECTORY ){ 564 | rc = LSM_IOERR_BKPT; /* Files only. */ 565 | break; 566 | } 567 | if ( DeleteFileW(zConverted) ){ 568 | rc = LSM_OK; /* Deleted OK. */ 569 | break; 570 | } 571 | if ( !win32RetryIoerr(pEnv, &nRetry) ){ 572 | rc = LSM_IOERR_BKPT; /* No more retries. */ 573 | break; 574 | } 575 | }while( 1 ); 576 | } 577 | lsmFree(pEnv, zConverted); 578 | return rc; 579 | } 580 | 581 | static int lsmWin32OsUnlink(lsm_env *pEnv, const char *zFile){ 582 | return win32Delete(pEnv, zFile); 583 | } 584 | 585 | #if !defined(win32IsLockBusy) 586 | #define win32IsLockBusy(a) (((a)==ERROR_LOCK_VIOLATION) || \ 587 | ((a)==ERROR_IO_PENDING)) 588 | #endif 589 | 590 | static int win32LockFile( 591 | Win32File *pWin32File, 592 | int iLock, 593 | int nLock, 594 | int eType 595 | ){ 596 | OVERLAPPED ovlp; 597 | 598 | assert( LSM_LOCK_UNLOCK==0 ); 599 | assert( LSM_LOCK_SHARED==1 ); 600 | assert( LSM_LOCK_EXCL==2 ); 601 | assert( eType>=LSM_LOCK_UNLOCK && eType<=LSM_LOCK_EXCL ); 602 | assert( nLock>=0 ); 603 | assert( iLock>0 && iLock<=32 ); 604 | 605 | memset(&ovlp, 0, sizeof(OVERLAPPED)); 606 | ovlp.Offset = (4096-iLock-nLock+1); 607 | if( eType>LSM_LOCK_UNLOCK ){ 608 | DWORD flags = LOCKFILE_FAIL_IMMEDIATELY; 609 | if( eType>=LSM_LOCK_EXCL ) flags |= LOCKFILE_EXCLUSIVE_LOCK; 610 | if( !LockFileEx(pWin32File->hFile, flags, 0, (DWORD)nLock, 0, &ovlp) ){ 611 | if( win32IsLockBusy(GetLastError()) ){ 612 | return LSM_BUSY; 613 | }else{ 614 | return LSM_IOERR_BKPT; 615 | } 616 | } 617 | }else{ 618 | if( !UnlockFileEx(pWin32File->hFile, 0, (DWORD)nLock, 0, &ovlp) ){ 619 | return LSM_IOERR_BKPT; 620 | } 621 | } 622 | return LSM_OK; 623 | } 624 | 625 | static int lsmWin32OsLock(lsm_file *pFile, int iLock, int eType){ 626 | Win32File *pWin32File = (Win32File *)pFile; 627 | return win32LockFile(pWin32File, iLock, 1, eType); 628 | } 629 | 630 | static int lsmWin32OsTestLock(lsm_file *pFile, int iLock, int nLock, int eType){ 631 | int rc; 632 | Win32File *pWin32File = (Win32File *)pFile; 633 | rc = win32LockFile(pWin32File, iLock, nLock, eType); 634 | if( rc!=LSM_OK ) return rc; 635 | win32LockFile(pWin32File, iLock, nLock, LSM_LOCK_UNLOCK); 636 | return LSM_OK; 637 | } 638 | 639 | static int lsmWin32OsShmMap(lsm_file *pFile, int iChunk, int sz, void **ppShm){ 640 | int rc; 641 | Win32File *pWin32File = (Win32File *)pFile; 642 | int iOffset = iChunk * sz; 643 | int iOffsetShift = iOffset % pWin32File->sysInfo.dwAllocationGranularity; 644 | int nNew = iChunk + 1; 645 | lsm_i64 nReq = nNew * sz; 646 | 647 | *ppShm = NULL; 648 | assert( sz>=0 ); 649 | assert( sz==LSM_SHM_CHUNK_SIZE ); 650 | if( iChunk>=pWin32File->nShm ){ 651 | LPHANDLE ahNew; 652 | LPVOID *apNew; 653 | LARGE_INTEGER fileSize; 654 | 655 | /* If the shared-memory file has not been opened, open it now. */ 656 | if( pWin32File->hShmFile==NULL ){ 657 | char *zShm = win32ShmFile(pWin32File); 658 | if( !zShm ) return LSM_NOMEM_BKPT; 659 | rc = win32Open(pWin32File->pEnv, zShm, 0, &pWin32File->hShmFile); 660 | lsmFree(pWin32File->pEnv, zShm); 661 | if( rc!=LSM_OK ){ 662 | return rc; 663 | } 664 | } 665 | 666 | /* If the shared-memory file is not large enough to contain the 667 | ** requested chunk, cause it to grow. */ 668 | memset(&fileSize, 0, sizeof(LARGE_INTEGER)); 669 | if( !GetFileSizeEx(pWin32File->hShmFile, &fileSize) ){ 670 | return LSM_IOERR_BKPT; 671 | } 672 | assert( fileSize.QuadPart>=0 ); 673 | if( fileSize.QuadParthShmFile, nReq); 675 | if( rc!=LSM_OK ){ 676 | return rc; 677 | } 678 | } 679 | 680 | ahNew = (LPHANDLE)lsmMallocZero(pWin32File->pEnv, sizeof(HANDLE) * nNew); 681 | if( !ahNew ) return LSM_NOMEM_BKPT; 682 | apNew = (LPVOID *)lsmMallocZero(pWin32File->pEnv, sizeof(LPVOID) * nNew); 683 | if( !apNew ){ 684 | lsmFree(pWin32File->pEnv, ahNew); 685 | return LSM_NOMEM_BKPT; 686 | } 687 | memcpy(ahNew, pWin32File->ahShm, sizeof(HANDLE) * pWin32File->nShm); 688 | memcpy(apNew, pWin32File->apShm, sizeof(LPVOID) * pWin32File->nShm); 689 | lsmFree(pWin32File->pEnv, pWin32File->ahShm); 690 | pWin32File->ahShm = ahNew; 691 | lsmFree(pWin32File->pEnv, pWin32File->apShm); 692 | pWin32File->apShm = apNew; 693 | pWin32File->nShm = nNew; 694 | } 695 | 696 | if( pWin32File->ahShm[iChunk]==NULL ){ 697 | HANDLE hMap; 698 | assert( nReq<=0xFFFFFFFF ); 699 | hMap = CreateFileMappingW(pWin32File->hShmFile, NULL, PAGE_READWRITE, 0, 700 | (DWORD)nReq, NULL); 701 | if( hMap==NULL ){ 702 | return LSM_IOERR_BKPT; 703 | } 704 | pWin32File->ahShm[iChunk] = hMap; 705 | } 706 | if( pWin32File->apShm[iChunk]==NULL ){ 707 | LPVOID pMap; 708 | pMap = MapViewOfFile(pWin32File->ahShm[iChunk], 709 | FILE_MAP_WRITE | FILE_MAP_READ, 0, 710 | iOffset - iOffsetShift, sz + iOffsetShift); 711 | if( pMap==NULL ){ 712 | return LSM_IOERR_BKPT; 713 | } 714 | pWin32File->apShm[iChunk] = pMap; 715 | } 716 | if( iOffsetShift!=0 ){ 717 | char *p = (char *)pWin32File->apShm[iChunk]; 718 | *ppShm = (void *)&p[iOffsetShift]; 719 | }else{ 720 | *ppShm = pWin32File->apShm[iChunk]; 721 | } 722 | return LSM_OK; 723 | } 724 | 725 | static void lsmWin32OsShmBarrier(void){ 726 | MemoryBarrier(); 727 | } 728 | 729 | static int lsmWin32OsShmUnmap(lsm_file *pFile, int bDelete){ 730 | Win32File *pWin32File = (Win32File *)pFile; 731 | 732 | if( pWin32File->hShmFile!=NULL ){ 733 | int i; 734 | for(i=0; inShm; i++){ 735 | if( pWin32File->apShm[i]!=NULL ){ 736 | UnmapViewOfFile(pWin32File->apShm[i]); 737 | pWin32File->apShm[i] = NULL; 738 | } 739 | if( pWin32File->ahShm[i]!=NULL ){ 740 | CloseHandle(pWin32File->ahShm[i]); 741 | pWin32File->ahShm[i] = NULL; 742 | } 743 | } 744 | CloseHandle(pWin32File->hShmFile); 745 | pWin32File->hShmFile = NULL; 746 | if( bDelete ){ 747 | char *zShm = win32ShmFile(pWin32File); 748 | if( zShm ){ win32Delete(pWin32File->pEnv, zShm); } 749 | lsmFree(pWin32File->pEnv, zShm); 750 | } 751 | } 752 | return LSM_OK; 753 | } 754 | 755 | #define MX_CLOSE_ATTEMPT 3 756 | static int lsmWin32OsClose(lsm_file *pFile){ 757 | int rc; 758 | int nRetry = 0; 759 | Win32File *pWin32File = (Win32File *)pFile; 760 | lsmWin32OsShmUnmap(pFile, 0); 761 | win32Unmap(pWin32File); 762 | do{ 763 | if( pWin32File->hFile==NULL ){ 764 | rc = LSM_IOERR_BKPT; 765 | break; 766 | } 767 | rc = CloseHandle(pWin32File->hFile); 768 | if( rc ){ 769 | pWin32File->hFile = NULL; 770 | rc = LSM_OK; 771 | break; 772 | } 773 | if( ++nRetry>=MX_CLOSE_ATTEMPT ){ 774 | rc = LSM_IOERR_BKPT; 775 | break; 776 | } 777 | }while( 1 ); 778 | lsmFree(pWin32File->pEnv, pWin32File->ahShm); 779 | lsmFree(pWin32File->pEnv, pWin32File->apShm); 780 | lsmFree(pWin32File->pEnv, pWin32File); 781 | return rc; 782 | } 783 | 784 | static int lsmWin32OsSleep(lsm_env *pEnv, int us){ 785 | unused_parameter(pEnv); 786 | return win32Sleep(us); 787 | } 788 | 789 | /**************************************************************************** 790 | ** Memory allocation routines. 791 | */ 792 | 793 | static void *lsmWin32OsMalloc(lsm_env *pEnv, size_t N){ 794 | assert( HeapValidate(GetProcessHeap(), 0, NULL) ); 795 | return HeapAlloc(GetProcessHeap(), 0, (SIZE_T)N); 796 | } 797 | 798 | static void lsmWin32OsFree(lsm_env *pEnv, void *p){ 799 | assert( HeapValidate(GetProcessHeap(), 0, NULL) ); 800 | if( p ){ 801 | HeapFree(GetProcessHeap(), 0, p); 802 | } 803 | } 804 | 805 | static void *lsmWin32OsRealloc(lsm_env *pEnv, void *p, size_t N){ 806 | unsigned char *m = (unsigned char *)p; 807 | assert( HeapValidate(GetProcessHeap(), 0, NULL) ); 808 | if( 1>N ){ 809 | lsmWin32OsFree(pEnv, p); 810 | return NULL; 811 | }else if( NULL==p ){ 812 | return lsmWin32OsMalloc(pEnv, N); 813 | }else{ 814 | #if 0 /* arguable: don't shrink */ 815 | SIZE_T sz = HeapSize(GetProcessHeap(), 0, m); 816 | if( sz>=(SIZE_T)N ){ 817 | return p; 818 | } 819 | #endif 820 | return HeapReAlloc(GetProcessHeap(), 0, m, N); 821 | } 822 | } 823 | 824 | static size_t lsmWin32OsMSize(lsm_env *pEnv, void *p){ 825 | assert( HeapValidate(GetProcessHeap(), 0, NULL) ); 826 | return (size_t)HeapSize(GetProcessHeap(), 0, p); 827 | } 828 | 829 | 830 | #ifdef LSM_MUTEX_WIN32 831 | /************************************************************************* 832 | ** Mutex methods for Win32 based systems. If LSM_MUTEX_WIN32 is 833 | ** missing then a no-op implementation of mutexes found below will be 834 | ** used instead. 835 | */ 836 | #include "windows.h" 837 | 838 | typedef struct Win32Mutex Win32Mutex; 839 | struct Win32Mutex { 840 | lsm_env *pEnv; 841 | CRITICAL_SECTION mutex; 842 | #ifdef LSM_DEBUG 843 | DWORD owner; 844 | #endif 845 | }; 846 | 847 | #ifndef WIN32_MUTEX_INITIALIZER 848 | # define WIN32_MUTEX_INITIALIZER { 0 } 849 | #endif 850 | 851 | #ifdef LSM_DEBUG 852 | # define LSM_WIN32_STATIC_MUTEX { 0, WIN32_MUTEX_INITIALIZER, 0 } 853 | #else 854 | # define LSM_WIN32_STATIC_MUTEX { 0, WIN32_MUTEX_INITIALIZER } 855 | #endif 856 | 857 | static int lsmWin32OsMutexStatic( 858 | lsm_env *pEnv, 859 | int iMutex, 860 | lsm_mutex **ppStatic 861 | ){ 862 | static volatile LONG initialized = 0; 863 | static Win32Mutex sMutex[2] = { 864 | LSM_WIN32_STATIC_MUTEX, 865 | LSM_WIN32_STATIC_MUTEX 866 | }; 867 | 868 | assert( iMutex==LSM_MUTEX_GLOBAL || iMutex==LSM_MUTEX_HEAP ); 869 | assert( LSM_MUTEX_GLOBAL==1 && LSM_MUTEX_HEAP==2 ); 870 | 871 | if( InterlockedCompareExchange(&initialized, 1, 0)==0 ){ 872 | int i; 873 | for(i=0; ipEnv = pEnv; 888 | InitializeCriticalSection(&pMutex->mutex); 889 | 890 | *ppNew = (lsm_mutex *)pMutex; 891 | return LSM_OK; 892 | } 893 | 894 | static void lsmWin32OsMutexDel(lsm_mutex *p){ 895 | Win32Mutex *pMutex = (Win32Mutex *)p; 896 | DeleteCriticalSection(&pMutex->mutex); 897 | lsmFree(pMutex->pEnv, pMutex); 898 | } 899 | 900 | static void lsmWin32OsMutexEnter(lsm_mutex *p){ 901 | Win32Mutex *pMutex = (Win32Mutex *)p; 902 | EnterCriticalSection(&pMutex->mutex); 903 | 904 | #ifdef LSM_DEBUG 905 | assert( pMutex->owner!=GetCurrentThreadId() ); 906 | pMutex->owner = GetCurrentThreadId(); 907 | assert( pMutex->owner==GetCurrentThreadId() ); 908 | #endif 909 | } 910 | 911 | static int lsmWin32OsMutexTry(lsm_mutex *p){ 912 | BOOL bRet; 913 | Win32Mutex *pMutex = (Win32Mutex *)p; 914 | bRet = TryEnterCriticalSection(&pMutex->mutex); 915 | #ifdef LSM_DEBUG 916 | if( bRet ){ 917 | assert( pMutex->owner!=GetCurrentThreadId() ); 918 | pMutex->owner = GetCurrentThreadId(); 919 | assert( pMutex->owner==GetCurrentThreadId() ); 920 | } 921 | #endif 922 | return !bRet; 923 | } 924 | 925 | static void lsmWin32OsMutexLeave(lsm_mutex *p){ 926 | Win32Mutex *pMutex = (Win32Mutex *)p; 927 | #ifdef LSM_DEBUG 928 | assert( pMutex->owner==GetCurrentThreadId() ); 929 | pMutex->owner = 0; 930 | assert( pMutex->owner!=GetCurrentThreadId() ); 931 | #endif 932 | LeaveCriticalSection(&pMutex->mutex); 933 | } 934 | 935 | #ifdef LSM_DEBUG 936 | static int lsmWin32OsMutexHeld(lsm_mutex *p){ 937 | Win32Mutex *pMutex = (Win32Mutex *)p; 938 | return pMutex ? pMutex->owner==GetCurrentThreadId() : 1; 939 | } 940 | static int lsmWin32OsMutexNotHeld(lsm_mutex *p){ 941 | Win32Mutex *pMutex = (Win32Mutex *)p; 942 | return pMutex ? pMutex->owner!=GetCurrentThreadId() : 1; 943 | } 944 | #endif 945 | /* 946 | ** End of Win32 mutex implementation. 947 | *************************************************************************/ 948 | #else 949 | /************************************************************************* 950 | ** Noop mutex implementation 951 | */ 952 | typedef struct NoopMutex NoopMutex; 953 | struct NoopMutex { 954 | lsm_env *pEnv; /* Environment handle (for xFree()) */ 955 | int bHeld; /* True if mutex is held */ 956 | int bStatic; /* True for a static mutex */ 957 | }; 958 | static NoopMutex aStaticNoopMutex[2] = { 959 | {0, 0, 1}, 960 | {0, 0, 1}, 961 | }; 962 | 963 | static int lsmWin32OsMutexStatic( 964 | lsm_env *pEnv, 965 | int iMutex, 966 | lsm_mutex **ppStatic 967 | ){ 968 | assert( iMutex>=1 && iMutex<=(int)array_size(aStaticNoopMutex) ); 969 | *ppStatic = (lsm_mutex *)&aStaticNoopMutex[iMutex-1]; 970 | return LSM_OK; 971 | } 972 | static int lsmWin32OsMutexNew(lsm_env *pEnv, lsm_mutex **ppNew){ 973 | NoopMutex *p; 974 | p = (NoopMutex *)lsmMallocZero(pEnv, sizeof(NoopMutex)); 975 | if( p ) p->pEnv = pEnv; 976 | *ppNew = (lsm_mutex *)p; 977 | return (p ? LSM_OK : LSM_NOMEM_BKPT); 978 | } 979 | static void lsmWin32OsMutexDel(lsm_mutex *pMutex) { 980 | NoopMutex *p = (NoopMutex *)pMutex; 981 | assert( p->bStatic==0 && p->pEnv ); 982 | lsmFree(p->pEnv, p); 983 | } 984 | static void lsmWin32OsMutexEnter(lsm_mutex *pMutex){ 985 | NoopMutex *p = (NoopMutex *)pMutex; 986 | assert( p->bHeld==0 ); 987 | p->bHeld = 1; 988 | } 989 | static int lsmWin32OsMutexTry(lsm_mutex *pMutex){ 990 | NoopMutex *p = (NoopMutex *)pMutex; 991 | assert( p->bHeld==0 ); 992 | p->bHeld = 1; 993 | return 0; 994 | } 995 | static void lsmWin32OsMutexLeave(lsm_mutex *pMutex){ 996 | NoopMutex *p = (NoopMutex *)pMutex; 997 | assert( p->bHeld==1 ); 998 | p->bHeld = 0; 999 | } 1000 | #ifdef LSM_DEBUG 1001 | static int lsmWin32OsMutexHeld(lsm_mutex *pMutex){ 1002 | NoopMutex *p = (NoopMutex *)pMutex; 1003 | return p ? p->bHeld : 1; 1004 | } 1005 | static int lsmWin32OsMutexNotHeld(lsm_mutex *pMutex){ 1006 | NoopMutex *p = (NoopMutex *)pMutex; 1007 | return p ? !p->bHeld : 1; 1008 | } 1009 | #endif 1010 | /***************************************************************************/ 1011 | #endif /* else LSM_MUTEX_NONE */ 1012 | 1013 | /* Without LSM_DEBUG, the MutexHeld tests are never called */ 1014 | #ifndef LSM_DEBUG 1015 | # define lsmWin32OsMutexHeld 0 1016 | # define lsmWin32OsMutexNotHeld 0 1017 | #endif 1018 | 1019 | lsm_env *lsm_default_env(void){ 1020 | static lsm_env win32_env = { 1021 | sizeof(lsm_env), /* nByte */ 1022 | 1, /* iVersion */ 1023 | /***** file i/o ******************/ 1024 | 0, /* pVfsCtx */ 1025 | lsmWin32OsFullpath, /* xFullpath */ 1026 | lsmWin32OsOpen, /* xOpen */ 1027 | lsmWin32OsRead, /* xRead */ 1028 | lsmWin32OsWrite, /* xWrite */ 1029 | lsmWin32OsTruncate, /* xTruncate */ 1030 | lsmWin32OsSync, /* xSync */ 1031 | lsmWin32OsSectorSize, /* xSectorSize */ 1032 | lsmWin32OsRemap, /* xRemap */ 1033 | lsmWin32OsFileid, /* xFileid */ 1034 | lsmWin32OsClose, /* xClose */ 1035 | lsmWin32OsUnlink, /* xUnlink */ 1036 | lsmWin32OsLock, /* xLock */ 1037 | lsmWin32OsTestLock, /* xTestLock */ 1038 | lsmWin32OsShmMap, /* xShmMap */ 1039 | lsmWin32OsShmBarrier, /* xShmBarrier */ 1040 | lsmWin32OsShmUnmap, /* xShmUnmap */ 1041 | /***** memory allocation *********/ 1042 | 0, /* pMemCtx */ 1043 | lsmWin32OsMalloc, /* xMalloc */ 1044 | lsmWin32OsRealloc, /* xRealloc */ 1045 | lsmWin32OsFree, /* xFree */ 1046 | lsmWin32OsMSize, /* xSize */ 1047 | /***** mutexes *********************/ 1048 | 0, /* pMutexCtx */ 1049 | lsmWin32OsMutexStatic, /* xMutexStatic */ 1050 | lsmWin32OsMutexNew, /* xMutexNew */ 1051 | lsmWin32OsMutexDel, /* xMutexDel */ 1052 | lsmWin32OsMutexEnter, /* xMutexEnter */ 1053 | lsmWin32OsMutexTry, /* xMutexTry */ 1054 | lsmWin32OsMutexLeave, /* xMutexLeave */ 1055 | lsmWin32OsMutexHeld, /* xMutexHeld */ 1056 | lsmWin32OsMutexNotHeld, /* xMutexNotHeld */ 1057 | /***** other *********************/ 1058 | lsmWin32OsSleep, /* xSleep */ 1059 | }; 1060 | return &win32_env; 1061 | } 1062 | 1063 | #endif 1064 | -------------------------------------------------------------------------------- /tests.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import tempfile 4 | import threading 5 | import unittest 6 | 7 | try: 8 | import lsm 9 | except ImportError: 10 | sys.stderr.write('Unable to import `lsm`. Make sure it is properly ' 11 | 'installed.\n') 12 | sys.stderr.flush() 13 | raise 14 | 15 | 16 | def b(s): 17 | return s.encode('utf-8') if not isinstance(s, bytes) else s 18 | 19 | 20 | class BaseTestLSM(unittest.TestCase): 21 | def setUp(self): 22 | self.filename = tempfile.mktemp() 23 | self.db = lsm.LSM(self.filename) 24 | 25 | def tearDown(self): 26 | if self.db.is_open: 27 | while self.db.transaction_depth > 0: 28 | self.db.rollback(False) 29 | self.db.close() 30 | if os.path.exists(self.filename): 31 | os.unlink(self.filename) 32 | 33 | def assertMissing(self, key): 34 | self.assertRaises(KeyError, lambda: self.db[key]) 35 | 36 | def assertBEqual(self, lhs, rhs): 37 | if isinstance(lhs, list): 38 | if lhs and isinstance(lhs[0], tuple): 39 | self.assertEqual(lhs, [tuple(b(si) for si in i) 40 | for i in rhs]) 41 | else: 42 | self.assertEqual(lhs, [b(i) for i in rhs]) 43 | else: 44 | self.assertEqual(lhs, b(rhs)) 45 | 46 | 47 | class TestLSM(BaseTestLSM): 48 | def test_db_open_close(self): 49 | self.db['foo'] = b('bar') 50 | 51 | def test_dict_api(self): 52 | self.db['k1'] = 'v1' 53 | self.db['k2'] = 'v2' 54 | self.db['k3'] = 'v3' 55 | self.assertBEqual(self.db['k1'], 'v1') 56 | self.assertBEqual(self.db['k3'], 'v3') 57 | self.assertMissing('k4') 58 | 59 | del self.db['k1'] 60 | self.assertMissing('k1') 61 | 62 | # No error is raised trying to delete a key that doesn't exist. 63 | del self.db['k1'] 64 | 65 | self.assertTrue('k2' in self.db) 66 | self.assertFalse('k1' in self.db) 67 | 68 | self.assertBEqual(self.db['k22', lsm.SEEK_GE], 'v3') 69 | self.assertBEqual(self.db['k22', lsm.SEEK_LE], 'v2') 70 | 71 | self.db.update({'foo': 'bar', 'nug': 'nizer'}) 72 | self.assertBEqual(self.db['foo'], 'bar') 73 | self.assertBEqual(self.db['nug'], 'nizer') 74 | 75 | def test_keys_values(self): 76 | for i in range(1, 5): 77 | self.db['k%s' % i] = 'v%s' % i 78 | 79 | keys = [key for key in self.db.keys()] 80 | self.assertBEqual(keys, ['k1', 'k2', 'k3', 'k4']) 81 | 82 | keys = [key for key in self.db.keys(True)] 83 | self.assertBEqual(keys, ['k4', 'k3', 'k2', 'k1']) 84 | 85 | values = [value for value in self.db.values()] 86 | self.assertBEqual(values, ['v1', 'v2', 'v3', 'v4']) 87 | 88 | values = [value for value in self.db.values(True)] 89 | self.assertBEqual(values, ['v4', 'v3', 'v2', 'v1']) 90 | 91 | def test_fetch(self): 92 | self.db['k1'] = 'v1' 93 | self.db['k2'] = 'v2' 94 | self.db['k3'] = 'v3' 95 | self.assertBEqual(self.db.fetch('k2'), 'v2') 96 | self.assertBEqual(self.db.fetch('k2', lsm.SEEK_LE), 'v2') 97 | self.assertBEqual(self.db.fetch('k2', lsm.SEEK_GE), 'v2') 98 | 99 | self.assertRaises(KeyError, self.db.fetch, 'k22') 100 | self.assertBEqual(self.db.fetch('k22', lsm.SEEK_LE), 'v2') 101 | self.assertBEqual(self.db.fetch('k22', lsm.SEEK_GE), 'v3') 102 | 103 | def test_fetch_bulk(self): 104 | self.db.update({'k1': 'v1', 'k2': 'v2', 'k3': 'v3', 'k4': 'v4'}) 105 | res = self.db.fetch_bulk(['k1', 'k2', 'k3', 'kx']) 106 | self.assertEqual(res, {'k1': b'v1', 'k2': b'v2', 'k3': b'v3'}) 107 | 108 | res = self.db.fetch_bulk(['k4']) 109 | self.assertEqual(res, {'k4': b'v4'}) 110 | 111 | res = self.db.fetch_bulk(['foo', 'bar']) 112 | self.assertEqual(res, {}) 113 | 114 | def assertIterEqual(self, i, expected): 115 | self.assertBEqual(list(i), expected) 116 | 117 | def test_fetch_range(self): 118 | results = [] 119 | for i in range(1, 10): 120 | self.db['k%s' % i] = 'v%s' % i 121 | results.append(('k%s' % i, 'v%s' % i)) 122 | 123 | res = self.db['k2':'k5'] 124 | self.assertIterEqual(res, [ 125 | ('k2', 'v2'), 126 | ('k3', 'v3'), 127 | ('k4', 'v4'), 128 | ('k5', 'v5'), 129 | ]) 130 | 131 | # Empty start. 132 | res = self.db[:'k3'] 133 | self.assertIterEqual(res, [ 134 | ('k1', 'v1'), 135 | ('k2', 'v2'), 136 | ('k3', 'v3'), 137 | ]) 138 | 139 | # Empty end. 140 | res = self.db['k7':] 141 | self.assertIterEqual(res, [ 142 | ('k7', 'v7'), 143 | ('k8', 'v8'), 144 | ('k9', 'v9'), 145 | ]) 146 | 147 | # Missing end. 148 | res = self.db['k7':'k88'] 149 | self.assertIterEqual(res, [ 150 | ('k7', 'v7'), 151 | ('k8', 'v8'), 152 | ]) 153 | 154 | # Missing start. 155 | res = self.db['k33':'k5'] 156 | self.assertIterEqual(res, [ 157 | ('k4', 'v4'), 158 | ('k5', 'v5'), 159 | ]) 160 | 161 | # Start exceeds highest key. 162 | res = self.db['xx':'yy'] 163 | self.assertIterEqual(res, []) 164 | 165 | res = self.db['xx':] 166 | self.assertIterEqual(res, []) 167 | 168 | # End preceds lowest key. 169 | res = self.db['aa':'bb'] 170 | self.assertIterEqual(res, []) 171 | 172 | res = self.db[:'bb'] 173 | self.assertIterEqual(res, []) 174 | 175 | res = self.db[:] 176 | self.assertIterEqual(res, results) 177 | 178 | def test_fetch_range_reverse(self): 179 | results = [] 180 | for i in range(1, 10): 181 | self.db['k%s' % i] = 'v%s' % i 182 | results.append(('k%s' % i, 'v%s' % i)) 183 | 184 | res = self.db['k5':'k2':True] 185 | self.assertIterEqual(res, [ 186 | ('k5', 'v5'), 187 | ('k4', 'v4'), 188 | ('k3', 'v3'), 189 | ('k2', 'v2'), 190 | ]) 191 | 192 | # Empty end. 193 | res = self.db['k7'::True] 194 | self.assertIterEqual(res, [ 195 | ('k9', 'v9'), 196 | ('k8', 'v8'), 197 | ('k7', 'v7'), 198 | ]) 199 | 200 | # Empty start. 201 | res = self.db[:'k3':True] 202 | self.assertIterEqual(res, [ 203 | ('k3', 'v3'), 204 | ('k2', 'v2'), 205 | ('k1', 'v1'), 206 | ]) 207 | 208 | # Missing start. 209 | res = self.db['k88':'k7':True] 210 | self.assertIterEqual(res, [ 211 | ('k8', 'v8'), 212 | ('k7', 'v7'), 213 | ]) 214 | 215 | # Missing end. 216 | res = self.db['k5':'k33':True] 217 | self.assertIterEqual(res, [ 218 | ('k5', 'v5'), 219 | ('k4', 'v4'), 220 | ]) 221 | 222 | # End exceeds highest key. 223 | res = self.db[:'xx':True] 224 | self.assertIterEqual(res, list(reversed(results))) 225 | 226 | # Start exceeds highest key. 227 | res = self.db['yy':'xx':True] 228 | self.assertIterEqual(res, []) 229 | 230 | res = self.db['xx'::True] 231 | self.assertIterEqual(res, []) 232 | 233 | # End preceds lowest key. 234 | res = self.db['bb':'aa':True] 235 | self.assertIterEqual(res, []) 236 | 237 | res = self.db[:'bb':True] 238 | self.assertIterEqual(res, []) 239 | 240 | # Missing both. 241 | res = self.db[::True] 242 | self.assertIterEqual(res, list(reversed(results))) 243 | 244 | def test_fetch_range_implicit(self): 245 | for i in range(1, 10): 246 | self.db['k%s' % i] = 'v%s' % i 247 | 248 | self.assertIterEqual(self.db['k7':'k4'], [ 249 | ('k7', 'v7'), 250 | ('k6', 'v6'), 251 | ('k5', 'v5'), 252 | ('k4', 'v4'), 253 | ]) 254 | 255 | self.assertIterEqual(self.db['k4':'k7'], [ 256 | ('k4', 'v4'), 257 | ('k5', 'v5'), 258 | ('k6', 'v6'), 259 | ('k7', 'v7'), 260 | ]) 261 | 262 | def test_delete_range(self): 263 | for i in range(1, 10): 264 | self.db['k%s' % i] = 'v%s' % i 265 | 266 | # delete_range does not include the start/end keys. 267 | del self.db['k3':'k7'] 268 | self.assertBEqual(self.db['k3'], 'v3') 269 | self.assertBEqual(self.db['k7'], 'v7') 270 | 271 | for key in ['k4', 'k5', 'k6']: 272 | self.assertMissing(key) 273 | 274 | # Missing start key. 275 | del self.db['k4':'k8'] 276 | self.assertBEqual(self.db['k8'], 'v8') 277 | self.assertMissing('k7') 278 | 279 | # Invalid start key. 280 | del self.db['k0':'k2'] 281 | self.assertBEqual(self.db['k2'], 'v2') 282 | self.assertMissing('k1') 283 | 284 | self.assertBEqual(self.db['k9'], 'v9') 285 | 286 | # Invalid end key. 287 | del self.db['k8':'xx'] 288 | self.assertMissing('k9') 289 | 290 | # Invalid start and end keys low. 291 | del self.db['aa':'bb'] 292 | 293 | with self.db.cursor() as cursor: 294 | accum = [val for val in cursor] 295 | self.assertBEqual(accum, [ 296 | ('k2', 'v2'), 297 | ('k3', 'v3'), 298 | ('k8', 'v8'), 299 | ]) 300 | 301 | # Invalid start and end keys high. 302 | del self.db['xx':'yy'] 303 | 304 | with self.db.cursor() as cursor: 305 | accum = [val for val in cursor] 306 | self.assertBEqual(accum, [ 307 | ('k2', 'v2'), 308 | ('k3', 'v3'), 309 | ('k8', 'v8'), 310 | ]) 311 | 312 | def test_iter_database(self): 313 | for i in range(1, 5): 314 | self.db['k%s' % i] = 'v%s' % i 315 | 316 | items = list(self.db) 317 | self.assertIterEqual(items, [ 318 | ('k1', 'v1'), 319 | ('k2', 'v2'), 320 | ('k3', 'v3'), 321 | ('k4', 'v4'), 322 | ]) 323 | 324 | items = list(reversed(self.db)) 325 | self.assertIterEqual(items, [ 326 | ('k4', 'v4'), 327 | ('k3', 'v3'), 328 | ('k2', 'v2'), 329 | ('k1', 'v1'), 330 | ]) 331 | 332 | def test_incr(self): 333 | self.assertEqual(self.db.incr('i0'), 1) 334 | self.assertEqual(self.db.incr('i0'), 2) 335 | self.assertEqual(self.db.incr('i0'), 3) 336 | 337 | def test_data_types(self): 338 | key = b('k\xe2\x80\x941') 339 | self.db[key] = key 340 | ret = self.db[key] 341 | self.assertEqual(ret, key) 342 | 343 | self.assertRaises(KeyError, lambda: self.db[1]) 344 | self.assertRaises(KeyError, lambda: self.db[1.0]) 345 | self.assertRaises(TypeError, lambda: self.db.insert(key, None)) 346 | 347 | 348 | class TestTransactions(BaseTestLSM): 349 | def assertDepth(self, value): 350 | self.assertEqual(self.db.transaction_depth, value) 351 | 352 | def test_transaction_apis(self): 353 | self.db.begin() 354 | self.db['k1'] = 'v1' 355 | self.assertTrue(self.db.rollback(keep_transaction=True)) 356 | self.assertDepth(1) 357 | self.assertMissing('k1') 358 | 359 | self.db['k2'] = 'v2' 360 | self.assertTrue(self.db.rollback(keep_transaction=False)) 361 | self.assertDepth(0) 362 | self.assertMissing('k2') 363 | 364 | # Cannot commit, no transaction is open. 365 | self.assertFalse(self.db.commit()) 366 | 367 | # Cannot rollback, no transaction is open. 368 | self.assertFalse(self.db.rollback()) 369 | 370 | self.db.begin() # 1. 371 | self.db['k1'] = 'v1' 372 | 373 | self.db.begin() # 2. 374 | self.db['k1'] = 'v1-1' 375 | self.db.rollback() 376 | self.assertBEqual(self.db['k1'], 'v1') 377 | self.assertDepth(2) 378 | self.db['k1'] = 'v1-2' 379 | self.db.commit() 380 | 381 | self.assertDepth(1) 382 | self.db.rollback(False) 383 | self.assertMissing('k1') 384 | self.assertDepth(0) 385 | 386 | def test_transaction_context_manager(self): 387 | with self.db.transaction(): 388 | self.db['k1'] = 'v1' 389 | self.assertDepth(1) 390 | 391 | self.assertBEqual(self.db['k1'], 'v1') 392 | self.assertDepth(0) 393 | 394 | with self.db.transaction() as txn: 395 | self.db['k2'] = 'v2' 396 | self.assertTrue(txn.rollback()) 397 | self.assertDepth(1) 398 | 399 | self.db['k3'] = 'v3' 400 | self.assertTrue(txn.rollback()) 401 | self.assertDepth(1) 402 | 403 | self.db['k4'] = 'v4' 404 | 405 | self.assertDepth(0) 406 | self.assertMissing('k2') 407 | self.assertMissing('k3') 408 | self.assertBEqual(self.db['k4'], 'v4') 409 | 410 | def test_transaction_nesting(self): 411 | with self.db.transaction() as txn1: 412 | with self.db.transaction() as txn2: 413 | self.db['k0'] = 'v0' 414 | 415 | with self.db.transaction() as txn3: 416 | self.db['k1'] = 'v1' 417 | del self.db['k0'] 418 | self.assertDepth(3) 419 | txn3.rollback() 420 | 421 | self.assertMissing('k1') 422 | self.assertBEqual(self.db['k0'], 'v0') 423 | self.db['k2'] = 'v2' 424 | del self.db['k0'] 425 | 426 | self.db['k3'] = 'v3' 427 | self.assertDepth(1) 428 | 429 | self.assertMissing('k0') 430 | self.assertMissing('k1') 431 | self.assertBEqual(self.db['k2'], 'v2') 432 | self.assertBEqual(self.db['k3'], 'v3') 433 | 434 | def test_transaction_nesting_2(self): 435 | with self.db.transaction() as txn1: 436 | self.db['k1'] = 'v1' 437 | with self.db.transaction() as txn2: 438 | self.db['k2'] = 'v2' 439 | txn2.commit() 440 | 441 | self.db['k2'] = 'v2-1' 442 | 443 | with self.db.transaction() as txn3: 444 | self.db['k2'] = 'v2-2' 445 | txn3.rollback() 446 | 447 | self.assertBEqual(self.db['k2'], 'v2-1') 448 | txn2.rollback() 449 | 450 | self.assertBEqual(self.db['k2'], 'v2') 451 | 452 | self.assertDepth(0) 453 | self.assertBEqual(self.db['k1'], 'v1') 454 | self.assertBEqual(self.db['k2'], 'v2') 455 | 456 | def test_transaction_decorator(self): 457 | class FuncError(Exception): 458 | pass 459 | 460 | @self.db.transaction() 461 | def txn_func(error_out=False, **kwargs): 462 | for key, value in kwargs.items(): 463 | self.db[key] = value 464 | if error_out: 465 | raise FuncError() 466 | 467 | txn_func(k1='v1', k2='v2') 468 | self.assertBEqual(self.db['k1'], 'v1') 469 | self.assertBEqual(self.db['k2'], 'v2') 470 | 471 | self.assertRaises(FuncError, txn_func, error_out=True, k1='v1-1') 472 | self.assertBEqual(self.db['k1'], 'v1') 473 | self.assertBEqual(self.db['k2'], 'v2') 474 | 475 | 476 | class TestCursors(BaseTestLSM): 477 | def setUp(self): 478 | super(TestCursors, self).setUp() 479 | self.db['bb'] = 'bbb' 480 | self.db['gg'] = 'ggg' 481 | self.db['aa'] = 'aaa' 482 | self.db['dd'] = 'ddd' 483 | self.db['zz'] = 'zzz' 484 | self.db['ee'] = 'eee' 485 | self.db['bbb'] = 'bbb' 486 | 487 | def test_cursor_simple(self): 488 | with self.db.cursor() as cursor: 489 | items = list(cursor) 490 | 491 | self.assertBEqual(items, [ 492 | ('aa', 'aaa'), 493 | ('bb', 'bbb'), 494 | ('bbb', 'bbb'), 495 | ('dd', 'ddd'), 496 | ('ee', 'eee'), 497 | ('gg', 'ggg'), 498 | ('zz', 'zzz'), 499 | ]) 500 | 501 | def test_cursor_reversed(self): 502 | with self.db.cursor(True) as cursor: 503 | items = list(cursor) 504 | 505 | self.assertBEqual(items, [ 506 | ('zz', 'zzz'), 507 | ('gg', 'ggg'), 508 | ('ee', 'eee'), 509 | ('dd', 'ddd'), 510 | ('bbb', 'bbb'), 511 | ('bb', 'bbb'), 512 | ('aa', 'aaa'), 513 | ]) 514 | 515 | def test_seek_and_iterate(self): 516 | with self.db.cursor() as cursor: 517 | cursor.seek('dd', lsm.SEEK_GE) 518 | items = list(cursor) 519 | 520 | self.assertBEqual(items, [ 521 | ('dd', 'ddd'), 522 | ('ee', 'eee'), 523 | ('gg', 'ggg'), 524 | ('zz', 'zzz'), 525 | ]) 526 | 527 | with self.db.cursor() as cursor: 528 | cursor.seek('dd', lsm.SEEK_EQ) 529 | try: 530 | cursor.next() 531 | except Exception as exc: 532 | if (sys.version_info > (3,0)): 533 | self.assertEqual(exc.args[0], 'Misuse') 534 | else: 535 | self.assertEqual(exc.message, 'Misuse') 536 | else: 537 | raise AssertionError('Mis-use exception not raised.') 538 | 539 | def test_seek_and_iterate_reverse(self): 540 | with self.db.cursor(True) as cursor: 541 | cursor.seek('dd', lsm.SEEK_LE) 542 | items = list(cursor) 543 | 544 | self.assertBEqual(items, [ 545 | ('dd', 'ddd'), 546 | ('bbb', 'bbb'), 547 | ('bb', 'bbb'), 548 | ('aa', 'aaa'), 549 | ]) 550 | 551 | with self.db.cursor() as cursor: 552 | cursor.seek('dd', lsm.SEEK_EQ) 553 | try: 554 | cursor.previous() 555 | except Exception as exc: 556 | if (sys.version_info > (3,0)): 557 | self.assertEqual(exc.args[0], 'Misuse') 558 | else: 559 | self.assertEqual(exc.message, 'Misuse') 560 | else: 561 | raise AssertionError('Mis-use exception not raised.') 562 | 563 | def test_seek_missing(self): 564 | with self.db.cursor() as cursor: 565 | self.assertRaises(KeyError, cursor.seek, 'missing') 566 | 567 | with self.db.cursor(True) as cursor: 568 | self.assertRaises(KeyError, cursor.seek, 'missing') 569 | 570 | def test_seek_missing_for_iteration(self): 571 | with self.db.cursor() as cursor: 572 | cursor.seek('cccc', lsm.SEEK_GE) 573 | self.assertBEqual(cursor.key(), 'dd') 574 | self.assertBEqual(cursor.value(), 'ddd') 575 | 576 | items = [item for item in cursor] 577 | self.assertBEqual(items, [ 578 | ('dd', 'ddd'), 579 | ('ee', 'eee'), 580 | ('gg', 'ggg'), 581 | ('zz', 'zzz'), 582 | ]) 583 | 584 | with self.db.cursor(True) as cursor: 585 | cursor.seek('cccc', lsm.SEEK_LE) 586 | self.assertBEqual(cursor.key(), 'bbb') 587 | self.assertBEqual(cursor.value(), 'bbb') 588 | 589 | items = [item for item in cursor] 590 | self.assertBEqual(items, [ 591 | ('bbb', 'bbb'), 592 | ('bb', 'bbb'), 593 | ('aa', 'aaa'), 594 | ]) 595 | 596 | def test_fetch_until(self): 597 | with self.db.cursor() as cursor: 598 | cursor.seek('bbb', lsm.SEEK_GE) 599 | items = [item for item in cursor.fetch_until('ee')] 600 | 601 | self.assertBEqual(items, [ 602 | ('bbb', 'bbb'), 603 | ('dd', 'ddd'), 604 | ('ee', 'eee'), 605 | ]) 606 | 607 | # Invalid end key. 608 | with self.db.cursor() as cursor: 609 | cursor.seek('bbb', lsm.SEEK_GE) 610 | items = [item for item in cursor.fetch_until('ef')] 611 | 612 | self.assertBEqual(items, [ 613 | ('bbb', 'bbb'), 614 | ('dd', 'ddd'), 615 | ('ee', 'eee'), 616 | ]) 617 | 618 | # Invalid start key. 619 | with self.db.cursor() as cursor: 620 | cursor.seek('cccc', lsm.SEEK_GE) 621 | items = [item for item in cursor.fetch_until('foo')] 622 | 623 | self.assertBEqual(items, [ 624 | ('dd', 'ddd'), 625 | ('ee', 'eee'), 626 | ]) 627 | 628 | # Start key precedes lowest key. 629 | with self.db.cursor() as cursor: 630 | cursor.seek('a', lsm.SEEK_GE) 631 | items = [item for item in cursor.fetch_until('bx')] 632 | 633 | self.assertBEqual(items, [ 634 | ('aa', 'aaa'), 635 | ('bb', 'bbb'), 636 | ('bbb', 'bbb'), 637 | ]) 638 | 639 | # End with key that exceeds highest key. 640 | with self.db.cursor() as cursor: 641 | cursor.seek('dd', lsm.SEEK_GE) 642 | items = [item for item in cursor.fetch_until('zzzzzz')] 643 | 644 | self.assertBEqual(items, [ 645 | ('dd', 'ddd'), 646 | ('ee', 'eee'), 647 | ('gg', 'ggg'), 648 | ('zz', 'zzz'), 649 | ]) 650 | 651 | def test_fetch_range(self): 652 | with self.db.cursor() as cursor: 653 | items = [item for item in cursor.fetch_range('bb', 'ee')] 654 | 655 | self.assertBEqual(items, [ 656 | ('bb', 'bbb'), 657 | ('bbb', 'bbb'), 658 | ('dd', 'ddd'), 659 | ('ee', 'eee'), 660 | ]) 661 | 662 | with self.db.cursor() as cursor: 663 | items = [item for item in cursor.fetch_range('a', 'cc')] 664 | 665 | self.assertBEqual(items, [ 666 | ('aa', 'aaa'), 667 | ('bb', 'bbb'), 668 | ('bbb', 'bbb'), 669 | ]) 670 | 671 | with self.db.cursor() as cursor: 672 | items = [item for item in cursor.fetch_range('foo', 'zzzz')] 673 | 674 | self.assertBEqual(items, [ 675 | ('gg', 'ggg'), 676 | ('zz', 'zzz'), 677 | ]) 678 | 679 | with self.db.cursor() as cursor: 680 | items = [item for item in cursor.fetch_range('zzzz', 'zzzzz')] 681 | 682 | self.assertEqual(items, []) 683 | 684 | with self.db.cursor() as cursor: 685 | items = [item for item in cursor.fetch_range('a', 'aA')] 686 | 687 | self.assertEqual(items, []) 688 | 689 | with self.db.cursor() as cursor: 690 | items = [item for item in cursor.fetch_range('eee', 'ba')] 691 | 692 | self.assertBEqual(items, [ 693 | ('bb', 'bbb'), 694 | ('bbb', 'bbb'), 695 | ('dd', 'ddd'), 696 | ('ee', 'eee'), 697 | ]) 698 | 699 | def test_fetch_range_reverse(self): 700 | with self.db.cursor(True) as cursor: 701 | items = [item for item in cursor.fetch_range('ee', 'bb')] 702 | 703 | self.assertBEqual(items, [ 704 | ('ee', 'eee'), 705 | ('dd', 'ddd'), 706 | ('bbb', 'bbb'), 707 | ('bb', 'bbb'), 708 | ]) 709 | 710 | with self.db.cursor(True) as cursor: 711 | items = [item for item in cursor.fetch_range('cc', 'a')] 712 | 713 | self.assertBEqual(items, [ 714 | ('bbb', 'bbb'), 715 | ('bb', 'bbb'), 716 | ('aa', 'aaa'), 717 | ]) 718 | 719 | with self.db.cursor(True) as cursor: 720 | items = [item for item in cursor.fetch_range('zzzz', 'foo')] 721 | 722 | self.assertBEqual(items, [ 723 | ('zz', 'zzz'), 724 | ('gg', 'ggg'), 725 | ]) 726 | 727 | with self.db.cursor(True) as cursor: 728 | items = [item for item in cursor.fetch_range('zzzzz', 'zzzz')] 729 | 730 | self.assertEqual(items, []) 731 | 732 | with self.db.cursor(True) as cursor: 733 | items = [item for item in cursor.fetch_range('aA', 'a')] 734 | 735 | self.assertEqual(items, []) 736 | 737 | with self.db.cursor(True) as cursor: 738 | items = [item for item in cursor.fetch_range('ba', 'eee')] 739 | 740 | self.assertBEqual(items, [ 741 | ('ee', 'eee'), 742 | ('dd', 'ddd'), 743 | ('bbb', 'bbb'), 744 | ('bb', 'bbb'), 745 | ]) 746 | 747 | def test_cursor_consumed(self): 748 | for reverse in (False, True): 749 | with self.db.cursor(reverse=reverse) as cursor: 750 | l1 = [item for item in cursor] 751 | l2 = [item for item in cursor] 752 | if reverse: 753 | cursor.last() 754 | else: 755 | cursor.first() 756 | k1 = [key for key in cursor.keys()] 757 | k2 = [key for key in cursor.keys()] 758 | if reverse: 759 | cursor.last() 760 | else: 761 | cursor.first() 762 | v1 = [value for value in cursor.values()] 763 | v2 = [value for value in cursor.values()] 764 | 765 | self.assertTrue(len(l1) == 7) 766 | self.assertEqual(l2, []) 767 | self.assertTrue(len(k1) == 7) 768 | self.assertEqual(k2, []) 769 | self.assertTrue(len(v1) == 7) 770 | self.assertEqual(v2, []) 771 | 772 | def test_keys_and_values(self): 773 | t_keys = ['aa', 'bb', 'bbb', 'dd', 'ee', 'gg', 'zz'] 774 | t_values = ['aaa', 'bbb', 'bbb', 'ddd', 'eee', 'ggg', 'zzz'] 775 | 776 | with self.db.cursor() as cursor: 777 | keys = [key for key in cursor.keys()] 778 | cursor.first() 779 | values = [value for value in cursor.values()] 780 | 781 | self.assertBEqual(keys, t_keys) 782 | self.assertBEqual(values, t_values) 783 | 784 | with self.db.cursor(True) as cursor: 785 | keys = [key for key in cursor.keys()] 786 | cursor.last() 787 | values = [value for value in cursor.values()] 788 | 789 | self.assertBEqual(keys, list(reversed(t_keys))) 790 | self.assertBEqual(values, list(reversed(t_values))) 791 | 792 | 793 | class TestLSMOptions(BaseTestLSM): 794 | def test_no_open(self): 795 | db = lsm.LSM('test.lsm', open_database=False) 796 | self.assertFalse(db.is_open) 797 | self.assertFalse(os.path.exists('test.lsm')) 798 | 799 | def test_default_options(self): 800 | self.assertEqual(self.db.page_size, 4096) 801 | self.assertEqual(self.db.block_size, 1024) 802 | self.assertEqual(self.db.multiple_processes, 1) 803 | self.assertEqual(self.db.readonly, 0) 804 | self.assertEqual(self.db.write_safety, 1) 805 | self.assertEqual(self.db.autoflush, 1024) 806 | self.assertEqual(self.db.autowork, 1) 807 | self.assertEqual(self.db.automerge, 4) 808 | self.assertEqual(self.db.autocheckpoint, 2048) 809 | #self.assertTrue(self.db.mmap in (0, 1)) 810 | self.assertEqual(self.db.transaction_log, 1) 811 | 812 | def test_file_options(self): 813 | self.db.close() 814 | os.unlink(self.filename) 815 | 816 | db = lsm.LSM(self.filename, page_size=1024, block_size=4096) 817 | self.assertEqual(db.page_size, 1024) 818 | self.assertEqual(db.block_size, 4096) 819 | 820 | # Page and block cannot be modified after creation. 821 | def set_page(): 822 | db.page_size = 8192 823 | def set_block(): 824 | db.block_size = 8192 825 | self.assertRaises(ValueError, set_page) 826 | self.assertRaises(ValueError, set_block) 827 | 828 | # We can, however, alter the safety level at any time. 829 | self.assertEqual(db.write_safety, lsm.SAFETY_NORMAL) 830 | db.write_safety = lsm.SAFETY_FULL 831 | self.assertEqual(db.write_safety, lsm.SAFETY_FULL) 832 | 833 | for i in range(10): 834 | db['k%s' % i] = 'v%s' % i 835 | 836 | self.assertBEqual(db['k0'], 'v0') 837 | self.assertBEqual(db['k9'], 'v9') 838 | db.close() 839 | 840 | db2 = lsm.LSM(self.filename, page_size=1024, block_size=4096, 841 | mmap=0, transaction_log=False, write_safety=0, 842 | multiple_processes=False) 843 | self.assertEqual(db2.page_size, 1024) 844 | self.assertEqual(db2.block_size, 4096) 845 | self.assertEqual(db2.mmap, 0) 846 | self.assertEqual(db2.transaction_log, 0) 847 | self.assertEqual(db2.write_safety, 0) 848 | self.assertEqual(db2.multiple_processes, 0) 849 | self.assertBEqual(db2['k0'], 'v0') 850 | self.assertBEqual(db2['k9'], 'v9') 851 | db2.close() 852 | 853 | def test_multithreading(self): 854 | def create_entries_thread(low, high): 855 | for i in range(low, high): 856 | self.db['k%02d' % i] = 'v%s' % i 857 | 858 | threads = [] 859 | for i in range(8): 860 | threads.append(threading.Thread( 861 | target=create_entries_thread, 862 | args=(i * 10, i * 10 + 10))) 863 | 864 | [t.start() for t in threads] 865 | [t.join() for t in threads] 866 | 867 | keys = [key for key in self.db.keys()] 868 | self.assertEqual(len(keys), 80) 869 | self.assertBEqual(keys[0], 'k00') 870 | self.assertBEqual(keys[-1], 'k79') 871 | 872 | expected = ['k%02d' % i for i in range(80)] 873 | self.assertBEqual(keys, expected) 874 | 875 | 876 | class TestLSMInfo(BaseTestLSM): 877 | def test_lsm_info(self): 878 | self.db.close() 879 | 880 | # Page size is 1KB. 881 | db = lsm.LSM(self.filename, page_size=1024, autocheckpoint=4) 882 | 883 | w0 = db.pages_written() 884 | r0 = db.pages_read() 885 | c0 = db.checkpoint_size() 886 | self.assertEqual(w0, 0) 887 | self.assertEqual(r0, 0) 888 | self.assertEqual(c0, 0) 889 | 890 | data = '0' * 1024 891 | for i in range(1024): 892 | db[str(i)] = data 893 | r = db[str(i)] 894 | 895 | w1 = db.pages_written() 896 | r1 = db.pages_read() 897 | c1 = db.checkpoint_size() 898 | self.assertEqual(w1, 974) 899 | self.assertEqual(r1, 0) # Not sure why not increasing... 900 | self.assertEqual(c1, 0) 901 | 902 | 903 | if __name__ == '__main__': 904 | unittest.main(argv=sys.argv) 905 | --------------------------------------------------------------------------------