├── README.md ├── kvkit ├── __init__.py ├── backends │ ├── __init__.py │ ├── berkeleydb.py │ ├── forest.py │ ├── helpers.py │ ├── kyoto.py │ ├── leveldb.py │ ├── rocks.py │ ├── sophia.py │ └── sqlite4.py ├── exceptions.py ├── graph.py ├── query.py └── tests.py ├── runtests.py └── setup.py /README.md: -------------------------------------------------------------------------------- 1 | # kvkit 2 | 3 | High-level Python toolkit for ordered key/value stores. 4 | 5 | Supports: 6 | 7 | * [BerkeleyDB](http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/downloads/index.html) via [bsddb3](https://www.jcea.es/programacion/pybsddb_doc/). 8 | * [KyotoCabinet](http://fallabs.com/kyotocabinet/) via [Python 2.x bindings](http://fallabs.com/kyotocabinet/pythonlegacydoc/). 9 | * [LevelDB](http://leveldb.org/) via [plyvel](https://plyvel.readthedocs.io/en/latest/). 10 | * [RocksDB](http://rocksdb.org/) via [pyrocksdb](https://pyrocksdb.readthedocs.io/en/v0.4/) 11 | * [Sqlite4 LSM DB](https://www.sqlite.org/src4/doc/trunk/www/lsmusr.wiki) via [python-lsm-db](https://lsm-db.readthedocs.io/en/latest/) 12 | 13 | Right now KyotoCabinet is the most well-supported database, but the SQLite4 LSM is also pretty robust. The other databases implement the minimal slicing APIs to enable the Model/Secondary Indexing APIs to work. 14 | 15 | This project should be considered **experimental**. 16 | 17 | ### Features 18 | 19 | * Store structured data models. 20 | * Secondary indexes and arbitrarily complex querying. 21 | * Graph database (Hexastore) with search pipeline. 22 | * High-level slicing APIs. 23 | 24 | ### Models 25 | 26 | `kvkit` provides a lightweight structured data model API. Individual fields on the model can be optionally typed, and also support secondary indexes. 27 | 28 | Field types: 29 | 30 | * `Field()`: simplest field type, treated as raw bytes. 31 | * `DateTimeField()`: store Python `datetime` objects. 32 | * `DateField()`: store Python `date` objects. 33 | * `LongField()`: store Python `int` and `long`. Values are encoded as an 8 byte `long long`, big-endian. 34 | * `FloatField()`: store Python `float`. Values are encoded as an 8 byte double-precision float, big-endian. 35 | 36 | A `Model` is composed of one or more fields, in addition to a required `id` field which stores an automatically-generated integer ID. 37 | 38 | `Model` classes are defined declaratively, a-la many popular Python ORMs: 39 | 40 | ```python 41 | 42 | # KyotoCabinet on-disk B-tree. 43 | db = TreeDB('address_book.kct') 44 | 45 | # Create a base model-class pointing at our db. 46 | class BaseModel(Model): 47 | class Meta: 48 | database = db 49 | 50 | class Contact(BaseModel): 51 | last_name = Field(index=True) 52 | first_name = Field(index=True) 53 | 54 | 55 | class PhoneNumber(BaseModel): 56 | contact_id = LongField(index=True) 57 | phone_number = Field() 58 | 59 | def get_contact(self): 60 | return Contact.load(self.contact_id) 61 | 62 | 63 | class Address(BaseModel): 64 | contact_id = LongField(index=True) 65 | street = Field() 66 | city = Field(index=True) 67 | state = Field(index=True) 68 | postal_code = Field() 69 | 70 | def get_contact(self): 71 | return Contact.load(self.contact_id) 72 | ``` 73 | 74 | To create a new contact and add a phone number for them, we might write: 75 | 76 | ```python 77 | 78 | huey = Contact.create( 79 | last_name='Leifer', 80 | first_name='Huey', 81 | dob=datetime.date(2011, 5, 1)) 82 | 83 | phone = PhoneNumber.create( 84 | contact_id=huey.id, 85 | phone_number='555-1234') 86 | ``` 87 | 88 | Let's say we need to look up Huey's phone number(s). We might write: 89 | 90 | ```python 91 | huey = Contact.get(Contact.first_name == 'Huey') 92 | phones = PhoneNumber.query(PhoneNumber.contact_id == huey.id) 93 | for phone in phones: 94 | print phone.phone_number 95 | ``` 96 | 97 | If there were more than one person named "Huey" in our database, we could be more specific by specifying additional query clauses: 98 | 99 | ```python 100 | 101 | huey_leifer = Contact.get( 102 | (Contact.first_name == 'Huey') & 103 | (Contact.last_name == 'Leifer')) 104 | ``` 105 | 106 | To query all contacts whose last name begins with "Le" we could write: 107 | 108 | ```python 109 | 110 | Contact.query(Contact.last_name.startswith('Le')) 111 | ``` 112 | 113 | If we wanted to express a range, such as "Le" -> "Mo", we could write: 114 | 115 | ```python 116 | 117 | Contact.query( 118 | (Contact.last_name >= 'Le') & 119 | (Contact.last_name <= 'Mo')) 120 | ``` 121 | 122 | Fields can be queried using the following operations: 123 | 124 | * `==` for equality 125 | * `<` and `<=` 126 | * `>` and `>=` 127 | * `!=` for inequality 128 | * `.startswith()` for prefix search 129 | 130 | Multiple clauses can be combined using set operations: 131 | 132 | * `&` for AND (intersection) 133 | * `|` for OR (union) 134 | 135 | ### Graph database (Hexastore) 136 | 137 | The graph database is based on an idea described in the Redis [secondary indexing documentation](http://redis.io/topics/indexes#representing-and-querying-graphs-using-an-hexastore). The idea is that the database will store triples of `subject`, `predicate` and `object`. These can be any application-specific values. For example, I might want to store my friends and some information about them: 138 | 139 | ```python 140 | 141 | db = CacheTreeDB() # KyotoCabinet in-memory B-tree 142 | graph = Hexastore(db) 143 | 144 | data = ( 145 | ('charlie', 'friends', 'huey'), 146 | ('charlie', 'friends', 'mickey'), 147 | ('charlie', 'friends', 'zaizee'), 148 | ('huey', 'friends', 'charlie'), 149 | ('huey', 'friends', 'zaizee'), 150 | ('zaizee', 'friends', 'huey'), 151 | ('charlie', 'lives', 'KS'), 152 | ('huey', 'lives', 'KS'), 153 | ('mickey', 'lives', 'KS'), 154 | ('zaizee', 'lives', 'MO'), 155 | ) 156 | graph.store_many(data) 157 | ``` 158 | 159 | To do a simple query asking who my friends are, I can write: 160 | 161 | ```python 162 | 163 | for result in graph.query(s='charlie', p='friends'): 164 | print result['o'] 165 | 166 | # prints huey, mickey, zaizee 167 | ``` 168 | 169 | I can also ask for other things, like all the people who live in Kansas: 170 | 171 | ```python 172 | 173 | for result in graph.query(p='lives', o='KS'): 174 | print result['s'] 175 | 176 | # prints charlie, huey, mickey 177 | ``` 178 | 179 | Things get especially interesting when you construct a pipeline using variables. Let's get all of my friends who live in Kansas: 180 | 181 | ```python 182 | 183 | X = graph.v.X # Create a variable reference. 184 | results = graph.search( 185 | ('charlie', 'friends', X), 186 | (X, 'lives', 'KS')) 187 | print results['X'] 188 | 189 | # prints set(['huey', 'mickey']) 190 | ``` 191 | 192 | In this query, we will use two variables, and answer the question "Who has friends who live in Missouri?" 193 | 194 | ```python 195 | 196 | X = graph.v.X 197 | Y = graph.v.Y 198 | results = graph.search( 199 | (X, 'lives', 'MO'), 200 | (Y, 'friends', X)) 201 | print results['Y'] 202 | 203 | # prints set(['charlie', 'huey']) 204 | # charlie and huey are friends with zaizee, who lives in MO. 205 | ``` 206 | 207 | ### Unified Slicing API 208 | 209 | `kvkit` provides unified indexing and slicing APIs. Slices obey the following rules: 210 | 211 | * Inclusive of both endpoints. 212 | * If the start key does not exist, the next-highest key will be used, if one exists. 213 | * If the end key does not exist, the next-lowest key will be used, if one exists. 214 | * Supports efficient iteration forwards or backwards. 215 | 216 | ```pycon 217 | 218 | >>> from kvkit import CacheTreeDB # KyotoCabinet in-memory B-tree 219 | >>> db = CacheTreeDB() 220 | 221 | >>> # Populate some data. 222 | >>> for key in ['aa', 'aa1', 'aa2', 'bb', 'cc', 'dd', 'ee']: 223 | ... db[key] = key 224 | ... 225 | 226 | >>> list(db['aa':'cc']) 227 | [('aa', 'aa'), ('aa1', 'aa1'), ('aa2', 'aa2'), ('bb', 'bb'), ('cc', 'cc')] 228 | 229 | >>> list(db['aa0':'cc2']) # Example where start & end do not exist. 230 | [('aa1', 'aa1'), ('aa2', 'aa2'), ('bb', 'bb'), ('cc', 'cc')] 231 | ``` 232 | 233 | In addition to slicing, all databases implement the following dictionary-like methods: 234 | 235 | * `update()` 236 | * `keys()` 237 | * `values()` 238 | * `items()` 239 | * `__setitem__` and `__delitem__` 240 | * `__iter__` 241 | 242 | All databases also implement: 243 | 244 | * `incr()` 245 | * `decr()` 246 | * `open()` 247 | * `close()` 248 | 249 | ### Installation 250 | 251 | `kvkit` can be installed from PyPI: 252 | 253 | ```console 254 | $ pip install kvkit 255 | ``` 256 | -------------------------------------------------------------------------------- /kvkit/__init__.py: -------------------------------------------------------------------------------- 1 | from kvkit.exceptions import DatabaseError 2 | 3 | try: 4 | from kvkit.backends.berkeleydb import BerkeleyDB 5 | except ImportError: 6 | pass 7 | 8 | try: 9 | from kvkit.backends.kyoto import CacheHashDB 10 | from kvkit.backends.kyoto import CacheTreeDB 11 | from kvkit.backends.kyoto import DirectoryHashDB 12 | from kvkit.backends.kyoto import DirectoryTreeDB 13 | from kvkit.backends.kyoto import HashDB 14 | from kvkit.backends.kyoto import PlainTextDB 15 | from kvkit.backends.kyoto import PrototypeHashDB 16 | from kvkit.backends.kyoto import PrototypeTreeDB 17 | from kvkit.backends.kyoto import StashDB 18 | from kvkit.backends.kyoto import TreeDB 19 | except ImportError: 20 | pass 21 | 22 | try: 23 | from kvkit.backends.leveldb import LevelDB 24 | except ImportError: 25 | pass 26 | 27 | try: 28 | from kvkit.backends.rocks import RocksDB 29 | except ImportError: 30 | pass 31 | 32 | try: 33 | from kvkit.backends.sqlite4 import LSM 34 | except ImportError: 35 | pass 36 | 37 | from kvkit.graph import Hexastore 38 | from kvkit.query import DateField 39 | from kvkit.query import DateTimeField 40 | from kvkit.query import Field 41 | from kvkit.query import FloatField 42 | from kvkit.query import LongField 43 | from kvkit.query import Model 44 | 45 | 46 | __version__ = '0.1.2' 47 | -------------------------------------------------------------------------------- /kvkit/backends/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/coleifer/kvkit/a3551c6fc6bac73ee0a0ab098611d1c9f1b8d089/kvkit/backends/__init__.py -------------------------------------------------------------------------------- /kvkit/backends/berkeleydb.py: -------------------------------------------------------------------------------- 1 | # Requires bsddb3 package. 2 | import bsddb3 3 | from bsddb3.db import DBNotFoundError 4 | 5 | from kvkit.backends.helpers import KVHelper 6 | 7 | 8 | class BerkeleyDB(KVHelper, bsddb3._DBWithCursor): 9 | def __init__(self, filename, flag='c', mode=0o666, btflags=0, 10 | cache_size=None, maxkeypage=None, minkeypage=None, 11 | page_size=None, lorder=None): 12 | 13 | self.filename = filename 14 | flags = bsddb3._checkflag(flag, filename) 15 | env = bsddb3._openDBEnv(cache_size) 16 | db = bsddb3.db.DB(env) 17 | if page_size is not None: 18 | db.set_pagesize(page_size) 19 | if lorder is not None: 20 | db.set_lorder(lorder) 21 | db.set_flags(btflags) 22 | if minkeypage is not None: 23 | db.set_bt_minkey(minkeypage) 24 | if maxkeypage is not None: 25 | db.set_bt_maxkey(maxkeypage) 26 | db.open(self.filename, bsddb3.db.DB_BTREE, flags, mode) 27 | super(BerkeleyDB, self).__init__(db) 28 | 29 | def __getitem__(self, key): 30 | if isinstance(key, slice): 31 | if key.start > key.stop or key.step: 32 | return self.get_slice_rev(key.start, key.stop) 33 | else: 34 | return self.get_slice(key.start, key.stop) 35 | else: 36 | return super(BerkeleyDB, self).__getitem__(key) 37 | 38 | def get_slice(self, start, end): 39 | try: 40 | key, value = self.set_location(start) 41 | except DBNotFoundError: 42 | raise StopIteration 43 | else: 44 | if key > end: 45 | raise StopIteration 46 | yield key, value 47 | 48 | while True: 49 | try: 50 | key, value = self.next() 51 | except DBNotFoundError: 52 | raise StopIteration 53 | else: 54 | if key > end: 55 | raise StopIteration 56 | yield key, value 57 | 58 | def get_slice_rev(self, start, end): 59 | if start is None or end is None: 60 | start, end = end, start 61 | 62 | if start is None: 63 | key, value = self.last() 64 | else: 65 | try: 66 | key, value = self.set_location(start) 67 | except DBNotFoundError: 68 | key, value = self.last() 69 | 70 | if start is None or key <= start: 71 | yield key, value 72 | 73 | while True: 74 | try: 75 | key, value = self.previous() 76 | except DBNotFoundError: 77 | raise StopIteration 78 | else: 79 | if key < end: 80 | raise StopIteration 81 | yield key, value 82 | -------------------------------------------------------------------------------- /kvkit/backends/forest.py: -------------------------------------------------------------------------------- 1 | import struct 2 | 3 | from forestdb import ForestDB as _ForestDB 4 | 5 | 6 | class ForestDB(_ForestDB): 7 | def __init__(self, filename): 8 | super(ForestDB, self).__init__(filename) 9 | self._kv = self.kv('default') 10 | 11 | def __getitem__(self, key): 12 | return self._kv[key] 13 | 14 | def __setitem__(self, key, value): 15 | self._kv[key] = value 16 | 17 | def __delitem__(self, key): 18 | del self._kv[key] 19 | 20 | def __contains__(self, key): 21 | return key in self._kv 22 | 23 | def update(self, _data_dict=None, **data): 24 | return self._kv.update(_data_dict, **data) 25 | 26 | def __len__(self): 27 | return len(self._kv) 28 | 29 | def cursor(self, *args, **kwargs): 30 | return self._kv.cursor(*args, **kwargs) 31 | 32 | def keys(self): 33 | return self._kv.keys() 34 | 35 | def values(self): 36 | return self._kv.values() 37 | 38 | def __iter__(self): 39 | return iter(self._kv) 40 | 41 | def __enter__(self): 42 | self.open() 43 | return self 44 | 45 | def __exit__(self, exc_type, exc_val, exc_tb): 46 | self.close() 47 | 48 | def incr(self, key, amount=1): 49 | try: 50 | value = self[key] 51 | except KeyError: 52 | value = amount 53 | else: 54 | value = struct.unpack('>q', value)[0] + amount 55 | self[key] = struct.pack('>q', value) 56 | return value 57 | 58 | def decr(self, key, amount=1): 59 | return self.incr(key, amount * -1) 60 | -------------------------------------------------------------------------------- /kvkit/backends/helpers.py: -------------------------------------------------------------------------------- 1 | import contextlib 2 | import struct 3 | 4 | 5 | def clean_key_slice(key): 6 | start = key.start 7 | stop = key.stop 8 | reverse = key.step 9 | first = start is None 10 | last = stop is None 11 | one_empty = (first and not last) or (last and not first) 12 | none_empty = not first and not last 13 | if reverse: 14 | if one_empty: 15 | start, stop = stop, start 16 | if none_empty and (start < stop): 17 | start, stop = stop, start 18 | if none_empty and start > stop: 19 | reverse = True 20 | return start, stop, reverse 21 | 22 | 23 | class KVHelper(object): 24 | def __enter__(self): 25 | self.open() 26 | return self 27 | 28 | def __exit__(self, exc_type, exc_val, exc_tb): 29 | self.close() 30 | 31 | def incr(self, key, amount=1): 32 | try: 33 | value = self[key] 34 | except KeyError: 35 | value = amount 36 | else: 37 | value = struct.unpack('>q', value)[0] + amount 38 | self[key] = struct.pack('>q', value) 39 | return value 40 | 41 | def decr(self, key, amount=1): 42 | return self.incr(key, amount * -1) 43 | 44 | @contextlib.contextmanager 45 | def transaction(self): 46 | yield 47 | -------------------------------------------------------------------------------- /kvkit/backends/kyoto.py: -------------------------------------------------------------------------------- 1 | # Requires kyotocabinet Python legacy bindings. 2 | import operator 3 | import os 4 | import struct 5 | 6 | import kyotocabinet as kc 7 | 8 | from kvkit.exceptions import DatabaseError 9 | from kvkit.backends.helpers import clean_key_slice 10 | 11 | 12 | # Generic modes. 13 | EXCEPTIONAL = kc.DB.GEXCEPTIONAL 14 | CONCURRENT = kc.DB.GCONCURRENT 15 | 16 | # Open options. 17 | READER = kc.DB.OREADER 18 | WRITER = kc.DB.OWRITER 19 | CREATE = kc.DB.OCREATE 20 | TRUNCATE = kc.DB.OTRUNCATE 21 | AUTOCOMMIT = kc.DB.OAUTOTRAN 22 | AUTOSYNC = kc.DB.OAUTOSYNC 23 | NOLOCK = kc.DB.ONOLOCK # Open without locking. 24 | TRYLOCK = kc.DB.OTRYLOCK # Open and lock without blocking. 25 | NOREPAIR = kc.DB.ONOREPAIR 26 | 27 | # Merge options. 28 | MERGE_OVERWRITE = kc.DB.MSET # Overwrite existing values. 29 | MERGE_PRESERVE = kc.DB.MADD # Keep existing values. 30 | MERGE_REPLACE = kc.DB.MREPLACE # Modify existing records only. 31 | MERGE_APPEND = kc.DB.MAPPEND # Append new values. 32 | 33 | # Special filenames 34 | DB_PROTOTYPE_HASH = '-' 35 | DB_PROTOTYPE_TREE = '+' 36 | DB_STASH = ':' 37 | DB_CACHE_HASH = '*' 38 | DB_CACHE_TREE = '%' 39 | 40 | # Special extensions 41 | DB_FILE_HASH = '.kch' 42 | DB_FILE_TREE = '.kct' 43 | DB_DIRECTORY_HASH = '.kcd' 44 | DB_DIRECTORY_TREE = '.kcf' 45 | DB_TEXT = '.kcx' 46 | 47 | NOP = kc.Visitor.NOP 48 | 49 | # Tuning parameters 50 | 51 | # Log parameters are supported by all databases. 52 | P_LOG = 'log' # Path to log file. 53 | P_LOGKINDS = 'logkinds' # debug, info, warn or error. 54 | P_LOGPX = 'logpx' # prefix for each log message. 55 | 56 | P_BUCKETS = 'bnum' # Buckets in hash table. 57 | 58 | P_CAPCNT = 'capcnt' # Sets the capacity by record number. 59 | P_CAPSIZ = 'capsiz' # Sets the capacity by memory usage. 60 | 61 | P_SIZE = 'psiz' # page size. 62 | P_PCCAP = 'pccap' # tune page cache. 63 | 64 | # Reduce memory at the expense of time. Linear means use linear linked-list 65 | # for hash collisions, saves 6 bytes/record. Compression should only be used 66 | # for values > 1KB. 67 | P_OPTS = 'opts' # s (small), l (linear), c (compress) 68 | 69 | P_ZCOMP = 'zcomp' # zlib, def (deflate), gz, lzo, lzma, arc 70 | P_ZKEY = 'zkey' # cipher key of compressor 71 | P_RCOMP = 'rcomp' # comp fn: lex, dec (decimal), lexdesc, decdesc 72 | P_APOW = 'apow' # alignment 73 | P_FPOW = 'fpow' # tune_fbp 74 | P_MSIZ = 'msiz' # tune map 75 | P_DFUNIT = 'dfunit' # tune defrag. 76 | 77 | """ 78 | All databases: log, logkinds, logpx 79 | Stash: bnum 80 | Cache hash: opts, bnum, zcomp, capcnt, capsiz, zkey 81 | Cache tree: opts, bnum, zcomp, zkey, psiz, rcomp, pccap 82 | File hash: apow, fpow, opts, bnum, msiz, dfunit, zcomp, zkey 83 | File tree: apow, fpow, opts, bnum, msiz, dfunit, zcomp, zkey, psiz, rcomp, 84 | pccap 85 | Dir hash: opts, zcomp, zkey. 86 | Dir tree: opts, zcomp, zkey, psiz, rcomp, pccap. 87 | Plain: n/a 88 | 89 | StashDB 90 | ------- 91 | bnum: default is ~1M. Should be 80% - 400% to total records. Collision 92 | chaining is linear linked list search. 93 | 94 | CacheHashDB 95 | ----------- 96 | bnum: default ~1M. Should be 50% - 400% of total records. Collision chaining 97 | is binary search. 98 | opts: useful to reduce memory at expense of time effciency. Use compression 99 | if the key and value of each record is > 1KB. 100 | cap_count and/or cap_size: used to keep memory usage constant by expiring 101 | old records. 102 | 103 | CacheTreeDB 104 | ----------- 105 | Inherits all tuning options from the CacheHashDB, since each node of the btree 106 | is serialized as a page-buffer and treated as a record in the cache hash db. 107 | 108 | page size: default is 8192 109 | page cache: default is 64MB 110 | comparator: default is lexical ordering 111 | 112 | HashDB 113 | ------ 114 | 115 | bnum: default ~1M. Suggested ratio is twice the total number of records, but 116 | can be anything from 100% - 400%. 117 | apow: Power of the alignment of record size. Default=3, so the address of 118 | each record is aligned to a multiple of 8 (1<<3) bytes. 119 | fpow: Power of the capacity of the free block pool. Default=10, rarely needs 120 | to be modified. 121 | msiz: Size of internal memory-mapped region. Default is 64MB. 122 | dfunit: Unit step number of auto-defragmentation. Auto-defrag is disabled by 123 | default. 124 | 125 | apow, fpow, opts and bnum *must* be specified before a DB is opened and 126 | cannot be changed after the fact. 127 | 128 | TreeDB 129 | ------ 130 | 131 | Inherits tuning parameters from the HashDB. 132 | 133 | page size: default is 8192 134 | page cache: default is 64MB 135 | comparator: default is lexical 136 | 137 | The default alignment is 256 (1<<8) and the default bucket number is ~64K. 138 | The bucket number should be calculated by the number of pages. Suggested 139 | ratio of bucket number is 10% of the number of records. 140 | 141 | page size must be specified before the DB is opened and cannot be changed. 142 | """ 143 | 144 | 145 | class Database(object): 146 | default_flags = kc.DB.OWRITER | kc.DB.OCREATE 147 | extension = None 148 | 149 | def __init__(self, filename, exceptional=False, concurrent=False, 150 | open_database=True, **opts): 151 | config = 0 152 | if exceptional: 153 | config |= EXCEPTIONAL 154 | if concurrent: 155 | config |= CONCURRENT 156 | 157 | if opts: 158 | opt_str = '#'.join([ 159 | '%s=%s' % (key, value) for key, value in opts.items()]) 160 | filename = '#'.join((filename, opt_str)) 161 | 162 | self.filename = filename 163 | self._config = config 164 | if self.extension and not self.filename.endswith(self.extension): 165 | self.filename = '%s%s' % (self.filename, self.extension) 166 | self.db = kc.DB(self._config) 167 | self._closed = True 168 | if open_database: 169 | self.open() 170 | 171 | def open(self, flags=None): 172 | if not self._closed: 173 | self.close() 174 | if flags is None: 175 | flags = self.default_flags 176 | if not self.db.open(self.filename, flags): 177 | raise DatabaseError(self.db.error()) 178 | return True 179 | 180 | def close(self): 181 | if not self.db.close(): 182 | raise DatabaseError(self.db.error()) 183 | self._closed = True 184 | return True 185 | 186 | def __enter__(self): 187 | self.open() 188 | return self 189 | 190 | def __exit__(self, exc_type, exc_val, exc_tb): 191 | try: 192 | self.close() 193 | except DatabaseError: 194 | pass 195 | 196 | def __setitem__(self, key, value): 197 | self.db.set(key, value) 198 | 199 | def __getitem__(self, key): 200 | if isinstance(key, (list, tuple)): 201 | return self.db.get_bulk(key, True) 202 | elif isinstance(key, slice): 203 | start, stop, reverse = clean_key_slice(key) 204 | if reverse: 205 | return self.get_slice_rev(start, stop) 206 | else: 207 | return self.get_slice(start, stop) 208 | else: 209 | value = self.db.get(key) 210 | if value is None: 211 | raise KeyError(key) 212 | return value 213 | 214 | def __delitem__(self, key): 215 | if isinstance(key, (list, tuple)): 216 | self.db.remove_bulk(key, True) 217 | elif isinstance(key, slice): 218 | pass 219 | else: 220 | self.db.remove(key) 221 | 222 | def __contains__(self, key): 223 | return self.db.check(key) != -1 224 | 225 | def __len__(self): 226 | return self.db.count() 227 | 228 | def update(self, _data=None, **kwargs): 229 | """ 230 | Update multiple records atomically. Returns the number of records 231 | updated, raising a `DatabaseError` on error. 232 | """ 233 | if _data: 234 | if kwargs: 235 | _data.update(kwargs) 236 | ret = self.db.set_bulk(_data, True) 237 | else: 238 | ret = self.db.set_bulk(kwargs, True) 239 | if ret < 0: 240 | raise DatabaseError('Error updating records: %s' % self.db.error()) 241 | return ret 242 | 243 | def pop(self, key=None): 244 | """ 245 | Remove the first record, or the record specified by the given key, 246 | returning the value. 247 | """ 248 | if key: 249 | ret = self.db.seize(key) 250 | else: 251 | ret = self.db.shift() 252 | 253 | if ret is None: 254 | raise KeyError(key) 255 | 256 | return ret 257 | 258 | def clear(self): 259 | """Remove all records, returning `True` on success.""" 260 | return self.db.clear() 261 | 262 | def flush(self, hard=True): 263 | """Synchronize to disk.""" 264 | return self.db.synchronize(hard) 265 | 266 | def add(self, key, value): 267 | """ 268 | Add the key/value pair to the database. If the key already exists, then 269 | no change is made to the existing value. 270 | 271 | Returns boolean indicating whether value was added. 272 | """ 273 | return self.db.add(key, value) 274 | 275 | def replace(self, key, value): 276 | """ 277 | Replace the value at the given key. If the key does not exist, then 278 | no change is made. 279 | 280 | Returns boolean indicating whether value was replaced. 281 | """ 282 | return self.db.replace(key, value) 283 | 284 | def append(self, key, value): 285 | """ 286 | Append the value to a pre-existing value at the given key. If no 287 | value exists, this is equivalent to set. 288 | """ 289 | return self.db.append(key, value) 290 | 291 | def cas(self, key, old, new): 292 | """ 293 | Conditionally set the new value for the given key, but only if the 294 | pre-existing value at the key equals `old`. 295 | 296 | Returns boolean indicating if the value was swapped. 297 | """ 298 | return self.db.cas(key, old, new) 299 | 300 | def copy_to_file(self, dest): 301 | """ 302 | Create a copy of the database, returns boolean indicating success. 303 | """ 304 | return self.db.copy(dest) 305 | 306 | def begin(self, hard=False): 307 | """ 308 | Begin a transaction. If `hard=True`, then the operation will be 309 | physically synchronized with the device. 310 | 311 | Returns boolean indicating success. 312 | """ 313 | return self.db.begin_transaction(hard) 314 | 315 | def commit(self): 316 | """ 317 | Commit a transaction. Returns boolean indicating success. 318 | """ 319 | return self.db.end_transaction(True) 320 | 321 | def rollback(self): 322 | """ 323 | Rollback a transaction. Returns boolean indicating success. 324 | """ 325 | return self.db.end_transaction(False) 326 | 327 | def transaction(self): 328 | return transaction(self) 329 | 330 | def match_prefix(self, prefix, max_records=-1): 331 | return self.db.match_prefix(prefix, max_records) 332 | 333 | def match_regex(self, regex, max_records=-1): 334 | return self.db.match_regex(regex, max_records) 335 | 336 | def match(self, query, acceptable_distance=1, utf8=False, max_records=-1): 337 | return self.db.match_similar( 338 | query, 339 | acceptable_distance, 340 | utf8, 341 | max_records) 342 | 343 | def lock(self, writable=False, processor=None): 344 | return self.db.occupy(writable, processor) 345 | 346 | def incr(self, key, n=1, initial=0): 347 | return self.db.increment(key, n, initial) 348 | 349 | def decr(self, key, n=1, initial=0): 350 | return self.db.increment(key, n * -1, initial) 351 | 352 | def cursor(self, reverse=False): 353 | return Cursor(self.db.cursor(), reverse) 354 | 355 | def atomic(self, hard=False): 356 | """ 357 | Perform transaction via function `fn`. 358 | """ 359 | def decorator(fn): 360 | def fn_wrapper(): 361 | result = fn() 362 | if result is False: 363 | return False 364 | return True 365 | 366 | def inner(): 367 | return self.db.transaction(fn_wrapper) 368 | 369 | return inner 370 | return decorator 371 | 372 | def process(self, fn): 373 | """ 374 | Process database using a function. The function should accept 375 | a key and value. 376 | """ 377 | return self.db.iterate(fn) 378 | 379 | def cursor_process(self, fn): 380 | """ 381 | Traverse records by cursor, using the given function. 382 | """ 383 | return self.db.cursor_process(fn) 384 | 385 | def process_items(self, fn, store_result=False): 386 | if store_result: 387 | accum = [] 388 | else: 389 | accum = None 390 | 391 | def process(cursor): 392 | cursor.jump() 393 | if store_result: 394 | def inner(key, value): 395 | accum.append(fn(key, value)) 396 | return NOP 397 | else: 398 | def inner(key, value): 399 | fn(key, value) 400 | return NOP 401 | while cursor.accept(inner): 402 | cursor.step() 403 | 404 | self.db.cursor_process(process) 405 | return accum 406 | 407 | def __iter__(self): 408 | return iter(self.db) 409 | 410 | def keys(self): 411 | return iter(self.db) 412 | 413 | def itervalues(self): 414 | with self.cursor() as cursor: 415 | for _, value in cursor: 416 | yield value 417 | 418 | def values(self): 419 | processor = lambda k, v: v 420 | return self.process_items(processor, True) 421 | 422 | def iteritems(self): 423 | with self.cursor() as cursor: 424 | for item in cursor: 425 | yield item 426 | 427 | def items(self): 428 | processor = lambda k, v: (k, v) 429 | return self.process_items(processor, True) 430 | 431 | def merge(self, databases, mode=MERGE_OVERWRITE): 432 | return self.db.merge([database.db for database in databases], mode) 433 | 434 | def __or__(self, rhs): 435 | return self.merge(rhs) 436 | 437 | def _get_int(self, key): 438 | return struct.unpack('>q', self[key])[0] 439 | 440 | def _set_int(self, key, value): 441 | self[key] = struct.pack('>q', value) 442 | 443 | def _get_float(self, key): 444 | return struct.unpack('>d', self[key])[0] 445 | 446 | def _set_float(self, key, value): 447 | self[key] = struct.pack('>d', value) 448 | 449 | def get_slice(self, start, end): 450 | if start and start > end: 451 | raise ValueError('%s must be less than or equal to %s.' % ( 452 | start, end)) 453 | 454 | with self.cursor() as cursor: 455 | if start is None: 456 | cursor.first() 457 | else: 458 | cursor.seek(start) 459 | 460 | for i, (k, v) in enumerate(cursor.fetch_until(end)): 461 | yield (k, v) 462 | 463 | def get_slice_rev(self, start, end): 464 | if start and start < end: 465 | raise ValueError('%s must be greater than or equal to %s.' % ( 466 | start, end)) 467 | 468 | # Iterating backwards requires a bit more hackery... 469 | with self.cursor(reverse=True) as cursor: 470 | if start is None: 471 | cursor.last() 472 | else: 473 | if not cursor.seek(start): 474 | cursor.last() 475 | 476 | # When seeking, kyotocabinet may go to the next highest matching 477 | # record. For backwards searches, we want the lowest without 478 | # going over as the start point. This bit corrects that. 479 | res = cursor.get() 480 | if res and start and res[0] > start: 481 | cursor._previous() 482 | 483 | for i, (k, v) in enumerate(cursor.fetch_until(end)): 484 | yield (k, v) 485 | 486 | 487 | class Cursor(object): 488 | def __init__(self, cursor, reverse=False): 489 | self._cursor = cursor 490 | self._reverse = reverse 491 | self._consumed = False 492 | 493 | def __enter__(self): 494 | if self._reverse: 495 | self.last() 496 | else: 497 | self.first() 498 | return self 499 | 500 | def __exit__(self, exc_type, exc_val, exc_tb): 501 | try: 502 | self.close() 503 | except: 504 | pass 505 | 506 | def close(self): 507 | self._cursor.disable() 508 | 509 | def first(self): 510 | self._consumed = False 511 | self._cursor.jump() 512 | 513 | def last(self): 514 | self._consumed = False 515 | self._cursor.jump_back() 516 | 517 | def seek(self, key): 518 | self._consumed = False 519 | return self._cursor.jump(key) 520 | 521 | def __iter__(self): 522 | self._consumed = False 523 | return self 524 | 525 | def next(self): 526 | if self._consumed: 527 | raise StopIteration 528 | 529 | key_value = self.get() 530 | if key_value is None: 531 | self._consumed = True 532 | raise StopIteration 533 | 534 | if self._reverse: 535 | self._consumed = not self._previous() 536 | else: 537 | self._consumed = not self._next() 538 | 539 | return key_value 540 | 541 | def set(self, value, step=False): 542 | return self._cursor.set_value(value, step) 543 | 544 | def get(self): 545 | return self._cursor.get() 546 | 547 | def remove(self): 548 | return self._cursor.remove() 549 | 550 | def pop(self): 551 | return self._cursor.seize() 552 | 553 | def _next(self): 554 | return self._cursor.step() 555 | 556 | def _previous(self): 557 | return self._cursor.step_back() 558 | 559 | def fetch_count(self, n): 560 | while n > 0: 561 | yield next(self) 562 | n -= 1 563 | 564 | def fetch_until(self, end_key): 565 | compare = operator.le if self._reverse else operator.ge 566 | for key, value in self: 567 | if compare(key, end_key): 568 | if key == end_key: 569 | yield (key, value) 570 | raise StopIteration 571 | else: 572 | yield (key, value) 573 | 574 | 575 | class _callable_context_manager(object): 576 | def __call__(self, fn): 577 | def inner(*args, **kwargs): 578 | with self: 579 | return fn(*args, **kwargs) 580 | return inner 581 | 582 | 583 | class transaction(_callable_context_manager): 584 | def __init__(self, db): 585 | self._db = db 586 | 587 | def __enter__(self): 588 | self._db.begin() 589 | 590 | def __exit__(self, exc_type, exc_val, exc_tb): 591 | if exc_type: 592 | self._db.rollback() 593 | else: 594 | try: 595 | self._db.commit() 596 | except: 597 | try: 598 | self._db.rollback() 599 | except: 600 | pass 601 | raise 602 | 603 | def commit(self): 604 | self._db.commit() 605 | self._db.begin() 606 | 607 | def rollback(self): 608 | self._db.rollback() 609 | self._db.begin() 610 | 611 | 612 | class HashDB(Database): 613 | # Persisten O(1) hash table, unordered. Key-level locking (rwlock). 614 | extension = DB_FILE_HASH 615 | 616 | 617 | class TreeDB(Database): 618 | # Persisten O(log N) B+ tree, ordered. Page-level locking (rwlock). 619 | extension = DB_FILE_TREE 620 | 621 | 622 | class DirectoryHashDB(Database): 623 | # Persisten O(?), unordered. Key-level locking (rwlock). 624 | extension = DB_DIRECTORY_HASH 625 | 626 | 627 | class DirectoryTreeDB(Database): 628 | # Persisten O(log N) B+ tree, ordered. Page-level locking (rwlock). 629 | extension = DB_DIRECTORY_TREE 630 | 631 | 632 | class PlainTextDB(Database): 633 | # Persisten O(?) plain text, stored order. Key-level locking (rwlock). 634 | extension = DB_TEXT 635 | 636 | 637 | class _FilenameDatabase(Database): 638 | filename = None 639 | 640 | def __init__(self, exceptional=False, concurrent=False, **opts): 641 | super(_FilenameDatabase, self).__init__( 642 | self.filename, 643 | exceptional, 644 | concurrent, 645 | **opts) 646 | 647 | 648 | class PrototypeHashDB(_FilenameDatabase): 649 | # Volatile O(1) hash table, unordered. DB locking. 650 | filename = DB_PROTOTYPE_HASH 651 | 652 | 653 | class PrototypeTreeDB(_FilenameDatabase): 654 | # Volatile O(log N) red-black tree, lexical order. DB locking. 655 | filename = DB_PROTOTYPE_TREE 656 | 657 | 658 | class StashDB(_FilenameDatabase): 659 | # Volatile O(1) hash table, unordered. Key-level locking (rwlock). 660 | filename = DB_STASH 661 | 662 | 663 | class CacheHashDB(_FilenameDatabase): 664 | # Volatile O(1) hash table, unordered. Key-level locking (mutex). 665 | filename = DB_CACHE_HASH 666 | 667 | 668 | class CacheTreeDB(_FilenameDatabase): 669 | # Volatile O(log N) B+ tree, ordered. Page-level locking (rwlock). 670 | filename = DB_CACHE_TREE 671 | -------------------------------------------------------------------------------- /kvkit/backends/leveldb.py: -------------------------------------------------------------------------------- 1 | # Requries plyvel. 2 | from contextlib import contextmanager 3 | import struct 4 | 5 | import plyvel 6 | 7 | from kvkit.backends.helpers import clean_key_slice 8 | from kvkit.backends.helpers import KVHelper 9 | 10 | 11 | class LevelDB(KVHelper): 12 | def __init__(self, filename, *args, **kwargs): 13 | self.filename = filename 14 | kwargs.setdefault('create_if_missing', True) 15 | self.db = plyvel.DB(filename, *args, **kwargs) 16 | self._closed = False 17 | 18 | def __getitem__(self, key): 19 | if isinstance(key, slice): 20 | start, stop, reverse = clean_key_slice(key) 21 | if reverse: 22 | # LevelDB uses slightly different meaning for start/stop when 23 | # reverse than kyotocabinet. 24 | start, stop = stop, start 25 | return self.db.iterator( 26 | start=start, 27 | stop=stop, 28 | include_start=True, 29 | include_stop=True, 30 | reverse=reverse) 31 | elif isinstance(key, (list, tuple)): 32 | pass 33 | else: 34 | res = self.db.get(key) 35 | if res is None: 36 | raise KeyError(key) 37 | return res 38 | 39 | def open(self): 40 | pass 41 | 42 | def close(self): 43 | if self._closed: 44 | return False 45 | 46 | self.db.close() 47 | self._closed = True 48 | return True 49 | 50 | def __setitem__(self, key, value): 51 | self.db.put(key, value) 52 | 53 | def __delitem__(self, key): 54 | self.db.delete(key) 55 | 56 | def update(self, _data=None, **kwargs): 57 | batch = self.db.write_batch() 58 | if _data: 59 | kwargs.update(_data) 60 | for key, value in kwargs.iteritems(): 61 | batch.put(key, value) 62 | batch.write() 63 | 64 | def keys(self): 65 | return (key for key in self.db.iterator(include_value=False)) 66 | 67 | def values(self): 68 | return (value for value in self.db.iterator(include_key=False)) 69 | 70 | def items(self): 71 | return (item for item in self.db) 72 | -------------------------------------------------------------------------------- /kvkit/backends/rocks.py: -------------------------------------------------------------------------------- 1 | # Requires pyrocksdb. 2 | from contextlib import contextmanager 3 | import struct 4 | 5 | # See https://pyrocksdb.readthedocs.io/en/latest/tutorial/index.html 6 | import rocksdb 7 | 8 | from kvkit.backends.helpers import clean_key_slice 9 | from kvkit.backends.helpers import KVHelper 10 | 11 | 12 | class RocksDB(KVHelper): 13 | def __init__(self, filename, *args, **kwargs): 14 | self.filename = filename 15 | kwargs.setdefault('create_if_missing', True) 16 | options = rocksdb.Options(**kwargs) 17 | self.db = rocksdb.DB(filename, options) 18 | self._closed = False 19 | 20 | def __getitem__(self, key): 21 | if isinstance(key, slice): 22 | start, stop, reverse = clean_key_slice(key) 23 | if reverse: 24 | return self.get_slice_rev(start, stop) 25 | else: 26 | return self.get_slice(start, stop) 27 | elif isinstance(key, (list, tuple)): 28 | return self.db.multi_get(key) 29 | else: 30 | res = self.db.get(key) 31 | if res is None: 32 | raise KeyError(key) 33 | return res 34 | 35 | def open(self): 36 | pass 37 | 38 | def close(self): 39 | if self._closed: 40 | return False 41 | 42 | self.db.close() 43 | self._closed = True 44 | return True 45 | 46 | def __setitem__(self, key, value): 47 | self.db.put(key, value) 48 | 49 | def __delitem__(self, key): 50 | self.db.delete(key) 51 | 52 | def update(self, _data=None, **kwargs): 53 | batch = rocksdb.WriteBatch() 54 | if _data: 55 | kwargs.update(_data) 56 | for key, value in kwargs.iteritems(): 57 | batch.put(key, value) 58 | self.db.write(batch) 59 | 60 | def keys(self): 61 | iterator = self.db.iterkeys() 62 | iterator.seek_to_first() 63 | for key in iterator: 64 | yield key 65 | 66 | def values(self): 67 | iterator = self.db.itervalues() 68 | iterator.seek_to_first() 69 | for value in iterator: 70 | yield value 71 | 72 | def items(self): 73 | iterator = self.db.iteritems() 74 | iterator.seek_to_first() 75 | for item in iterator: 76 | yield item 77 | 78 | def get_slice(self, start, end): 79 | iterator = self.db.iteritems() 80 | if start is None: 81 | iterator.seek_to_first() 82 | else: 83 | iterator.seek(start) 84 | 85 | while True: 86 | key, value = iterator.next() 87 | if key > end: 88 | raise StopIteration 89 | yield key, value 90 | 91 | def get_slice_rev(self, start, end): 92 | iterator = reversed(self.db.iteritems()) 93 | if start is None: 94 | iterator.seek_to_last() 95 | else: 96 | iterator.seek(start) 97 | 98 | try: 99 | key, value = iterator.next() 100 | except StopIteration: 101 | iterator.seek_to_last() 102 | key, value = iterator.next() 103 | 104 | if key <= start or start is None: 105 | yield key, value 106 | 107 | while True: 108 | key, value = iterator.next() 109 | if key < end: 110 | raise StopIteration 111 | yield key, value 112 | -------------------------------------------------------------------------------- /kvkit/backends/sophia.py: -------------------------------------------------------------------------------- 1 | import struct 2 | 3 | from sophy import SimpleDatabase 4 | 5 | 6 | class Sophia(SimpleDatabase): 7 | def __enter__(self): 8 | self.open() 9 | return self 10 | 11 | def __exit__(self, exc_type, exc_val, exc_tb): 12 | self.close() 13 | 14 | def incr(self, key, amount=1): 15 | try: 16 | value = self[key] 17 | except KeyError: 18 | value = amount 19 | else: 20 | value = struct.unpack('>q', value)[0] + amount 21 | self[key] = struct.pack('>q', value) 22 | return value 23 | 24 | def decr(self, key, amount=1): 25 | return self.incr(key, amount * -1) 26 | -------------------------------------------------------------------------------- /kvkit/backends/sqlite4.py: -------------------------------------------------------------------------------- 1 | # Requires python-lsm-db 2 | # No action necessary as `LSM` already implements the appropriate interfaces. 3 | from lsm import LSM 4 | -------------------------------------------------------------------------------- /kvkit/exceptions.py: -------------------------------------------------------------------------------- 1 | class DatabaseError(Exception): 2 | pass 3 | -------------------------------------------------------------------------------- /kvkit/graph.py: -------------------------------------------------------------------------------- 1 | # Hexastore. 2 | import itertools 3 | import json 4 | 5 | 6 | class _VariableGenerator(object): 7 | def __getattr__(self, name): 8 | return Variable(name) 9 | 10 | def __call__(self, name): 11 | return Variable(name) 12 | 13 | 14 | class Hexastore(object): 15 | 16 | def __init__(self, database, prefix='', serialize=json.dumps, 17 | deserialize=json.loads): 18 | self.database = database 19 | self.prefix = prefix 20 | self.serialize = serialize 21 | self.deserialize = deserialize 22 | self.v = _VariableGenerator() 23 | 24 | def _data_for_storage(self, s, p, o): 25 | serialized = self.serialize({ 26 | 's': s, 27 | 'p': p, 28 | 'o': o}) 29 | 30 | data = {} 31 | for key in self.keys_for_values(s, p, o): 32 | data[key] = serialized 33 | 34 | return data 35 | 36 | def store(self, s, p, o): 37 | return self.database.update(self._data_for_storage(s, p, o)) 38 | 39 | def store_many(self, items): 40 | data = {} 41 | for item in items: 42 | data.update(self._data_for_storage(*item)) 43 | return self.database.update(data) 44 | 45 | def delete(self, s, p, o): 46 | for key in self.keys_for_values(s, p, o): 47 | del self.database[key] 48 | 49 | def keys_for_values(self, s, p, o): 50 | zipped = zip('spo', (s, p, o)) 51 | for ((p1, v1), (p2, v2), (p3, v3)) in itertools.permutations(zipped): 52 | yield '::'.join(( 53 | self.prefix, 54 | ''.join((p1, p2, p3)), 55 | v1, 56 | v2, 57 | v3)) 58 | 59 | def keys_for_query(self, s=None, p=None, o=None): 60 | parts = [self.prefix] 61 | key = lambda parts: '::'.join(parts) 62 | 63 | if s and p and o: 64 | parts.extend(('spo', s, p, o)) 65 | return key(parts), None 66 | elif s and p: 67 | parts.extend(('spo', s, p)) 68 | elif s and o: 69 | parts.extend(('sop', s, o)) 70 | elif p and o: 71 | parts.extend(('pos', p, o)) 72 | elif s: 73 | parts.extend(('spo', s)) 74 | elif p: 75 | parts.extend(('pso', p)) 76 | elif o: 77 | parts.extend(('osp', o)) 78 | return key(parts + ['']), key(parts + ['\xff']) 79 | 80 | def query(self, s=None, p=None, o=None): 81 | start, end = self.keys_for_query(s, p, o) 82 | deserialize = self.deserialize 83 | if end is None: 84 | try: 85 | yield deserialize(self.database[start]) 86 | except KeyError: 87 | raise StopIteration 88 | else: 89 | for key, value in self.database[start:end]: 90 | yield deserialize(value) 91 | 92 | def v(self, name): 93 | return Variable(name) 94 | 95 | def search(self, *conditions): 96 | results = {} 97 | 98 | for condition in conditions: 99 | if isinstance(condition, tuple): 100 | query = dict(zip('spo', condition)) 101 | else: 102 | query = condition.copy() 103 | materialized = {} 104 | targets = [] 105 | 106 | for part in ('s', 'p', 'o'): 107 | if isinstance(query[part], Variable): 108 | variable = query.pop(part) 109 | materialized[part] = set() 110 | targets.append((variable, part)) 111 | 112 | # Potentially rather than popping all the variables, we could use 113 | # the result values from a previous condition and do O(results) 114 | # loops looking for a single variable. 115 | for result in self.query(**query): 116 | ok = True 117 | for var, part in targets: 118 | if var in results and result[part] not in results[var]: 119 | ok = False 120 | break 121 | 122 | if ok: 123 | for var, part in targets: 124 | materialized[part].add(result[part]) 125 | 126 | for var, part in targets: 127 | if var in results: 128 | results[var] &= materialized[part] 129 | else: 130 | results[var] = materialized[part] 131 | 132 | return dict((var.name, vals) for (var, vals) in results.items()) 133 | 134 | 135 | class Variable(object): 136 | __slots__ = ['name'] 137 | 138 | def __init__(self, name): 139 | self.name = name 140 | 141 | def __repr__(self): 142 | return '' % (self.name) 143 | -------------------------------------------------------------------------------- /kvkit/query.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import pickle 3 | import struct 4 | 5 | 6 | class Node(object): 7 | # Node in a query tree. 8 | def __init__(self): 9 | self.negated = False 10 | 11 | def _e(op, inv=False): 12 | """ 13 | Lightweight factory which returns a method that builds an Expression 14 | consisting of the left-hand and right-hand operands, using `op`. 15 | """ 16 | def inner(self, rhs): 17 | if inv: 18 | return Expression(rhs, op, self) 19 | return Expression(self, op, rhs) 20 | return inner 21 | 22 | __and__ = _e('AND') 23 | __or__ = _e('OR') 24 | __rand__ = _e('AND', inv=True) 25 | __ror__ = _e('OR', inv=True) 26 | __eq__ = _e('=') 27 | __ne__ = _e('!=') 28 | __lt__ = _e('<') 29 | __le__ = _e('<=') 30 | __gt__ = _e('>') 31 | __ge__ = _e('>=') 32 | 33 | def startswith(self, prefix): 34 | return Expression(self, 'startswith', prefix) 35 | 36 | 37 | class Expression(Node): 38 | def __init__(self, lhs, op, rhs): 39 | self.lhs = lhs 40 | self.op = op 41 | self.rhs = rhs 42 | 43 | def __repr__(self): 44 | return '' % (self.lhs, self.op, self.rhs) 45 | 46 | 47 | class Field(Node): 48 | _counter = 0 49 | 50 | def __init__(self, index=False, default=None): 51 | self.index = index 52 | self.default = default 53 | self.model = None 54 | self.name = None 55 | self._order = Field._counter 56 | Field._counter += 1 57 | 58 | def __repr__(self): 59 | return '<%s: %s.%s>' % ( 60 | type(self), 61 | self.model._meta.name, 62 | self.name) 63 | 64 | def bind(self, model, name): 65 | self.model = model 66 | self.name = name 67 | setattr(self.model, self.name, FieldDescriptor(self)) 68 | 69 | def clone(self): 70 | field = type(self)(index=self.index, default=self.default) 71 | field.model = self.model 72 | field.name = self.name 73 | return field 74 | 75 | def db_value(self, value): 76 | return value 77 | 78 | def python_value(self, value): 79 | return value 80 | 81 | 82 | class DateTimeField(Field): 83 | def db_value(self, value): 84 | if value: 85 | return value.strftime('%Y-%m-%d %H:%M:%S.%f') 86 | 87 | def python_value(self, value): 88 | if value: 89 | return datetime.datetime.strptime(value, '%Y-%m-%d %H:%M:%S.%f') 90 | 91 | 92 | class DateField(Field): 93 | def db_value(self, value): 94 | if value: 95 | return value.strftime('%Y-%m-%d') 96 | 97 | def python_value(self, value): 98 | if value: 99 | pieces = [int(piece) for piece in value.split('-')] 100 | return datetime.date(*pieces) 101 | 102 | 103 | class LongField(Field): 104 | def db_value(self, value): 105 | return struct.pack('>q', value) if value is not None else '' 106 | 107 | def python_value(self, value): 108 | if value: 109 | return struct.unpack('>q', value)[0] 110 | 111 | 112 | class FloatField(Field): 113 | def db_value(self, value): 114 | return struct.pack('>d', value) if value is not None else '' 115 | 116 | def python_value(self, value): 117 | if value: 118 | return struct.unpack('>d', value)[0] 119 | 120 | 121 | class FieldDescriptor(object): 122 | def __init__(self, field): 123 | self.field = field 124 | self.name = self.field.name 125 | 126 | def __get__(self, instance, instance_type=None): 127 | if instance is not None: 128 | return instance._data.get(self.name) 129 | return self.field 130 | 131 | def __set__(self, instance, value): 132 | instance._data[self.name] = value 133 | 134 | 135 | class DeclarativeMeta(type): 136 | def __new__(cls, name, bases, attrs): 137 | if bases == (object,): 138 | return super(DeclarativeMeta, cls).__new__(cls, name, bases, attrs) 139 | 140 | database = None 141 | fields = {} 142 | serialize = None 143 | 144 | # Inherit fields from parent classes. 145 | for base in bases: 146 | if not hasattr(base, '_meta'): 147 | continue 148 | 149 | for field in base._meta.sorted_fields: 150 | if field.name not in fields: 151 | fields[field.name] = field.clone() 152 | 153 | if database is None and base._meta.database is not None: 154 | database = base._meta.database 155 | if serialize is None: 156 | serialize = base._meta.serialize 157 | 158 | # Introspect all declared fields. 159 | for key, value in attrs.items(): 160 | if isinstance(value, Field): 161 | fields[key] = value 162 | 163 | # Read metadata configuration. 164 | declared_meta = attrs.pop('Meta', None) 165 | if declared_meta: 166 | if getattr(declared_meta, 'database', None) is not None: 167 | database = declared_meta.database 168 | if getattr(declared_meta, 'serialize', None) is not None: 169 | serialize = declared_meta.serialize 170 | 171 | # Always have an `id` field. 172 | if 'id' not in fields: 173 | fields['id'] = LongField() 174 | 175 | if serialize is None: 176 | serialize = True 177 | 178 | attrs['_meta'] = Metadata(name, database, fields, serialize) 179 | model = super(DeclarativeMeta, cls).__new__(cls, name, bases, attrs) 180 | 181 | # Bind fields to model. 182 | for name, field in fields.items(): 183 | field.bind(model, name) 184 | 185 | # Process 186 | model._meta.prepared() 187 | 188 | return model 189 | 190 | 191 | class Metadata(object): 192 | def __init__(self, model_name, database, fields, serialize): 193 | self.model_name = model_name 194 | self.database = database 195 | self.fields = fields 196 | self.serialize = serialize 197 | 198 | self.name = model_name.lower() 199 | self.sequence = 'id_seq:%s' % self.name 200 | 201 | self.defaults = {} 202 | self.defaults_callable = {} 203 | 204 | def prepared(self): 205 | self.sorted_fields = sorted( 206 | [field for field in self.fields.values()], 207 | key=lambda field: field._order) 208 | 209 | # Populate index attributes. 210 | self.indexed_fields = set() 211 | self.indexed_field_objects = [] 212 | self.indexes = {} 213 | for field in self.sorted_fields: 214 | if field.index: 215 | self.indexed_fields.add(field.name) 216 | self.indexed_field_objects.append(field) 217 | self.indexes[field.name] = Index(self.database, field) 218 | 219 | for field in self.sorted_fields: 220 | if callable(field.default): 221 | self.defaults_callable[field.name] = field.default 222 | elif field.default: 223 | self.defaults[field.name] = field.default 224 | 225 | def next_id(self): 226 | return self.database.incr(self.sequence) 227 | 228 | def get_instance_key(self, instance_id): 229 | return '%s:%s' % (self.name, instance_id) 230 | 231 | 232 | def with_metaclass(meta, base=object): 233 | return meta('newbase', (base,), {}) 234 | 235 | 236 | class Model(with_metaclass(DeclarativeMeta)): 237 | def __init__(self, **kwargs): 238 | self._data = self._meta.defaults.copy() 239 | for key, value in self._meta.defaults_callable.items(): 240 | self._data[key] = value() 241 | self._data.update(kwargs) 242 | 243 | @classmethod 244 | def create(cls, **kwargs): 245 | instance = cls(**kwargs) 246 | instance.save() 247 | return instance 248 | 249 | @classmethod 250 | def load(cls, primary_key): 251 | return cls(**cls._read_model_data(primary_key)) 252 | 253 | def save(self, atomic=True): 254 | if atomic: 255 | with self._meta.database.transaction(): 256 | self._save() 257 | else: 258 | self._save() 259 | 260 | def _save(self): 261 | # If we are updating an existing object, load the original data 262 | # so we can correctly update any indexes. 263 | original_data = None 264 | if self.id and self._meta.indexes: 265 | original_data = type(self)._read_indexed_data(self.id) 266 | 267 | # Save the actual model data. 268 | self._save_model_data() 269 | 270 | # Update any secondary indexes. 271 | self._update_indexes(original_data) 272 | 273 | def _save_model_data(self): 274 | database = self._meta.database 275 | 276 | # Generate the next ID in sequence if no ID is set. 277 | if not self.id: 278 | self.id = self._meta.next_id() 279 | 280 | # Retrieve the primary key identifying this model instance. 281 | key = self._meta.get_instance_key(self.id) 282 | 283 | if self._meta.serialize: 284 | # Store all model data serialized in a single record. 285 | database[key] = pickle.dumps(self._data) 286 | else: 287 | # Store model data in discrete records, one per field. 288 | for field in self._meta.sorted_fields: 289 | field_key = '%s:%s' % (key, field.name) 290 | value = field.db_value(getattr(self, field.name)) 291 | database[field_key] = value or '' 292 | 293 | def _update_indexes(self, original_data): 294 | database = self._meta.database 295 | primary_key = self.id 296 | 297 | for field, index in self._meta.indexes.items(): 298 | # Retrieve the value of the indexed field. 299 | value = getattr(self, field) 300 | 301 | # If the value differs from what was previously stored, remove 302 | # the old value. 303 | if original_data is not None and original_data[field] != value: 304 | index.delete(value, primary_key) 305 | 306 | # Store the value in the index. 307 | index.store(value, primary_key) 308 | index.store_endpoint() 309 | 310 | @classmethod 311 | def _read_model_data(cls, primary_key, fields=None): 312 | key = cls._meta.get_instance_key(primary_key) 313 | if cls._meta.serialize: 314 | # For serialized models, simply grab all data. 315 | data = pickle.loads(cls._meta.database[key]) 316 | else: 317 | # Load the model data from each field. 318 | data = {} 319 | fields = fields or cls._meta.sorted_fields 320 | for field in fields: 321 | field_key = '%s:%s' % (key, field.name) 322 | data[field.name] = field.python_value( 323 | cls._meta.database[field_key]) 324 | return data 325 | 326 | @classmethod 327 | def _read_indexed_data(cls, primary_key): 328 | return cls._read_model_data( 329 | primary_key, 330 | cls._meta.indexed_field_objects) 331 | 332 | def delete(self, atomic=True): 333 | if atomic: 334 | with self._meta.database.transaction(): 335 | self._delete() 336 | else: 337 | self._delete() 338 | 339 | def _delete(self): 340 | database = self._meta.database 341 | 342 | key = self._meta.get_instance_key(self.id) 343 | if self._meta.serialize: 344 | del database[key] 345 | else: 346 | # Save model data to discrete fields. 347 | for field in self._meta.sorted_fields: 348 | field_key = '%s:%s' % (key, field.name) 349 | del database[field_key] 350 | 351 | for field, index in self._meta.indexes.items(): 352 | index.delete(getattr(self, field), self.id) 353 | 354 | @classmethod 355 | def get(cls, expr): 356 | results = cls.query(expr) 357 | if results: 358 | return results[0] 359 | 360 | @classmethod 361 | def query(cls, expr): 362 | def dfs(expr): 363 | lhs = expr.lhs 364 | rhs = expr.rhs 365 | if isinstance(lhs, Expression): 366 | lhs = dfs(lhs) 367 | if isinstance(rhs, Expression): 368 | rhs = dfs(rhs) 369 | 370 | if isinstance(lhs, Field): 371 | index = cls._meta.indexes[lhs.name] 372 | return set(index.query(rhs, expr.op)) 373 | elif expr.op == 'AND': 374 | return set(lhs) & set(rhs) 375 | elif expr.op == 'OR': 376 | return set(lhs) | set(rhs) 377 | else: 378 | raise ValueError('Unable to execute query, unexpected type.') 379 | 380 | id_list = dfs(expr) 381 | return [cls.load(primary_key) for primary_key in sorted(id_list)] 382 | 383 | 384 | class Index(object): 385 | def __init__(self, database, field): 386 | self.database = database 387 | self.field = field 388 | self.name = 'idx:%s:%s' % (field.model._meta.name, field.name) 389 | self.stop_key = '%s\xff\xff\xff' % self.name 390 | self.convert_pk = field.model.id.db_value 391 | 392 | def get_key(self, value, primary_key): 393 | return '%s\xff%s\xff%s' % ( 394 | self.name, 395 | self.field.db_value(value) or '', 396 | self.convert_pk(primary_key)) 397 | 398 | def get_prefix(self, value=None, closed=False): 399 | if value is None: 400 | return '%s\xff' % self.name 401 | else: 402 | return '%s\xff%s%s' % ( 403 | self.name, 404 | self.field.db_value(value), 405 | '\xff' if closed else '') 406 | 407 | def store(self, value, primary_key): 408 | self.database[self.get_key(value, primary_key)] = str(primary_key) 409 | 410 | def delete(self, value, primary_key): 411 | del self.database[self.get_key(value, primary_key)] 412 | 413 | def store_endpoint(self): 414 | self.database[self.stop_key] = '' 415 | 416 | def query(self, value, operation): 417 | if operation == '=': 418 | start_key = self.get_prefix(value, closed=True) 419 | end_key = start_key + '\xff' 420 | return [value for key, value in self.database[start_key:end_key]] 421 | elif operation in ('<', '<='): 422 | start_key = self.get_prefix() 423 | end_key = self.get_prefix(value) + '\xff' 424 | if operation == '<=': 425 | end_key += '\xff' 426 | results = self.database[start_key:end_key] 427 | return [value for key, value in results] 428 | elif operation in ('>', '>='): 429 | start_key = self.stop_key 430 | end_key = self.get_prefix(value) 431 | if operation == '>': 432 | end_key += '\xff\xff' 433 | results = self.database[start_key:end_key] 434 | return [value for i, (key, value) in enumerate(results) if i > 0] 435 | elif operation == '!=': 436 | match = self.get_prefix(value, closed=True) 437 | start_key = self.get_prefix() 438 | end_key = self.stop_key 439 | results = self.database[start_key:end_key] 440 | return [v for k, v in results if not k.startswith(match)][:-1] 441 | elif operation == 'startswith': 442 | start_key = self.get_prefix(value) 443 | end_key = start_key + '\xff\xff' 444 | return [value for key, value in self.database[start_key:end_key]] 445 | -------------------------------------------------------------------------------- /kvkit/tests.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import gc 3 | import os 4 | import shutil 5 | import struct 6 | import sys 7 | import tempfile 8 | import unittest 9 | 10 | from kvkit.backends.kyoto import * 11 | from kvkit.backends.kyoto import _FilenameDatabase 12 | from kvkit.graph import * 13 | from kvkit.query import * 14 | 15 | try: 16 | from kvkit.backends.berkeleydb import BerkeleyDB 17 | except ImportError: 18 | BerkeleyDB = None 19 | 20 | try: 21 | from kvkit.backends.forest import ForestDB 22 | except ImportError: 23 | ForestDB = None 24 | 25 | try: 26 | from kvkit.backends.leveldb import LevelDB 27 | except ImportError: 28 | LevelDB = None 29 | 30 | try: 31 | from kvkit.backends.rocks import RocksDB 32 | except ImportError: 33 | RocksDB = None 34 | 35 | try: 36 | from kvkit.backends.sophia import Sophia 37 | except ImportError: 38 | Sophia = None 39 | 40 | try: 41 | from kvkit.backends.sqlite4 import LSM 42 | except ImportError: 43 | LSM = None 44 | 45 | 46 | class BaseTestCase(unittest.TestCase): 47 | database_class = None 48 | 49 | def setUp(self): 50 | self.db = self.create_db() 51 | 52 | def tearDown(self): 53 | if self.db is not None: 54 | try: 55 | self.db.close() 56 | except Exception: 57 | pass 58 | finally: 59 | self.delete_db() 60 | 61 | def delete_db(self): 62 | if not isinstance(self.db, _FilenameDatabase): 63 | if os.path.isfile(self.db.filename): 64 | os.unlink(self.db.filename) 65 | elif os.path.isdir(self.db.filename): 66 | shutil.rmtree(self.db.filename) 67 | 68 | def create_db(self): 69 | if self.database_class is not None: 70 | db = self.database_class('test.db') 71 | return db 72 | 73 | def create_rows(self, n): 74 | for i in range(n): 75 | self.db['k%s' % i] = i 76 | 77 | 78 | class KVKitTests(object): 79 | def test_storage(self): 80 | self.db['k1'] = 'v1' 81 | self.db['k2'] = 'v2' 82 | self.assertEqual(self.db['k1'], 'v1') 83 | self.assertEqual(self.db['k2'], 'v2') 84 | self.assertRaises(KeyError, lambda: self.db['k3']) 85 | 86 | self.assertEqual(self.db[['k1', 'k2', 'k3']], { 87 | 'k1': 'v1', 88 | 'k2': 'v2', 89 | }) 90 | self.assertEqual(self.db[['kx', 'ky']], {}) 91 | 92 | self.assertTrue('k1' in self.db) 93 | self.assertFalse('k3' in self.db) 94 | self.assertEqual(len(self.db), 2) 95 | 96 | self.assertEqual(self.db.pop('k1'), 'v1') 97 | self.assertRaises(KeyError, lambda: self.db.pop('k1')) 98 | 99 | self.assertEqual(self.db.pop(), ('k2', 'v2')) 100 | self.assertEqual(len(self.db), 0) 101 | 102 | data = dict(('k%s' % i, i) for i in range(0, 40, 5)) 103 | self.db.update(**data) 104 | self.assertEqual(len(self.db), 8) 105 | 106 | self.assertEqual(sorted(self.db.match_prefix('k2')), ['k20', 'k25']) 107 | self.assertEqual( 108 | sorted(self.db.match_regex('k[2,3]5')), 109 | ['k25', 'k35']) 110 | self.assertEqual( 111 | sorted(self.db.match('k2')), 112 | ['k0', 'k20', 'k25', 'k5']) 113 | 114 | self.assertTrue(self.db.append('k0', 'xx')) 115 | self.assertTrue(self.db.append('kx', 'huey')) 116 | self.assertEqual(self.db['k0'], '0xx') 117 | self.assertEqual(self.db['kx'], 'huey') 118 | 119 | self.assertTrue(self.db.replace('kx', 'mickey')) 120 | self.assertEqual(self.db['kx'], 'mickey') 121 | self.assertFalse(self.db.replace('kz', 'foo')) 122 | self.assertRaises(KeyError, lambda: self.db['kz']) 123 | 124 | self.assertEqual(self.db.incr('ct'), 1) 125 | self.assertEqual(self.db.incr('ct', 2), 3) 126 | self.assertEqual(self.db.decr('ct', 4), -1) 127 | 128 | def test_store_types(self): 129 | self.db._set_int('i1', 1337) 130 | self.db._set_int('i2', 0) 131 | 132 | self.db._set_float('f1', 3.14159) 133 | self.db._set_float('f2', 0) 134 | 135 | self.assertEqual(self.db._get_int('i1'), 1337) 136 | self.assertEqual(self.db._get_int('i2'), 0) 137 | 138 | self.assertEqual(self.db._get_float('f1'), 3.14159) 139 | self.assertEqual(self.db._get_float('f2'), 0.) 140 | 141 | def test_atomic(self): 142 | @self.db.atomic() 143 | def atomic_succeed(): 144 | self.db['k1'] = 'v1' 145 | 146 | @self.db.atomic() 147 | def atomic_fail(): 148 | self.db['k2'] = 'v2' 149 | return False 150 | 151 | atomic_succeed() 152 | self.assertEqual(self.db['k1'], 'v1') 153 | 154 | atomic_fail() 155 | self.assertFalse('k2' in self.db) 156 | 157 | def test_transaction(self): 158 | with self.db.transaction(): 159 | self.db['k1'] = 'v1' 160 | self.db['k2'] = 'v2' 161 | 162 | data = {'k1': 'v1', 'k2': 'v2'} 163 | self.assertEqual(self.db[['k1', 'k2']], data) 164 | 165 | def failed_transaction(): 166 | with self.db.transaction(): 167 | self.db['k3'] = 'v3' 168 | raise Exception() 169 | 170 | self.assertRaises(Exception, failed_transaction) 171 | self.assertFalse('k3' in self.db) 172 | 173 | @self.db.transaction() 174 | def f_succeed(): 175 | self.db['kx'] = 'x' 176 | 177 | @self.db.transaction() 178 | def f_fail(): 179 | self.db['kz'] = 'z' 180 | raise Exception() 181 | 182 | f_succeed() 183 | self.assertEqual(self.db['kx'], 'x') 184 | 185 | self.assertRaises(Exception, f_fail) 186 | self.assertRaises(KeyError, lambda: self.db['kz']) 187 | 188 | def test_cursors(self): 189 | self.create_rows(4) 190 | with self.db.cursor() as cursor: 191 | items = list(cursor) 192 | self.assertEqual(items, [ 193 | ('k0', '0'), 194 | ('k1', '1'), 195 | ('k2', '2'), 196 | ('k3', '3'), 197 | ]) 198 | 199 | cursor = self.db.cursor() 200 | cursor.seek('k2') 201 | results = list(cursor) 202 | self.assertEqual(results, [('k2', '2'), ('k3', '3')]) 203 | cursor.close() 204 | 205 | with self.db.cursor() as cursor: 206 | cursor.seek('k1') 207 | results = cursor.fetch_until('k3') 208 | self.assertEqual( 209 | list(results), [('k1', '1'), ('k2', '2'), ('k3', '3')]) 210 | 211 | cursor.seek('k0') 212 | results = cursor.fetch_count(2) 213 | self.assertEqual(list(results), [('k0', '0'), ('k1', '1')]) 214 | 215 | cursor.seek('k1') 216 | results = cursor.fetch_until('k222') 217 | self.assertEqual(list(results), [('k1', '1'), ('k2', '2')]) 218 | 219 | def test_process(self): 220 | self.db.update(k1='v1', k2='v2') 221 | def to_upper(key, value): 222 | return value.upper() 223 | 224 | self.assertTrue(self.db.process(to_upper)) 225 | self.assertEqual(self.db['k1'], 'V1') 226 | self.assertEqual(self.db['k2'], 'V2') 227 | 228 | def test_dict_iteration(self): 229 | self.db.update(k1='v1', k2='v2', k3='v3') 230 | self.assertEqual(sorted(self.db.keys()), ['k1', 'k2', 'k3']) 231 | self.assertEqual(sorted(self.db.values()), ['v1', 'v2', 'v3']) 232 | self.assertEqual(sorted(self.db.itervalues()), ['v1', 'v2', 'v3']) 233 | self.assertEqual(sorted(self.db.items()), [ 234 | ('k1', 'v1'), ('k2', 'v2'), ('k3', 'v3')]) 235 | self.assertEqual(sorted(self.db.iteritems()), [ 236 | ('k1', 'v1'), ('k2', 'v2'), ('k3', 'v3')]) 237 | 238 | 239 | class SliceTests(object): 240 | def assertSlice(self, s, expected): 241 | self.assertEqual(list(s), [(val, val) for val in expected]) 242 | 243 | def create_slice_data(self): 244 | data = ('aa', 'aa1', 'aa2', 'bb', 'cc', 'dd', 'ee', 'ff') 245 | for value in data: 246 | self.db[value] = value 247 | 248 | def test_slices(self): 249 | self.create_slice_data() 250 | 251 | # Endpoints both exist. 252 | s = self.db['aa':'cc'] 253 | self.assertSlice(s, ['aa', 'aa1', 'aa2', 'bb', 'cc']) 254 | 255 | # Missing start. 256 | s = self.db['aa0':'cc'] 257 | self.assertSlice(s, ['aa1', 'aa2', 'bb', 'cc']) 258 | 259 | # Missing end. 260 | s = self.db['aa1':'cc2'] 261 | self.assertSlice(s, ['aa1', 'aa2', 'bb', 'cc']) 262 | 263 | # Missing both. 264 | s = self.db['aa0':'cc2'] 265 | self.assertSlice(s, ['aa1', 'aa2', 'bb', 'cc']) 266 | 267 | # Start precedes first key. 268 | s = self.db['\x01':'aa2'] 269 | self.assertSlice(s, ['aa', 'aa1', 'aa2']) 270 | 271 | # End exceeds last key. 272 | s = self.db['dd':'zz'] 273 | self.assertSlice(s, ['dd', 'ee', 'ff']) 274 | 275 | def test_slice_reverse(self): 276 | self.create_slice_data() 277 | 278 | # Endpoints both exist. 279 | s = self.db['ff':'cc'] 280 | self.assertSlice(s, ['ff', 'ee', 'dd', 'cc']) 281 | 282 | # Missing end. 283 | s = self.db['cc':'aa0'] 284 | self.assertSlice(s, ['cc', 'bb', 'aa2', 'aa1']) 285 | 286 | # Missing start. 287 | s = self.db['cc2':'aa1'] 288 | self.assertSlice(s, ['cc', 'bb', 'aa2', 'aa1']) 289 | 290 | # Missing both. 291 | s = self.db['cc2':'aa0'] 292 | self.assertSlice(s, ['cc', 'bb', 'aa2', 'aa1']) 293 | 294 | # Start exceeds last key. 295 | s = self.db['zz':'cc'] 296 | self.assertSlice(s, ['ff', 'ee', 'dd', 'cc']) 297 | 298 | # End precedes first key. 299 | s = self.db['bb':'\x01'] 300 | self.assertSlice(s, ['bb', 'aa2', 'aa1', 'aa']) 301 | 302 | # Start is almost to the last key. 303 | s = self.db['ef':'cc'] 304 | self.assertSlice(s, ['ee', 'dd', 'cc']) 305 | 306 | def test_slice_start_end(self): 307 | self.create_slice_data() 308 | 309 | s = self.db[:'bb'] 310 | self.assertSlice(s, ['aa', 'aa1', 'aa2', 'bb']) 311 | 312 | s = self.db[:'cc':True] 313 | self.assertSlice(s, ['cc', 'bb', 'aa2', 'aa1', 'aa']) 314 | 315 | s = self.db['cc'::True] 316 | self.assertSlice(s, ['ff', 'ee', 'dd', 'cc']) 317 | 318 | 319 | class ModelTests(object): 320 | def setUp(self): 321 | super(ModelTests, self).setUp() 322 | 323 | class Person(Model): 324 | first = Field(index=True) 325 | last = Field(index=True) 326 | dob = DateField(index=True) 327 | 328 | class Meta: 329 | database = self.db 330 | serialize = False 331 | 332 | class Note(Model): 333 | content = Field() 334 | timestamp = DateTimeField(default=datetime.datetime.now) 335 | 336 | class Meta: 337 | database = self.db 338 | 339 | class Numeric(Model): 340 | x = LongField(index=True) 341 | y = FloatField(index=True) 342 | z = DateField(index=True) 343 | 344 | class Meta: 345 | database = self.db 346 | 347 | self.Person = Person 348 | self.Note = Note 349 | self.Numeric = Numeric 350 | 351 | def test_model_operations(self): 352 | huey = self.Person.create( 353 | first='huey', 354 | last='leifer', 355 | dob=datetime.date(2010, 1, 2)) 356 | self.assertEqual(huey.first, 'huey') 357 | self.assertEqual(huey.last, 'leifer') 358 | self.assertEqual(huey.dob, datetime.date(2010, 1, 2)) 359 | self.assertEqual(huey.id, 1) 360 | 361 | ziggy = self.Person.create( 362 | first='ziggy', 363 | dob=datetime.date(2011, 2, 3)) 364 | self.assertEqual(ziggy.first, 'ziggy') 365 | self.assertEqual(ziggy.last, None) 366 | self.assertEqual(ziggy.id, 2) 367 | 368 | huey_db = self.Person.load(1) 369 | self.assertEqual(huey_db.first, 'huey') 370 | self.assertEqual(huey_db.last, 'leifer') 371 | self.assertEqual(huey_db.dob, datetime.date(2010, 1, 2)) 372 | self.assertEqual(huey_db.id, 1) 373 | 374 | ziggy_db = self.Person.load(2) 375 | self.assertEqual(ziggy_db.first, 'ziggy') 376 | self.assertEqual(ziggy_db.last, '') 377 | self.assertEqual(ziggy_db.dob, datetime.date(2011, 2, 3)) 378 | self.assertEqual(ziggy_db.id, 2) 379 | 380 | keys_1 = set(self.db.keys()) 381 | huey_db.delete() 382 | keys_2 = set(self.db.keys()) 383 | diff = keys_1 - keys_2 384 | one = struct.pack('>q', 1) 385 | two = struct.pack('>q', 2) 386 | self.assertEqual(diff, set([ 387 | 'person:1:first', 'person:1:last', 'person:1:dob', 'person:1:id', 388 | 'idx:person:first\xffhuey\xff%s' % one, 389 | 'idx:person:last\xffleifer\xff%s' % one, 390 | 'idx:person:dob\xff2010-01-02\xff%s' % one])) 391 | self.assertEqual(keys_2, set([ 392 | 'person:2:first', 'person:2:last', 'person:2:dob', 'person:2:id', 393 | 'idx:person:first\xffziggy\xff%s' % two, 394 | 'idx:person:last\xff\xff%s' % two, 395 | 'idx:person:dob\xff2011-02-03\xff%s' % two, 'id_seq:person', 396 | 'idx:person:first\xff\xff\xff', 397 | 'idx:person:last\xff\xff\xff', 398 | 'idx:person:dob\xff\xff\xff', 399 | ])) 400 | 401 | def test_model_serialized(self): 402 | note = self.Note.create(content='note 1') 403 | self.assertTrue(note.timestamp is not None) 404 | self.assertEqual(note.id, 1) 405 | 406 | note_db = self.Note.load(note.id) 407 | self.assertEqual(note_db.content, 'note 1') 408 | self.assertEqual(note_db.timestamp, note.timestamp) 409 | self.assertEqual(note_db.id, 1) 410 | 411 | note2 = self.Note.create(content='note 2') 412 | keys_1 = set(self.db.keys()) 413 | note.delete() 414 | keys_2 = set(self.db.keys()) 415 | diff = keys_1 - keys_2 416 | self.assertEqual(diff, set(['note:1'])) 417 | self.assertEqual(keys_1, set([ 418 | 'id_seq:note', 'note:1', 'note:2'])) 419 | 420 | def _create_people(self): 421 | people = ( 422 | ('huey', 'leifer'), 423 | ('mickey', 'leifer'), 424 | ('zaizee', 'owen'), 425 | ('beanie', 'owen'), 426 | ('scout', 'owen'), 427 | ) 428 | for first, last in people: 429 | self.Person.create(first=first, last=last) 430 | 431 | def assertPeople(self, expr, first_names): 432 | results = self.Person.query(expr) 433 | self.assertEqual([person.first for person in results], first_names) 434 | 435 | def test_query(self): 436 | self._create_people() 437 | 438 | self.assertPeople(self.Person.last == 'leifer', ['huey', 'mickey']) 439 | self.assertPeople( 440 | self.Person.last == 'owen', 441 | ['zaizee', 'beanie', 'scout']) 442 | 443 | def test_get(self): 444 | self._create_people() 445 | huey = self.Person.get(self.Person.first == 'huey') 446 | self.assertEqual(huey.first, 'huey') 447 | self.assertEqual(huey.last, 'leifer') 448 | 449 | zaizee = self.Person.get( 450 | (self.Person.first == 'zaizee') & 451 | (self.Person.last == 'owen')) 452 | self.assertEqual(zaizee.first, 'zaizee') 453 | self.assertEqual(zaizee.last, 'owen') 454 | 455 | self.assertIsNone(self.Person.get(self.Person.first == 'not here')) 456 | 457 | def test_query_tree(self): 458 | self._create_people() 459 | 460 | expr = (self.Person.last == 'leifer') | (self.Person.first == 'scout') 461 | self.assertPeople(expr, ['huey', 'mickey', 'scout']) 462 | 463 | expr = ( 464 | (self.Person.last == 'leifer') | 465 | (self.Person.first == 'scout') | 466 | (self.Person.first >= 'z')) 467 | self.assertPeople(expr, ['huey', 'mickey', 'zaizee', 'scout']) 468 | 469 | def test_less_than(self): 470 | self._create_people() 471 | 472 | # Less than an existing value. 473 | expr = (self.Person.first < 'mickey') 474 | self.assertPeople(expr, ['huey', 'beanie']) 475 | 476 | # Less than or equal to an existing value. 477 | expr = (self.Person.first <= 'mickey') 478 | self.assertPeople(expr, ['huey', 'mickey', 'beanie']) 479 | 480 | # Less than a non-existant value. 481 | expr = (self.Person.first < 'nuggie') 482 | self.assertPeople(expr, ['huey', 'mickey', 'beanie']) 483 | 484 | # Less than or equal to a non-existant value. 485 | expr = (self.Person.first <= 'nuggie') 486 | self.assertPeople(expr, ['huey', 'mickey', 'beanie']) 487 | 488 | def test_greater_than(self): 489 | self._create_people() 490 | 491 | # Greater than an existing value. 492 | expr = (self.Person.first > 'mickey') 493 | self.assertPeople(expr, ['zaizee', 'scout']) 494 | 495 | # Greater than or equal to an existing value. 496 | expr = (self.Person.first >= 'mickey') 497 | self.assertPeople(expr, ['mickey', 'zaizee', 'scout']) 498 | 499 | # Greater than a non-existant value. 500 | expr = (self.Person.first > 'nuggie') 501 | self.assertPeople(expr, ['zaizee', 'scout']) 502 | 503 | # Greater than or equal to a non-existant value. 504 | expr = (self.Person.first >= 'nuggie') 505 | self.assertPeople(expr, ['zaizee', 'scout']) 506 | 507 | def test_startswith(self): 508 | names = ('aaa', 'aab', 'abb', 'bbb', 'ba') 509 | for name in names: 510 | self.Person.create(first=name, last=name) 511 | 512 | self.assertPeople( 513 | self.Person.last.startswith('a'), 514 | ['aaa', 'aab', 'abb']) 515 | 516 | self.assertPeople(self.Person.last.startswith('aa'), ['aaa', 'aab']) 517 | self.assertPeople(self.Person.last.startswith('aaa'), ['aaa']) 518 | self.assertPeople(self.Person.last.startswith('aaaa'), []) 519 | self.assertPeople(self.Person.last.startswith('b'), ['bbb', 'ba']) 520 | self.assertPeople(self.Person.last.startswith('bb'), ['bbb']) 521 | self.assertPeople(self.Person.last.startswith('c'), []) 522 | 523 | def create_numeric(self): 524 | values = ( 525 | (1, 2.0, datetime.date(2015, 1, 2)), 526 | (2, 3.0, datetime.date(2015, 1, 3)), 527 | (3, 4.0, datetime.date(2015, 1, 4)), 528 | (10, 10.0, datetime.date(2015, 1, 10)), 529 | (11, 11.0, datetime.date(2015, 1, 11)), 530 | ) 531 | for x, y, z in values: 532 | self.Numeric.create(x=x, y=y, z=z) 533 | 534 | def assertNumeric(self, expr, xs): 535 | query = self.Numeric.query(expr) 536 | self.assertEqual([n.x for n in query], xs) 537 | 538 | def test_query_numeric(self): 539 | self.create_numeric() 540 | 541 | X = self.Numeric.x 542 | Y = self.Numeric.y 543 | 544 | self.assertNumeric(X == 3, [3]) 545 | self.assertNumeric(X < 3, [1, 2]) 546 | self.assertNumeric(X <= 3, [1, 2, 3]) 547 | self.assertNumeric(X > 3, [10, 11]) 548 | self.assertNumeric(X >= 3, [3, 10, 11]) 549 | 550 | # Missing values. 551 | self.assertNumeric(X < 4, [1, 2, 3]) 552 | self.assertNumeric(X <= 4, [1, 2, 3]) 553 | self.assertNumeric(X > 4, [10, 11]) 554 | self.assertNumeric(X >= 4, [10, 11]) 555 | 556 | # Higher than largest. 557 | self.assertNumeric(X > 11, []) 558 | self.assertNumeric(X > 12, []) 559 | self.assertNumeric(X >= 12, []) 560 | 561 | # Lower than smallest. 562 | self.assertNumeric(X < 1, []) 563 | self.assertNumeric(self.Numeric.x < 0, []) # XXX: ?? 564 | self.assertNumeric(self.Numeric.x <= 0, []) 565 | 566 | # Floats. 567 | self.assertNumeric(Y == 4.0, [3]) 568 | self.assertNumeric(Y < 4.0, [1, 2]) 569 | self.assertNumeric(Y <= 4.0, [1, 2, 3]) 570 | self.assertNumeric(Y > 4.0, [10, 11]) 571 | self.assertNumeric(Y >= 4.0, [3, 10, 11]) 572 | 573 | # Missing values. 574 | self.assertNumeric(Y < 5.0, [1, 2, 3]) 575 | self.assertNumeric(Y <= 5.0, [1, 2, 3]) 576 | self.assertNumeric(Y > 5.04, [10, 11]) 577 | self.assertNumeric(Y >= 5.04, [10, 11]) 578 | 579 | # Higher than largest. 580 | self.assertNumeric(Y > 11., []) 581 | self.assertNumeric(Y > 11.1, []) 582 | self.assertNumeric(Y >= 11.1, []) 583 | 584 | # Lower than smallest. 585 | self.assertNumeric(Y < 2.0, []) 586 | self.assertNumeric(self.Numeric.y < 0, []) # XXX: ?? 587 | self.assertNumeric(self.Numeric.y <= 0, []) 588 | 589 | def test_query_numeric_complex(self): 590 | # 1, 2, 3, 10, 11 --- 2., 3., 4., 10., 11. 591 | self.create_numeric() 592 | 593 | X = self.Numeric.x 594 | Y = self.Numeric.y 595 | 596 | expr = ((X <= 2) | (Y > 9)) 597 | self.assertNumeric(expr, [1, 2, 10, 11]) 598 | 599 | expr = ( 600 | ((X < 1) | (Y > 10)) | 601 | ((X > 14) & (Y < 1)) | 602 | (X == 3)) 603 | self.assertNumeric(expr, [3, 11]) 604 | 605 | expr = ((X != 2) & (X != 3)) 606 | self.assertNumeric(expr, [1, 10, 11]) 607 | 608 | 609 | class GraphTests(object): 610 | def setUp(self): 611 | super(GraphTests, self).setUp() 612 | self.H = Hexastore(self.db) 613 | 614 | def create_graph_data(self): 615 | data = ( 616 | ('charlie', 'likes', 'huey'), 617 | ('charlie', 'likes', 'mickey'), 618 | ('charlie', 'likes', 'zaizee'), 619 | ('charlie', 'is', 'human'), 620 | ('connor', 'likes', 'huey'), 621 | ('connor', 'likes', 'mickey'), 622 | ('huey', 'eats', 'catfood'), 623 | ('huey', 'is', 'cat'), 624 | ('mickey', 'eats', 'anything'), 625 | ('mickey', 'is', 'dog'), 626 | ('zaizee', 'eats', 'catfood'), 627 | ('zaizee', 'is', 'cat'), 628 | ) 629 | self.H.store_many(data) 630 | 631 | def test_search_extended(self): 632 | self.create_graph_data() 633 | X = self.H.v.x 634 | Y = self.H.v.y 635 | Z = self.H.v.z 636 | result = self.H.search( 637 | (X, 'likes', Y), 638 | (Y, 'is', 'cat'), 639 | (Z, 'likes', Y)) 640 | self.assertEqual(result['x'], set(['charlie', 'connor'])) 641 | self.assertEqual(result['y'], set(['huey', 'zaizee'])) 642 | self.assertEqual(result['z'], set(['charlie', 'connor'])) 643 | 644 | self.H.store_many(( 645 | ('charlie', 'likes', 'connor'), 646 | ('connor', 'likes', 'charlie'), 647 | ('connor', 'is', 'baby'), 648 | ('connor', 'is', 'human'), 649 | ('nash', 'is', 'baby'), 650 | ('nash', 'is', 'human'), 651 | ('connor', 'lives', 'ks'), 652 | ('nash', 'lives', 'nv'), 653 | ('charlie', 'lives', 'ks'))) 654 | 655 | result = self.H.search( 656 | ('charlie', 'likes', X), 657 | (X, 'is', 'baby'), 658 | (X, 'lives', 'ks')) 659 | self.assertEqual(result, {'x': set(['connor'])}) 660 | 661 | result = self.H.search( 662 | (X, 'is', 'baby'), 663 | (X, 'likes', Y), 664 | (Y, 'lives', 'ks')) 665 | self.assertEqual(result, { 666 | 'x': set(['connor']), 667 | 'y': set(['charlie']), 668 | }) 669 | 670 | def assertTriples(self, result, expected): 671 | result = list(result) 672 | self.assertEqual(len(result), len(expected)) 673 | for i1, i2 in zip(result, expected): 674 | self.assertEqual( 675 | (i1['s'], i1['p'], i1['o']), i2) 676 | 677 | def test_query(self): 678 | self.create_graph_data() 679 | res = self.H.query('charlie', 'likes') 680 | self.assertTriples(res, ( 681 | ('charlie', 'likes', 'huey'), 682 | ('charlie', 'likes', 'mickey'), 683 | ('charlie', 'likes', 'zaizee'), 684 | )) 685 | 686 | res = self.H.query(p='is', o='cat') 687 | self.assertTriples(res, ( 688 | ('huey', 'is', 'cat'), 689 | ('zaizee', 'is', 'cat'), 690 | )) 691 | 692 | res = self.H.query(s='huey') 693 | self.assertTriples(res, ( 694 | ('huey', 'eats', 'catfood'), 695 | ('huey', 'is', 'cat'), 696 | )) 697 | 698 | res = self.H.query(o='huey') 699 | self.assertTriples(res, ( 700 | ('charlie', 'likes', 'huey'), 701 | ('connor', 'likes', 'huey'), 702 | )) 703 | 704 | def test_search(self): 705 | self.create_graph_data() 706 | X = self.H.v('x') 707 | result = self.H.search( 708 | {'s': 'charlie', 'p': 'likes', 'o': X}, 709 | {'s': X, 'p': 'eats', 'o': 'catfood'}, 710 | {'s': X, 'p': 'is', 'o': 'cat'}) 711 | self.assertEqual(result, {'x': set(['huey', 'zaizee'])}) 712 | 713 | def test_search_simple(self): 714 | self.create_friends() 715 | X = self.H.v('x') 716 | result = self.H.search({'s': X, 'p': 'friend', 'o': 'charlie'}) 717 | self.assertEqual(result, {'x': set(['huey', 'zaizee'])}) 718 | 719 | def test_search_2var(self): 720 | self.create_friends() 721 | X = self.H.v('x') 722 | Y = self.H.v('y') 723 | 724 | result = self.H.search( 725 | {'s': X, 'p': 'friend', 'o': 'charlie'}, 726 | {'s': Y, 'p': 'friend', 'o': X}) 727 | self.assertEqual(result, { 728 | 'x': set(['huey']), 729 | 'y': set(['charlie']), 730 | }) 731 | 732 | result = self.H.search( 733 | ('charlie', 'friend', X), 734 | (X, 'friend', Y), 735 | (Y, 'friend', 'nuggie')) 736 | self.assertEqual(result, { 737 | 'x': set(['huey']), 738 | 'y': set(['mickey']), 739 | }) 740 | 741 | result = self.H.search( 742 | ('huey', 'friend', X), 743 | (X, 'friend', Y)) 744 | self.assertEqual(result['y'], set(['huey', 'nuggie'])) 745 | 746 | def test_search_mutual(self): 747 | self.create_friends() 748 | X = self.H.v('x') 749 | Y = self.H.v('y') 750 | 751 | result = self.H.search( 752 | {'s': X, 'p': 'friend', 'o': Y}, 753 | {'s': Y, 'p': 'friend', 'o': X}) 754 | self.assertEqual(result['y'], set(['charlie', 'huey'])) 755 | 756 | def create_friends(self): 757 | data = ( 758 | ('charlie', 'friend', 'huey'), 759 | ('huey', 'friend', 'charlie'), 760 | ('huey', 'friend', 'mickey'), 761 | ('zaizee', 'friend', 'charlie'), 762 | ('zaizee', 'friend', 'mickey'), 763 | ('mickey', 'friend', 'nuggie'), 764 | ) 765 | for item in data: 766 | self.H.store(*item) 767 | 768 | 769 | class HashTests(KVKitTests, BaseTestCase): 770 | database_class = HashDB 771 | 772 | 773 | class TreeTests(KVKitTests, GraphTests, ModelTests, SliceTests, BaseTestCase): 774 | database_class = TreeDB 775 | 776 | 777 | class CacheHashTests(KVKitTests, BaseTestCase): 778 | database_class = CacheHashDB 779 | 780 | 781 | class CacheTreeTests(KVKitTests, GraphTests, ModelTests, SliceTests, 782 | BaseTestCase): 783 | database_class = CacheTreeDB 784 | 785 | 786 | if BerkeleyDB: 787 | class BerkeleyDBTests(SliceTests, GraphTests, ModelTests, BaseTestCase): 788 | database_class = BerkeleyDB 789 | 790 | 791 | if LevelDB: 792 | class LevelDBTests(SliceTests, GraphTests, ModelTests, BaseTestCase): 793 | database_class = LevelDB 794 | 795 | 796 | if LSM: 797 | class LSMTests(SliceTests, GraphTests, ModelTests, BaseTestCase): 798 | database_class = LSM 799 | 800 | 801 | if ForestDB: 802 | class ForestDBTests(SliceTests, GraphTests, ModelTests, BaseTestCase): 803 | database_class = ForestDB 804 | 805 | if Sophia: 806 | class SophiaTests(SliceTests, GraphTests, ModelTests, BaseTestCase): 807 | database_class = Sophia 808 | _test_dir = '/tmp/sophia-db' 809 | 810 | def create_db(self): 811 | return self.database_class(os.path.join(self._test_dir, 'test-db')) 812 | 813 | def delete_db(self): 814 | self.db.close() 815 | if os.path.exists(self._test_dir): 816 | shutil.rmtree(self._test_dir) 817 | 818 | 819 | if RocksDB: 820 | # RocksDB does not implement an actual `close()` method, so we cannot 821 | # reliably re-use the same database file due to locks hanging around. 822 | # For that reason, each test needs to either re-use the same DB or use 823 | # a new db file. I opted for the latter. 824 | class RocksDBTests(SliceTests, GraphTests, ModelTests, BaseTestCase): 825 | database_class = RocksDB 826 | 827 | def create_db(self): 828 | return self.database_class(tempfile.mktemp()) 829 | 830 | 831 | if __name__ == '__main__': 832 | unittest.main(argv=sys.argv) 833 | -------------------------------------------------------------------------------- /runtests.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import optparse 4 | import sys 5 | import unittest 6 | 7 | from kvkit import tests 8 | 9 | def runtests(cases=None): 10 | if cases: 11 | suite = unittest.TestLoader().loadTestsFromNames(cases) 12 | else: 13 | suite = unittest.TestLoader().loadTestsFromModule(tests) 14 | result = unittest.TextTestRunner(verbosity=2).run(suite) 15 | if result.failures: 16 | sys.exit(1) 17 | elif result.errors: 18 | sys.exit(2) 19 | sys.exit(0) 20 | 21 | if __name__ == '__main__': 22 | parser = optparse.OptionParser() 23 | opt = parser.add_option 24 | opt('-b', '--berkeleydb', dest='berkeleydb', action='store_true') 25 | opt('-H', '--kyoto-hash', dest='kyoto_hash', action='store_true') 26 | opt('-k', '--kyoto', dest='kyoto', action='store_true') 27 | opt('-l', '--lsm', dest='lsm', action='store_true') 28 | opt('-m', '--minimal', dest='minimal', action='store_true') 29 | opt('-r', '--rocksdb', dest='rocksdb', action='store_true') 30 | opt('-s', '--sophia', dest='sophia', action='store_true') 31 | opt('-T', '--kyoto-tree', dest='kyoto_tree', action='store_true') 32 | opt('-v', '--leveldb', dest='leveldb', action='store_true') 33 | 34 | options, args = parser.parse_args() 35 | cases = set() 36 | if options.minimal: 37 | cases = ['TreeTests'] 38 | else: 39 | if options.berkeleydb: 40 | cases.add('BerkeleyDBTests') 41 | if options.kyoto: 42 | cases.update(('HashTests', 'TreeTests', 'CacheHashTests', 43 | 'CacheTreeTests')) 44 | if options.kyoto_hash: 45 | cases.update(('HashTests', 'CacheHashTests')) 46 | if options.kyoto_tree: 47 | cases.update(('TreeTests', 'CacheTreeTests')) 48 | if options.leveldb: 49 | cases.add('LevelDBTests') 50 | if options.lsm: 51 | cases.add('LSMTests') 52 | if options.sophia: 53 | cases.add('SophiaTests') 54 | if options.rocksdb: 55 | cases.add('RocksDBTests') 56 | 57 | cases = ['kvkit.tests.%s' % case for case in sorted(cases)] 58 | runtests(cases) 59 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import os 2 | from setuptools import setup, find_packages 3 | 4 | 5 | setup( 6 | name='kvkit', 7 | version=__import__('kvkit').__version__, 8 | description='high-level python toolkit for ordered key/value stores', 9 | author='Charles Leifer', 10 | author_email='coleifer@gmail.com', 11 | url='http://github.com/coleifer/kvkit/', 12 | packages=find_packages(), 13 | package_data = { 14 | 'kvkit': [ 15 | ], 16 | }, 17 | classifiers=[ 18 | 'Development Status :: 4 - Beta', 19 | 'Environment :: Web Environment', 20 | 'Intended Audience :: Developers', 21 | 'License :: OSI Approved :: MIT License', 22 | 'Operating System :: OS Independent', 23 | 'Programming Language :: Python', 24 | ], 25 | test_suite='runtests.runtests', 26 | ) 27 | --------------------------------------------------------------------------------