├── .gitignore ├── LICENSE ├── README.markdown ├── benchmark ├── benchmark_fixed_list.py └── benchmark_python_redis.py ├── fixedlist └── __init__.py └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | 5 | # C extensions 6 | *.so 7 | 8 | # Distribution / packaging 9 | .Python 10 | env/ 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | lib/ 17 | lib64/ 18 | parts/ 19 | sdist/ 20 | var/ 21 | *.egg-info/ 22 | .installed.cfg 23 | *.egg 24 | 25 | # PyInstaller 26 | # Usually these files are written by a python script from a template 27 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 28 | *.manifest 29 | *.spec 30 | 31 | # Installer logs 32 | pip-log.txt 33 | pip-delete-this-directory.txt 34 | 35 | # Unit test / coverage reports 36 | htmlcov/ 37 | .tox/ 38 | .coverage 39 | .cache 40 | nosetests.xml 41 | coverage.xml 42 | 43 | # Translations 44 | *.mo 45 | *.pot 46 | 47 | # Django stuff: 48 | *.log 49 | 50 | # Sphinx documentation 51 | docs/_build/ 52 | 53 | # PyBuilder 54 | target/ 55 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2018 Doist 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /README.markdown: -------------------------------------------------------------------------------- 1 | fixedlist: fast performance fixed list for Redis 2 | ================================================ 3 | 4 | This Python library makes it possible to implement a fast fixed list structure for Redis with following properties: 5 | 6 | * Fast inserts, updates and fetches 7 | * 2x faster perfomance than pure Redis implementation 8 | * 1.4x less memory footprint (due to gziped data) 9 | * No duplicates inside the list 10 | 11 | Requires Redis 2.6+ and newest version of redis-py. 12 | 13 | 14 | Installation 15 | ============ 16 | 17 | Can be installed very easily via 18 | 19 | $ pip install fixedlist 20 | 21 | 22 | Benchmark 23 | ========= 24 | 25 | Testing fixedlist against a pure Redis implementaiton of a fixedlist yields good results. 26 | 27 | The benchmarks are in the `benchmark/` folder and they store 300 lists with 200 values each. The lists are int and strings lists. 28 | 29 | * fixedlist is about 2.4x faster (47s vs. 117s on a Mac Book Pro from 2014) 30 | * fixedlist uses 1.4x less memory (1.24MB vs. 1.84MB) 31 | 32 | The benchmark is run in a following way: 33 | 34 | * time python benchmark/benchmark_fixed_list.py 35 | * time python benchmark/benchmark_python_redis.py 36 | 37 | Redis stats are fetched from `redis-cli info` command. 38 | 39 | 40 | Examples 41 | ======== 42 | 43 | Setting things up: 44 | 45 | ```python 46 | import fixedlist 47 | fixedlist.set_redis(redis_host='locahost', redis_port=6380) 48 | ``` 49 | 50 | Add a value to a list: 51 | 52 | ```python 53 | fixedlist.add('hello', 'world') 54 | assert fixedlist.get('hello') == ['world'] 55 | ``` 56 | 57 | Add mutliple values to multiple keys at once: 58 | 59 | ```python 60 | fixedlist.add(['hello1', 'hello2'], ['world1', 'world2']) 61 | assert fixedlist.get('hello1') == ['world1', 'world2'] 62 | ``` 63 | 64 | Fetch multiple at once: 65 | 66 | ```python 67 | assert fixedlist.get_multi(['hello1', 'hello2']) ==\ 68 | {'hello1': ['world1', 'world2'], 69 | 'hello2': ['world1', 'world2']} 70 | ``` 71 | 72 | Remove a value: 73 | 74 | ```python 75 | fixedlist.remove('hello1', 'world1') 76 | assert fixedlist.get('hello1') == ['world2'] 77 | ``` 78 | 79 | Handle duplicates 80 | 81 | ```python 82 | fixedlist.empty('hello') 83 | fixedlist.add('hello', 'world') 84 | fixedlist.add('hello', 'world') 85 | assert fixedlist.get('hello') == ['world'] 86 | ``` 87 | 88 | 89 | Full API 90 | ======== 91 | 92 | Redis related: 93 | 94 | * `fixedlist.set_redis(system_name='default', redis_host='localhost', redis_port=6379, **redis_kws)`: Setup Redis 95 | * `fixedlist.get_redis(system='default')`: Return a Redis client to `system` 96 | 97 | List fetches: 98 | 99 | * `fixedlist.get(list_key, system='default')`: Return values for `list_key` 100 | * `fixedlist.get_multi(list_keys, system='default')`: Return a dictionary of values for `list_keys` 101 | 102 | List adding/setting: 103 | 104 | * `fixedlist.add(list_keys, values_to_add, system='default', limit=200)`: Add `values_to_add` to `list_keys`. Limit the size of `list_keys` to `limit` 105 | * `fixedlist.set(list_keys, values, system='default')`: Set `list_keys` to `values` 106 | 107 | List removing/resetting: 108 | 109 | * `fixedlist.remove(list_keys, values_to_remove, system='default')`: Remove `values_to_remove` from `list_keys` 110 | * `fixedlist.empty(list_keys, system='default')`: Empty the values of `list_keys` 111 | * `fixedlist.varnish(list_keys, system='default')`: Delete `list_keys` keys 112 | 113 | Caveat 114 | ====== 115 | 116 | Values should not contain the raw `~` character as this is used internally as 117 | a list separator. 118 | 119 | Copyright 120 | ========= 121 | 122 | 2018 by Doist Ltd. 123 | 124 | 125 | Developer 126 | ========= 127 | 128 | [Amir Salihefendic](http://amix.dk) 129 | 130 | 131 | License 132 | ======= 133 | 134 | MIT 135 | -------------------------------------------------------------------------------- /benchmark/benchmark_fixed_list.py: -------------------------------------------------------------------------------- 1 | import random 2 | import string 3 | 4 | import fixedlist 5 | 6 | KEYS = 300 7 | VALUES = 250 8 | 9 | def main(): 10 | fixedlist.set_redis(redis_host='0.0.0.0', redis_port=6380) 11 | 12 | # Ints 13 | for i in xrange(0, KEYS): 14 | list_key = 'test_list_%s' % i 15 | fixedlist.varnish(list_key) 16 | for i in range(0, VALUES): 17 | fixedlist.add(list_key, [i, i+1]) 18 | assert len(fixedlist.get(list_key)) <= 200 19 | 20 | # Strings 21 | for i in xrange(0, KEYS): 22 | list_key = 'test_list_%s' % i 23 | fixedlist.varnish(list_key) 24 | for i in range(0, VALUES): 25 | fixedlist.add(list_key, [randomword(6), randomword(6)]) 26 | assert len(fixedlist.get(list_key)) <= 200 27 | 28 | def randomword(length): 29 | return ''.join(random.choice(string.lowercase) for i in range(length)) 30 | 31 | if __name__ == '__main__': 32 | main() 33 | -------------------------------------------------------------------------------- /benchmark/benchmark_python_redis.py: -------------------------------------------------------------------------------- 1 | import random 2 | import string 3 | 4 | import fixedlist 5 | 6 | KEYS = 300 7 | VALUES = 250 8 | 9 | def main(): 10 | fixedlist.set_redis(redis_host='0.0.0.0', redis_port=6380) 11 | rcli = fixedlist.get_redis() 12 | 13 | # Ints 14 | for i in xrange(0, KEYS): 15 | list_key = 'test_list_i_%s' % i 16 | rcli.delete(list_key) 17 | for i in range(0, VALUES): 18 | list_add(rcli, list_key, [i, i+1]) 19 | assert len(list_get(rcli, list_key)) <= 200 20 | 21 | # Strings 22 | for i in xrange(0, KEYS): 23 | list_key = 'test_list_s_%s' % i 24 | rcli.delete(list_key) 25 | for i in range(0, VALUES): 26 | list_add(rcli, list_key, [randomword(6), randomword(6)]) 27 | assert len(list_get(rcli, list_key)) <= 200 28 | 29 | def list_add(rcli, list_key, values, limit=200): 30 | with rcli.pipeline() as p: 31 | p.watch(list_key) 32 | for value in values: 33 | p.lrem(list_key, value) 34 | p.lpush(list_key, value) 35 | p.ltrim(list_key, 0, limit-1) 36 | p.unwatch() 37 | 38 | def list_get(rcli, list_key, limit=200): 39 | return rcli.lrange(list_key, 0, limit) 40 | 41 | def randomword(length): 42 | return ''.join(random.choice(string.lowercase) for i in range(length)) 43 | 44 | if __name__ == '__main__': 45 | main() 46 | -------------------------------------------------------------------------------- /fixedlist/__init__.py: -------------------------------------------------------------------------------- 1 | import types 2 | import zlib 3 | import redis 4 | 5 | from redis import WatchError 6 | 7 | DEFAULT_RETRIES = 5 8 | 9 | 10 | #--- System specific ---------------------------------------------- 11 | REDIS_SYSTEMS = {} 12 | 13 | def set_redis(system_name='default', redis_host='localhost', 14 | redis_port=6379, db=0, **kw): 15 | """Setup a system.""" 16 | REDIS_SYSTEMS[system_name] = redis.Redis(host=redis_host, 17 | port=redis_port, 18 | db=db, **kw) 19 | 20 | def get_redis(system='default'): 21 | return REDIS_SYSTEMS.get(system) 22 | 23 | 24 | #--- List functions ---------------------------------------------- 25 | def get(list_key, system='default'): 26 | """Return list with key `list_key`. 27 | 28 | If a list isn't created, then an empty list is returned. 29 | """ 30 | result = get_redis(system).get(list_key) 31 | return _decode_list_row(result) 32 | 33 | 34 | def get_multi(list_keys, system='default'): 35 | """Return a dictionary of lists with keys `list_keys`. 36 | 37 | If a list isn't created, then an entry with an empty list is returned. 38 | """ 39 | result = {} 40 | 41 | results = get_redis(system).mget(list_keys) 42 | 43 | for i, key in enumerate(list_keys): 44 | result[key] = _decode_list_row(results[i]) 45 | 46 | return result 47 | 48 | 49 | 50 | def _multi_list_op(list_keys, values_to_add, modify, check=None, system='default', limit=200, retries=DEFAULT_RETRIES): 51 | if not check: 52 | check = lambda k, l, vta: True 53 | 54 | list_keys = _ensure_list(list_keys) 55 | values_to_add = _ensure_list(values_to_add) 56 | 57 | r = get_redis(system) 58 | with r.pipeline() as p: 59 | while retries > 0: 60 | try: 61 | for k in list_keys: 62 | p.watch(k) 63 | current_values = {} 64 | for k in list_keys: 65 | current_values[k] = _decode_list_row(p.get(k)) 66 | 67 | should_update = False 68 | for key, current_list in current_values.iteritems(): 69 | if check(key, current_list, values_to_add): 70 | should_update = True 71 | break 72 | 73 | if not should_update: 74 | p.unwatch() 75 | return 76 | 77 | p.multi() 78 | for key, current_list in current_values.iteritems(): 79 | new_list = modify(key, list(current_list), values_to_add, limit) 80 | if current_list != new_list: 81 | p.set(key, _encode_list(new_list)) 82 | p.execute() 83 | return current_values 84 | except WatchError: 85 | retries -= 1 86 | continue 87 | 88 | def add(list_keys, values_to_add, system='default', limit=200, retries=DEFAULT_RETRIES): 89 | """Add `values_to_add` to lists with keys `list_keys`. 90 | 91 | Example:: 92 | 93 | >>> fixed_list.add(['key1', 'key3'], ['some value']) 94 | >>> fixed_list.add('key1', ['some value']) 95 | """ 96 | def check(k, l, vta): 97 | x = any(val not in l for val in vta) 98 | return x 99 | 100 | def modify(k, l, vta, limit): 101 | only_new = [ val for val in vta if val not in l ] 102 | if len(only_new) > 0: 103 | l.extend(only_new) 104 | return l[-limit:] 105 | _multi_list_op(list_keys, values_to_add, modify, check, system, limit, retries) 106 | 107 | 108 | def remove(list_keys, values_to_remove, system='default', retries=DEFAULT_RETRIES): 109 | """Remove `values_to_remove` from lists with keys `list_keys`. 110 | 111 | Example:: 112 | 113 | >>> fixed_list.add(['key1', 'key3'], ['some value']) 114 | >>> fixed_list.add('key1', ['some value']) 115 | """ 116 | 117 | def check(k, l, vtr): 118 | return any(val in l for val in vtr ) 119 | 120 | def modify(k, l, vtr, limit): 121 | start_len = len(l) 122 | for v in vtr: 123 | try: 124 | l.remove(v) 125 | except: 126 | pass 127 | if start_len > len(l): 128 | return l 129 | 130 | return _multi_list_op(list_keys, values_to_remove, modify, check, system, None, retries) 131 | 132 | def set(list_keys, values, system='default'): 133 | """Set lists with keys `list_keys` to `values`. 134 | 135 | Example:: 136 | 137 | >>> fixed_list.set(['key1', 'key3'], ['1', '2']) 138 | >>> fixed_list.set('key1', ['1', '2']) 139 | """ 140 | list_keys = _ensure_list(list_keys) 141 | values = _ensure_list(values) 142 | values = _encode_list(values) 143 | 144 | if not list_keys: 145 | return 146 | 147 | r = get_redis(system) 148 | with r.pipeline() as p: 149 | p.multi() 150 | for key in list_keys: 151 | p.set(key, values) 152 | p.execute() 153 | 154 | return True 155 | 156 | 157 | def empty(list_keys, system='default'): 158 | """Reset lists with keys `list_keys`. 159 | 160 | Example:: 161 | 162 | >>> fixed_list.list_empty(['test']) 163 | """ 164 | set(list_keys, [], system) 165 | return True 166 | 167 | 168 | def varnish(list_keys, system='default'): 169 | """Delete lists completely with keys `list_keys`.""" 170 | list_keys = _ensure_list(list_keys) 171 | 172 | if not list_keys: 173 | return 174 | 175 | r = get_redis(system) 176 | with r.pipeline() as p: 177 | p.multi() 178 | for key in list_keys: 179 | p.delete(key) 180 | p.execute() 181 | return True 182 | 183 | 184 | #--- Internal ---------------------------------------------- 185 | def _decode_list_row(value): 186 | if not value: 187 | return [] 188 | value = zlib.decompress(value) 189 | return [ v for v in value.split(r'~') if v ] 190 | 191 | def _encode_list(list_value): 192 | if not list_value: 193 | return '' 194 | 195 | v_encoded = [] 196 | for v in list_value: 197 | v_encoded.append('%s~' % v) 198 | result = ''.join(v_encoded) 199 | return zlib.compress(result) 200 | 201 | def _ensure_list(some_value): 202 | if type(some_value) == types.ListType: 203 | return some_value 204 | else: 205 | return [some_value] 206 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # Copyright (c) 2007 Qtrac Ltd. All rights reserved. 3 | # This module is free software: you can redistribute it and/or modify 4 | # it under the terms of the GNU General Public License as published by 5 | # the Free Software Foundation, either version 3 of the License, or (at 6 | # your option) any later version. 7 | 8 | from setuptools import setup 9 | 10 | setup(name='fixedlist', 11 | version = '1.0', 12 | author="amix", 13 | author_email="amix@amix.dk", 14 | url="http://www.amix.dk/", 15 | install_requires = ['redis>=2.7.1'], 16 | classifiers=[ 17 | "Development Status :: 5 - Production/Stable", 18 | "Intended Audience :: Developers", 19 | "License :: OSI Approved :: BSD License", 20 | "Operating System :: OS Independent", 21 | "Programming Language :: Python", 22 | "Topic :: Software Development :: Libraries :: Python Modules", 23 | ], 24 | packages=['fixedlist'], 25 | include_package_data=True, 26 | zip_safe=False, 27 | platforms=["Any"], 28 | license="BSD", 29 | keywords='redis fixed list small list short list', 30 | description="Fast performance fixed list for Redis", 31 | long_description="""\ 32 | fixedlist 33 | --------- 34 | This Python library makes it possible to implement a fast fixed list structure for Redis with following properties: 35 | 36 | * Fixed size of the list 37 | * Fast inserts, updates and fetches 38 | * Small memory footprint with gziped data 39 | * No duplicates inside the list 40 | 41 | Requires Redis 2.6+ and newest version of redis-py. 42 | 43 | 44 | Installation 45 | ------------ 46 | 47 | Can be installed very easily via:: 48 | 49 | $ pip install fixedlist 50 | 51 | For more help look at https://github.com/Doist/fixedlist 52 | 53 | Copyright: 2015 by Doist Ltd. 54 | 55 | Developer: Amir Salihefendic ( http://amix.dk ) 56 | 57 | License: BSD""") 58 | --------------------------------------------------------------------------------