├── .gitignore ├── LICENSE ├── README.markdown ├── redis_ds ├── __init__.py ├── redis_config.py ├── redis_dict.py ├── redis_hash_dict.py ├── redis_list.py ├── redis_set.py ├── serialization.py └── tests.py ├── requirements.txt └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | *~ 2 | *.pyc 3 | build 4 | dist 5 | env 6 | *.egg-info 7 | *#* -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 Will Larson 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. -------------------------------------------------------------------------------- /README.markdown: -------------------------------------------------------------------------------- 1 | [redis]: http://code.google.com/p/redis/ "Redis" 2 | [redis-py]: http://github.com/andymccurdy/redis-py "Redis-Py" 3 | 4 | *If you'd like to be a contributor, send me a note! I'd be more than glad to pass this over to a more active maintainer.* 5 | 6 | This package exposes [Redis][redis] backed datastructures which bring together 7 | standard Python syntax with persistent in-memory storage. Redis operations 8 | are all atomic as well, meaning that a thoughtful approach can use them 9 | concurrently on multiple machines. 10 | 11 | Utimately, a trivial wrapper around [Redis-Py][redis-py]. 12 | 13 | 14 | ## Simple Installation 15 | 16 | The simplest approach to installation is: 17 | 18 | pip install redis 19 | pip install -e git+https://github.com/lethain/Redis-Python-Datastructures.git#egg=redis_ds 20 | 21 | ## Installation for development 22 | 23 | For development, you should checkout the repository, 24 | and install it into a [virtualenv](http://www.virtualenv.org/en/latest/): 25 | 26 | # get the code 27 | git clone https://github.com/lethain/Redis-Python-Datastructures.git 28 | cd Redis-Python-Datastructures.git 29 | 30 | # create and activate a virtualenv if you don't have one already 31 | virtualenv env 32 | . ./activate 33 | 34 | # install it 35 | pip install -r requirements.txt 36 | python setup.py develop 37 | 38 | 39 | ## Running tests 40 | 41 | With some embarassment, the tests currently run against local Redis using keys 42 | prefixed with "test_rds.", and taking great pains not to delete any other keys, 43 | so if you have a toy Redis node, the tests will not cause any harm, but you really 44 | shouldn't run the tests against a Redis node you care deeply about. 45 | 46 | Run them via: 47 | 48 | python src/redis_ds/tests.py 49 | 50 | The tests really ought to be run against a mocked out version of Redis, but that 51 | work hasn't been done yet. 52 | 53 | 54 | # Usage 55 | 56 | This section covers how to use this library to interface with Redis. 57 | 58 | 59 | ## Dictionary via Redis Strings 60 | 61 | Using the entire Redis cluster as a dictionary: 62 | 63 | 64 | >>> from redis_ds.redis_dict import RedisDict 65 | >>> x = RedisDict() 66 | >>> x 67 | {} 68 | >>> x['a'] = 100 69 | >>> x 70 | {'a': '100'} 71 | >>> x['a'] 72 | '100' 73 | >>> x['b'] 74 | >>> len(x) 75 | 1 76 | 77 | ## Dictionaries via Redis Hashes 78 | 79 | Using Redis hashes we can store multiple dictionaries in 80 | one Redis server. 81 | 82 | >>> from redis_ds.redis_hash_dict import RedisHashDict 83 | >>> x = RedisHashDict("some_hash_key") 84 | >>> x 85 | {} 86 | >>> x['a'] = 100 87 | >>> x 88 | {'a': '100'} 89 | >>> x['a'] 90 | '100' 91 | >>> x['b'] 92 | >>> len(x) 93 | 1 94 | 95 | ## Lists via Redis Lists 96 | 97 | We also have a kind-of-sort-off implementation of a list which 98 | certainly doesn't have the full flexibility of a Python list, 99 | but is persistent, synchronized and sharable. 100 | 101 | >>> from redis_ds.redis_list import RedisList 102 | >>> x = RedisList("my-list") 103 | >>> x 104 | RedisList([]) 105 | >>> x.append("a") 106 | 1 107 | >>> x.append("b") 108 | 2 109 | >>> x 110 | RedisList(['a', 'b']) 111 | >>> x[0] 112 | 'a' 113 | >>> x.pop() 114 | 'b' 115 | >>> x 116 | RedisList(['a']) 117 | 118 | It also provides access to blocking versions of pop, which 119 | with a little creativity you can use to create a message queue 120 | with workers. 121 | 122 | >>> x.pop(blocking=True) 123 | 'a' 124 | 125 | Woohoo. 126 | 127 | ## Sets 128 | 129 | Sets are also available thanks to work by [@hhuuggoo](https://github.com/hhuuggoo): 130 | 131 | >>> from redis_ds.redis_set import RedisSet 132 | >>> x = RedisSet() 133 | >>> x.add("a") 134 | >>> x.add("a") 135 | >>> x.add("b") 136 | >>> x.add("b") 137 | >>> len(x) 138 | 2 139 | >>> 'a' in x 140 | True 141 | >>> 'c' in x 142 | False 143 | >>> x.pop() 144 | 'a' 145 | >>> len(x) 146 | 1 147 | 148 | 149 | ## Serializing Values Stored in Redis 150 | 151 | Thanks to work by [@hhuuggoo](https://github.com/hhuuggoo), this library also 152 | supports serializing values before storing them in Redis. Each class has a 153 | serialized equivalent, for example the above hashmap example becomes: 154 | 155 | >>> from redis_ds.redis_hash_dict import PickleRedisHashDict 156 | >>> y = PickleRedisHashDict('some_other_key') 157 | >>> y 158 | {} 159 | >>> y['a'] = {'obj': 'ect'} 160 | >>> y 161 | {'a': {'obj': 'ect'}} 162 | >>> y['a']['obj'] 163 | 'ect' 164 | 165 | The same can be done using JSON instead of Pickle by changing it to: 166 | 167 | >>> from redis_ds.redis_hash_dict import JSONRedisHashDict 168 | 169 | and so on. The same is true for ``RedisList`` which has ``PickleRedisList`` 170 | and ``JSONRedisList``, and so on. -------------------------------------------------------------------------------- /redis_ds/__init__.py: -------------------------------------------------------------------------------- 1 | "Module for redis datastructures." 2 | __version__ = '0.1' 3 | -------------------------------------------------------------------------------- /redis_ds/redis_config.py: -------------------------------------------------------------------------------- 1 | """ 2 | Monkey patch configuration here... 3 | """ 4 | import redis 5 | 6 | CLIENT = redis.Redis() 7 | -------------------------------------------------------------------------------- /redis_ds/redis_dict.py: -------------------------------------------------------------------------------- 1 | """ 2 | This is module contains RedisDict, which allows users to interact with 3 | Redis strings using the standard Python dictionary syntax. 4 | 5 | Note that this uses an entire Redis database to back the dictionary, 6 | not a Redis hashmap. If you prefer an interface to a hashmap, the 7 | ``redis_hash_dict`` file does just that. 8 | """ 9 | import UserDict 10 | import redis_ds.redis_config as redis_config 11 | from redis_ds.serialization import PassThroughSerializer, PickleSerializer, JSONSerializer 12 | 13 | 14 | class RedisDict(UserDict.DictMixin, PassThroughSerializer): 15 | "Dictionary interface to Redis database." 16 | def __init__(self, redis_client=redis_config.CLIENT): 17 | """ 18 | Parameters: 19 | - redis_client: configured redis_client to use for all requests. 20 | should be fine to monkey patch this to set the 21 | default settings for your environment... 22 | """ 23 | self._client = redis_client 24 | 25 | def keys(self, pattern="*"): 26 | "Keys for Redis dictionary." 27 | return self._client.keys(pattern) 28 | 29 | def __len__(self): 30 | "Number of key-value pairs in dictionary/database." 31 | return self._client.dbsize() 32 | 33 | def __getitem__(self, key): 34 | "Retrieve a value by key." 35 | return self.deserialize(self._client.get(key)) 36 | 37 | def __setitem__(self, key, val): 38 | "Set a value by key." 39 | val = self.serialize(val) 40 | return self._client.set(key, val) 41 | 42 | def __delitem__(self, key): 43 | "Ensure deletion of a key from dictionary." 44 | return self._client.delete(key) 45 | 46 | def __contains__(self, key): 47 | "Check if database contains a specific key." 48 | return self._client.exists(key) 49 | 50 | def get(self, key, default=None): 51 | "Retrieve a key's value from the database falling back to a default." 52 | return self.__getitem__(key) or default 53 | 54 | 55 | class PickleRedisDict(RedisDict, PickleSerializer): 56 | "Serialize redis dictionary values via pickle." 57 | pass 58 | 59 | 60 | class JSONRedisDict(RedisDict, JSONSerializer): 61 | "Serialize redis dictionary values via JSON." 62 | pass 63 | -------------------------------------------------------------------------------- /redis_ds/redis_hash_dict.py: -------------------------------------------------------------------------------- 1 | """ 2 | Module contains RedisHashDict, which allows users to interact with Redis hashes 3 | as if they were Python dictionaries. 4 | """ 5 | import redis_ds.redis_config as redis_config 6 | from redis_ds.serialization import PassThroughSerializer, PickleSerializer, JSONSerializer 7 | import UserDict 8 | 9 | 10 | class RedisHashDict(UserDict.DictMixin, PassThroughSerializer): 11 | "A dictionary interface to Redis hashmaps." 12 | def __init__(self, hash_key, redis_client=redis_config.CLIENT): 13 | "Initialize the redis hashmap dictionary interface." 14 | self._client = redis_client 15 | self.hash_key = hash_key 16 | 17 | def keys(self): 18 | "Return all keys in the Redis hashmap." 19 | return self._client.hkeys(self.hash_key) 20 | 21 | def __len__(self): 22 | "Number of key-value pairs in the Redis hashmap." 23 | return self._client.hlen(self.hash_key) 24 | 25 | def __getitem__(self, key): 26 | "Retrieve a value from the hashmap." 27 | return self.deserialize(self._client.hget(self.hash_key, key)) 28 | 29 | def __setitem__(self, key, val): 30 | "Set a key's value in the hashmap." 31 | val = self.serialize(val) 32 | return self._client.hset(self.hash_key, key, val) 33 | 34 | def __delitem__(self, key): 35 | "Ensure a key does not exist in the hashmap." 36 | return self._client.hdel(self.hash_key, key) 37 | 38 | def __contains__(self, key): 39 | "Check if a key exists within the hashmap." 40 | return self._client.hexists(self.hash_key, key) 41 | 42 | def get(self, key, default=None): 43 | "Retrieve a key's value or a default value if the key does not exist." 44 | return self.__getitem__(key) or default 45 | 46 | 47 | class PickleRedisHashDict(RedisHashDict, PickleSerializer): 48 | "Serialize hashmap values using pickle." 49 | pass 50 | 51 | 52 | class JSONRedisHashDict(RedisHashDict, JSONSerializer): 53 | "Serialize hashmap values using JSON." 54 | pass 55 | -------------------------------------------------------------------------------- /redis_ds/redis_list.py: -------------------------------------------------------------------------------- 1 | "A pythonic interface to a Redis dictionary." 2 | import redis_ds.redis_config as redis_config 3 | from redis_ds.serialization import PassThroughSerializer, PickleSerializer, JSONSerializer 4 | 5 | 6 | class RedisList(PassThroughSerializer): 7 | "Interface to a Redis list." 8 | def __init__(self, list_key, redis_client=redis_config.CLIENT): 9 | "Initialize interface." 10 | self._client = redis_client 11 | self.list_key = list_key 12 | 13 | def __len__(self): 14 | "Number of values in list." 15 | return self._client.llen(self.list_key) 16 | 17 | def __getitem__(self, key): 18 | "Retrieve a value by index or values by slice syntax." 19 | if type(key) == int: 20 | return self.deserialize(self._client.lindex(self.list_key, key)) 21 | elif hasattr(key, 'start') and hasattr(key, 'stop'): 22 | start = key.start or 0 23 | stop = key.stop or -1 24 | values = self._client.lrange(self.list_key, start, stop) 25 | return [self.deserialize(value) for value in values] 26 | else: 27 | raise IndexError 28 | 29 | def __setitem__(self, pos, val): 30 | "Set the value at a position." 31 | val = self.serialize(val) 32 | return self._client.lset(self.list_key, pos, val) 33 | 34 | def append(self, val, head=False): 35 | "Append a value to list to rear or front." 36 | val = self.serialize(val) 37 | if head: 38 | return self._client.lpush(self.list_key, val) 39 | else: 40 | return self._client.rpush(self.list_key, val) 41 | 42 | def pop(self, head=False, blocking=False): 43 | "Remove an value from head or tail of list." 44 | if head and blocking: 45 | return self.deserialize(self._client.blpop(self.list_key)[1]) 46 | elif head: 47 | return self.deserialize(self._client.lpop(self.list_key)) 48 | elif blocking: 49 | return self.deserialize(self._client.brpop(self.list_key)[1]) 50 | else: 51 | return self.deserialize(self._client.rpop(self.list_key)) 52 | 53 | def __unicode__(self): 54 | "Represent entire list." 55 | return u"RedisList(%s)" % (self[0:-1],) 56 | 57 | def __repr__(self): 58 | "Represent entire list." 59 | return self.__unicode__() 60 | 61 | 62 | class PickleRedisList(RedisList, PickleSerializer): 63 | "Serialize Redis List values via Pickle." 64 | pass 65 | 66 | 67 | class JSONRedisList(RedisList, JSONSerializer): 68 | "Serialize Redis List values via JSON." 69 | pass 70 | -------------------------------------------------------------------------------- /redis_ds/redis_set.py: -------------------------------------------------------------------------------- 1 | "A Pythonic interface to a Redis set." 2 | import redis_ds.redis_config as redis_config 3 | from redis_ds.serialization import PassThroughSerializer, PickleSerializer, JSONSerializer 4 | 5 | 6 | class RedisSet(PassThroughSerializer): 7 | "An object which behaves like a Python set, but which is based by Redis." 8 | def __init__(self, set_key, redis_client=redis_config.CLIENT): 9 | "Initialize the set." 10 | self._client = redis_client 11 | self.set_key = set_key 12 | 13 | def __len__(self): 14 | "Number of values in the set." 15 | return self._client.scard(self.set_key) 16 | 17 | def add(self, val): 18 | "Add a value to the set." 19 | val = self.serialize(val) 20 | self._client.sadd(self.set_key, val) 21 | 22 | def update(self, vals): 23 | "Idempotently add multiple values to the set." 24 | vals = [self.serialize(x) for x in vals] 25 | self._client.sadd(self.set_key, *vals) 26 | 27 | def __contains__(self, val): 28 | "Check if a value is a member of a set." 29 | return self._client.sismember(self.set_key, val) 30 | 31 | def pop(self): 32 | "Remove and return a value from the set." 33 | return self.deserialize(self._client.spop(self.set_key)) 34 | 35 | def remove(self, val): 36 | "Remove a specific value from the set." 37 | self._client.srem(self.set_key, self.serialize(val)) 38 | 39 | def __unicode__(self): 40 | "Represent all members in a set." 41 | objs = self._client.smembers(self.set_key) 42 | objs = [self.deserialize(x) for x in objs] 43 | return u"RedisSet(%s)" % (objs,) 44 | 45 | def __repr__(self): 46 | "Represent all members in a set." 47 | return self.__unicode__() 48 | 49 | 50 | class PickleRedisSet(RedisSet, PickleSerializer): 51 | "Pickle values stored in set." 52 | pass 53 | 54 | 55 | class JSONRedisSet(RedisSet, JSONSerializer): 56 | "JSON values stored in set." 57 | pass 58 | 59 | -------------------------------------------------------------------------------- /redis_ds/serialization.py: -------------------------------------------------------------------------------- 1 | "Mixins for serializing objects." 2 | import json 3 | import cPickle as pickle 4 | 5 | 6 | class PassThroughSerializer(object): 7 | "Don't serialize." 8 | def serialize(self, obj): 9 | "Support for serializing objects stored in Redis." 10 | return obj 11 | 12 | def deserialize(self, obj): 13 | "Support for deserializing objects stored in Redis." 14 | return obj 15 | 16 | 17 | class PickleSerializer(PassThroughSerializer): 18 | "Serialize values using pickle." 19 | def serialize(self, obj): 20 | return pickle.dumps(obj) 21 | 22 | def deserialize(self, obj): 23 | "Deserialize values using pickle." 24 | return pickle.loads(obj) 25 | 26 | 27 | class JSONSerializer(PassThroughSerializer): 28 | "Serialize values using JSON." 29 | def serialize(self, obj): 30 | return json.dumps(obj) 31 | 32 | def deserialize(self, obj): 33 | "Deserialize values using JSON." 34 | return json.loads(obj) 35 | 36 | -------------------------------------------------------------------------------- /redis_ds/tests.py: -------------------------------------------------------------------------------- 1 | "Tests for redis datastructures." 2 | import unittest 3 | from redis_ds.redis_dict import RedisDict, PickleRedisDict, JSONRedisDict 4 | from redis_ds.redis_hash_dict import RedisHashDict, PickleRedisHashDict, JSONRedisHashDict 5 | from redis_ds.redis_list import RedisList, PickleRedisList, JSONRedisList 6 | from redis_ds.redis_set import RedisSet, PickleRedisSet, JSONRedisSet 7 | 8 | 9 | class TestRedisDatastructures(unittest.TestCase): 10 | "Test the various data structures." 11 | prefix = "test_rds" 12 | 13 | def test_redis_dict(self): 14 | "Test the redis dict implementation." 15 | key = "%s.dict" % self.prefix 16 | for class_impl in (RedisDict, PickleRedisDict, JSONRedisDict): 17 | rd = class_impl() 18 | del rd[key] 19 | init_size = len(rd) 20 | self.assertFalse(key in rd) 21 | rd[key] = 10 22 | self.assertTrue(key in rd) 23 | self.assertEqual(len(rd), init_size + 1) 24 | 25 | # pass through serialize loses type information, whereas 26 | # the other serializers retain type correctly, hence the 27 | # ambiguity in this test 28 | self.assertTrue(rd[key] in ('10', 10)) 29 | del rd[key] 30 | self.assertFalse(key in rd) 31 | self.assertEqual(len(rd), init_size) 32 | 33 | def test_redis_hash_dict(self): 34 | "Test the redis hash dict implementation." 35 | hash_key = "%s.hash_dict" % self.prefix 36 | key = "hello" 37 | 38 | # ensure dictionary isn't here 39 | rd = RedisDict() 40 | del rd[hash_key] 41 | 42 | for class_impl in (RedisHashDict, PickleRedisHashDict, JSONRedisHashDict): 43 | rhd = class_impl(hash_key) 44 | self.assertEqual(len(rhd), 0) 45 | self.assertFalse(key in rhd) 46 | rhd[key] = 10 47 | self.assertTrue(key in rhd) 48 | self.assertEqual(len(rhd), 1) 49 | 50 | # pass through serialize loses type information, whereas 51 | # the other serializers retain type correctly, hence the 52 | # ambiguity in this test 53 | self.assertTrue(rhd[key] in ('10', 10)) 54 | del rhd[key] 55 | self.assertFalse(key in rhd) 56 | self.assertEqual(len(rhd), 0) 57 | 58 | def test_redis_list(self): 59 | "Test the redis hash dict implementation." 60 | list_key = "%s.list" % self.prefix 61 | 62 | # ensure list isn't here 63 | rd = RedisDict() 64 | del rd[list_key] 65 | 66 | for class_impl in (RedisList, PickleRedisList, JSONRedisList): 67 | rl = class_impl(list_key) 68 | self.assertEqual(len(rl), 0) 69 | rl.append("a") 70 | rl.append("b") 71 | self.assertEqual(len(rl), 2) 72 | self.assertEquals(rl[0], "a") 73 | self.assertEquals(rl[-1], "b") 74 | self.assertEquals(rl[:1], ["a", "b"]) 75 | self.assertEquals(rl[:], ["a", "b"]) 76 | self.assertEquals(rl.pop(), "b") 77 | self.assertEquals(rl.pop(), "a") 78 | 79 | def test_redis_set(self): 80 | "Test redis set." 81 | set_key = "%s.list" % self.prefix 82 | 83 | # ensure set isn't here 84 | rd = RedisDict() 85 | del rd[set_key] 86 | 87 | for class_impl in (RedisSet, PickleRedisSet, JSONRedisSet): 88 | rs = class_impl(set_key) 89 | self.assertEquals(len(rs), 0) 90 | rs.add("a") 91 | rs.add("a") 92 | rs.update(("a", "b")) 93 | self.assertEquals(len(rs), 2) 94 | self.assertTrue(rs.pop() in ("a", "b")) 95 | self.assertEquals(len(rs), 1) 96 | self.assertTrue(rs.pop() in ("a", "b")) 97 | self.assertEquals(len(rs), 0) 98 | 99 | 100 | 101 | 102 | if __name__ == '__main__': 103 | unittest.main() 104 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | redis>=2.8.0 2 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | try: 3 | from setuptools import setup, find_packages 4 | except ImportError: 5 | from ez_setup import use_setuptools 6 | use_setuptools() 7 | from setuptools import setup, find_packages 8 | import redis_ds 9 | 10 | setup(name='redis_ds', 11 | version=redis_ds.__version__, 12 | description='simple python datastructure wrappings for redis', 13 | author='Will Larson', 14 | author_email='lethain@gmail.com', 15 | url='http://github.com/lethain/Redis-Python-Datastructures', 16 | packages=find_packages(), 17 | include_package_data=True, 18 | zip_safe=False, 19 | ) 20 | --------------------------------------------------------------------------------