├── .github └── PULL_REQUEST_TEMPLATE.md ├── .gitignore ├── .pyup.yml ├── .travis.yml ├── CHANGELOG.rst ├── MANIFEST.in ├── README.rst ├── dev-requirements.txt ├── redis_schematics ├── __init__.py ├── exceptions.py ├── patches.py └── utils.py ├── requirements.txt ├── setup.py ├── tests ├── __init__.py └── test_integration.py └── tox.ini /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | ## Type 2 | 3 | - Bug fix (issue #) 4 | - Feature (issue #) 5 | - Docs 6 | - Tests 7 | 8 | ## Status 9 | 10 | - WIP 11 | - Done 12 | 13 | ## Tasks 14 | 15 | - [ ] Unit tests 16 | - [ ] Integration tests 17 | - [ ] Documentation 18 | - [ ] Changelog 19 | 20 | ## Description 21 | 22 | *Insert short description* 23 | 24 | ## Related to 25 | 26 | - Any project/issue 27 | 28 | ## Needs Release 29 | 30 | - Major 31 | - Minor 32 | - Bugfix 33 | - No 34 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | 49 | # Translations 50 | *.mo 51 | *.pot 52 | 53 | # Django stuff: 54 | *.log 55 | local_settings.py 56 | 57 | # Flask stuff: 58 | instance/ 59 | .webassets-cache 60 | 61 | # Scrapy stuff: 62 | .scrapy 63 | 64 | # Sphinx documentation 65 | docs/_build/ 66 | 67 | # PyBuilder 68 | target/ 69 | 70 | # Jupyter Notebook 71 | .ipynb_checkpoints 72 | 73 | # pyenv 74 | .python-version 75 | 76 | # celery beat schedule file 77 | celerybeat-schedule 78 | 79 | # SageMath parsed files 80 | *.sage.py 81 | 82 | # dotenv 83 | .env 84 | 85 | # virtualenv 86 | .venv 87 | venv/ 88 | ENV/ 89 | 90 | # Spyder project settings 91 | .spyderproject 92 | .spyproject 93 | 94 | # Rope project settings 95 | .ropeproject 96 | 97 | # mkdocs documentation 98 | /site 99 | 100 | # mypy 101 | .mypy_cache/ 102 | 103 | # Editor swap files 104 | *.swo 105 | *.swp 106 | 107 | # Jenkins test reports 108 | junit*.xml 109 | -------------------------------------------------------------------------------- /.pyup.yml: -------------------------------------------------------------------------------- 1 | # autogenerated pyup.io config file 2 | # see https://pyup.io/docs/configuration/ for all available options 3 | 4 | schedule: every day 5 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | python: 3.6 3 | services: 4 | - redis-server 5 | before_install: 6 | - python --version 7 | install: 8 | - pip install tox 9 | script: 10 | - tox -e $TOX_ENV 11 | after_success: 12 | - pip install coveralls 13 | - coveralls 14 | matrix: 15 | include: 16 | - python: 3.6 17 | env: TOX_ENV=py36 18 | - env: 19 | - TOX_ENV=flake8 20 | - python: 3.6 21 | env: 22 | - TOX_ENV=black 23 | - python: 3.7 24 | env: TOX_ENV=py37 25 | -------------------------------------------------------------------------------- /CHANGELOG.rst: -------------------------------------------------------------------------------- 1 | 2 | 0.3.2 (unreleased) 3 | ------------------ 4 | 5 | - Nothing changed yet. 6 | 7 | 8 | 0.3.0 (2020-02-14) 9 | ------------------ 10 | 11 | - Deprecate Python 2 and migrate to Python 3. 12 | 13 | 14 | 0.2.2 (2019-03-13) 15 | ------------------ 16 | 17 | **Features** 18 | 19 | - Optional JSON serialization. 20 | 21 | **Bug Fixes** 22 | 23 | - Broken tests with py3. 24 | 25 | 26 | 0.2.1 (2017-09-14) 27 | ------------------ 28 | 29 | **Bug Fixes** 30 | 31 | - Update setup url. 32 | 33 | 34 | 0.2.0 (2017-09-14) 35 | ------------------ 36 | 37 | **Features** 38 | 39 | - HashRedisMixin to store all objects on a single hash. 40 | - ``delete_all()`` and ``delete_filter()`` methods. 41 | 42 | **Bug Fixes** 43 | 44 | - Allow non-string id's to be used as primary keys. 45 | - Crash on ``all()`` with empty lists. 46 | 47 | 48 | 0.1.0 (2017-09-12) 49 | ------------------ 50 | 51 | - Initial release. 52 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include *.rst 2 | include *.txt 3 | include Makefile 4 | include tox.ini 5 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Redis Schematics 2 | ================ 3 | 4 | *Provides Redis persistence to Schematics models with cutomizable abstraction levels.* 5 | 6 | |travis| 7 | 8 | .. |travis| image:: https://travis-ci.org/loggi/redis-schematics.svg?branch=master 9 | :target: https://travis-ci.org/loggi/redis-schematics 10 | 11 | 12 | Installing 13 | ---------- 14 | 15 | Using pip:: 16 | 17 | pip install redis_schematics 18 | 19 | 20 | Understanding Persistence layers 21 | -------------------------------- 22 | 23 | There are several ways to implement complex objects persitence on a key-value-set 24 | database such as redis. The best way to do it depends on your application constraints. 25 | We think that providing a good storage model for your application is to allow you 26 | to choose which abstraction you want to use. Below you can find a comparison on different 27 | provided abstraction layers. 28 | 29 | *Currently we only support a SimpleRedisMixin and SimpleRedisModel, but you can 30 | use BaseRedisMixin to build your own persistance layers.* 31 | 32 | 33 | **SimpleRedisMixin** 34 | 35 | Add Redis persistance to an object using a simple approach. Each object 36 | correnspond to a single key on redis prefixed with the object namespace, 37 | which correnponds to a serialized object. To use this mixin you just need 38 | to declare a primary key as on the example below. 39 | 40 | You may use this Mixin when you have frequent matches on primary key and set 41 | operations, unique expires, hard memory contraints or just wants a 1-1 object-key 42 | approach. You may not use this Mixin if you need performance on ``filter``, ``all`` 43 | and ``get`` on non primary key operations. 44 | 45 | **HashRedisMixin** 46 | 47 | Add Redis persistance to an object using a single hash approach. Each type 48 | correnspond to a single key on redis containing a hash set with every instance 49 | as an entry on the set which contains a serialized object. 50 | 51 | You may use this Mixin when you have frequent matches on primary key, set and 52 | all operations, hard memory contraints or wants a single key approach. 53 | You may not use this Mixin if you need performance on ``filter`` and ``get`` on 54 | non primary key operations. 55 | 56 | 57 | Quickstart 58 | ---------- 59 | 60 | **Creating models with persistence** 61 | 62 | Note: you should include a pk, but don't bother setting it's value manually. 63 | We can infer it from an ``id`` field or by setting a tuple of field names using 64 | ``__unique_together__``. 65 | 66 | 67 | .. code-block:: python 68 | 69 | from datetime import datetime, timedelta 70 | 71 | from redis import StrictRedis 72 | from redis_schematics import SimpleRedisMixin 73 | from schematics import models, types 74 | 75 | 76 | class IceCreamModel(models.Model, SimpleRedisMixin): 77 | pk = types.StringType() # Just include a pk 78 | id = types.StringType() 79 | flavour = types.StringType() 80 | amount_kg = types.IntType() 81 | best_before = types.DateTimeType() 82 | 83 | 84 | **Setting on Redis** 85 | 86 | Saving is simple as ``set()``. 87 | 88 | .. code-block:: python 89 | 90 | vanilla = IceCreamModel(dict( 91 | id='vanilla', 92 | flavour='Sweet Vanilla', 93 | amount_kg=42, 94 | best_before=datetime.now() + timedelta(days=2), 95 | )) 96 | 97 | chocolate = IceCreamModel(dict( 98 | id='chocolate', 99 | flavour='Delicious Chocolate', 100 | amount_kg=12, 101 | best_before=datetime.now() + timedelta(days=3), 102 | )) 103 | 104 | vanilla.set() 105 | chocolate.set() 106 | 107 | **Getting from Redis** 108 | 109 | There are two basic ways to get an element from Redis: by pk or by value. 110 | You can use the classmethods ``match_for_pk(pk)`` or ``match_for_values(**Kwargs)`` 111 | or just simply ``match(**kwargs)`` to let us choose which one. Notice that the 112 | performance from both methods is a lot different, so you may avoid matching 113 | for values on high performance environments. You may also use refresh to reload 114 | an object from storage if it has been modified. 115 | 116 | .. code-block:: python 117 | 118 | IceCreamModel.match_for_pk('vanilla') 119 | IceCreamModel.match_for_values(amount__gte=30) 120 | 121 | IceCreamModel.match(id='vanilla') # match on pk 122 | IceCreamModel.match(best_before__gte=datetime.now()) # match on values 123 | 124 | vanilla.refresh() 125 | 126 | 127 | **Fetching all and filtering** 128 | 129 | You can also use ``all()`` to deserialize all and filters. Notice that 130 | this invlolves deserializing all stored objects. 131 | 132 | .. code-block:: python 133 | 134 | IceCreamModel.all() 135 | IceCreamModel.filter(amount__gte=30) 136 | 137 | 138 | **Deleting and expiring** 139 | 140 | To remove objects, you can set ``__expire__`` or use the ``delete()`` method. 141 | Notice that expires work differently on single key and multiple keys approaches. 142 | 143 | .. code-block:: python 144 | 145 | class MyVolatileModel(models.Model, SimpleRedisMixin): 146 | __expire__ = 3600 # model expire (in seconds) 147 | pk = types.StringType() 148 | 149 | vanilla.delete() 150 | 151 | 152 | JSON 153 | ---- 154 | 155 | If you want json serialization, you have at least two options: 156 | 157 | 1. Patch the default serializer. 158 | 2. Write a custom JSONEncoder. 159 | 160 | We've implemented a handy patch funtion, you need to add this 161 | code to somewhere at the top of everything to automagically add 162 | json serialization capabilities: 163 | 164 | .. code:: python 165 | 166 | from redis_schematics.patches import patch_json 167 | patch_json() 168 | 169 | .. note:: 170 | 171 | Eventually ``__json__`` will be added to the stdlib, see 172 | https://bugs.python.org/issue27362 173 | 174 | Roadmap 175 | ------- 176 | 177 | We are still ``0.x``, but we are very close to a stable API. Check `our roadmap `_ for a glance of what's missing. 178 | -------------------------------------------------------------------------------- /dev-requirements.txt: -------------------------------------------------------------------------------- 1 | flake8==3.7.9 2 | mock==4.0.2 3 | pytest==5.4.1 4 | pytest-cache==1.0 5 | pytest-cover==3.0.0 6 | pytest-sugar==0.9.2 7 | pytest-xdist==1.31.0 8 | zest.releaser==6.20.1 9 | tox==3.14.5 10 | -------------------------------------------------------------------------------- /redis_schematics/__init__.py: -------------------------------------------------------------------------------- 1 | # encoding: utf8 2 | 3 | """ 4 | Redis is and Python are an awesome combination. And they deserve the right 5 | amount of abstraction both can provide. Not more and not less. The objective 6 | here is not to provide an ORM-like interface for redis because redis is not 7 | a relational database. 8 | """ 9 | 10 | from __future__ import absolute_import 11 | 12 | import uuid 13 | import json 14 | 15 | from schematics import models, types 16 | from redis_schematics.exceptions import ( 17 | NotFound, 18 | MultipleFound, 19 | StrictPerformanceException, 20 | ) 21 | from redis_schematics.utils import group_filters_by_suffix, match_filters 22 | 23 | 24 | class BaseRedisMixin(object): 25 | """ 26 | Base mixin to add Redis persistance to schematics models. 27 | """ 28 | 29 | __redis_client__ = None 30 | """Redis client to be used by this mixin.""" 31 | 32 | __expire__ = None 33 | """Object expire time in seconds. Default(None) is no expire.""" 34 | 35 | __unique_args__ = None 36 | __strict_performance__ = False 37 | 38 | @staticmethod 39 | def __serializer__(obj): 40 | """Method used to serialize to string prior to dumping complex objects.""" 41 | return json.dumps(obj) 42 | 43 | @staticmethod 44 | def __deserializer__(dump): 45 | """Method used to deserialize to string prior to loading complex objects.""" 46 | return json.loads(dump) 47 | 48 | @property 49 | def key(self): 50 | """Key for the object on the redis database.""" 51 | return self.__key_pattern__(self.__primary_key__) 52 | 53 | @property 54 | def __prefix__(self): 55 | """Prefix used on for this object keys on the redis database.""" 56 | return self.__class__.__name__ 57 | 58 | @property 59 | def __primary_key__(self): 60 | """Callable that returns an unique identifier for an object on a 61 | given namespace. By default it uses pk, id and __unique_args__.""" 62 | if getattr(self, "pk", None): 63 | return str(self.pk) 64 | 65 | if getattr(self, "id", None): 66 | return str(self.id) 67 | 68 | if self.__unique_args__: 69 | return ".".join([getattr(self, const) for const in self.__unique_args__]) 70 | 71 | def __key_pattern__(self, *args): 72 | """Pattern used to build a key or a query for a given object.""" 73 | namespace = [] 74 | 75 | if self.__prefix__: 76 | namespace.append(self.__prefix__) 77 | 78 | if args: 79 | namespace += list(args) 80 | 81 | return ":".join(namespace) 82 | 83 | def __json__(self): 84 | """ 85 | If you want json serialization, you have at least two options: 86 | 1. Patch the default serializer. 87 | 2. Write a custom JSONEncoder. 88 | redis-schematics comes with a handy patch funtion, you need to add this 89 | code, to somewhere at the top of everything to automagically add 90 | json serialization capabilities: 91 | from redis_schematics.patches import patch_json 92 | patch_json() 93 | Note: Eventually `__json__` will be added to the stdlib, see 94 | https://bugs.python.org/issue27362 95 | """ 96 | return self.to_primitive() 97 | 98 | 99 | class SimpleRedisMixin(BaseRedisMixin): 100 | """ 101 | Add Redis persistance to an object using a simple approach. Each object 102 | correnspond to a single key on redis prefixed with the object namespace, 103 | which correnponds to a serialized object. To use this mixin you just need 104 | to declare a primary key such as: 105 | 106 | ..code::python 107 | pk = types.StringType() 108 | 109 | You may use this Mixin when you have frequent match on primary key and set 110 | operations, unique expires, hard memory contraints or just wants a 1-1 object-key 111 | approach. You may not use this Mixin if you need performance on filter, all 112 | and get on non primary key operations. 113 | """ 114 | 115 | @classmethod 116 | def match(cls, **kwargs): 117 | """ 118 | Gets an element from storage matching any arguments. 119 | Returns an instance if only one object is found. 120 | 121 | Args: 122 | **kwargs: filter attributes matching the object to search. 123 | 124 | Returns: 125 | (SimpleRedisMixin): Schema object instance. 126 | 127 | Raises: 128 | (NotFound): Object matching query does not exist. 129 | (MultipleFound): Multiple objects matching query found. 130 | (StrictPerformanceException): Low performance query not allowed by the 131 | current configuration. 132 | 133 | Perfomance: 134 | On primary key: O(1) 135 | On non-primary fields: O(n*k) where n is the size of the database and 136 | k is the is the number of kwargs. 137 | """ 138 | 139 | schema = cls().import_data(kwargs) 140 | 141 | if schema.__primary_key__: 142 | return cls.match_for_pk(schema.__primary_key__) 143 | 144 | return cls.match_for_values(**kwargs) 145 | 146 | @classmethod 147 | def match_for_pk(cls, pk): 148 | """ 149 | Gets an element from storage using its primary key. Returns an instance if 150 | an object is found. 151 | 152 | Args: 153 | pk(str): object primary key. 154 | 155 | Returns: 156 | (SimpleRedisMixin): Schema object instance. 157 | 158 | Raises: 159 | (NotFound): Object matching query does not exist. 160 | 161 | Perfomance: O(1) 162 | """ 163 | schema = cls({"pk": pk}) 164 | result = cls.__redis_client__.get(schema.key) 165 | 166 | if result is None: 167 | raise NotFound() 168 | 169 | obj = schema.__deserializer__(result) 170 | return schema.import_data(obj) 171 | 172 | @classmethod 173 | def match_for_values(cls, **kwargs): 174 | """ 175 | Gets an element from storage matching value arguments. 176 | Returns an instance if only one object is found. 177 | 178 | Args: 179 | **kwargs: filter attributes matching the object to search. 180 | 181 | Returns: 182 | (SimpleRedisMixin): Schema object instance. 183 | 184 | Raises: 185 | (NotFound): Object matching query does not exist. 186 | (MultipleFound): Multiple objects matching query found. 187 | (StrictPerformanceException): Low performance query not allowed by the 188 | current configuration. 189 | 190 | Perfomance: O(n*k) where n is the size of the database 191 | and k is the is the number of kwargs. 192 | """ 193 | 194 | query = cls.filter(**kwargs) 195 | 196 | if len(query) == 0: 197 | raise NotFound() 198 | 199 | elif len(query) > 1: 200 | raise MultipleFound() 201 | 202 | return query[0] 203 | 204 | @classmethod 205 | def _all_keys(cls): 206 | """ 207 | Gets all the keys from elements on the storage. 208 | 209 | Returns: 210 | List(str): Schema object instances. 211 | 212 | Perfomance: O(n) where n is the size of the database. 213 | """ 214 | pattern = cls().__key_pattern__("*") 215 | return cls.__redis_client__.keys(pattern) 216 | 217 | @classmethod 218 | def all(cls): 219 | """ 220 | Gets all elements from storage. 221 | 222 | Returns: 223 | List(SimpleRedisMixin): Schema object instances. 224 | 225 | Raises: 226 | (StrictPerformanceException): Low performance query not allowed by the 227 | current configuration. 228 | 229 | Perfomance: O(n) where n is the size of the database. 230 | """ 231 | if cls.__strict_performance__: 232 | raise StrictPerformanceException() 233 | 234 | keys = cls._all_keys() 235 | 236 | if not keys: 237 | return [] 238 | 239 | results = cls.__redis_client__.mget(*keys) 240 | return [cls().import_data(cls.__deserializer__(r)) for r in results] 241 | 242 | @classmethod 243 | def filter(cls, **kwargs): 244 | """ 245 | Gets all elements from storage matching value arguments. 246 | Returns an instance if only one object is found. 247 | 248 | Args: 249 | **kwargs: filter attributes matching objects to search. 250 | 251 | Returns: 252 | List(SimpleRedisMixin): Schema object instances. 253 | 254 | Raises: 255 | (StrictPerformanceException): Low performance query not allowed by the 256 | current configuration. 257 | 258 | Perfomance: O(n*k) where n is the size of the database 259 | and k is the is the number of kwargs. 260 | """ 261 | if cls.__strict_performance__: 262 | raise StrictPerformanceException() 263 | 264 | filter_group = group_filters_by_suffix(kwargs) 265 | return [r for r in cls.all() if match_filters(r, filter_group)] 266 | 267 | @classmethod 268 | def delete_all(cls, **kwargs): 269 | """ 270 | Deletes all elements from storage. 271 | 272 | Returns: 273 | int: Number of deleted elements. 274 | 275 | Perfomance: O(n) where n is the size of the database. 276 | """ 277 | keys = cls._all_keys() 278 | 279 | if not keys: 280 | return 0 281 | 282 | return cls.__redis_client__.delete(*keys) 283 | 284 | @classmethod 285 | def delete_filter(cls, **kwargs): 286 | """ 287 | Deletes elements from storage matching filters. 288 | 289 | Returns: 290 | int: Number of deleted elements. 291 | 292 | Perfomance: O(n*k) where n is the size of the database 293 | and k is the is the number of kwargs. 294 | """ 295 | matching = cls.filter(**kwargs) 296 | 297 | keys = [r.key for r in matching] 298 | 299 | if not keys: 300 | return 0 301 | 302 | return cls.__redis_client__.delete(*keys) 303 | 304 | def set(self): 305 | """ 306 | Sets the element on storage. 307 | 308 | Perfomance: O(1) 309 | """ 310 | self.pk = self.__primary_key__ or str(uuid.uuid4) 311 | 312 | self.__redis_client__.set( 313 | self.key, self.__serializer__(self.to_primitive()), self.__expire__ 314 | ) 315 | 316 | def refresh(self): 317 | """ 318 | Updates current object from storage. 319 | 320 | Raises: 321 | (NotFound): Object was deleted meanwhile. 322 | 323 | Perfomance: O(1) 324 | """ 325 | result = self.__redis_client__.get(self.key) 326 | 327 | if result is None: 328 | raise NotFound() 329 | 330 | self.import_data(self.__deserializer__(result)) 331 | 332 | def delete(self): 333 | """ 334 | Deletes the element from storage. 335 | 336 | Perfomance: O(1) 337 | """ 338 | self.__redis_client__.delete(self.key) 339 | 340 | 341 | class HashRedisMixin(BaseRedisMixin): 342 | """ 343 | Add Redis persistance to an object using a single hash approach. Each type 344 | correnspond to a single key on redis containing a hash set with every instance 345 | as an entry on the set which contains a serialized object. 346 | To use this mixin you just need to declare a primary key such as: 347 | 348 | ..code::python 349 | pk = types.StringType() 350 | 351 | You may use this Mixin when you have frequent match on primary key, set and 352 | all operations, hard memory contraints or wants a single key approach. 353 | You may not use this Mixin if you need performance on filter and get on 354 | non primary key operations. 355 | """ 356 | 357 | @property 358 | def __set_key__(self): 359 | return self.__key_pattern__() 360 | 361 | @classmethod 362 | def match(cls, **kwargs): 363 | """ 364 | Gets an element from storage matching any arguments. 365 | Returns an instance if only one object is found. 366 | 367 | Args: 368 | **kwargs: filter attributes matching the object to search. 369 | 370 | Returns: 371 | (SimpleRedisMixin): Schema object instance. 372 | 373 | Raises: 374 | (NotFound): Object matching query does not exist. 375 | (MultipleFound): Multiple objects matching query found. 376 | (StrictPerformanceException): Low performance query not allowed by the 377 | current configuration. 378 | 379 | Perfomance: 380 | On primary key: O(1) 381 | On non-primary fields: O(n*k) where n is the number of elements in the 382 | set and k is the is the number of kwargs. 383 | """ 384 | 385 | schema = cls().import_data(kwargs) 386 | 387 | if schema.__primary_key__: 388 | return cls.match_for_pk(schema.__primary_key__) 389 | 390 | return cls.match_for_values(**kwargs) 391 | 392 | @classmethod 393 | def match_for_pk(cls, pk): 394 | """ 395 | Gets an element from storage using its primary key. Returns an instance if 396 | an object is found. 397 | 398 | Args: 399 | pk(str): object primary key. 400 | 401 | Returns: 402 | (SimpleRedisMixin): Schema object instance. 403 | 404 | Raises: 405 | (NotFound): Object matching query does not exist. 406 | 407 | Perfomance: O(1) 408 | """ 409 | schema = cls({"pk": pk}) 410 | result = cls.__redis_client__.hget(schema.__set_key__, schema.key) 411 | 412 | if result is None: 413 | raise NotFound() 414 | 415 | obj = schema.__deserializer__(result) 416 | return schema.import_data(obj) 417 | 418 | @classmethod 419 | def match_for_values(cls, **kwargs): 420 | """ 421 | Gets an element from storage matching value arguments. 422 | Returns an instance if only one object is found. 423 | 424 | Args: 425 | **kwargs: filter attributes matching the object to search. 426 | 427 | Returns: 428 | (SimpleRedisMixin): Schema object instance. 429 | 430 | Raises: 431 | (NotFound): Object matching query does not exist. 432 | (MultipleFound): Multiple objects matching query found. 433 | (StrictPerformanceException): Low performance query not allowed by the 434 | current configuration. 435 | 436 | Perfomance: O(n*k) where n is the number of elements in the 437 | set and k is the is the number of kwargs. 438 | """ 439 | 440 | query = cls.filter(**kwargs) 441 | 442 | if len(query) == 0: 443 | raise NotFound() 444 | 445 | elif len(query) > 1: 446 | raise MultipleFound() 447 | 448 | return query[0] 449 | 450 | @classmethod 451 | def _all_keys(cls): 452 | """ 453 | Gets all the keys from elements on the storage. 454 | 455 | Returns: 456 | List(str): Schema object instances. 457 | 458 | Perfomance: O(n) where n is the number of elements. 459 | """ 460 | schema = cls() 461 | return cls.__redis_client__.hgetall(schema.__set_key__).keys() 462 | 463 | @classmethod 464 | def all(cls): 465 | """ 466 | Gets all elements from storage. 467 | 468 | Returns: 469 | List(SimpleRedisMixin): Schema object instances. 470 | 471 | Perfomance: O(n) where n is the number of elements. 472 | """ 473 | schema = cls() 474 | results = cls.__redis_client__.hgetall(schema.__set_key__).values() 475 | return [cls().import_data(cls.__deserializer__(r)) for r in results] 476 | 477 | @classmethod 478 | def filter(cls, **kwargs): 479 | """ 480 | Gets all elements from storage matching value arguments. 481 | Returns an instance if only one object is found. 482 | 483 | Args: 484 | **kwargs: filter attributes matching objects to search. 485 | 486 | Returns: 487 | List(SimpleRedisMixin): Schema object instances. 488 | 489 | Raises: 490 | (StrictPerformanceException): Low performance query not allowed by the 491 | current configuration. 492 | 493 | Perfomance: O(n*k) where n is the number of elements in the 494 | set and k is the is the number of kwargs. 495 | """ 496 | if cls.__strict_performance__: 497 | raise StrictPerformanceException() 498 | 499 | filter_group = group_filters_by_suffix(kwargs) 500 | return [r for r in cls.all() if match_filters(r, filter_group)] 501 | 502 | @classmethod 503 | def delete_all(cls, **kwargs): 504 | """ 505 | Deletes all elements from storage. 506 | 507 | Returns: 508 | int: Number of deleted elements. 509 | 510 | Perfomance: O(1). 511 | """ 512 | schema = cls() 513 | return cls.__redis_client__.delete(schema.__set_key__) 514 | 515 | @classmethod 516 | def delete_filter(cls, **kwargs): 517 | """ 518 | Deletes elements from storage matching filters. 519 | 520 | Returns: 521 | int: Number of deleted elements. 522 | 523 | Perfomance: O(n*k) where n is the number of elements in the 524 | set and k is the is the number of kwargs. 525 | """ 526 | matching = cls.filter(**kwargs) 527 | 528 | keys = [r.key for r in matching] 529 | 530 | if not keys: 531 | return 0 532 | 533 | schema = cls() 534 | return cls.__redis_client__.hdel(schema.__set_key__, *keys) 535 | 536 | def set(self): 537 | """ 538 | Sets the element on storage. 539 | 540 | Perfomance: O(1) 541 | """ 542 | self.pk = self.__primary_key__ or str(uuid.uuid4) 543 | 544 | self.__redis_client__.hset( 545 | self.__set_key__, self.key, self.__serializer__(self.to_primitive()) 546 | ) 547 | self.__redis_client__.expire(self.__set_key__, self.__expire__) 548 | 549 | def refresh(self): 550 | """ 551 | Updates current object from storage. 552 | 553 | Raises: 554 | (NotFound): Object was deleted meanwhile. 555 | 556 | Perfomance: O(1) 557 | """ 558 | result = self.__redis_client__.hget(self.__set_key__, self.key) 559 | 560 | if result is None: 561 | raise NotFound() 562 | 563 | self.import_data(self.__deserializer__(result)) 564 | 565 | def delete(self): 566 | """ 567 | Deletes the element from storage. 568 | 569 | Perfomance: O(1) 570 | """ 571 | self.__redis_client__.hdel(self.__set_key__, self.key) 572 | 573 | 574 | class SimpleRedisModel(models.Model, SimpleRedisMixin): 575 | """Shortcut to a Model using SimpleRedisMixin.""" 576 | 577 | pk = types.StringType() 578 | 579 | 580 | class HashRedisModel(models.Model, HashRedisMixin): 581 | """Shortcut to a Model using HashRedisMixin.""" 582 | 583 | pk = types.StringType() 584 | -------------------------------------------------------------------------------- /redis_schematics/exceptions.py: -------------------------------------------------------------------------------- 1 | # encoding: utf8 2 | 3 | from __future__ import absolute_import 4 | 5 | 6 | class RedisMixinException(Exception): 7 | pass 8 | 9 | 10 | class NotFound(KeyError, RedisMixinException): 11 | """Object not found found for a single object query.""" 12 | 13 | pass 14 | 15 | 16 | class MultipleFound(KeyError, RedisMixinException): 17 | """Multiple objects found for a single object query.""" 18 | 19 | pass 20 | 21 | 22 | class StrictPerformanceException(RedisMixinException): 23 | """Avoid performance issues on due to programming errors.""" 24 | 25 | pass 26 | -------------------------------------------------------------------------------- /redis_schematics/patches.py: -------------------------------------------------------------------------------- 1 | def patch_json(): 2 | """ 3 | Patch json default encoder to globally try to find and call a ``__json__`` 4 | method inside classes before raising "TypeError: Object of type 'X' is not 5 | JSON serializable" 6 | """ 7 | from json import JSONEncoder 8 | 9 | def _default(self, obj): 10 | return getattr(obj.__class__, "__json__", _default.default)(obj) 11 | 12 | _default.default = JSONEncoder().default 13 | JSONEncoder.default = _default 14 | -------------------------------------------------------------------------------- /redis_schematics/utils.py: -------------------------------------------------------------------------------- 1 | # encoding: utf8 2 | 3 | from __future__ import absolute_import 4 | 5 | 6 | FILTER_OPS = { 7 | "__gt": lambda s, r: s > r, 8 | "__lt": lambda s, r: s < r, 9 | "__gte": lambda s, r: s >= r, 10 | "__lte": lambda s, r: s <= r, 11 | "__eq": lambda s, r: s == r, 12 | "__not": lambda s, r: s != r, 13 | "__in": lambda s, r: s in r, 14 | "__exclude": lambda s, r: s not in r, 15 | } 16 | 17 | 18 | def group_filters_by_suffix(filters): 19 | 20 | used = {} 21 | 22 | def get_suffix(suffix): 23 | suffixed = {k: v for k, v in filters.items() if k.endswith(suffix)} 24 | cleaned = {k.replace(suffix, ""): v for k, v in suffixed.items()} 25 | used.update(suffixed) 26 | return cleaned 27 | 28 | filter_group = {k: get_suffix(k) for k in FILTER_OPS} 29 | filter_group["__eq"].update(**{k: v for k, v in filters.items() if k not in used}) 30 | return filter_group 31 | 32 | 33 | def match_filters(obj, filter_group): 34 | for suffix, category_filters in filter_group.items(): 35 | operator = FILTER_OPS[suffix] 36 | 37 | for k, v in category_filters.items(): 38 | if not operator(getattr(obj, k, None), v): 39 | return False 40 | 41 | return True 42 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | schematics==2.1.0 2 | redis==3.4.1 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import codecs 2 | import os 3 | from setuptools import setup, find_packages 4 | 5 | 6 | here = os.path.abspath(os.path.dirname(__file__)) 7 | 8 | 9 | REQUIREMENTS = [ 10 | 'schematics', 11 | 'redis', 12 | ] 13 | 14 | TEST_REQUIREMENTS = [ 15 | 'flake8', 16 | 'mock', 17 | 'tox', 18 | 'pytest', 19 | 'pytest-cache', 20 | 'pytest-cover', 21 | 'pytest-sugar', 22 | 'pytest-xdist', 23 | ] 24 | 25 | with codecs.open(os.path.join(here, 'README.rst'), encoding='utf-8') as f: 26 | README = f.read() 27 | 28 | 29 | setup(name='redis_schematics', 30 | version='0.3.2.dev0', 31 | description='Redis storage backend for schematics.', 32 | long_description=README, 33 | license='Apache License (2.0)', 34 | author='Gabriela Surita', 35 | author_email='gsurita@loggi.com', 36 | url='https://github.com/loggi/redis-schematics', 37 | classifiers=[ 38 | 'Intended Audience :: Developers', 39 | 'Operating System :: OS Independent', 40 | 'Programming Language :: Python', 41 | 'Programming Language :: Python :: 3', 42 | 'Programming Language :: Python :: 3.4', 43 | 'Programming Language :: Python :: 3.5', 44 | 'Programming Language :: Python :: 3.6', 45 | 'Programming Language :: Python :: 3.7', 46 | ], 47 | keywords=[ 48 | 'loggi', 49 | 'schematics', 50 | 'redis', 51 | ], 52 | packages=find_packages(), 53 | include_package_data=True, 54 | zip_safe=False, 55 | install_requires=REQUIREMENTS, 56 | test_suite='tests', 57 | tests_require=TEST_REQUIREMENTS+REQUIREMENTS) 58 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/loggi/redis-schematics/ecb1a808baf8296536d433bebe05ab836443b728/tests/__init__.py -------------------------------------------------------------------------------- /tests/test_integration.py: -------------------------------------------------------------------------------- 1 | # encoding: utf8 2 | 3 | from __future__ import absolute_import 4 | 5 | import json 6 | from datetime import datetime 7 | from unittest import TestCase 8 | 9 | from redis import StrictRedis 10 | from schematics import types, models 11 | 12 | from redis_schematics import HashRedisMixin, SimpleRedisMixin 13 | from redis_schematics.exceptions import NotFound 14 | 15 | 16 | client = StrictRedis(host="localhost", port=6379, db=4) 17 | 18 | 19 | class TestModel(models.Model): 20 | __redis_client__ = client 21 | __expire__ = 120 22 | 23 | pk = types.StringType() 24 | id = types.IntType(required=True) 25 | name = types.StringType() 26 | created = types.DateTimeType() 27 | good_number = types.IntType() 28 | 29 | 30 | class BaseModelStorageTest(object): 31 | @property 32 | def raw_value(self): 33 | raise NotImplementedError() 34 | 35 | @property 36 | def stored(self): 37 | raise NotImplementedError() 38 | 39 | def test_set(self): 40 | assert self.stored == self.schema.to_primitive() 41 | 42 | def test_match(self): 43 | result = self.TestModel.match(id=123) 44 | assert self.stored == result.to_primitive() 45 | 46 | result = self.TestModel.match(pk="123") 47 | assert self.stored == result.to_primitive() 48 | 49 | result = self.TestModel.match(good_number=42) 50 | assert self.stored == result.to_primitive() 51 | 52 | result = self.TestModel.match(good_number__lt=43) 53 | assert self.stored == result.to_primitive() 54 | 55 | result = self.TestModel.match(good_number__gt=41) 56 | assert self.stored == result.to_primitive() 57 | 58 | def test_match_on_non_existing(self): 59 | self.assertRaises(NotFound, self.TestModel.match, id=321) 60 | self.assertRaises(NotFound, self.TestModel.match, pk="321") 61 | self.assertRaises(NotFound, self.TestModel.match, good_number__gt=42) 62 | self.assertRaises(NotFound, self.TestModel.match, good_number__lt=42) 63 | 64 | def test_match_for_pk(self): 65 | result = self.TestModel.match_for_pk("123") 66 | assert self.stored == result.to_primitive() 67 | 68 | def test_match_for_values(self): 69 | result = self.TestModel.match_for_values(name="Bar") 70 | assert self.stored == result.to_primitive() 71 | 72 | def test_all(self): 73 | result = self.TestModel.all() 74 | assert self.stored == result[0].to_primitive() 75 | 76 | def test_all_on_non_existing(self): 77 | self.schema.delete() 78 | result = self.TestModel.all() 79 | assert result == [] 80 | 81 | def test_filter(self): 82 | result = self.TestModel.filter(good_number__lt=43) 83 | assert self.stored == result[0].to_primitive() 84 | result = self.TestModel.filter(good_number__lt=42) 85 | assert result == [] 86 | 87 | def test_delete(self): 88 | self.schema.delete() 89 | assert self.raw_value is None 90 | 91 | def test_delete_all(self): 92 | assert self.TestModel.delete_all() == 1 93 | assert self.raw_value is None 94 | 95 | def test_delete_all_on_non_existing(self): 96 | self.schema.delete() 97 | assert self.TestModel.delete_all() == 0 98 | assert self.raw_value is None 99 | 100 | def test_delete_filter(self): 101 | assert self.TestModel.delete_filter(name="Bla") == 0 102 | assert self.raw_value 103 | assert self.TestModel.delete_filter(name="Bar") == 1 104 | assert self.raw_value is None 105 | 106 | def test_delete_filter_on_non_existing(self): 107 | self.schema.delete() 108 | assert self.TestModel.delete_filter() == 0 109 | assert self.raw_value is None 110 | 111 | def test_json_serialization(self): 112 | from redis_schematics.patches import patch_json 113 | 114 | patch_json() 115 | data = self.TestModel.all() 116 | data_json = json.dumps(data) 117 | expected = json.dumps([x.to_primitive() for x in data]) 118 | assert data_json == expected 119 | 120 | def tearDown(self): 121 | client.delete("TestSimpleModel:123") 122 | 123 | 124 | class SimpleModelStorageTest(BaseModelStorageTest, TestCase): 125 | class TestSimpleModel(TestModel, SimpleRedisMixin): 126 | pass 127 | 128 | def setUp(self): 129 | self.TestModel = self.TestSimpleModel 130 | self.schema = self.TestModel( 131 | {"id": 123, "name": "Bar", "created": datetime.now(), "good_number": 42} 132 | ) 133 | self.schema.set() 134 | 135 | def tearDown(self): 136 | client.delete("TestSimpleModel:123") 137 | 138 | @property 139 | def raw_value(self): 140 | return client.get("TestSimpleModel:123") 141 | 142 | @property 143 | def stored(self): 144 | return json.loads(self.raw_value.decode("utf-8")) 145 | 146 | 147 | class HashModelStorageTest(BaseModelStorageTest, TestCase): 148 | class TestHashModel(TestModel, HashRedisMixin): 149 | pass 150 | 151 | def setUp(self): 152 | self.TestModel = self.TestHashModel 153 | self.schema = self.TestModel( 154 | {"id": 123, "name": "Bar", "created": datetime.now(), "good_number": 42} 155 | ) 156 | self.schema.set() 157 | 158 | def tearDown(self): 159 | client.delete("TestHashModel") 160 | 161 | @property 162 | def raw_value(self): 163 | return client.hget("TestHashModel", "TestHashModel:123") 164 | 165 | @property 166 | def stored(self): 167 | return json.loads(self.raw_value.decode("utf-8")) 168 | 169 | 170 | class HashModelDynamicKeyStorageTest(BaseModelStorageTest, TestCase): 171 | class TestHashModel(TestModel, HashRedisMixin): 172 | hash_id = types.StringType(default="SubKey") 173 | 174 | @property 175 | def __set_key__(self): 176 | return self.__key_pattern__(self.hash_id) 177 | 178 | def setUp(self): 179 | self.TestModel = self.TestHashModel 180 | self.schema = self.TestModel( 181 | {"id": 123, "name": "Bar", "created": datetime.now(), "good_number": 42} 182 | ) 183 | self.schema.set() 184 | 185 | def tearDown(self): 186 | client.delete("TestHashModel:SubKey") 187 | 188 | @property 189 | def raw_value(self): 190 | return client.hget("TestHashModel:SubKey", "TestHashModel:123") 191 | 192 | @property 193 | def stored(self): 194 | return json.loads(self.raw_value.decode("utf-8")) 195 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py36,py37,flake8 3 | 4 | [testenv] 5 | commands = 6 | py.test tests --cov-report term-missing --cov redis_schematics --tb=line -v --junitxml=junit-{envname}.xml {posargs} 7 | deps = 8 | -rdev-requirements.txt 9 | -rrequirements.txt 10 | 11 | install_command = pip install -U {packages} {posargs} 12 | 13 | [testenv:flake8] 14 | basepython = python 15 | commands = flake8 redis_schematics/*.py tests 16 | deps = flake8 17 | 18 | [testenv:black] 19 | basepython = python 20 | commands = black --check redis_schematics tests 21 | deps = black 22 | 23 | [flake8] 24 | max-line-length = 99 25 | --------------------------------------------------------------------------------