├── .gitignore ├── LICENSE ├── README.rst └── ormcache ├── __init__.py ├── exceptions.py ├── manager.py ├── models.py ├── query.py └── utils.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | /build 3 | /dist -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2009, David Cramer 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 5 | 6 | * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 7 | * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 8 | 9 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Django ORM Cache 2 | ================ 3 | 4 | This project is an attempt to provide a low-level, somewhat magical, approach to handling row-level object caches. 5 | 6 | The project page, including all descriptions and reference material, is still very much a work in progress. If you are interested in contributing to the projects, in code, in ideas, or fixing our ever so awesome project outline, please get in touch with one of the project admins (see below). 7 | 8 | Summary 9 | ------- 10 | 11 | In brief, it's goal is to store only unique objects in a cache key, instead of storing groups of objects in many different locations. This would make updates to a cache object as simple as updating a single cache instance, and also automatically handle updating any lists that hold that object when it is removed. 12 | 13 | Approach 14 | -------- 15 | 16 | The approach consists of several major pieces of functionality: 17 | 18 | * Unique row-level object cache keys. 19 | * Lists of cache keys (e.g. a QuerySet). 20 | * Model-based dependency via signals. 21 | * Row-based dependency via reverse mapping. 22 | 23 | The caching layer will also provide a few additional items: 24 | 25 | * Handling cache stampedes via pre-expiration timestamps. 26 | * Namespace versioning and expirations. 27 | 28 | Row-level Caches 29 | ---------------- 30 | 31 | The primary goal of the project was originally to handle invalidation of many instances of a single object. Typical caching setups would have you create copies of a single object, and place it in many cache keys. The goal of this project is to replace those many copies with a single unique instance, thus making updates and invalidation a much simpler task. 32 | 33 | CachedModel 34 | ----------- 35 | 36 | The row-level cache is primarily managed by the ``CachedModel`` class. This class override several key methods on the normal Model class: 37 | 38 | * save() -- upon saving it automatically updates the cache instance. 39 | * delete() -- upon delete (optional maybe? useless in memcache) it removes the cache instance. 40 | * objects -- a CacheManager instance. 41 | * nocache -- the default manager instance. 42 | 43 | Usage:: 44 | 45 | from ormcache.models import CachedModel 46 | class Article(CachedModel): 47 | ... 48 | 49 | QuerySet Caches 50 | --------------- 51 | 52 | One key problem with having a unique instance of a cache is managing pointers to that instance, as well as the efficiency of those pointers. This approach will simply store a set of pointers (primary keys) to which the backend would automatically query for and invalidate as needed. 53 | 54 | CacheManager 55 | ------------ 56 | 57 | The QuerySet caching consists of one key component, the CacheManager. It's responsibility is to handle the unique storage and retrieval of QuerySet caches. It is also in charge of informing the parent class (CachedModel) of any invalidation warnings. 58 | 59 | The code itself should work exactly like the default Manager and QuerySet with a few additional methods: 60 | 61 | * clean() -- removes the queryset; executes but does not perform any SQL or cache.get methods). 62 | * reset() -- resets the queryset; executes the sql and updates the cache. 63 | * execute() -- forces execution of the current query; no return value. 64 | * cache(key=, timeout=) -- changes several options of the current cache instance. 65 | 66 | One thing to note is that we may possibly be able to get rid of clean, or merge it with reset. The execute() method exists because reset does not force execution of the queryset (maybe this should be changed?). 67 | 68 | Fetching Sets 69 | ------------- 70 | 71 | The biggest use of the CacheManager will come in the former of sets. A set is simply a list of cache key pointers. For efficieny this will be stored in a custom format for Django models:: 72 | 73 | (ModelClass, (*pks), (*select_related fields), key length) 74 | 75 | ModelClass 76 | ########## 77 | 78 | The first item in our cache key is the ModelClass in which the list is pointing to. We may need a way to handle unions but that's up to further discussion. The ModelClass is needed to reference where the pks go. 79 | 80 | Pointers 81 | ######## 82 | The second item, is our list of pointers, or primary keys in this use-case. This could be anything from a list of your standard id integers, to a group of long strings. 83 | 84 | Relations 85 | ######### 86 | 87 | The third item, our select_related fields. These are needed to ensure that we can follow the depth when querying. 88 | 89 | e.g. 90 | We do MyModel.objects.all().select_related('user'). We then fetch the key which says its these 10 pks, from MyModel. We now need to also know that user was involved so we can fetch that in batch (vs it automatically doing 1 cache call per key). 91 | 92 | So the route would be: 93 | 94 | * Pull MyModel's cache key. 95 | * Batch pull all pks referenced from queryset. 96 | * Group all fields by their ModelClass, and then either: 97 | 1. Do a cachedqueryset call on them (which would be looking up another list of values, possibly reusable, bad idea?) 98 | 2. Pull (as much at a time) the caches in bulk. 99 | 100 | Upon failure of any cache pull it would fall back to the ORM, request all the rows it can (for a single model, in a single query), and then set those in the cache. If any one of those rows was not found in the database, it would throw up an Invalidation Warning, which, by default, would expire that parent object and repoll the database for it's dataset. 101 | 102 | Key Length 103 | ########## 104 | 105 | The key length is something which was brought up recently. Memcached has limits to how much you can store in a single key. We can work around those. The key length would simply tell the CacheManager how many keys are used for that single list. 106 | 107 | Model Dependencies 108 | ------------------ 109 | 110 | Model dependencies would be handling via registration in a way such as how signals are handled. 111 | 112 | Object Dependencies 113 | ------------------- 114 | 115 | Object dependencies will be handled by storing a reverse mapping to keys. These dependencies would simply be attached (in a separate key) to the original object in which they are dependent on. 116 | 117 | References 118 | ---------- 119 | 120 | * [http://www.davidcramer.net/code/61/handling-cache-invalidation.html Handling Cache Invalidation] by David Cramer -------------------------------------------------------------------------------- /ormcache/__init__.py: -------------------------------------------------------------------------------- 1 | from manager import CacheManager 2 | from models import CachedModel 3 | -------------------------------------------------------------------------------- /ormcache/exceptions.py: -------------------------------------------------------------------------------- 1 | class CachedModelException(Exception): pass 2 | 3 | 4 | # Our invalidation classes 5 | class CacheInvalidationWarning(CachedModelException): pass 6 | 7 | class CacheMissingWarning(CacheInvalidationWarning): 8 | """ 9 | CacheMissingWarning is thrown when we're trying to fetch a queryset 10 | and it's missing objects in the database. 11 | """ 12 | pass 13 | 14 | class CacheExpiredWarning(CacheInvalidationWarning): 15 | """ 16 | CacheExpiredWarning is thrown when we're trying to fetch from the cache 17 | but the pre-expiration has been hit. 18 | """ 19 | pass 20 | -------------------------------------------------------------------------------- /ormcache/manager.py: -------------------------------------------------------------------------------- 1 | from django.db.models.manager import Manager 2 | from query import CachedQuerySet 3 | 4 | class CacheManager(Manager): 5 | """ 6 | A manager to store and retrieve cached objects using CACHE_BACKEND 7 | 8 | -- the key prefix for all cached objects on this model. [default: db_table] 9 | -- in seconds, the maximum time before data is invalidated. [default: DEFAULT_CACHE_TIME] 10 | """ 11 | def __init__(self, *args, **kwargs): 12 | self.key_prefix = kwargs.pop('key_prefix', None) 13 | self.timeout = kwargs.pop('timeout', None) 14 | super(CacheManager, self).__init__(*args, **kwargs) 15 | 16 | def get_query_set(self): 17 | return CachedQuerySet(model=self.model, timeout=self.timeout, key_prefix=self.key_prefix) 18 | 19 | def cache(self, *args, **kwargs): 20 | return self.get_query_set().cache(*args, **kwargs) 21 | 22 | def clean(self, *args, **kwargs): 23 | # Use reset instead if you are using memcached, as clean makes no sense (extra bandwidth when 24 | # memcached will automatically clean iself). 25 | return self.get_query_set().clean(*args, **kwargs) 26 | 27 | def reset(self, *args, **kwargs): 28 | return self.get_query_set().reset(*args, **kwargs) 29 | -------------------------------------------------------------------------------- /ormcache/models.py: -------------------------------------------------------------------------------- 1 | from django.db.models.manager import Manager 2 | from django.db.models.base import ModelBase, Model 3 | from django.core.exceptions import ObjectDoesNotExist, MultipleObjectsReturned 4 | from django.db.models.fields import FieldDoesNotExist 5 | from django.db.models.options import Options 6 | from django.db.models import signals 7 | from django.db.models.loading import register_models, get_model 8 | from django.dispatch import dispatcher 9 | from django.utils.functional import curry 10 | from django.conf import settings 11 | 12 | from django.core.cache import cache 13 | 14 | import types 15 | import sys 16 | 17 | from manager import CacheManager 18 | from utils import get_cache_key_for_pk 19 | 20 | DEFAULT_CACHE_TIME = 60*60*60 # the maximum an item should be in the cache 21 | 22 | # Signals rundown: 23 | # .cache(expire_on=['create', 'update', 'delete']) 24 | # use namespaces possible so the cache key becomes key_name:expire_namespace(not always present):hash 25 | 26 | # for example, a call with no expires: 27 | # db_table:hash 28 | 29 | # a call with a delete expires 30 | # db_table:0,0,0:hash 31 | 32 | # the numbers represent our current namespace level for the 3 expiration methods 33 | # in order to do this, we'd have to actually store another cache key per model 34 | # and to support threading, query that cache key everytime we do any cache queryset 35 | # hit 36 | # e.g. cache.get('ns:db_table') = 0,0,0 37 | 38 | # when a new row is created, we'd set that to 1,0,0 39 | # which would invalidate anything that had a create expiration set because the key is 40 | # now invalid, because the namespace changed. 41 | 42 | # if you only had create expirations set 43 | # your namespace would be :0: -- its all about the queryset call, you still have to 44 | # call it the same way throughout your code 45 | 46 | # We can also add a table namespace, which says "delete everything" so our 47 | # cache key now becomes db_table:ns_count:0,0,0:hash 48 | # where the 0,0,0: is optional 49 | 50 | # ns_count would be stored in the same ns:db_table key and starts at 0 51 | # this would most likely only be incremented if you did a push to your site 52 | # and needed to say wipe all articles because the dataset changed. 53 | 54 | class CachedModelBase(ModelBase): 55 | # TODO: find a way to not overwrite __new__ like this 56 | def __new__(cls, name, bases, attrs): 57 | # If this isn't a subclass of CachedModel, don't do anything special. 58 | try: 59 | if not filter(lambda b: issubclass(b, CachedModel), bases): 60 | return super(CachedModelBase, cls).__new__(cls, name, bases, attrs) 61 | except NameError: 62 | # 'CachedModel' isn't defined yet, meaning we're looking at Django's own 63 | # Model class, defined below. 64 | return super(CachedModelBase, cls).__new__(cls, name, bases, attrs) 65 | 66 | # Create the class. 67 | new_class = type.__new__(cls, name, bases, {'__module__': attrs.pop('__module__')}) 68 | new_class.add_to_class('_meta', Options(attrs.pop('Meta', None))) 69 | new_class.add_to_class('DoesNotExist', types.ClassType('DoesNotExist', (ObjectDoesNotExist,), {})) 70 | new_class.add_to_class('MultipleObjectsReturned', 71 | types.ClassType('MultipleObjectsReturned', (MultipleObjectsReturned, ), {})) 72 | 73 | # Build complete list of parents 74 | for base in bases: 75 | # TODO: Checking for the presence of '_meta' is hackish. 76 | if '_meta' in dir(base): 77 | new_class._meta.parents.append(base) 78 | new_class._meta.parents.extend(base._meta.parents) 79 | 80 | 81 | if getattr(new_class._meta, 'app_label', None) is None: 82 | # Figure out the app_label by looking one level up. 83 | # For 'django.contrib.sites.models', this would be 'sites'. 84 | model_module = sys.modules[new_class.__module__] 85 | new_class._meta.app_label = model_module.__name__.split('.')[-2] 86 | 87 | # Bail out early if we have already created this class. 88 | m = get_model(new_class._meta.app_label, name, False) 89 | if m is not None: 90 | return m 91 | 92 | # Add all attributes to the class. 93 | for obj_name, obj in attrs.items(): 94 | new_class.add_to_class(obj_name, obj) 95 | 96 | # Add Fields inherited from parents 97 | for parent in new_class._meta.parents: 98 | for field in parent._meta.fields: 99 | # Only add parent fields if they aren't defined for this class. 100 | try: 101 | new_class._meta.get_field(field.name) 102 | except FieldDoesNotExist: 103 | field.contribute_to_class(new_class, field.name) 104 | 105 | new_class._prepare() 106 | 107 | register_models(new_class._meta.app_label, new_class) 108 | # Because of the way imports happen (recursively), we may or may not be 109 | # the first class for this model to register with the framework. There 110 | # should only be one class for each model, so we must always return the 111 | # registered version. 112 | return get_model(new_class._meta.app_label, name, False) 113 | 114 | class CachedModel(Model): 115 | """ 116 | docstring for CachedModel 117 | """ 118 | __metaclass__ = CachedModelBase 119 | 120 | # objects = CacheManager() 121 | # nocache = Manager() 122 | 123 | # Maybe this would work? 124 | @classmethod 125 | def _prepare(cls): 126 | # TODO: How do we extend the parent classes classmethod properly? 127 | # super(CachedModel, cls)._prepare() errors 128 | opts = cls._meta 129 | opts._prepare(cls) 130 | 131 | if opts.order_with_respect_to: 132 | cls.get_next_in_order = curry(cls._get_next_or_previous_in_order, is_next=True) 133 | cls.get_previous_in_order = curry(cls._get_next_or_previous_in_order, is_next=False) 134 | setattr(opts.order_with_respect_to.rel.to, 'get_%s_order' % cls.__name__.lower(), curry(method_get_order, cls)) 135 | setattr(opts.order_with_respect_to.rel.to, 'set_%s_order' % cls.__name__.lower(), curry(method_set_order, cls)) 136 | 137 | # Give the class a docstring -- its definition. 138 | if cls.__doc__ is None: 139 | cls.__doc__ = "%s(%s)" % (cls.__name__, ", ".join([f.attname for f in opts.fields])) 140 | 141 | if hasattr(cls, 'get_absolute_url'): 142 | cls.get_absolute_url = curry(get_absolute_url, opts, cls.get_absolute_url) 143 | 144 | cls.add_to_class('objects', CacheManager()) 145 | cls.add_to_class('nocache', Manager()) 146 | cls.add_to_class('_default_manager', cls.nocache) 147 | signals.class_prepared.send(sender=cls) 148 | 149 | @staticmethod 150 | def _get_cache_key_for_pk(model, pk): 151 | return get_cache_key_for_pk(model, pk) 152 | 153 | @property 154 | def cache_key(self): 155 | return self._get_cache_key_for_pk(self.__class__, self.pk) 156 | 157 | def save(self, *args, **kwargs): 158 | cache.set(self._get_cache_key_for_pk(self.__class__, self.pk), self) 159 | super(CachedModel, self).save(*args, **kwargs) 160 | 161 | def delete(self, *args, **kwargs): 162 | # TODO: create an option that tells the model whether or not it should 163 | # do a cache.delete when the object is deleted. For memcached we 164 | # wouldn't care about deleting. 165 | cache.delete(self._get_cache_key_for_pk(self.__class__, self.pk)) 166 | super(CachedModel, self).delete(*args, **kwargs) 167 | -------------------------------------------------------------------------------- /ormcache/query.py: -------------------------------------------------------------------------------- 1 | from django.db.models.query import QuerySet, ITER_CHUNK_SIZE 2 | from django.db import backend, connection 3 | from django.core.cache import cache 4 | from django.conf import settings 5 | 6 | from utils import get_cache_key_for_pk 7 | from exceptions import CacheMissingWarning 8 | 9 | # TODO: if the query is passing pks then we need to make it pull the cache key from the model 10 | # and try to fetch that first 11 | # if there are additional filters to apply beyond pks we then filter those after we're already pulling the pks 12 | 13 | # TODO: should we also run these additional filters each time we pull back a ref list to check for validation? 14 | 15 | # TODO: all related field calls need to be removed and replaced with cache key sets of some sorts 16 | # (just remove the join and make it do another qs.filter(pk__in) to pull them, which would do a many cache get callb) 17 | 18 | DEFAULT_CACHE_TIME = 60*60*24 # 24 hours 19 | 20 | class FauxCachedQuerySet(list): 21 | """ 22 | We generate a FauxCachedQuerySet when we are returning a 23 | CachedQuerySet from a CachedModel. 24 | """ 25 | pass 26 | 27 | class CachedQuerySet(QuerySet): 28 | """ 29 | Extends the QuerySet object and caches results via CACHE_BACKEND. 30 | """ 31 | def __init__(self, model=None, key_prefix=None, timeout=None, key_name=None, *args, **kwargs): 32 | self._cache_keys = {} 33 | self._cache_reset = False 34 | self._cache_clean = False 35 | if key_prefix: 36 | self.cache_key_prefix = key_prefix 37 | else: 38 | if model: 39 | self.cache_key_prefix = model._meta.db_table 40 | else: 41 | self.cache_key_prefix = '' 42 | self.cache_key_name = key_name 43 | if timeout: 44 | self.cache_timeout = timeout 45 | else: 46 | self.cache_timeout = getattr(cache, 'default_timeout', getattr(settings, 'DEFAULT_CACHE_TIME', DEFAULT_CACHE_TIME)) 47 | QuerySet.__init__(self, model, *args, **kwargs) 48 | 49 | def _clone(self, klass=None, **kwargs): 50 | c = QuerySet._clone(self, klass, **kwargs) 51 | c._cache_clean = kwargs.pop('_cache_clean', self._cache_clean) 52 | c._cache_reset = kwargs.pop('_cache_reset', self._cache_reset) 53 | c.cache_key_prefix = kwargs.pop('cache_key_prefix', self.cache_key_prefix) 54 | c.cache_timeout = kwargs.pop('cache_timeout', self.cache_timeout) 55 | c._cache_keys = {} 56 | return c 57 | 58 | def _get_sorted_clause_key(self): 59 | return (isinstance(i, basestring) and i.lower().replace('`', '').replace("'", '') or str(tuple(sorted(i))) for i in self._get_sql_clause()) 60 | 61 | def _get_cache_key(self, extra=''): 62 | # TODO: Need to figure out if this is the best use. 63 | # Maybe we should use extra for cache_key_name, extra was planned for use 64 | # in things like .count() as it's a different cache key than the normal queryset, 65 | # but that also doesn't make sense because theoretically count() is already different 66 | # sql so the sorted_sql_clause should have figured that out. 67 | if self.cache_key_name is not None: 68 | return '%s:%s' % (self.cache_key_prefix, self.cache_key_name) 69 | if extra not in self._cache_keys: 70 | self._cache_keys[extra] = '%s:%s:%s' % (self.cache_key_prefix, str(hash(''.join(self._get_sorted_clause_key()))), extra) 71 | return self._cache_keys[extra] 72 | 73 | def _prepare_queryset_for_cache(self, queryset): 74 | """ 75 | This is where the magic happens. We need to first see if our result set 76 | is in the cache. If it isn't, we need to do the query and set the cache 77 | to (ModelClass, (*,), (*,), ). 78 | """ 79 | # TODO: make this split up large sets of data based on an option 80 | # and sets the last param, keys, to how many datasets are stored 81 | # in the cache to regenerate. 82 | keys = tuple(obj.pk for obj in queryset) 83 | if self._select_related: 84 | if not self._max_related_depth: 85 | fields = [f.name for f in opts.fields if f.rel and not f.null] 86 | else: 87 | # TODO: handle depth relate lookups 88 | fields = () 89 | else: 90 | fields = () 91 | 92 | return (queryset[0].__class__, keys, fields, 1) 93 | 94 | def _get_queryset_from_cache(self, cache_object): 95 | """ 96 | We transform the cache storage into an actual QuerySet object 97 | automagickly handling the keys depth and select_related fields (again, 98 | using the recursive methods of CachedQuerySet). 99 | 100 | We effectively would just be doing a cache.multi_get(*pks), grabbing 101 | the pks for each releation, e.g. user, and then doing a 102 | CachedManager.objects.filter() on them. This also then makes that 103 | queryset reusable. So the question is, should that queryset have been 104 | reusable? It could be invalidated by some other code which we aren't 105 | tieing directly into the parent queryset so maybe we can't do the 106 | objects.filter() query here and we have to do it internally. 107 | """ 108 | # TODO: make this work for people who have, and who don't have, instance caching 109 | model, keys, fields, length = cache_object 110 | 111 | results = self._get_objects_for_keys(model, keys) 112 | 113 | if fields: 114 | # TODO: optimize this so it's only one get_many call instead of one per select_related field 115 | # XXX: this probably isn't handling depth beyond 1, didn't test even depth of 1 yet 116 | for f in fields: 117 | field = model._meta.get_field(f) 118 | field_results = dict((r.id, r) for r in self._get_objects_for_keys(f.rel.to, [getattr(r, field.db_column) for r in results])) 119 | for r in results: 120 | setattr(r, f.name, field_results[getattr(r, field.db_column)]) 121 | return results 122 | 123 | def _get_objects_for_keys(self, model, keys): 124 | # First we fetch any keys that we can from the cache 125 | results = cache.get_many([get_cache_key_for_pk(model, k) for k in keys]).values() 126 | 127 | # Now we need to compute which keys weren't present in the cache 128 | missing = [k for k in results.iterkeys() if not results[k]] 129 | 130 | # We no longer need to know what the keys were so turn it into a list 131 | results = list(results) 132 | # Query for any missing objects 133 | # TODO: should this only be doing the cache.set if it's from a CachedModel? 134 | # if not then we need to expire it, hook signals? 135 | objects = list(model._default_manager.filter(pk__in=missing)) 136 | for o in objects: 137 | cache.set(o.cache_key, o) 138 | results.extend(objects) 139 | 140 | # Do a simple len() lookup (maybe we shouldn't rely on it returning the right 141 | # number of objects 142 | cnt = len(missing) - len(objects) 143 | if cnt: 144 | raise CacheMissingWarning("%d objects missing in the database" % (cnt,)) 145 | return results 146 | 147 | def _get_data(self): 148 | ck = self._get_cache_key() 149 | if self._result_cache is None or self._cache_clean or self._cache_reset: 150 | if self._cache_clean: 151 | cache.delete(ck) 152 | return 153 | if self._cache_reset: 154 | result_cache = None 155 | else: 156 | result_cache = cache.get(ck) 157 | if result_cache is None: 158 | # We need to lookup the initial table queryset, without related 159 | # fields selected. We then need to loop through each field which 160 | # should be selected and doing another CachedQuerySet() call for 161 | # each set of data. 162 | 163 | # This will allow it to transparently, and recursively, handle 164 | # all calls to the cache. 165 | 166 | # We will use _prepare_queryset_for_cache to store it in the 167 | # the cache, and _get_queryset_from_cache to pull it. 168 | 169 | # Maybe we should override getstate and setstate instead? 170 | 171 | # We first have to remove select_related values from the QuerySet 172 | # as we don't want to pull these in to the dataset as they may already exist 173 | # in memory. 174 | 175 | # TODO: create a function that works w/ our patch and Django trunk which will 176 | # grab the select_related fields for us given X model and (Y list or N depth). 177 | 178 | # TODO: find a clean way to say "is this only matching pks?" if it is we wont 179 | # need to store a result set in memory but we'll need to apply the filters by hand. 180 | qs = QuerySet._clone(QuerySet(), **self.__dict__) 181 | self._result_cache = qs._get_data() 182 | self._cache_reset = False 183 | cache.set(ck, self._prepare_queryset_for_cache(self._result_cache), self.cache_timeout*60) 184 | else: 185 | try: 186 | self._result_cache = self._get_queryset_from_cache(result_cache) 187 | except CacheMissingWarning: 188 | # When an object is missing we reset the cached list. 189 | # TODO: this should be some kind of option at a global and model level. 190 | return self.reset()._get_data() 191 | return FauxCachedQuerySet(self._result_cache) 192 | 193 | def execute(self): 194 | """ 195 | Forces execution on the queryset 196 | """ 197 | self._get_data() 198 | return self 199 | 200 | def get(self, *args, **kwargs): 201 | """ 202 | Performs the SELECT and returns a single object matching the given 203 | keyword arguments. 204 | """ 205 | if self._cache_clean: 206 | clone = self.filter(*args, **kwargs) 207 | if not clone._order_by: 208 | clone._order_by = () 209 | cache.delete(self._get_cache_key()) 210 | else: 211 | return QuerySet.get(self, *args, **kwargs) 212 | 213 | def clean(self): 214 | """ 215 | Removes queryset from the cache upon execution. 216 | """ 217 | return self._clone(_cache_clean=True) 218 | 219 | def count(self): 220 | return QuerySet.count(self) 221 | count = cache.get(self._get_cache_key('count')) 222 | if count is None: 223 | count = int(QuerySet.count(self)) 224 | cache.set(self._get_cache_key('count'), count, self.cache_timeout) 225 | return count 226 | 227 | def cache(self, *args, **kwargs): 228 | """ 229 | Overrides CacheManager's options for this QuerySet. 230 | 231 | -- the key prefix for all cached objects 232 | on this model. [default: db_table] 233 | -- in seconds, the maximum time before data is 234 | invalidated. 235 | -- the key suffix for this cached queryset 236 | useful if you want to cache the same queryset with two expiration 237 | methods. 238 | """ 239 | return self._clone(cache_key_prefix=kwargs.pop('key_prefix', self.cache_key_prefix), cache_timeout=kwargs.pop('timeout', self.cache_timeout), cache_key_name=kwargs.pop('key_name', self.cache_key_name)) 240 | 241 | def reset(self): 242 | """ 243 | Updates the queryset in the cache upon execution. 244 | """ 245 | return self._clone(_cache_reset=True) 246 | 247 | def values(self, *fields): 248 | return self._clone(klass=CachedValuesQuerySet, _fields=fields) 249 | 250 | # need a better way to do this.. (will mix-ins work?) 251 | class CachedValuesQuerySet(CachedQuerySet): 252 | def __init__(self, *args, **kwargs): 253 | super(CachedQuerySet, self).__init__(*args, **kwargs) 254 | # select_related isn't supported in values(). 255 | self._select_related = False 256 | 257 | def iterator(self): 258 | try: 259 | select, sql, params = self._get_sql_clause() 260 | except EmptyResultSet: 261 | raise StopIteration 262 | 263 | # self._fields is a list of field names to fetch. 264 | if self._fields: 265 | #columns = [self.model._meta.get_field(f, many_to_many=False).column for f in self._fields] 266 | if not self._select: 267 | columns = [self.model._meta.get_field(f, many_to_many=False).column for f in self._fields] 268 | else: 269 | columns = [] 270 | for f in self._fields: 271 | if f in [field.name for field in self.model._meta.fields]: 272 | columns.append( self.model._meta.get_field(f, many_to_many=False).column ) 273 | elif not self._select.has_key( f ): 274 | raise FieldDoesNotExist, '%s has no field named %r' % ( self.model._meta.object_name, f ) 275 | 276 | field_names = self._fields 277 | else: # Default to all fields. 278 | columns = [f.column for f in self.model._meta.fields] 279 | field_names = [f.column for f in self.model._meta.fields] 280 | 281 | select = ['%s.%s' % (backend.quote_name(self.model._meta.db_table), backend.quote_name(c)) for c in columns] 282 | 283 | # Add any additional SELECTs. 284 | if self._select: 285 | select.extend(['(%s) AS %s' % (quote_only_if_word(s[1]), backend.quote_name(s[0])) for s in self._select.items()]) 286 | 287 | if getattr(self, '_db_use_master', False): 288 | cursor = connection.write_cursor() 289 | else: 290 | cursor = connection.read_cursor() 291 | cursor.execute("SELECT " + (self._distinct and "DISTINCT " or "") + ",".join(select) + sql, params) 292 | while 1: 293 | rows = cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE) 294 | if not rows: 295 | raise StopIteration 296 | for row in rows: 297 | yield dict(zip(field_names, row)) 298 | 299 | def _clone(self, klass=None, **kwargs): 300 | c = super(CachedValuesQuerySet, self)._clone(klass, **kwargs) 301 | c._fields = self._fields[:] 302 | return c 303 | -------------------------------------------------------------------------------- /ormcache/utils.py: -------------------------------------------------------------------------------- 1 | def get_cache_key_for_pk(model, pk): 2 | return '%s:%s' % (model._meta.db_table, pk) 3 | --------------------------------------------------------------------------------