├── .gitignore ├── LICENSE ├── MANIFEST.in ├── README.rst ├── background_task ├── __init__.py ├── admin.py ├── management │ ├── __init__.py │ └── commands │ │ ├── __init__.py │ │ └── process_tasks.py ├── models.py ├── tasks.py └── tests.py ├── setup.py └── tests ├── run_coverage.sh ├── run_tests.sh └── test_settings.py /.gitignore: -------------------------------------------------------------------------------- 1 | .coverage 2 | *.pyc 3 | *.swp 4 | .DS_Store 5 | /django_background_task.egg-info 6 | /html_coverage 7 | /bin 8 | /lib 9 | /.Python 10 | /src 11 | /include 12 | /dist 13 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2010, John Montgomery. 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without modification, 5 | are permitted provided that the following conditions are met: 6 | 7 | 1. Redistributions of source code must retain the above copyright notice, 8 | this list of conditions and the following disclaimer. 9 | 10 | 2. Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in the 12 | documentation and/or other materials provided with the distribution. 13 | 14 | 3. Neither the name of Django Bakground Task nor the names of its 15 | contributors may be used to endorse or promote products derived from 16 | this software without specific prior written permission. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 19 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 20 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 21 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR 22 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 23 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 24 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 25 | ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 26 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 27 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.rst 2 | include LICENSE 3 | recursive-include tests * -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | ====================== 2 | Django Background Task 3 | ====================== 4 | 5 | Django Background Task is a databased-backed work queue for Django_, loosely based around Ruby's DelayedJob_ library. 6 | 7 | In Django Background Task, all tasks are implemented as functions (or any other callable). 8 | 9 | There are two parts to using background tasks: 10 | 11 | * creating the task functions and registering them with the scheduler 12 | * setup a cron task (or long running process) to execute the tasks 13 | 14 | Creating and registering tasks 15 | ============================== 16 | 17 | To register a task use the background decorator:: 18 | 19 | from background_task import background 20 | from django.contrib.auth.models import User 21 | 22 | @background(schedule=60) 23 | def notify_user(user_id): 24 | # lookup user by id and send them a message 25 | user = User.objects.get(pk=user_id) 26 | user.email_user('Here is a notification', 'You have been notified') 27 | 28 | This will convert the notify_user into a background task function. When you call it from regular code it will actually create a Task object and stores it in the database. The database then contains serialised information about which function actually needs running later on. This does place limits on the parameters that can be passed when calling the function - they must all be serializable as JSON. Hence why in the example above a user_id is passed rather than a User object. 29 | 30 | Calling notify_user as normal will schedule the original function to be run 60 seconds from now:: 31 | 32 | notify_user(user.id) 33 | 34 | This is the default schedule time (as set in the decorator), but it can be overridden:: 35 | 36 | notify_user(user.id, schedule=90) # 90 seconds from now 37 | notify_user(user.id, schedule=timedelta(minutes=20)) # 20 minutes from now 38 | notify_user(user.id, schedule=datetime.now()) # at a specific time 39 | 40 | Running tasks 41 | ============= 42 | 43 | There is a management command to run tasks that have been scheduled:: 44 | 45 | python manage.py process_tasks 46 | 47 | This will simply poll the database queue every few seconds to see if there is a new task to run. 48 | 49 | NB: to aid the management task in finding the registered tasks it is best to put them in a file called 'tasks.py'. You can put them elsewhere, but you have to ensure that they will be imported so the decorator can register them with the scheduler. By putting them in tasks.py they will be auto-discovered and the file automatically imported by the management command. 50 | 51 | The process_tasks management command has the following options: 52 | 53 | * `duration` - Run task for this many seconds (0 or less to run forever) - default is 0 54 | * `sleep` - Sleep for this many seconds before checking for new tasks (if none were found) - default is 5 55 | * `log-file` - Log file destination 56 | * `log-std` - Redirect stdout and stderr to the logging system 57 | * `log-level` - Set logging level (CRITICAL, ERROR, WARNING, INFO, DEBUG) 58 | 59 | You can use the `duration` option for simple process control, by running the management command via a cron job and setting the duration to the time till cron calls the command again. This way if the command fails it will get restarted by the cron job later anyway. It also avoids having to worry about resource/memory leaks too much. The alternative is to use a grown-up program like supervisord_ to handle this for you. 60 | 61 | Settings 62 | ======== 63 | 64 | There are two settings that can be set in your `settings.py` file. 65 | 66 | * `MAX_ATTEMPTS` - controls how many times a task will be attempted (default 25) 67 | * `MAX_RUN_TIME` - maximum possible task run time, after which tasks will be unlocked and tried again (default 3600 seconds) 68 | 69 | Task errors 70 | =========== 71 | 72 | Tasks are retried if they fail and the error recorded in last_error (and logged). A task is retried as it may be a temporary issue, such as a transient network problem. However each time a task is retried it is retried later and later, using an exponential back off, based on the number of attempts:: 73 | 74 | (attempts ** 4) + 5 75 | 76 | This means that initially the task will be tried again a few seconds later. After four attempts the task is tried again 261 seconds later (about four minutes). At twenty five attempts the task will not be tried again for nearly four days! It is not unheard of for a transient error to last a long time and this behavior is intended to stop tasks that are triggering errors constantly (i.e. due to a coding error) form dominating task processing. You should probably monitor the task queue to check for tasks that have errors. After `MAX_ATTEMPTS` the task will be marked as failed and will not be rescheduled again. 77 | 78 | 79 | .. _Django: http://www.djangoproject.com/ 80 | .. _DelayedJob: http://github.com/tobi/delayed_job 81 | .. _supervisord: http://supervisord.org/ 82 | 83 | -------------------------------------------------------------------------------- /background_task/__init__.py: -------------------------------------------------------------------------------- 1 | VERSION = (0, 1, 8) 2 | __version__ = '.'.join(map(str, VERSION)) 3 | 4 | def background(*arg, **kw): 5 | from tasks import tasks 6 | return tasks.background(*arg, **kw) 7 | -------------------------------------------------------------------------------- /background_task/admin.py: -------------------------------------------------------------------------------- 1 | from django.contrib import admin 2 | 3 | from models import Task 4 | 5 | class TaskAdmin(admin.ModelAdmin): 6 | display_filter = ['task_name'] 7 | list_display = ['task_name', 'task_params', 'run_at', 'priority', 'attempts'] 8 | 9 | admin.site.register(Task, TaskAdmin) -------------------------------------------------------------------------------- /background_task/management/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lilspikey/django-background-task/f0678b05bc17dcf09662c4ec7b06c74aca4dae5c/background_task/management/__init__.py -------------------------------------------------------------------------------- /background_task/management/commands/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lilspikey/django-background-task/f0678b05bc17dcf09662c4ec7b06c74aca4dae5c/background_task/management/commands/__init__.py -------------------------------------------------------------------------------- /background_task/management/commands/process_tasks.py: -------------------------------------------------------------------------------- 1 | from django.core.management.base import BaseCommand 2 | import time 3 | from optparse import make_option 4 | import logging 5 | import sys 6 | 7 | from background_task.tasks import tasks, autodiscover 8 | 9 | class Command(BaseCommand): 10 | LOG_LEVELS = ['CRITICAL', 'ERROR', 'WARNING', 'INFO', 'DEBUG'] 11 | 12 | help = 'Run tasks that are scheduled to run on the queue' 13 | option_list = BaseCommand.option_list + ( 14 | make_option('--duration', 15 | action='store', 16 | dest='duration', 17 | type='int', 18 | default=0, 19 | help='Run task for this many seconds (0 or less to run forever) - default is 0'), 20 | make_option('--sleep', 21 | action='store', 22 | dest='sleep', 23 | type='float', 24 | default=5.0, 25 | help='Sleep for this many seconds before checking for new tasks (if none were found) - default is 5'), 26 | make_option('--log-file', 27 | action='store', 28 | dest='log_file', 29 | help='Log file destination'), 30 | make_option('--log-std', 31 | action='store_true', 32 | dest='log_std', 33 | help='Redirect stdout and stderr to the logging system'), 34 | make_option('--log-level', 35 | action='store', 36 | type='choice', 37 | choices=LOG_LEVELS, 38 | dest='log_level', 39 | help='Set logging level (%s)' % ', '.join(LOG_LEVELS)), 40 | ) 41 | 42 | def _configure_logging(self, log_level, log_file, log_std): 43 | 44 | if log_level: 45 | log_level = getattr(logging, log_level) 46 | 47 | config = {} 48 | if log_level: 49 | config['level'] = log_level 50 | if log_file: 51 | config['filename'] = log_file 52 | 53 | if config: 54 | logging.basicConfig(**config) 55 | 56 | if log_std: 57 | class StdOutWrapper(object): 58 | def write(self, s): 59 | logging.info(s) 60 | class StdErrWrapper(object): 61 | def write(self, s): 62 | logging.error(s) 63 | sys.stdout = StdOutWrapper() 64 | sys.stderr = StdErrWrapper() 65 | 66 | 67 | def handle(self, *args, **options): 68 | log_level = options.pop('log_level', None) 69 | log_file = options.pop('log_file', None) 70 | log_std = options.pop('log_std', False) 71 | duration = options.pop('duration', 0) 72 | sleep = options.pop('sleep', 5.0) 73 | 74 | self._configure_logging(log_level, log_file, log_std) 75 | 76 | autodiscover() 77 | 78 | start_time = time.time() 79 | 80 | while (duration <= 0) or (time.time() - start_time) <= duration: 81 | if not tasks.run_next_task(): 82 | logging.debug('waiting for tasks') 83 | time.sleep(sleep) 84 | -------------------------------------------------------------------------------- /background_task/models.py: -------------------------------------------------------------------------------- 1 | from django.db import models 2 | from django.db.models import Q 3 | from django.conf import settings 4 | 5 | from datetime import timedelta 6 | from hashlib import sha1 7 | import traceback 8 | from StringIO import StringIO 9 | import logging 10 | 11 | try: 12 | import json # Django >= 1.6 13 | except ImportError: 14 | from django.utils import simplejson as json # Django <= 1.5 15 | 16 | try: 17 | from django.utils import timezone 18 | datetime_now = timezone.now 19 | except ImportError: 20 | from datetime import datetime 21 | datetime_now = datetime.now 22 | 23 | 24 | # inspired by http://github.com/tobi/delayed_job 25 | 26 | 27 | class TaskManager(models.Manager): 28 | 29 | def find_available(self): 30 | now = datetime_now() 31 | qs = self.unlocked(now) 32 | ready = qs.filter(run_at__lte=now, failed_at=None) 33 | return ready.order_by('-priority', 'run_at') 34 | 35 | def unlocked(self, now): 36 | max_run_time = getattr(settings, 'MAX_RUN_TIME', 3600) 37 | qs = self.get_query_set() 38 | expires_at = now - timedelta(seconds=max_run_time) 39 | unlocked = Q(locked_by=None) | Q(locked_at__lt=expires_at) 40 | return qs.filter(unlocked) 41 | 42 | def new_task(self, task_name, args=None, kwargs=None, 43 | run_at=None, priority=0): 44 | args = args or () 45 | kwargs = kwargs or {} 46 | if run_at is None: 47 | run_at = datetime_now() 48 | 49 | task_params = json.dumps((args, kwargs)) 50 | task_hash = sha1(task_name + task_params).hexdigest() 51 | 52 | return Task(task_name=task_name, 53 | task_params=task_params, 54 | task_hash=task_hash, 55 | priority=priority, 56 | run_at=run_at) 57 | 58 | 59 | class Task(models.Model): 60 | # the "name" of the task/function to be run 61 | task_name = models.CharField(max_length=255, db_index=True) 62 | # the json encoded parameters to pass to the task 63 | task_params = models.TextField() 64 | # a sha1 hash of the name and params, to lookup already scheduled tasks 65 | task_hash = models.CharField(max_length=40, db_index=True) 66 | 67 | # what priority the task has 68 | priority = models.IntegerField(default=0, db_index=True) 69 | # when the task should be run 70 | run_at = models.DateTimeField(db_index=True) 71 | 72 | # how many times the task has been tried 73 | attempts = models.IntegerField(default=0, db_index=True) 74 | # when the task last failed 75 | failed_at = models.DateTimeField(db_index=True, null=True, blank=True) 76 | # details of the error that occurred 77 | last_error = models.TextField(blank=True) 78 | 79 | # details of who's trying to run the task at the moment 80 | locked_by = models.CharField(max_length=64, db_index=True, 81 | null=True, blank=True) 82 | locked_at = models.DateTimeField(db_index=True, null=True, blank=True) 83 | 84 | objects = TaskManager() 85 | 86 | def params(self): 87 | args, kwargs = json.loads(self.task_params) 88 | # need to coerce kwargs keys to str 89 | kwargs = dict((str(k), v) for k, v in kwargs.items()) 90 | return args, kwargs 91 | 92 | def lock(self, locked_by): 93 | now = datetime_now() 94 | unlocked = Task.objects.unlocked(now).filter(pk=self.pk) 95 | updated = unlocked.update(locked_by=locked_by, locked_at=now) 96 | if updated: 97 | return Task.objects.get(pk=self.pk) 98 | return None 99 | 100 | def _extract_error(self, type, err, tb): 101 | file = StringIO() 102 | traceback.print_exception(type, err, tb, None, file) 103 | return file.getvalue() 104 | 105 | def reschedule(self, type, err, traceback): 106 | self.last_error = self._extract_error(type, err, traceback) 107 | max_attempts = getattr(settings, 'MAX_ATTEMPTS', 25) 108 | 109 | if self.attempts >= max_attempts: 110 | self.failed_at = datetime_now() 111 | logging.warn('Marking task %s as failed', self) 112 | else: 113 | self.attempts += 1 114 | backoff = timedelta(seconds=(self.attempts ** 4) + 5) 115 | self.run_at = datetime_now() + backoff 116 | logging.warn('Rescheduling task %s for %s later at %s', self, 117 | backoff, self.run_at) 118 | 119 | # and unlock 120 | self.locked_by = None 121 | self.locked_at = None 122 | 123 | self.save() 124 | 125 | def save(self, *arg, **kw): 126 | # force NULL rather than empty string 127 | self.locked_by = self.locked_by or None 128 | return super(Task, self).save(*arg, **kw) 129 | 130 | def __unicode__(self): 131 | return u'Task(%s)' % self.task_name 132 | 133 | class Meta: 134 | db_table = 'background_task' 135 | -------------------------------------------------------------------------------- /background_task/tasks.py: -------------------------------------------------------------------------------- 1 | from models import Task, datetime_now 2 | 3 | import os 4 | import logging 5 | import sys 6 | from datetime import datetime, timedelta 7 | from django.db import transaction 8 | from django.utils.importlib import import_module 9 | 10 | 11 | class Tasks(object): 12 | def __init__(self): 13 | self._tasks = {} 14 | self._runner = DBTaskRunner() 15 | 16 | def background(self, name=None, schedule=None): 17 | ''' 18 | decorator to turn a regular function into 19 | something that gets run asynchronously in 20 | the background, at a later time 21 | ''' 22 | 23 | # see if used as simple decorator 24 | # where first arg is the function to be decorated 25 | fn = None 26 | if name and callable(name): 27 | fn = name 28 | name = None 29 | 30 | def _decorator(fn): 31 | _name = name 32 | if not _name: 33 | _name = '%s.%s' % (fn.__module__, fn.__name__) 34 | proxy = TaskProxy(_name, fn, schedule, self._runner) 35 | self._tasks[_name] = proxy 36 | return proxy 37 | 38 | if fn: 39 | return _decorator(fn) 40 | 41 | return _decorator 42 | 43 | def run_task(self, task_name, args, kwargs): 44 | task = self._tasks[task_name] 45 | task.task_function(*args, **kwargs) 46 | 47 | def run_next_task(self): 48 | return self._runner.run_next_task(self) 49 | 50 | 51 | class TaskSchedule(object): 52 | SCHEDULE = 0 53 | RESCHEDULE_EXISTING = 1 54 | CHECK_EXISTING = 2 55 | 56 | def __init__(self, run_at=None, priority=None, action=None): 57 | self._run_at = run_at 58 | self._priority = priority 59 | self._action = action 60 | 61 | @classmethod 62 | def create(self, schedule): 63 | if isinstance(schedule, TaskSchedule): 64 | return schedule 65 | priority = None 66 | run_at = None 67 | action = None 68 | if schedule: 69 | if isinstance(schedule, (int, timedelta, datetime)): 70 | run_at = schedule 71 | else: 72 | run_at = schedule.get('run_at', None) 73 | priority = schedule.get('priority', None) 74 | action = schedule.get('action', None) 75 | return TaskSchedule(run_at=run_at, priority=priority, action=action) 76 | 77 | def merge(self, schedule): 78 | params = {} 79 | for name in ['run_at', 'priority', 'action']: 80 | attr_name = '_%s' % name 81 | value = getattr(self, attr_name, None) 82 | if value is None: 83 | params[name] = getattr(schedule, attr_name, None) 84 | else: 85 | params[name] = value 86 | return TaskSchedule(**params) 87 | 88 | @property 89 | def run_at(self): 90 | run_at = self._run_at or datetime_now() 91 | if isinstance(run_at, int): 92 | run_at = datetime_now() + timedelta(seconds=run_at) 93 | if isinstance(run_at, timedelta): 94 | run_at = datetime_now() + run_at 95 | return run_at 96 | 97 | @property 98 | def priority(self): 99 | return self._priority or 0 100 | 101 | @property 102 | def action(self): 103 | return self._action or TaskSchedule.SCHEDULE 104 | 105 | def __repr__(self): 106 | return 'TaskSchedule(run_at=%s, priority=%s)' % (self._run_at, 107 | self._priority) 108 | 109 | def __eq__(self, other): 110 | return self._run_at == other._run_at \ 111 | and self._priority == other._priority \ 112 | and self._action == other._action 113 | 114 | 115 | class DBTaskRunner(object): 116 | ''' 117 | Encapsulate the model related logic in here, in case 118 | we want to support different queues in the future 119 | ''' 120 | 121 | def __init__(self): 122 | self.worker_name = str(os.getpid()) 123 | 124 | def schedule(self, task_name, args, kwargs, run_at=None, 125 | priority=0, action=TaskSchedule.SCHEDULE): 126 | '''Simply create a task object in the database''' 127 | 128 | task = Task.objects.new_task(task_name, args, kwargs, 129 | run_at, priority) 130 | 131 | if action != TaskSchedule.SCHEDULE: 132 | task_hash = task.task_hash 133 | unlocked = Task.objects.unlocked(datetime_now()) 134 | existing = unlocked.filter(task_hash=task_hash) 135 | if action == TaskSchedule.RESCHEDULE_EXISTING: 136 | updated = existing.update(run_at=run_at, priority=priority) 137 | if updated: 138 | return 139 | elif action == TaskSchedule.CHECK_EXISTING: 140 | if existing.count(): 141 | return 142 | 143 | task.save() 144 | 145 | @transaction.autocommit 146 | def get_task_to_run(self): 147 | tasks = Task.objects.find_available()[:5] 148 | for task in tasks: 149 | # try to lock task 150 | locked_task = task.lock(self.worker_name) 151 | if locked_task: 152 | return locked_task 153 | return None 154 | 155 | @transaction.autocommit 156 | def run_task(self, tasks, task): 157 | try: 158 | logging.info('Running %s', task) 159 | args, kwargs = task.params() 160 | tasks.run_task(task.task_name, args, kwargs) 161 | # task done, so can delete it 162 | task.delete() 163 | logging.info('Ran task and deleting %s', task) 164 | except Exception: 165 | t, e, traceback = sys.exc_info() 166 | logging.warn('Rescheduling %s', task, exc_info=(t, e, traceback)) 167 | task.reschedule(t, e, traceback) 168 | del traceback 169 | 170 | def run_next_task(self, tasks): 171 | # we need to commit to make sure 172 | # we can see new tasks as they arrive 173 | task = self.get_task_to_run() 174 | transaction.commit() 175 | if task: 176 | self.run_task(tasks, task) 177 | transaction.commit() 178 | return True 179 | else: 180 | return False 181 | 182 | 183 | class TaskProxy(object): 184 | def __init__(self, name, task_function, schedule, runner): 185 | self.name = name 186 | self.now = self.task_function = task_function 187 | self.runner = runner 188 | self.schedule = TaskSchedule.create(schedule) 189 | 190 | def __call__(self, *args, **kwargs): 191 | schedule = kwargs.pop('schedule', None) 192 | schedule = TaskSchedule.create(schedule).merge(self.schedule) 193 | run_at = schedule.run_at 194 | priority = schedule.priority 195 | action = schedule.action 196 | self.runner.schedule(self.name, args, kwargs, run_at, priority, action) 197 | 198 | def __unicode__(self): 199 | return u'TaskProxy(%s)' % self.name 200 | 201 | tasks = Tasks() 202 | 203 | 204 | def autodiscover(): 205 | '''autodiscover tasks.py files in much the same way as admin app''' 206 | import imp 207 | from django.conf import settings 208 | 209 | for app in settings.INSTALLED_APPS: 210 | try: 211 | app_path = import_module(app).__path__ 212 | except AttributeError: 213 | continue 214 | try: 215 | imp.find_module('tasks', app_path) 216 | except ImportError: 217 | continue 218 | 219 | import_module("%s.tasks" % app) 220 | -------------------------------------------------------------------------------- /background_task/tests.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from django.test import TransactionTestCase 3 | from django.conf import settings 4 | 5 | from datetime import timedelta, datetime 6 | 7 | from background_task.tasks import tasks, TaskSchedule, TaskProxy 8 | from background_task.models import Task, datetime_now 9 | from background_task import background 10 | 11 | _recorded = [] 12 | 13 | 14 | def empty_task(): 15 | pass 16 | 17 | 18 | def record_task(*arg, **kw): 19 | _recorded.append((arg, kw)) 20 | 21 | 22 | class TestBackgroundDecorator(unittest.TestCase): 23 | 24 | def test_get_proxy(self): 25 | proxy = tasks.background()(empty_task) 26 | self.assertNotEqual(proxy, empty_task) 27 | self.assertTrue(isinstance(proxy, TaskProxy)) 28 | 29 | # and alternate form 30 | proxy = tasks.background(empty_task) 31 | self.assertNotEqual(proxy, empty_task) 32 | self.assertTrue(isinstance(proxy, TaskProxy)) 33 | 34 | def test_default_name(self): 35 | proxy = tasks.background()(empty_task) 36 | self.assertEqual(proxy.name, 'background_task.tests.empty_task') 37 | 38 | proxy = tasks.background()(record_task) 39 | self.assertEqual(proxy.name, 'background_task.tests.record_task') 40 | 41 | proxy = tasks.background(empty_task) 42 | #print proxy 43 | self.assertTrue(isinstance(proxy, TaskProxy)) 44 | self.assertEqual(proxy.name, 'background_task.tests.empty_task') 45 | 46 | def test_specified_name(self): 47 | proxy = tasks.background(name='mytask')(empty_task) 48 | self.assertEqual(proxy.name, 'mytask') 49 | 50 | def test_task_function(self): 51 | proxy = tasks.background()(empty_task) 52 | self.assertEqual(proxy.task_function, empty_task) 53 | 54 | proxy = tasks.background()(record_task) 55 | self.assertEqual(proxy.task_function, record_task) 56 | 57 | def test_default_schedule(self): 58 | proxy = tasks.background()(empty_task) 59 | self.assertEqual(TaskSchedule(), proxy.schedule) 60 | 61 | def test_schedule(self): 62 | proxy = tasks.background(schedule=10)(empty_task) 63 | self.assertEqual(TaskSchedule(run_at=10), proxy.schedule) 64 | 65 | def test__unicode__(self): 66 | proxy = tasks.background()(empty_task) 67 | self.assertEqual(u'TaskProxy(background_task.tests.empty_task)', 68 | unicode(proxy)) 69 | 70 | def test_shortcut(self): 71 | '''check shortcut to decorator works''' 72 | proxy = background()(empty_task) 73 | self.failIfEqual(proxy, empty_task) 74 | self.assertEqual(proxy.task_function, empty_task) 75 | 76 | 77 | class TestTaskProxy(unittest.TestCase): 78 | 79 | def setUp(self): 80 | super(TestTaskProxy, self).setUp() 81 | self.proxy = tasks.background()(record_task) 82 | 83 | def test_run_task(self): 84 | tasks.run_task(self.proxy.name, [], {}) 85 | self.assertEqual(((), {}), _recorded.pop()) 86 | 87 | tasks.run_task(self.proxy.name, ['hi'], {}) 88 | self.assertEqual((('hi',), {}), _recorded.pop()) 89 | 90 | tasks.run_task(self.proxy.name, [], {'kw': 1}) 91 | self.assertEqual(((), {'kw': 1}), _recorded.pop()) 92 | 93 | 94 | class TestTaskSchedule(unittest.TestCase): 95 | 96 | def test_priority(self): 97 | self.assertEqual(0, TaskSchedule().priority) 98 | self.assertEqual(0, TaskSchedule(priority=0).priority) 99 | self.assertEqual(1, TaskSchedule(priority=1).priority) 100 | self.assertEqual(2, TaskSchedule(priority=2).priority) 101 | 102 | def _within_one_second(self, d1, d2): 103 | self.failUnless(isinstance(d1, datetime)) 104 | self.failUnless(isinstance(d2, datetime)) 105 | self.failUnless(abs(d1 - d2) <= timedelta(seconds=1)) 106 | 107 | def test_run_at(self): 108 | for schedule in [None, 0, timedelta(seconds=0)]: 109 | now = datetime_now() 110 | run_at = TaskSchedule(run_at=schedule).run_at 111 | self._within_one_second(run_at, now) 112 | 113 | now = datetime_now() 114 | run_at = TaskSchedule(run_at=now).run_at 115 | self._within_one_second(run_at, now) 116 | 117 | fixed_dt = datetime_now() + timedelta(seconds=60) 118 | run_at = TaskSchedule(run_at=fixed_dt).run_at 119 | self._within_one_second(run_at, fixed_dt) 120 | 121 | run_at = TaskSchedule(run_at=90).run_at 122 | self._within_one_second(run_at, datetime_now() + timedelta(seconds=90)) 123 | 124 | run_at = TaskSchedule(run_at=timedelta(seconds=35)).run_at 125 | self._within_one_second(run_at, datetime_now() + timedelta(seconds=35)) 126 | 127 | def test_create(self): 128 | fixed_dt = datetime_now() + timedelta(seconds=10) 129 | schedule = TaskSchedule.create({'run_at': fixed_dt}) 130 | self.assertEqual(schedule.run_at, fixed_dt) 131 | self.assertEqual(0, schedule.priority) 132 | self.assertEqual(TaskSchedule.SCHEDULE, schedule.action) 133 | 134 | schedule = {'run_at': fixed_dt, 'priority': 2, 135 | 'action': TaskSchedule.RESCHEDULE_EXISTING} 136 | schedule = TaskSchedule.create(schedule) 137 | self.assertEqual(schedule.run_at, fixed_dt) 138 | self.assertEqual(2, schedule.priority) 139 | self.assertEqual(TaskSchedule.RESCHEDULE_EXISTING, schedule.action) 140 | 141 | schedule = TaskSchedule.create(0) 142 | self._within_one_second(schedule.run_at, datetime_now()) 143 | 144 | schedule = TaskSchedule.create(10) 145 | self._within_one_second(schedule.run_at, 146 | datetime_now() + timedelta(seconds=10)) 147 | 148 | schedule = TaskSchedule.create(TaskSchedule(run_at=fixed_dt)) 149 | self.assertEqual(schedule.run_at, fixed_dt) 150 | self.assertEqual(0, schedule.priority) 151 | self.assertEqual(TaskSchedule.SCHEDULE, schedule.action) 152 | 153 | def test_merge(self): 154 | default = TaskSchedule(run_at=10, priority=2, 155 | action=TaskSchedule.RESCHEDULE_EXISTING) 156 | schedule = TaskSchedule.create(20).merge(default) 157 | 158 | self._within_one_second(datetime_now() + timedelta(seconds=20), 159 | schedule.run_at) 160 | self.assertEqual(2, schedule.priority) 161 | self.assertEqual(TaskSchedule.RESCHEDULE_EXISTING, schedule.action) 162 | 163 | schedule = TaskSchedule.create({'priority': 0}).merge(default) 164 | self._within_one_second(datetime_now() + timedelta(seconds=10), 165 | schedule.run_at) 166 | self.assertEqual(0, schedule.priority) 167 | self.assertEqual(TaskSchedule.RESCHEDULE_EXISTING, schedule.action) 168 | 169 | action = TaskSchedule.CHECK_EXISTING 170 | schedule = TaskSchedule.create({'action': action}).merge(default) 171 | self._within_one_second(datetime_now() + timedelta(seconds=10), 172 | schedule.run_at) 173 | self.assertEqual(2, schedule.priority) 174 | self.assertEqual(action, schedule.action) 175 | 176 | def test_repr(self): 177 | self.assertEqual('TaskSchedule(run_at=10, priority=0)', 178 | repr(TaskSchedule(run_at=10, priority=0))) 179 | 180 | 181 | class TestSchedulingTasks(TransactionTestCase): 182 | 183 | def test_background_gets_scheduled(self): 184 | self.result = None 185 | 186 | @tasks.background(name='test_background_gets_scheduled') 187 | def set_result(result): 188 | self.result = result 189 | 190 | # calling set_result should now actually create a record in the db 191 | set_result(1) 192 | 193 | all_tasks = Task.objects.all() 194 | self.assertEqual(1, all_tasks.count()) 195 | task = all_tasks[0] 196 | self.assertEqual('test_background_gets_scheduled', task.task_name) 197 | self.assertEqual('[[1], {}]', task.task_params) 198 | 199 | def test_reschedule_existing(self): 200 | 201 | reschedule_existing = TaskSchedule.RESCHEDULE_EXISTING 202 | 203 | @tasks.background(name='test_reschedule_existing', 204 | schedule=TaskSchedule(action=reschedule_existing)) 205 | def reschedule_fn(): 206 | pass 207 | 208 | # this should only end up with one task 209 | # and it should be scheduled for the later time 210 | reschedule_fn() 211 | reschedule_fn(schedule=90) 212 | 213 | all_tasks = Task.objects.all() 214 | self.assertEqual(1, all_tasks.count()) 215 | task = all_tasks[0] 216 | self.assertEqual('test_reschedule_existing', task.task_name) 217 | 218 | # check task is scheduled for later on 219 | now = datetime_now() 220 | self.failUnless(now + timedelta(seconds=89) < task.run_at) 221 | self.failUnless(now + timedelta(seconds=91) > task.run_at) 222 | 223 | def test_check_existing(self): 224 | 225 | check_existing = TaskSchedule.CHECK_EXISTING 226 | 227 | @tasks.background(name='test_check_existing', 228 | schedule=TaskSchedule(action=check_existing)) 229 | def check_fn(): 230 | pass 231 | 232 | # this should only end up with the first call 233 | # scheduled 234 | check_fn() 235 | check_fn(schedule=90) 236 | 237 | all_tasks = Task.objects.all() 238 | self.assertEqual(1, all_tasks.count()) 239 | task = all_tasks[0] 240 | self.assertEqual('test_check_existing', task.task_name) 241 | 242 | # check new task is scheduled for the earlier time 243 | now = datetime_now() 244 | self.failUnless(now - timedelta(seconds=1) < task.run_at) 245 | self.failUnless(now + timedelta(seconds=1) > task.run_at) 246 | 247 | 248 | class TestTaskRunner(TransactionTestCase): 249 | 250 | def setUp(self): 251 | super(TestTaskRunner, self).setUp() 252 | self.runner = tasks._runner 253 | 254 | def test_get_task_to_run_no_tasks(self): 255 | self.failIf(self.runner.get_task_to_run()) 256 | 257 | def test_get_task_to_run(self): 258 | task = Task.objects.new_task('mytask', (1), {}) 259 | task.save() 260 | self.failUnless(task.locked_by is None) 261 | self.failUnless(task.locked_at is None) 262 | 263 | locked_task = self.runner.get_task_to_run() 264 | self.failIf(locked_task is None) 265 | self.failIf(locked_task.locked_by is None) 266 | self.assertEqual(self.runner.worker_name, locked_task.locked_by) 267 | self.failIf(locked_task.locked_at is None) 268 | self.assertEqual('mytask', locked_task.task_name) 269 | 270 | 271 | class TestTaskModel(TransactionTestCase): 272 | 273 | def test_lock_uncontested(self): 274 | task = Task.objects.new_task('mytask') 275 | task.save() 276 | self.failUnless(task.locked_by is None) 277 | self.failUnless(task.locked_at is None) 278 | 279 | locked_task = task.lock('mylock') 280 | self.assertEqual('mylock', locked_task.locked_by) 281 | self.failIf(locked_task.locked_at is None) 282 | self.assertEqual(task.pk, locked_task.pk) 283 | 284 | def test_lock_contested(self): 285 | # locking should actually look at db, not object 286 | # in memory 287 | task = Task.objects.new_task('mytask') 288 | task.save() 289 | self.failIf(task.lock('mylock') is None) 290 | 291 | self.failUnless(task.lock('otherlock') is None) 292 | 293 | def test_lock_expired(self): 294 | settings.MAX_RUN_TIME = 60 295 | task = Task.objects.new_task('mytask') 296 | task.save() 297 | locked_task = task.lock('mylock') 298 | 299 | # force expire the lock 300 | expire_by = timedelta(seconds=(settings.MAX_RUN_TIME + 2)) 301 | locked_task.locked_at = locked_task.locked_at - expire_by 302 | locked_task.save() 303 | 304 | # now try to get the lock again 305 | self.failIf(task.lock('otherlock') is None) 306 | 307 | def test__unicode__(self): 308 | task = Task.objects.new_task('mytask') 309 | self.assertEqual(u'Task(mytask)', unicode(task)) 310 | 311 | 312 | class TestTasks(TransactionTestCase): 313 | 314 | def setUp(self): 315 | super(TestTasks, self).setUp() 316 | 317 | settings.MAX_RUN_TIME = 60 318 | settings.MAX_ATTEMPTS = 25 319 | 320 | @tasks.background(name='set_fields') 321 | def set_fields(**fields): 322 | for key, value in fields.items(): 323 | setattr(self, key, value) 324 | 325 | @tasks.background(name='throws_error') 326 | def throws_error(): 327 | raise RuntimeError("an error") 328 | 329 | self.set_fields = set_fields 330 | self.throws_error = throws_error 331 | 332 | def test_run_next_task_nothing_scheduled(self): 333 | self.failIf(tasks.run_next_task()) 334 | 335 | def test_run_next_task_one_task_scheduled(self): 336 | self.set_fields(worked=True) 337 | self.failIf(hasattr(self, 'worked')) 338 | 339 | self.failUnless(tasks.run_next_task()) 340 | 341 | self.failUnless(hasattr(self, 'worked')) 342 | self.failUnless(self.worked) 343 | 344 | def test_run_next_task_several_tasks_scheduled(self): 345 | self.set_fields(one='1') 346 | self.set_fields(two='2') 347 | self.set_fields(three='3') 348 | 349 | for i in range(3): 350 | self.failUnless(tasks.run_next_task()) 351 | 352 | self.failIf(tasks.run_next_task()) # everything should have been run 353 | 354 | for field, value in [('one', '1'), ('two', '2'), ('three', '3')]: 355 | self.failUnless(hasattr(self, field)) 356 | self.assertEqual(value, getattr(self, field)) 357 | 358 | def test_run_next_task_error_handling(self): 359 | self.throws_error() 360 | 361 | all_tasks = Task.objects.all() 362 | self.assertEqual(1, all_tasks.count()) 363 | original_task = all_tasks[0] 364 | 365 | # should run, but trigger error 366 | self.failUnless(tasks.run_next_task()) 367 | 368 | all_tasks = Task.objects.all() 369 | self.assertEqual(1, all_tasks.count()) 370 | 371 | failed_task = all_tasks[0] 372 | # should have an error recorded 373 | self.failIfEqual('', failed_task.last_error) 374 | self.failUnless(failed_task.failed_at is None) 375 | self.assertEqual(1, failed_task.attempts) 376 | 377 | # should have been rescheduled for the future 378 | # and no longer locked 379 | self.failUnless(failed_task.run_at > original_task.run_at) 380 | self.failUnless(failed_task.locked_by is None) 381 | self.failUnless(failed_task.locked_at is None) 382 | 383 | def test_run_next_task_does_not_run_locked(self): 384 | self.set_fields(locked=True) 385 | self.failIf(hasattr(self, 'locked')) 386 | 387 | all_tasks = Task.objects.all() 388 | self.assertEqual(1, all_tasks.count()) 389 | original_task = all_tasks[0] 390 | original_task.lock('lockname') 391 | 392 | self.failIf(tasks.run_next_task()) 393 | 394 | self.failIf(hasattr(self, 'locked')) 395 | all_tasks = Task.objects.all() 396 | self.assertEqual(1, all_tasks.count()) 397 | 398 | def test_run_next_task_unlocks_after_MAX_RUN_TIME(self): 399 | self.set_fields(lock_overridden=True) 400 | 401 | all_tasks = Task.objects.all() 402 | self.assertEqual(1, all_tasks.count()) 403 | original_task = all_tasks[0] 404 | locked_task = original_task.lock('lockname') 405 | 406 | self.failIf(tasks.run_next_task()) 407 | 408 | self.failIf(hasattr(self, 'lock_overridden')) 409 | 410 | # put lot time into past 411 | expire_by = timedelta(seconds=(settings.MAX_RUN_TIME + 2)) 412 | locked_task.locked_at = locked_task.locked_at - expire_by 413 | locked_task.save() 414 | 415 | # so now we should be able to override the lock 416 | # and run the task 417 | self.failUnless(tasks.run_next_task()) 418 | self.assertEqual(0, Task.objects.count()) 419 | 420 | self.failUnless(hasattr(self, 'lock_overridden')) 421 | self.failUnless(self.lock_overridden) 422 | 423 | def test_default_schedule_used_for_run_at(self): 424 | 425 | @tasks.background(name='default_schedule_used_for_run_at', schedule=60) 426 | def default_schedule_used_for_time(): 427 | pass 428 | 429 | now = datetime_now() 430 | default_schedule_used_for_time() 431 | 432 | all_tasks = Task.objects.all() 433 | self.assertEqual(1, all_tasks.count()) 434 | task = all_tasks[0] 435 | 436 | self.failUnless(now < task.run_at) 437 | self.failUnless((task.run_at - now) <= timedelta(seconds=61)) 438 | self.failUnless((task.run_at - now) >= timedelta(seconds=59)) 439 | 440 | def test_default_schedule_used_for_priority(self): 441 | 442 | @tasks.background(name='default_schedule_used_for_priority', 443 | schedule={'priority': 2}) 444 | def default_schedule_used_for_priority(): 445 | pass 446 | 447 | now = datetime_now() 448 | default_schedule_used_for_priority() 449 | 450 | all_tasks = Task.objects.all() 451 | self.assertEqual(1, all_tasks.count()) 452 | task = all_tasks[0] 453 | self.assertEqual(2, task.priority) 454 | 455 | def test_non_default_schedule_used(self): 456 | default_run_at = datetime_now() + timedelta(seconds=90) 457 | 458 | @tasks.background(name='non_default_schedule_used', 459 | schedule={'run_at': default_run_at, 'priority': 2}) 460 | def default_schedule_used_for_priority(): 461 | pass 462 | 463 | run_at = datetime_now().replace(microsecond=0) + timedelta(seconds=60) 464 | default_schedule_used_for_priority(schedule=run_at) 465 | 466 | all_tasks = Task.objects.all() 467 | self.assertEqual(1, all_tasks.count()) 468 | task = all_tasks[0] 469 | self.assertEqual(run_at, task.run_at) 470 | 471 | def test_failed_at_set_after_MAX_ATTEMPTS(self): 472 | @tasks.background(name='test_failed_at_set_after_MAX_ATTEMPTS') 473 | def failed_at_set_after_MAX_ATTEMPTS(): 474 | raise RuntimeError('failed') 475 | 476 | failed_at_set_after_MAX_ATTEMPTS() 477 | 478 | available = Task.objects.find_available() 479 | self.assertEqual(1, available.count()) 480 | task = available[0] 481 | 482 | self.failUnless(task.failed_at is None) 483 | 484 | task.attempts = settings.MAX_ATTEMPTS 485 | task.save() 486 | 487 | # task should be scheduled to run now 488 | # but will be marked as failed straight away 489 | self.failUnless(tasks.run_next_task()) 490 | 491 | available = Task.objects.find_available() 492 | self.assertEqual(0, available.count()) 493 | 494 | all_tasks = Task.objects.all() 495 | self.assertEqual(1, all_tasks.count()) 496 | task = all_tasks[0] 497 | 498 | self.failIf(task.failed_at is None) 499 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_packages 2 | 3 | version = __import__('background_task').__version__ 4 | 5 | setup( 6 | name='django-background-task', 7 | version=version, 8 | description='Database backed asynchronous task queue', 9 | long_description=open('README.rst').read(), 10 | author='John Montgomery', 11 | author_email='john@littlespikeyland.com', 12 | url='http://github.com/lilspikey/django-background-task', 13 | license='BSD', 14 | packages=find_packages(exclude=['ez_setup']), 15 | include_package_data=True, 16 | zip_safe=True, 17 | classifiers=[ 18 | 'Development Status :: 4 - Beta', 19 | 'Environment :: Web Environment', 20 | 'Framework :: Django', 21 | 'Intended Audience :: Developers', 22 | 'License :: OSI Approved :: BSD License', 23 | 'Operating System :: OS Independent', 24 | 'Programming Language :: Python', 25 | 'Topic :: Software Development :: Libraries :: Python Modules', 26 | ], 27 | ) 28 | -------------------------------------------------------------------------------- /tests/run_coverage.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # run from parent directory (e.g. tests/run_tests.sh) 3 | django-admin.py test_coverage background_task --pythonpath=. --pythonpath=tests --settings=test_settings -------------------------------------------------------------------------------- /tests/run_tests.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # run from parent directory (e.g. tests/run_tests.sh) 3 | django-admin.py test background_task --pythonpath=. --pythonpath=tests --settings=test_settings -------------------------------------------------------------------------------- /tests/test_settings.py: -------------------------------------------------------------------------------- 1 | import sys 2 | 3 | DEBUG = True 4 | TEMPLATE_DEBUG = DEBUG 5 | DATABASE_ENGINE = 'sqlite3' 6 | DATABASE_NAME = ':memory:' 7 | 8 | DATABASES = { 9 | 'default': { 10 | 'ENGINE': 'django.db.backends.%s' % DATABASE_ENGINE, 11 | 'NAME': DATABASE_NAME, 12 | 'USER': '', # Not used with sqlite3. 13 | 'PASSWORD': '', # Not used with sqlite3. 14 | } 15 | } 16 | 17 | 18 | INSTALLED_APPS = [ 'background_task' ] 19 | 20 | 21 | if 'test_coverage' in sys.argv: 22 | # http://pypi.python.org/pypi/django-coverage 23 | INSTALLED_APPS.append('django_coverage') 24 | COVERAGE_REPORT_HTML_OUTPUT_DIR = 'html_coverage' 25 | COVERAGE_MODULE_EXCLUDES = [] 26 | 27 | --------------------------------------------------------------------------------