├── requirements.txt ├── tests ├── __init__.py ├── run_all.sh ├── test_sqs_backend.py ├── test_utils.py ├── test_redis_backend.py ├── test_custom_backend.py ├── test_locmem_backend.py ├── test_workers.py ├── test_gator.py └── test_tasks.py ├── alligator ├── backends │ ├── __init__.py │ ├── locmem_backend.py │ ├── sqs_backend.py │ ├── redis_backend.py │ └── sqlite_backend.py ├── constants.py ├── __init__.py ├── exceptions.py ├── utils.py ├── workers.py ├── tasks.py └── gator.py ├── setup.cfg ├── docs ├── requirements.txt ├── reference │ ├── gator.rst │ ├── tasks.rst │ ├── utils.rst │ ├── workers.rst │ ├── exceptions.rst │ └── constants.rst ├── releasenotes │ ├── 0.8.0.rst │ ├── 1.0.0-beta-2.rst │ ├── 0.5.0.rst │ ├── release_notes_template.txt │ ├── 0.5.1.rst │ ├── 1.0.0-beta-1.rst │ ├── 0.6.0.rst │ ├── 0.7.0.rst │ ├── 0.10.0.rst │ ├── 1.0.0-alpha-3.rst │ ├── 0.9.0.rst │ ├── 0.7.1.rst │ ├── 1.0.0-alpha-2.rst │ ├── 1.0.0-alpha-1.rst │ └── 1.0.0.rst ├── index.rst ├── security.rst ├── migration-to-1.0.rst ├── installing.rst ├── contributing.rst ├── Makefile ├── bestpractices.rst ├── make.bat ├── conf.py ├── extending.rst └── tutorial.rst ├── requirements-tests.txt ├── AUTHORS ├── MANIFEST.in ├── .gitignore ├── .travis.yml ├── .readthedocs.yml ├── bin └── latergator.py ├── pyproject.toml ├── setup.py ├── LICENSE └── README.rst /requirements.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /alligator/backends/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [wheel] 2 | universal = 1 3 | -------------------------------------------------------------------------------- /docs/requirements.txt: -------------------------------------------------------------------------------- 1 | sphinxcontrib-napoleon 2 | sphinx_rtd_theme 3 | -------------------------------------------------------------------------------- /requirements-tests.txt: -------------------------------------------------------------------------------- 1 | -r requirements.txt 2 | 3 | pytest 4 | pytest-cov 5 | redis 6 | boto3 7 | -------------------------------------------------------------------------------- /AUTHORS: -------------------------------------------------------------------------------- 1 | Primary authors: 2 | 3 | * Daniel Lindsley 4 | 5 | Contributors: 6 | 7 | * Peter Bengtsson (peterbe) 8 | * Philippe Ombredanne (pombredanne) 9 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | recursive-include bin * 2 | recursive-include docs * 3 | include AUTHORS 4 | include LICENSE 5 | include README.rst 6 | include requirements.txt 7 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | *.pyc 3 | __pycache__ 4 | 5 | .vscode 6 | poetry.lock 7 | 8 | *.egg-info 9 | build 10 | dist 11 | docs/_build 12 | env 13 | htmlcov 14 | 15 | .coverage 16 | -------------------------------------------------------------------------------- /docs/reference/gator.rst: -------------------------------------------------------------------------------- 1 | .. ref-gator 2 | 3 | =============== 4 | alligator.gator 5 | =============== 6 | 7 | .. automodule:: alligator.gator 8 | :members: 9 | :undoc-members: 10 | -------------------------------------------------------------------------------- /docs/reference/tasks.rst: -------------------------------------------------------------------------------- 1 | .. ref-tasks 2 | 3 | =============== 4 | alligator.tasks 5 | =============== 6 | 7 | .. automodule:: alligator.tasks 8 | :members: 9 | :undoc-members: 10 | -------------------------------------------------------------------------------- /docs/reference/utils.rst: -------------------------------------------------------------------------------- 1 | .. ref-utils 2 | 3 | =============== 4 | alligator.utils 5 | =============== 6 | 7 | .. automodule:: alligator.utils 8 | :members: 9 | :undoc-members: 10 | -------------------------------------------------------------------------------- /docs/reference/workers.rst: -------------------------------------------------------------------------------- 1 | .. ref-workers 2 | 3 | ================= 4 | alligator.workers 5 | ================= 6 | 7 | .. automodule:: alligator.workers 8 | :members: 9 | :undoc-members: 10 | -------------------------------------------------------------------------------- /alligator/constants.py: -------------------------------------------------------------------------------- 1 | # Statuses a task can be in. 2 | WAITING = 0 3 | SUCCESS = 1 4 | FAILED = 2 5 | DELAYED = 3 6 | RETRYING = 4 7 | CANCELED = 5 8 | 9 | # The default queue name for Alligator. 10 | ALL = "all" 11 | -------------------------------------------------------------------------------- /docs/reference/exceptions.rst: -------------------------------------------------------------------------------- 1 | .. ref-exceptions 2 | 3 | ==================== 4 | alligator.exceptions 5 | ==================== 6 | 7 | .. automodule:: alligator.exceptions 8 | :members: 9 | :undoc-members: 10 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | python: 3 | - "3.6" 4 | - "3.7" 5 | - "3.8" 6 | services: 7 | - redis-server 8 | # command to install dependencies 9 | install: 10 | - pip install -r requirements-tests.txt 11 | # command to run tests 12 | script: 13 | - tests/run_all.sh 14 | -------------------------------------------------------------------------------- /.readthedocs.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | 3 | # sphinx: 4 | # configuration: docs/conf.py 5 | 6 | python: 7 | version: 3.7 8 | install: 9 | - requirements: docs/requirements.txt 10 | - method: pip 11 | path: . 12 | extra_requirements: 13 | - docs 14 | system_packages: true 15 | -------------------------------------------------------------------------------- /docs/releasenotes/0.8.0.rst: -------------------------------------------------------------------------------- 1 | alligator 0.8.0 2 | =============== 3 | 4 | :date: 2015-01-02 5 | 6 | This release adds the abililty to cancel a task before it is processed. 7 | 8 | 9 | Features 10 | -------- 11 | 12 | * Added task cancellation. (SHA:`981503a`, SHA:`c3b26ee` & SHA:`34fe5ab`) 13 | 14 | 15 | Bugfixes 16 | -------- 17 | 18 | * None 19 | -------------------------------------------------------------------------------- /docs/releasenotes/1.0.0-beta-2.rst: -------------------------------------------------------------------------------- 1 | alligator 1.0.0-beta-2 2 | ====================== 3 | 4 | :date: 2020-09-10 5 | 6 | This release incorporates all the changes from the prior 7 | 1.0.0-beta-X releases, 1.0.0-alpha-X releases, as well as bugfixes. 8 | 9 | 10 | Bugfixes 11 | -------- 12 | 13 | * Revised how logging is setup by the `Worker`. (SHA: `d52a877`) 14 | -------------------------------------------------------------------------------- /docs/releasenotes/0.5.0.rst: -------------------------------------------------------------------------------- 1 | alligator 0.5.0 2 | =============== 3 | 4 | :date: 2015-01-01 5 | 6 | The initial public release of Alligator! 7 | 8 | 9 | Features 10 | -------- 11 | 12 | * Working codes. 13 | * Fully passing tests. 14 | * Locmem support. 15 | * Redis support. 16 | * Install, tutorial & API docs. 17 | 18 | 19 | Bugfixes 20 | -------- 21 | 22 | None 23 | -------------------------------------------------------------------------------- /docs/releasenotes/release_notes_template.txt: -------------------------------------------------------------------------------- 1 | alligator 1.0.0-alpha-3 2 | ======================= 3 | 4 | :date: 2020-09-?? 5 | 6 | This release adds support for ___ . 7 | 8 | 9 | Features 10 | -------- 11 | 12 | * ___ . (SHA: ``) 13 | 14 | 15 | Bugfixes 16 | -------- 17 | 18 | * ___ . (SHA: ``) 19 | 20 | 21 | Documentation 22 | ------------- 23 | 24 | * ___ . (SHA: ``) 25 | -------------------------------------------------------------------------------- /docs/releasenotes/0.5.1.rst: -------------------------------------------------------------------------------- 1 | alligator 0.5.1 2 | =============== 3 | 4 | :date: 2015-01-01 5 | 6 | This was a bugfix release that corrected some packaging issues with Alligator 7 | 0.5.0. 8 | 9 | 10 | Features 11 | -------- 12 | 13 | * None 14 | 15 | 16 | Bugfixes 17 | -------- 18 | 19 | * Fixed a packaging error that resulted in none of the backends being included. 20 | (SHA:`0e00b3b`) 21 | -------------------------------------------------------------------------------- /bin/latergator.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import sys 3 | 4 | from alligator import Gator, Worker 5 | 6 | 7 | def main(dsn): 8 | gator = Gator(dsn) 9 | 10 | worker = Worker(gator) 11 | worker.run_forever() 12 | 13 | 14 | if __name__ == "__main__": 15 | if len(sys.argv) < 2: 16 | print("Usage: python latergator.py ") 17 | sys.exit(1) 18 | 19 | dsn = sys.argv[1] 20 | main(dsn) 21 | -------------------------------------------------------------------------------- /docs/releasenotes/1.0.0-beta-1.rst: -------------------------------------------------------------------------------- 1 | alligator 1.0.0-beta-1 2 | ====================== 3 | 4 | :date: 2020-09-07 5 | 6 | This release incorporates all the changes from the 1.0.0-alpha-X releases, as 7 | well as bugfixes. 8 | 9 | 10 | Bugfixes 11 | -------- 12 | 13 | * Bugfixes for failed tasks & consistent pop behavior. (SHA: `9d91309`) 14 | 15 | 16 | Documentation 17 | ------------- 18 | 19 | * Added more to the Migration Guide. (SHA: `6e48da4`) 20 | -------------------------------------------------------------------------------- /docs/reference/constants.rst: -------------------------------------------------------------------------------- 1 | .. ref-constants 2 | 3 | =================== 4 | alligator.constants 5 | =================== 6 | 7 | A set of constants included with ``alligator``. 8 | 9 | 10 | Task Constants 11 | ============== 12 | 13 | **WAITING** = ``0`` 14 | 15 | **SUCCESS** = ``1`` 16 | 17 | **FAILED** = ``2`` 18 | 19 | **DELAYED** = ``3`` 20 | 21 | **RETRYING** = ``4`` 22 | 23 | 24 | Queue Constants 25 | =============== 26 | 27 | **ALL** = ``all`` 28 | -------------------------------------------------------------------------------- /docs/releasenotes/0.6.0.rst: -------------------------------------------------------------------------------- 1 | alligator 0.6.0 2 | =============== 3 | 4 | :date: 2015-01-01 5 | 6 | This was a bugfix release that added Travis CI support & corrected Python 3 7 | errors in the Redis backend. 8 | 9 | 10 | Features 11 | -------- 12 | 13 | * Added Travis CI integration for testing. (SHA:`a570391` & SHA:`cef31d3`) 14 | 15 | 16 | Bugfixes 17 | -------- 18 | 19 | * Fixed the Python 3 support in the Redis client. (SHA:`d255427` & SHA:`58cdf3c`) 20 | -------------------------------------------------------------------------------- /docs/releasenotes/0.7.0.rst: -------------------------------------------------------------------------------- 1 | alligator 0.7.0 2 | =============== 3 | 4 | :date: 2015-01-01 5 | 6 | This release added support for the `Beanstalk`_ queue, bringing the supported 7 | backends to three! 8 | 9 | .. _`Beanstalk`: http://kr.github.io/beanstalkd/ 10 | 11 | 12 | Features 13 | -------- 14 | 15 | * Added Beanstalk support. (SHA:`a7edf0c`) 16 | * Added docs for the Beanstalk support. (SHA:`5c49994`) 17 | 18 | 19 | Bugfixes 20 | -------- 21 | 22 | * None 23 | -------------------------------------------------------------------------------- /docs/releasenotes/0.10.0.rst: -------------------------------------------------------------------------------- 1 | alligator 0.10.0 2 | ================ 3 | 4 | :date: 2015-03-31 5 | 6 | This release adds graceful handling of the SIGINT signal, allowing the 7 | in-progress task to complete before exiting. It also fixes a couple bugs 8 | regarding the test suite. 9 | 10 | 11 | Features 12 | -------- 13 | 14 | * Graceful handling of SIGINT, allowing an in-progress task to finish. 15 | (SHA: `3a05c08`) 16 | 17 | 18 | Bugfixes 19 | -------- 20 | 21 | * Moved an import to allow tests to pass without boto present. (SHA: `6f92b5b`) 22 | * Fixed a broken versioning test. (SHA: `1c0ce02`) 23 | -------------------------------------------------------------------------------- /docs/releasenotes/1.0.0-alpha-3.rst: -------------------------------------------------------------------------------- 1 | alligator 1.0.0-alpha-3 2 | ======================= 3 | 4 | :date: 2020-09-06 5 | 6 | This release adds support for a SQLite backend, as well as bugfixes for the 7 | delayed/scheduled tasks feature. 8 | 9 | 10 | Features 11 | -------- 12 | 13 | * Added official SQLite support. (SHA: `7ff4236`) 14 | 15 | 16 | Bugfixes 17 | -------- 18 | 19 | * Fixed several bugs around delayed/scheduled tasks. (SHA: `2e51867`) 20 | * Fixed the SQS backend to delete messages & trimming delays to an int. 21 | (SHA: `cd6c4e2`) 22 | 23 | 24 | Documentation 25 | ------------- 26 | 27 | * Fixed the Redis & SQS backend docstrings. (SHA: `4fe15e7`) 28 | -------------------------------------------------------------------------------- /alligator/__init__.py: -------------------------------------------------------------------------------- 1 | from .constants import WAITING, SUCCESS, FAILED, ALL 2 | from .gator import Gator 3 | from .tasks import Task 4 | from .workers import Worker 5 | 6 | __author__ = "Daniel Lindsley" 7 | __version__ = (1, 0, 0) 8 | __license__ = "BSD" 9 | 10 | 11 | def version(): 12 | """ 13 | Returns a human-readable version string. 14 | 15 | For official releases, it will follow a semver style (e.g. ``1.2.7``). 16 | For dev versions, it will have the semver style first, followed by 17 | hyphenated qualifiers (e.g. ``1.2.7-dev``). 18 | 19 | Returns a string. 20 | """ 21 | short = ".".join([str(bit) for bit in __version__[:3]]) 22 | return "-".join([short] + [str(bit) for bit in __version__[3:]]) 23 | -------------------------------------------------------------------------------- /tests/run_all.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | export ALLIGATOR_SLOW=true 3 | 4 | echo 'Locmem & SQLite Tests' 5 | export ALLIGATOR_CONN='locmem://' 6 | pytest -s -v --cov=alligator --cov-report=html tests 7 | echo 8 | echo 9 | 10 | echo 'Redis Tests' 11 | export ALLIGATOR_CONN='redis://localhost:6379/9' 12 | pytest -s -v tests/test_redis_backend.py 13 | echo 14 | echo 15 | 16 | if [[ ! -z "${ALLIGATOR_TESTS_INCLUDE_SQS}" ]]; then 17 | echo 'SQS Tests' 18 | echo 'Tests will take ~60 seconds before running, due to PurgeQueue operation restrictions...' 19 | echo 'Please be patient.' 20 | export ALLIGATOR_CONN='sqs://us-west-2/' 21 | pytest -s -v tests/test_sqs_backend.py 22 | echo 23 | echo 24 | else 25 | echo 'Skipping SQS...' 26 | fi 27 | -------------------------------------------------------------------------------- /alligator/exceptions.py: -------------------------------------------------------------------------------- 1 | class AlligatorException(Exception): 2 | """ 3 | A base exception for all Alligator errors. 4 | """ 5 | 6 | pass 7 | 8 | 9 | class TaskFailed(AlligatorException): 10 | """ 11 | Raised when a task fails. 12 | """ 13 | 14 | pass 15 | 16 | 17 | class UnknownModuleError(AlligatorException): 18 | """ 19 | Thrown when trying to import an unknown module for a task. 20 | """ 21 | 22 | pass 23 | 24 | 25 | class UnknownCallableError(AlligatorException): 26 | """ 27 | Thrown when trying to import an unknown attribute from a module for a task. 28 | """ 29 | 30 | pass 31 | 32 | 33 | class MultipleDelayError(AlligatorException): 34 | """ 35 | Thrown when more than one delay option is provided. 36 | """ 37 | 38 | pass 39 | -------------------------------------------------------------------------------- /docs/releasenotes/0.9.0.rst: -------------------------------------------------------------------------------- 1 | alligator 0.9.0 2 | =============== 3 | 4 | :date: 2015-01-11 5 | 6 | This release adds support for Amazon SQS as a queue backend. It also fixes a bug 7 | regarding the usage of ``latergator.py`` & corrects some of the documentation. 8 | 9 | Thanks to the following contributors: 10 | 11 | * peterbe 12 | * pombredanne 13 | 14 | 15 | Features 16 | -------- 17 | 18 | * Added the SQS backend. (SHA: `1eedb4b` & SHA: `3e413d7`) 19 | 20 | 21 | Bugfixes 22 | -------- 23 | 24 | * Corrected a copy-paste foul regarding the ``locmem`` backend in the tutorial. 25 | (SHA: `ec6c222`) 26 | * Added the correct hashbang to the ``latergator.py`` script. (SHA: `9fb99ea`) 27 | * Fixed a typo in the README. (SHA: `4b59a92`) 28 | * Added docs on setting a prefix for queue names in testing. (SHA: `a07958c`) 29 | -------------------------------------------------------------------------------- /docs/releasenotes/0.7.1.rst: -------------------------------------------------------------------------------- 1 | alligator 0.7.1 2 | =============== 3 | 4 | :date: 2015-01-02 5 | 6 | This release adds lots of documentation & release notes, as well as bugfixes to 7 | make ``Workers`` less unnecessarily busy & fixes argument handling in 8 | ``latergator.py``. 9 | 10 | 11 | Features 12 | -------- 13 | 14 | * Tutorial improvements. (SHA:`981503a`, SHA:`c3b26ee` & SHA:`34fe5ab`) 15 | * Added the Best Practices docs. (SHA:`0d9ea2e`) 16 | * Filled in the Extending docs. (SHA:`b0cf130`, SHA:`1d472d5` & SHA:`8eb158c`) 17 | 18 | 19 | Bugfixes 20 | -------- 21 | 22 | * Eventually fixed Travis CI support. (SHA:`f264b59`, SHA:`a8833d8`, 23 | SHA:`3f74b25`, SHA:`3968cef` & SHA:`f6358e4`) 24 | * Fixed how unnecessarily busy ``Worker.run_forever()`` was. (SHA:`7b242fc`) 25 | * Fixed the argument handling in ``latergator.py``. (SHA:`18c5e0a`) 26 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [tool.poetry] 2 | name = "alligator" 3 | version = "1.0.0" 4 | description = "Simple offline task queues." 5 | authors = ["Daniel Lindsley "] 6 | license = "BSD-3-Clause" 7 | readme = "README.rst" 8 | repository = "https://github.com/toastdriven/alligator" 9 | documentation = "https://alligator.readthedocs.io/en/latest/" 10 | keywords = [ 11 | "queues", 12 | "tasks", 13 | "offline", 14 | ] 15 | 16 | [tool.poetry.dependencies] 17 | python = "^3.6" 18 | 19 | [tool.poetry.dev-dependencies] 20 | pytest = "^6.0.1" 21 | pytest-cov = "^2.10.1" 22 | redis = "^3.5.3" 23 | boto3 = "^1.14.55" 24 | sphinx = "^3.2.1" 25 | sphinxcontrib-napoleon = "^0.7" 26 | 27 | [tool.poetry.scripts] 28 | # latergator = 'bin/latergator.py' 29 | 30 | [build-system] 31 | requires = ["poetry>=0.12"] 32 | build-backend = "poetry.masonry.api" 33 | -------------------------------------------------------------------------------- /docs/releasenotes/1.0.0-alpha-2.rst: -------------------------------------------------------------------------------- 1 | alligator 1.0.0-alpha-2 2 | ======================= 3 | 4 | :date: 2020-09-05 5 | 6 | This release adds support for delayed/scheduled tasks. 7 | 8 | 9 | Features 10 | -------- 11 | 12 | * Added delayed/scheduled task support. (SHA: `da33d1a`, `a9f2d58`, & 13 | `9266d20`) 14 | * Added ``poetry`` support. (SHA: `2643b40`) 15 | 16 | 17 | Bugfixes 18 | -------- 19 | 20 | * Fixed the executable permission on ``latergator``. (SHA: `2d176a4`) 21 | * Formatted all the code with `black`. (SHA: `3cff2cc`) 22 | 23 | 24 | Documentation 25 | ------------- 26 | 27 | * Added 1.0.0-alpha-1 release notes. (SHA: `e83c391`) 28 | * Added the start of the "Migration to 1.0" guide. (SHA: `cbab984`) 29 | * Switched the docstring style. (SHA: `e5a6a40`) 30 | * Added delayed/scheduled tasks documentation. (SHA: `87c0165`) 31 | * Various documentation fixes. (SHA: `2459956`) 32 | -------------------------------------------------------------------------------- /docs/releasenotes/1.0.0-alpha-1.rst: -------------------------------------------------------------------------------- 1 | alligator 1.0.0-alpha-1 2 | ======================= 3 | 4 | :date: 2020-09-04 5 | 6 | This release modernizes the codebase, and drops Python 2 as well as 7 | Beanstalk/``beanstalkc`` support. 8 | 9 | 10 | Changes 11 | ------- 12 | 13 | * Removed Beanstalk support. (SHA: `368db30`) 14 | 15 | `Note:` This is due to the lack of Python 3 support in the ``beanstalkc`` 16 | library. If this changes in the future, we'll re-evaluate adding it back 17 | in. 18 | 19 | * Updated the SQS backend to `boto3`. (SHA: `ee80d69` & `c6fd671`) 20 | 21 | 22 | Bugfixes 23 | -------- 24 | 25 | * Changed ``Task.async`` to ``Task.is_async``, due to ``async`` being a 26 | reserved word in modern Python 3. (SHA: `bf98511`) 27 | * Updated all dependencies to more current versions. (SHA: `1d368e3`) 28 | * Fixed the Redis backend's tests. (SHA: `e793aa9`) 29 | * Fixed the worker process to correctly exit during testing. (SHA: `9b7b486`) 30 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | ========= 2 | Alligator 3 | ========= 4 | 5 | Simple offline task queues. For Python. 6 | 7 | `"See you later, alligator."` 8 | 9 | 10 | Guide 11 | ===== 12 | 13 | .. toctree:: 14 | :maxdepth: 2 15 | 16 | installing 17 | tutorial 18 | bestpractices 19 | extending 20 | 21 | migration-to-1.0 22 | 23 | contributing 24 | security 25 | 26 | 27 | API Reference 28 | ============= 29 | 30 | .. toctree:: 31 | :glob: 32 | 33 | reference/* 34 | 35 | 36 | Release Notes 37 | ============= 38 | 39 | .. toctree:: 40 | :maxdepth: 1 41 | 42 | releasenotes/1.0.0 43 | releasenotes/1.0.0-beta-2 44 | releasenotes/1.0.0-beta-1 45 | releasenotes/1.0.0-alpha-3 46 | releasenotes/1.0.0-alpha-2 47 | releasenotes/1.0.0-alpha-1 48 | releasenotes/0.10.0 49 | releasenotes/0.9.0 50 | releasenotes/0.8.0 51 | releasenotes/0.7.1 52 | releasenotes/0.7.0 53 | releasenotes/0.6.0 54 | releasenotes/0.5.1 55 | releasenotes/0.5.0 56 | 57 | 58 | Indices and tables 59 | ================== 60 | 61 | * :ref:`genindex` 62 | * :ref:`modindex` 63 | * :ref:`search` 64 | 65 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | from setuptools import setup 4 | 5 | 6 | setup( 7 | name="alligator", 8 | version="1.0.0", 9 | description="Simple offline task queues.", 10 | author="Daniel Lindsley", 11 | author_email="daniel@toastdriven.com", 12 | url="http://github.com/toastdriven/alligator/", 13 | long_description=open("README.rst", "r").read(), 14 | packages=["alligator", "alligator/backends"], 15 | include_package_data=True, 16 | zip_safe=False, 17 | scripts=["bin/latergator.py"], 18 | requires=[], 19 | install_requires=[], 20 | tests_require=["pytest", "coverage", "pytest-cov", "redis", "boto"], 21 | classifiers=[ 22 | "Development Status :: 5 - Production/Stable", 23 | "Environment :: Web Environment", 24 | "Environment :: No Input/Output (Daemon)", 25 | "Intended Audience :: Developers", 26 | "License :: OSI Approved :: BSD License", 27 | "Operating System :: OS Independent", 28 | "Programming Language :: Python", 29 | "Programming Language :: Python :: 2", 30 | "Programming Language :: Python :: 3", 31 | "Topic :: Utilities", 32 | ], 33 | ) 34 | -------------------------------------------------------------------------------- /docs/security.rst: -------------------------------------------------------------------------------- 1 | .. _security: 2 | 3 | ======== 4 | Security 5 | ======== 6 | 7 | Alligator takes security seriously. By default, it: 8 | 9 | * does not access your filesystem in any way. 10 | * only handles JSON-serializable data. 11 | * only imports code available to your ``PYTHONPATH``. 12 | 13 | While no known vulnerabilities exist, all software has bugs & Alligator is no 14 | exception. 15 | 16 | If you believe you have found a security-related issue, please **DO NOT SUBMIT 17 | AN ISSUE/PULL REQUEST**. This would be a public disclosure & would allow for 18 | 0-day exploits. 19 | 20 | Instead, please send an email to "daniel@toastdriven.com" & include the 21 | following information: 22 | 23 | * A description of the problem/suggestion. 24 | * How to recreate the bug. 25 | * If relevant, including the versions of your: 26 | 27 | * Python interpreter 28 | * Web framework (if applicable) 29 | * Alligator 30 | * Optionally of the other dependencies involved 31 | 32 | Please bear in mind that I'm not a security expert/researcher, so a layman's 33 | description of the issue is very important. 34 | 35 | Upon reproduction of the exploit, steps will be taken to fix the issue, release 36 | a new version & make users aware of the need to upgrade. Proper credit for the 37 | discovery of the issue will be granted via the AUTHORS file & other mentions. 38 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2014, Daniel Lindsley 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without modification, 5 | are permitted provided that the following conditions are met: 6 | 7 | 1. Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 10 | 2. Redistributions in binary form must reproduce the above copyright notice, 11 | this list of conditions and the following disclaimer in the documentation 12 | and/or other materials provided with the distribution. 13 | 14 | 3. Neither the name of the alligator nor the names of its contributors may be 15 | used to endorse or promote products derived from this software without 16 | specific prior written permission. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 19 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 20 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 21 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 22 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 23 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 24 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 25 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 26 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 27 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28 | -------------------------------------------------------------------------------- /tests/test_sqs_backend.py: -------------------------------------------------------------------------------- 1 | import os 2 | import time 3 | import unittest 4 | 5 | 6 | CONN_STRING = os.environ.get("ALLIGATOR_CONN") 7 | 8 | 9 | @unittest.skipIf(not CONN_STRING.startswith("sqs:"), "Skipping SQS tests") 10 | class SQSTestCase(unittest.TestCase): 11 | def setUp(self): 12 | super(SQSTestCase, self).setUp() 13 | 14 | from alligator.backends.sqs_backend import Client as SQSClient 15 | 16 | self.backend = SQSClient(CONN_STRING) 17 | 18 | def test_all(self): 19 | # Just reach in & clear things out. 20 | # This sucks, but is an AWS requirement (only once every 60 seconds). 21 | self.backend.drop_all("all") 22 | time.sleep(61) 23 | 24 | self.assertEqual(self.backend.conn_string, CONN_STRING) 25 | 26 | self.assertEqual(self.backend.len("all"), 0) 27 | self.assertEqual(self.backend.len("something"), 0) 28 | 29 | self.backend.push("all", "hello", '{"whee": 1}') 30 | self.backend.push("all", "hello", '{"whee": 2}') 31 | 32 | data = self.backend.pop("all") 33 | self.assertEqual(data, '{"whee": 1}') 34 | time.sleep(31) 35 | self.assertEqual(self.backend.len("all"), 1) 36 | 37 | with self.assertRaises(NotImplementedError): 38 | self.backend.get("all", "world") 39 | 40 | # Push a delayed task. 41 | shortly = int(time.time()) + 10 42 | self.backend.push("all", "hello", '{"whee": 3}', delay_until=shortly) 43 | time.sleep(31) 44 | self.assertEqual(self.backend.len("all"), 2) 45 | 46 | self.backend.drop_all("all") 47 | time.sleep(61) 48 | -------------------------------------------------------------------------------- /tests/test_utils.py: -------------------------------------------------------------------------------- 1 | import re 2 | import unittest 3 | 4 | from alligator import __version__, version 5 | from alligator.exceptions import UnknownModuleError, UnknownCallableError 6 | from alligator.utils import ( 7 | determine_module, 8 | determine_name, 9 | import_module, 10 | import_attr, 11 | ) 12 | 13 | 14 | class UtilsTestCase(unittest.TestCase): 15 | def test_import_module(self): 16 | module = import_module("random") 17 | self.assertEqual(module.__name__, "random") 18 | 19 | with self.assertRaises(UnknownModuleError) as cm: 20 | import_module("nopenopeNOPE") 21 | 22 | def test_import_attr(self): 23 | choice = import_attr("random", "choice") 24 | self.assertEqual(choice.__name__, "choice") 25 | 26 | with self.assertRaises(UnknownCallableError) as cm: 27 | import_attr("random", "yolo") 28 | 29 | def test_determine_module(self): 30 | from alligator.backends.locmem_backend import Client 31 | 32 | self.assertEqual( 33 | determine_module(Client), "alligator.backends.locmem_backend" 34 | ) 35 | 36 | def test_determine_name(self): 37 | from alligator.backends.locmem_backend import Client 38 | 39 | self.assertEqual(determine_name(import_module), "import_module") 40 | self.assertEqual(determine_name(Client), "Client") 41 | self.assertEqual(determine_name(lambda x: x), "") 42 | 43 | def test_version(self): 44 | semver = re.compile(r"[\d]+\.[\d]+\.[\d]+") 45 | 46 | v = version() 47 | self.assertTrue(semver.match(v)) 48 | 49 | if len(__version__) > 3: 50 | self.assertTrue(__version__[3] in v) 51 | -------------------------------------------------------------------------------- /tests/test_redis_backend.py: -------------------------------------------------------------------------------- 1 | import os 2 | import redis 3 | import time 4 | import unittest 5 | 6 | from alligator.backends.redis_backend import Client as RedisClient 7 | 8 | 9 | CONN_STRING = os.environ.get("ALLIGATOR_CONN") 10 | 11 | 12 | @unittest.skipIf(not CONN_STRING.startswith("redis:"), "Skipping Redis tests") 13 | class RedisTestCase(unittest.TestCase): 14 | def setUp(self): 15 | super(RedisTestCase, self).setUp() 16 | self.backend = RedisClient(CONN_STRING) 17 | 18 | # Just reach in & clear things out. 19 | self.backend.conn.flushdb() 20 | 21 | def test_init(self): 22 | self.assertEqual(self.backend.conn_string, CONN_STRING) 23 | self.assertTrue(isinstance(self.backend.conn, redis.StrictRedis)) 24 | 25 | def test_len(self): 26 | self.assertEqual(self.backend.len("all"), 0) 27 | self.assertEqual(self.backend.len("something"), 0) 28 | 29 | def test_drop_all(self): 30 | self.backend.push("all", "hello", '{"whee": 1}') 31 | self.backend.push("all", "world", '{"whee": 2}') 32 | 33 | self.assertEqual(self.backend.len("all"), 2) 34 | self.backend.drop_all("all") 35 | self.assertEqual(self.backend.len("all"), 0) 36 | 37 | def test_push(self): 38 | self.assertEqual(self.backend.len("all"), 0) 39 | 40 | self.backend.push("all", "hello", '{"whee": 1}') 41 | self.assertEqual(self.backend.len("all"), 1) 42 | 43 | def test_pop(self): 44 | self.backend.push("all", "hello", '{"whee": 1}') 45 | time.sleep(1) 46 | 47 | data = self.backend.pop("all") 48 | self.assertEqual(data, '{"whee": 1}') 49 | self.assertEqual(self.backend.len("all"), 0) 50 | 51 | def test_get(self): 52 | self.backend.push("all", "hello", '{"whee": 1}') 53 | self.backend.push("all", "world", '{"whee": 2}') 54 | 55 | data = self.backend.get("all", "world") 56 | self.assertEqual(data, '{"whee": 2}') 57 | self.assertEqual(self.backend.len("all"), 1) 58 | -------------------------------------------------------------------------------- /alligator/utils.py: -------------------------------------------------------------------------------- 1 | import importlib 2 | 3 | from .exceptions import UnknownModuleError, UnknownCallableError 4 | 5 | 6 | def determine_module(func): 7 | """ 8 | Given a function, returns the Python dotted path of the module it comes 9 | from. 10 | 11 | Ex:: 12 | 13 | from random import choice 14 | determine_module(choice) # Returns 'random' 15 | 16 | Args: 17 | func (callable): The callable 18 | 19 | Returns: 20 | str: Dotted path string 21 | """ 22 | return func.__module__ 23 | 24 | 25 | def determine_name(func): 26 | """ 27 | Given a function, returns the name of the function. 28 | 29 | Ex:: 30 | 31 | from random import choice 32 | determine_name(choice) # Returns 'choice' 33 | 34 | Args: 35 | func (callable): The callable 36 | 37 | Returns: 38 | str: Name string 39 | """ 40 | if hasattr(func, "__name__"): 41 | return func.__name__ 42 | elif hasattr(func, "__class__"): 43 | return func.__class__.__name__ 44 | 45 | # This shouldn't be possible, but blow up if so. 46 | raise AttributeError("Provided callable '{}' has no name.".format(func)) 47 | 48 | 49 | def import_module(module_name): 50 | """ 51 | Given a dotted Python path, imports & returns the module. 52 | 53 | If not found, raises ``UnknownModuleError``. 54 | 55 | Ex:: 56 | 57 | mod = import_module('random') 58 | 59 | Args: 60 | module_name (str): The dotted Python path 61 | 62 | Returns: 63 | module: The imported module 64 | """ 65 | try: 66 | return importlib.import_module(module_name) 67 | except ImportError as err: 68 | raise UnknownModuleError(str(err)) 69 | 70 | 71 | def import_attr(module_name, attr_name): 72 | """ 73 | Given a dotted Python path & an attribute name, imports the module & 74 | returns the attribute. 75 | 76 | If not found, raises ``UnknownCallableError``. 77 | 78 | Ex:: 79 | 80 | choice = import_attr('random', 'choice') 81 | 82 | Args: 83 | module_name (str): The dotted Python path 84 | attr_name (str): The attribute name 85 | 86 | Returns: 87 | attribute 88 | """ 89 | module = import_module(module_name) 90 | 91 | try: 92 | return getattr(module, attr_name) 93 | except AttributeError as err: 94 | raise UnknownCallableError(str(err)) 95 | -------------------------------------------------------------------------------- /tests/test_custom_backend.py: -------------------------------------------------------------------------------- 1 | import os 2 | import unittest 3 | from unittest import mock 4 | 5 | from alligator.backends.sqlite_backend import Client as SQLiteClient 6 | from alligator.constants import ALL 7 | from alligator.gator import Gator, Options 8 | from alligator.tasks import Task 9 | 10 | 11 | def add(a, b): 12 | return a + b 13 | 14 | 15 | class CustomBackendTestCase(unittest.TestCase): 16 | def setUp(self): 17 | super(CustomBackendTestCase, self).setUp() 18 | self.conn_string = "sqlite:///tmp/alligator_test.db" 19 | 20 | try: 21 | os.unlink("/tmp/alligator_test.db") 22 | except OSError: 23 | pass 24 | 25 | self.gator = Gator(self.conn_string, backend_class=SQLiteClient) 26 | self.gator.backend.setup_tables() 27 | 28 | def test_everything(self): 29 | self.assertEqual(self.gator.backend.len(ALL), 0) 30 | 31 | t1 = self.gator.task(add, 1, 3) 32 | t2 = self.gator.task(add, 5, 7) 33 | t3 = self.gator.task(add, 3, 13) 34 | t4 = self.gator.task(add, 9, 4) 35 | 36 | self.assertEqual(self.gator.backend.len(ALL), 4) 37 | 38 | task_1 = self.gator.pop() 39 | self.assertEqual(task_1.result, 4) 40 | 41 | task_3 = self.gator.get(t3.task_id) 42 | self.assertEqual(task_3.result, 16) 43 | 44 | task_2 = self.gator.pop() 45 | self.assertEqual(task_2.result, 12) 46 | 47 | self.assertEqual(self.gator.backend.len(ALL), 1) 48 | 49 | self.gator.backend.drop_all(ALL) 50 | self.assertEqual(self.gator.backend.len(ALL), 0) 51 | 52 | @mock.patch("time.time") 53 | def test_delay_until(self, mock_time): 54 | mock_time.return_value = 12345678 55 | 56 | self.assertEqual(self.gator.backend.len(ALL), 0) 57 | 58 | with self.gator.options(delay_until=12345777): 59 | t1 = self.gator.task(add, 2, 2) 60 | 61 | with self.gator.options(delay_until=12345999): 62 | t2 = self.gator.task(add, 3, 8) 63 | 64 | with self.gator.options(delay_until=12345678): 65 | t3 = self.gator.task(add, 4, 11) 66 | 67 | with self.gator.options(): 68 | t4 = self.gator.task(add, 7, 1) 69 | 70 | self.assertEqual(self.gator.backend.len(ALL), 4) 71 | 72 | task_1 = self.gator.pop() 73 | self.assertEqual(task_1.result, 4) 74 | 75 | mock_time.return_value = 123499999 76 | task_2 = self.gator.pop() 77 | self.assertEqual(task_2.result, 11) 78 | -------------------------------------------------------------------------------- /docs/releasenotes/1.0.0.rst: -------------------------------------------------------------------------------- 1 | alligator 1.0.0 2 | =============== 3 | 4 | :date: 2020-10-28 5 | 6 | This marks the official 1.0.0 release of ``alligator``. 7 | 8 | Major changes include: 9 | 10 | * Added scheduled & delayed task support 11 | * An official SQLite backend 12 | * Removed Beanstalk support 13 | * Upgraded to ``boto3`` 14 | * General modernization 15 | 16 | This also marks the beginning of API stability for the 1.X.X series of 17 | releases. While new features & bugfixes will continue to be added, no 18 | public method signatures will be changed in **backward-incompatible** ways. 19 | 20 | This means you should be able to upgrade between versions within the 1.X.X 21 | series without having to worry about ``alligator``'s API changing & breaking 22 | code that depends on it. 23 | 24 | Lastly, there are no changes from the ``1.0.0-beta-2`` build. 25 | 26 | What follows are the *combined* release notes for the prior 27 | 1.0.0-beta-X & 1.0.0-alpha-X releases. 28 | 29 | Enjoy & happy queuing! 30 | 31 | 32 | Features 33 | -------- 34 | 35 | * Added official SQLite support. (SHA: `7ff4236`) 36 | * Added delayed/scheduled task support. (SHA: `da33d1a`, `a9f2d58`, & 37 | `9266d20`) 38 | * Added ``poetry`` support. (SHA: `2643b40`) 39 | * Removed Beanstalk support. (SHA: `368db30`) 40 | 41 | `Note:` This is due to the lack of Python 3 support in the ``beanstalkc`` 42 | library. If this changes in the future, we'll re-evaluate adding it back 43 | in. 44 | 45 | * Updated the SQS backend to `boto3`. (SHA: `ee80d69` & `c6fd671`) 46 | 47 | 48 | Bugfixes 49 | -------- 50 | 51 | * Revised how logging is setup by the `Worker`. (SHA: `d52a877`) 52 | * Bugfixes for failed tasks & consistent pop behavior. (SHA: `9d91309`) 53 | * Fixed several bugs around delayed/scheduled tasks. (SHA: `2e51867`) 54 | * Fixed the SQS backend to delete messages & trimming delays to an int. 55 | (SHA: `cd6c4e2`) 56 | * Fixed the executable permission on ``latergator``. (SHA: `2d176a4`) 57 | * Formatted all the code with `black`. (SHA: `3cff2cc`) 58 | * Changed ``Task.async`` to ``Task.is_async``, due to ``async`` being a 59 | reserved word in modern Python 3. (SHA: `bf98511`) 60 | * Updated all dependencies to more current versions. (SHA: `1d368e3`) 61 | * Fixed the Redis backend's tests. (SHA: `e793aa9`) 62 | * Fixed the worker process to correctly exit during testing. (SHA: `9b7b486`) 63 | 64 | 65 | Documentation 66 | ------------- 67 | 68 | * Added more to the Migration Guide. (SHA: `6e48da4`) 69 | * Fixed the Redis & SQS backend docstrings. (SHA: `4fe15e7`) 70 | * Added 1.0.0-alpha-1 release notes. (SHA: `e83c391`) 71 | * Added the start of the "Migration to 1.0" guide. (SHA: `cbab984`) 72 | * Switched the docstring style. (SHA: `e5a6a40`) 73 | * Added delayed/scheduled tasks documentation. (SHA: `87c0165`) 74 | * Various documentation fixes. (SHA: `2459956`) 75 | -------------------------------------------------------------------------------- /docs/migration-to-1.0.rst: -------------------------------------------------------------------------------- 1 | .. _migration-to-1.0: 2 | 3 | ====================== 4 | Migration to 1.0 Guide 5 | ====================== 6 | 7 | In the process of going from ``0.10.0`` to ``1.0.0``, a couple things changed 8 | that are not backward-compatible. If you were relying on a previous version, 9 | this is what's needed to update: 10 | 11 | 12 | Switch to Python 3 13 | ================== 14 | 15 | It's time. Make the jump to Python 3. 16 | 17 | If not, the ``0.10.0`` will continue to be available on PyPI. 18 | 19 | 20 | Dropped Beanstalk Support 21 | ========================= 22 | 23 | Unfortunately, the `underlying library`_ (``beanstalkc``) never completed the 24 | transition to Python 3. 25 | 26 | As a result, for now, Beanstalk support has been removed. Please consider 27 | either switching backends or staying on the `0.10.0` release. 28 | 29 | .. _`underlying library`: https://github.com/earl/beanstalkc 30 | 31 | 32 | Update References from ``Task.async`` to ``Task.is_async`` 33 | ========================================================== 34 | 35 | In Python 3, ``async`` is a reserved word. 36 | 37 | It can be search & replaced with ``is_async``. Look for places where you're 38 | manually constructing ``Task`` objects or using ``gator.options(async=...)``. 39 | 40 | The behavior is the same, just a changed name. 41 | 42 | 43 | Redis Backend: Recreate Queues 44 | ============================== 45 | 46 | This one is a little painful. You'll need to allow your existing queues to 47 | drain of tasks with your *existing* Alligator install, then run the following: 48 | 49 | .. code:: python 50 | 51 | # If your Redis instance (or database) are specifically for Alligator... 52 | >>> gator.backend.conn.flushdb() 53 | 54 | # If you have other co-mingled data in the Redis instance/database, run: 55 | >>> gator.backend.conn.delete("all") 56 | # ...and repeat for any other queue names you have. 57 | 58 | After this is done, you can upgrade Alligator to `1.0.0` & simply proceed 59 | as normal. 60 | 61 | The reason is due to adding support for delayed/scheduled tasks. Within 62 | Redis, the queue keys switched from a plain `list` to a `sorted set`. As a 63 | result, the commands sent to Redis aren't compatible between `0.10.0` & 64 | `1.0.0`. 65 | 66 | Backward compatibility will be maintained throughout the `1.X.X` series, so 67 | changes like this shouldn't occur again for a long time. 68 | 69 | 70 | Custom Backends: Add ``delay_until`` Support 71 | ============================================ 72 | 73 | As part of the the ``1.0.0`` release, support was added for delayed/scheduled 74 | tasks. If your backend can support timestamps, you can add support if it's 75 | desirable. 76 | 77 | Changes needed: 78 | 79 | * In the ``Client.push``, add the ``, delay_until=None`` argument at the end 80 | of the signature & set it on the backend. 81 | * In the ``Client.pop``, add filtering to check for timestamps less than the 82 | current time. 83 | -------------------------------------------------------------------------------- /docs/installing.rst: -------------------------------------------------------------------------------- 1 | .. _installing: 2 | 3 | ==================== 4 | Installing Alligator 5 | ==================== 6 | 7 | Installation of Alligator itself is a relatively simple affair. For the most 8 | recent stable release, simply use pip_ to run:: 9 | 10 | $ pip install alligator 11 | 12 | Alternately, you can download the latest development source from Github:: 13 | 14 | $ git clone https://github.com/toastdriven/alligator.git 15 | $ cd alligator 16 | $ python setup.py install 17 | 18 | .. _pip: http://pip-installer.org/ 19 | 20 | 21 | Queue Backends 22 | ============== 23 | 24 | Alligator includes a ``Local Memory Client``, which is useful for development 25 | or testing (no setup required). However, this is not very scalable. 26 | 27 | For production use, you should install one of the following servers used for 28 | queuing: 29 | 30 | 31 | Redis 32 | ----- 33 | 34 | A in-memory data structure server, it offers excellent speed as well as being 35 | a frequently-already-installed server. Official releases can be found at 36 | http://redis.io/download. 37 | 38 | You'll also need to install the ``redis`` package:: 39 | 40 | $ pip install redis 41 | 42 | You can also install via other package managers:: 43 | 44 | # On Mac with Homebrew 45 | $ brew install redis 46 | 47 | # On Ubuntu 48 | $ sudo aptitude install redis 49 | 50 | 51 | SQS 52 | --- 53 | 54 | `Amazon SQS`_ is a queue service created by Amazon Web Services. It works well 55 | in large-scale environments or if you're already using other AWS services. 56 | 57 | It has the benefit of not requiring an installed setup, only an AWS account & 58 | a credit card, making it the easiest of the production queues to setup. 59 | 60 | You'll need to install the ``boto3`` packages:: 61 | 62 | $ pip install 'boto3>=1.12.0' 63 | 64 | .. warning:: 65 | 66 | SQS works differently than the other queues in a couple ways: 67 | 68 | 1. It does **NOT** support custom ``task_id``s. You can still set them 69 | and it will be preseved in the task, but the backend will overwrite 70 | your ``task_id`` choice once it's in the queue. 71 | 2. You must **manually** create queues! Alligator will not auto-create them 72 | (to save on requests performed & therefore how much you are billed). 73 | Create the ``Gator`` queues at the AWS console before trying to use them. 74 | 3. It does **NOT** support ``gator.get(...)``, as this functionality is not 75 | supported by the SQS service itself. 76 | 77 | It's also an excellent choice at large volumes, but you should be aware of 78 | the shortcomings. 79 | 80 | .. _`Amazon SQS`: https://aws.amazon.com/sqs/ 81 | 82 | 83 | SQLite 84 | ------ 85 | 86 | A file-backed database. It's fast, lightweight & easy to work with. 87 | Python 3 ships with built-in support & there's no server to run. Suitable 88 | for small/light loads & simple setups (or development). 89 | 90 | You can also install via other package managers:: 91 | 92 | # On Mac with Homebrew 93 | $ brew install sqlite 94 | 95 | # On Ubuntu 96 | $ sudo aptitude install sqlite3 97 | -------------------------------------------------------------------------------- /docs/contributing.rst: -------------------------------------------------------------------------------- 1 | .. _contributing: 2 | 3 | ============ 4 | Contributing 5 | ============ 6 | 7 | Alligator is open-source and, as such, grows (or shrinks) & improves in part 8 | due to the community. Below are some guidelines on how to help with the project. 9 | 10 | 11 | Philosophy 12 | ========== 13 | 14 | * Alligator is BSD-licensed. All contributed code must be either 15 | 16 | * the original work of the author, contributed under the BSD, or... 17 | * work taken from another project released under a BSD-compatible license. 18 | 19 | * GPL'd (or similar) works are not eligible for inclusion. 20 | * Alligator's git master branch should always be stable, production-ready & 21 | passing all tests. 22 | * Major releases (1.x.x) are commitments to backward-compatibility of the 23 | public APIs. Any documented API should ideally not change between major 24 | releases. The exclusion to this rule is in the event of a security 25 | issue. 26 | * Minor releases (x.3.x) are for the addition of substantial features or major 27 | bugfixes. 28 | * Patch releases (x.x.4) are for minor features or bugfixes. 29 | 30 | 31 | Guidelines For Reporting An Issue/Feature 32 | ========================================= 33 | 34 | So you've found a bug or have a great idea for a feature. Here's the steps you 35 | should take to help get it added/fixed in Alligator: 36 | 37 | * First, check to see if there's an existing issue/pull request for the 38 | bug/feature. All issues are at https://github.com/toastdriven/alligator/issues 39 | and pull reqs are at https://github.com/toastdriven/alligator/pulls. 40 | * If there isn't one there, please file an issue. The ideal report includes: 41 | 42 | * A description of the problem/suggestion. 43 | * How to recreate the bug. 44 | * If relevant, including the versions of your: 45 | 46 | * Python interpreter 47 | * Web framework (if applicable) 48 | * Alligator 49 | * Optionally of the other dependencies involved 50 | 51 | * Ideally, creating a pull request with a (failing) test case demonstrating 52 | what's wrong. This makes it easy for us to reproduce & fix the problem. 53 | Instructions for running the tests are at :doc:`index` 54 | 55 | 56 | Guidelines For Contributing Code 57 | ================================ 58 | 59 | If you're ready to take the plunge & contribute back some code/docs, the 60 | process should look like: 61 | 62 | * Fork the project on GitHub into your own account. 63 | * Clone your copy of Alligator. 64 | * Make a new branch in git & commit your changes there. 65 | * Push your new branch up to GitHub. 66 | * Again, ensure there isn't already an issue or pull request out there on it. 67 | If there is & you feel you have a better fix, please take note of the issue 68 | number & mention it in your pull request. 69 | * Create a new pull request (based on your branch), including what the 70 | problem/feature is, versions of your software & referencing any related 71 | issues/pull requests. 72 | 73 | In order to be merged into Alligator, contributions must have the following: 74 | 75 | * A solid patch that: 76 | 77 | * is clear. 78 | * works across all supported versions of Python. 79 | * follows the existing style of the code base (mostly PEP-8). 80 | * comments included as needed. 81 | 82 | * A test case that demonstrates the previous flaw that now passes 83 | with the included patch. 84 | * If it adds/changes a public API, it must also include documentation 85 | for those changes. 86 | * Must be appropriately licensed (see "Philosophy"). 87 | * Adds yourself to the AUTHORS file. 88 | 89 | If your contribution lacks any of these things, they will have to be added 90 | by a core contributor before being merged into Alligator proper, which may take 91 | additional time. 92 | -------------------------------------------------------------------------------- /tests/test_locmem_backend.py: -------------------------------------------------------------------------------- 1 | import os 2 | import unittest 3 | from unittest import mock 4 | 5 | from alligator.backends.locmem_backend import Client as LocmemClient 6 | 7 | 8 | CONN_STRING = os.environ.get("ALLIGATOR_CONN") 9 | 10 | 11 | @unittest.skipIf( 12 | not CONN_STRING.startswith("locmem:"), "Skipping Locmem tests" 13 | ) 14 | class LocmemTestCase(unittest.TestCase): 15 | def setUp(self): 16 | super(LocmemTestCase, self).setUp() 17 | self.backend = LocmemClient(CONN_STRING) 18 | 19 | # Just reach in & clear things out. 20 | LocmemClient.queues = {} 21 | LocmemClient.task_data = {} 22 | 23 | def test_init(self): 24 | self.assertEqual(LocmemClient.queues, {}) 25 | self.assertEqual(LocmemClient.task_data, {}) 26 | 27 | def test_len(self): 28 | LocmemClient.queues = { 29 | "all": [["a", None], ["b", 12345678], ["c", None]] 30 | } 31 | self.assertEqual(self.backend.len("all"), 3) 32 | self.assertEqual(self.backend.len("something"), 0) 33 | 34 | def test_drop_all(self): 35 | LocmemClient.queues = { 36 | "all": [["a", None], ["b", 12345678], ["c", None]] 37 | } 38 | LocmemClient.task_data = { 39 | "a": {"whatev": True}, 40 | "b": "grump", 41 | "d": "another", 42 | } 43 | 44 | self.backend.drop_all("all") 45 | self.assertEqual(LocmemClient.queues, {"all": []}) 46 | self.assertEqual(LocmemClient.task_data, {"d": "another"}) 47 | 48 | def test_push(self): 49 | self.assertEqual(LocmemClient.queues, {}) 50 | self.assertEqual(LocmemClient.task_data, {}) 51 | 52 | self.backend.push("all", "hello", {"whee": 1}) 53 | self.assertEqual(LocmemClient.queues, {"all": [["hello", None]]}) 54 | self.assertEqual(LocmemClient.task_data, {"hello": {"whee": 1}}) 55 | 56 | def test_push_delayed(self): 57 | self.assertEqual(LocmemClient.queues, {}) 58 | self.assertEqual(LocmemClient.task_data, {}) 59 | 60 | self.backend.push("all", "hello", {"whee": 1}, delay_until=12345798) 61 | self.assertEqual(LocmemClient.queues, {"all": [["hello", 12345798]]}) 62 | self.assertEqual(LocmemClient.task_data, {"hello": {"whee": 1}}) 63 | 64 | def test_pop(self): 65 | self.backend.push("all", "hello", {"whee": 1}) 66 | 67 | data = self.backend.pop("all") 68 | self.assertEqual(data, {"whee": 1}) 69 | self.assertEqual(LocmemClient.queues, {"all": []}) 70 | self.assertEqual(LocmemClient.task_data, {}) 71 | 72 | @mock.patch("time.time") 73 | def test_pop_skip_delayed(self, mock_time): 74 | mock_time.return_value = 12345678 75 | 76 | self.backend.push("all", "hello", {"whee": 1}, delay_until=12345798) 77 | self.backend.push("all", "hallo", {"whoo": 2}) 78 | 79 | # Here, we're checking to make sure a task that's waiting for a 80 | # "future" time isn't pulled off the queue. 81 | data = self.backend.pop("all") 82 | self.assertEqual(data, {"whoo": 2}) 83 | self.assertEqual(LocmemClient.queues, {"all": [["hello", 12345798]]}) 84 | self.assertEqual(LocmemClient.task_data, {"hello": {"whee": 1}}) 85 | 86 | def test_get(self): 87 | self.backend.push("all", "hello", {"whee": 1}) 88 | self.backend.push("all", "world", {"whee": 2}) 89 | 90 | data = self.backend.get("all", "world") 91 | self.assertEqual(data, {"whee": 2}) 92 | self.assertEqual(LocmemClient.queues, {"all": [["hello", None]]}) 93 | self.assertEqual(LocmemClient.task_data, {"hello": {"whee": 1}}) 94 | 95 | # Try a non-existent one. 96 | data = self.backend.get("all", "nopenopenope") 97 | self.assertEqual(data, None) 98 | -------------------------------------------------------------------------------- /alligator/backends/locmem_backend.py: -------------------------------------------------------------------------------- 1 | import math 2 | import time 3 | 4 | 5 | class Client(object): 6 | queues = {} 7 | task_data = {} 8 | 9 | def __init__(self, conn_string): 10 | """ 11 | An in-memory `Client`. Useful for development & testing. 12 | 13 | Likely not particularly thread-safe. 14 | 15 | Args: 16 | conn_string (str): The DSN. Ignored. 17 | """ 18 | # We ignore the conn_string, since everything is happening in-memory. 19 | pass 20 | 21 | def len(self, queue_name): 22 | """ 23 | Returns the length of the queue. 24 | 25 | Args: 26 | queue_name (str): The name of the queue. Usually handled by the 27 | `Gator` instance. 28 | 29 | Returns: 30 | int: The length of the queue 31 | """ 32 | return len(self.__class__.queues.get(queue_name, [])) 33 | 34 | def drop_all(self, queue_name): 35 | """ 36 | Drops all the task in the queue. 37 | 38 | Args: 39 | queue_name (str): The name of the queue. Usually handled by the 40 | `Gator` instance. 41 | """ 42 | cls = self.__class__ 43 | 44 | for task_id, _ in cls.queues.get(queue_name, []): 45 | cls.task_data.pop(task_id, None) 46 | 47 | cls.queues[queue_name] = [] 48 | 49 | def push(self, queue_name, task_id, data, delay_until=None): 50 | """ 51 | Pushes a task onto the queue. 52 | 53 | Args: 54 | queue_name (str): The name of the queue. Usually handled by the 55 | `Gator` instance. 56 | task_id (str): The identifier of the task. 57 | data (str): The relevant data for the task. 58 | delay_until (float): Optional. The Unix timestamp to delay 59 | execution of the task until. Default is `None` (no delay). 60 | 61 | Returns: 62 | str: The task's ID 63 | """ 64 | cls = self.__class__ 65 | cls.queues.setdefault(queue_name, []) 66 | cls.queues[queue_name].append([task_id, delay_until]) 67 | cls.task_data[task_id] = data 68 | return task_id 69 | 70 | def pop(self, queue_name): 71 | """ 72 | Pops a task off the queue. 73 | 74 | Args: 75 | queue_name (str): The name of the queue. Usually handled by the 76 | `Gator` instance. 77 | 78 | Returns: 79 | str: The data for the task. 80 | """ 81 | cls = self.__class__ 82 | queue = cls.queues.get(queue_name, []) 83 | now = math.floor(time.time()) 84 | 85 | for offset, task_info in enumerate(queue): 86 | task_id, delay_until = task_info[0], task_info[1] 87 | 88 | # Check for a delay. 89 | if delay_until is not None: 90 | if now < delay_until: 91 | continue 92 | 93 | # We've found one we can process. 94 | queue.pop(offset) 95 | return cls.task_data.pop(task_id, None) 96 | 97 | def get(self, queue_name, task_id): 98 | """ 99 | Pops a specific task off the queue by identifier. 100 | 101 | Args: 102 | queue_name: The name of the queue. Usually handled by the 103 | `Gator` instance. 104 | task_id (str): The identifier of the task. 105 | 106 | Returns: 107 | str: The data for the task. 108 | """ 109 | # This method is *very* non-thread-safe. 110 | cls = self.__class__ 111 | queue = cls.queues.get(queue_name, []) 112 | 113 | for offset, task_info in enumerate(queue): 114 | if task_info[0] == task_id: 115 | queue.pop(offset) 116 | return cls.task_data.pop(task_id, None) 117 | -------------------------------------------------------------------------------- /alligator/backends/sqs_backend.py: -------------------------------------------------------------------------------- 1 | import time 2 | from urllib.parse import urlparse 3 | 4 | import boto3 5 | from botocore.config import Config 6 | 7 | 8 | class Client(object): 9 | def __init__(self, conn_string): 10 | """ 11 | A Amazon SQS-based ``Client``. 12 | 13 | Args: 14 | conn_string (str): The DSN. The region is parsed out of it. 15 | Should be of the format ``sqs://region-name/`` 16 | """ 17 | self.conn_string = conn_string 18 | bits = urlparse(self.conn_string) 19 | self.conn = self.get_connection(region=bits.hostname) 20 | self._queue = None 21 | 22 | def get_connection(self, region): 23 | """ 24 | Returns a ``SQSConnection`` connection instance. 25 | """ 26 | config = Config(region_name=region) 27 | return boto3.resource("sqs", config=config) 28 | 29 | def _get_queue(self, queue_name): 30 | if self._queue is None: 31 | self._queue = self.conn.get_queue_by_name(QueueName=queue_name) 32 | 33 | return self._queue 34 | 35 | def len(self, queue_name): 36 | """ 37 | Returns the length of the queue. 38 | 39 | Args: 40 | queue_name (str): The name of the queue. Usually handled by the 41 | ``Gator`` instance. 42 | 43 | Returns: 44 | int: The length of the queue 45 | """ 46 | queue = self._get_queue(queue_name) 47 | queue.load() 48 | return int(queue.attributes.get("ApproximateNumberOfMessages", 0)) 49 | 50 | def drop_all(self, queue_name): 51 | """ 52 | Drops all the task in the queue. 53 | 54 | Args: 55 | queue_name (str): The name of the queue. Usually handled by the 56 | ``Gator`` instance. 57 | """ 58 | queue = self._get_queue(queue_name) 59 | queue.purge() 60 | 61 | def push(self, queue_name, task_id, data, delay_until=None): 62 | """ 63 | Pushes a task onto the queue. 64 | 65 | Args: 66 | queue_name (str): The name of the queue. Usually handled by the 67 | ``Gator`` instance. 68 | task_id (str): The identifier of the task. 69 | data (str): The relevant data for the task. 70 | delay_until (float): Optional. The Unix timestamp to delay 71 | processing of the task until. Default is `None`. 72 | 73 | Returns: 74 | str: The task ID. 75 | """ 76 | kwargs = { 77 | "MessageBody": data, 78 | } 79 | 80 | if delay_until is not None: 81 | now = time.time() 82 | delay_by = delay_until - now 83 | 84 | if delay_by > 0: 85 | kwargs["DelaySeconds"] = int(delay_by) 86 | 87 | # SQS doesn't let you specify a task id. 88 | queue = self._get_queue(queue_name) 89 | res = queue.send_message(**kwargs) 90 | return res.get("MessageId") 91 | 92 | def pop(self, queue_name): 93 | """ 94 | Pops a task off the queue. 95 | 96 | Args: 97 | queue_name (str): The name of the queue. Usually handled by the 98 | ``Gator`` instance. 99 | 100 | Returns: 101 | str: The data for the task. 102 | """ 103 | queue = self._get_queue(queue_name) 104 | messages = queue.receive_messages(MaxNumberOfMessages=1) 105 | 106 | if messages: 107 | message = messages[0] 108 | data = message.body 109 | message.delete() 110 | return data 111 | 112 | def get(self, queue_name, task_id): 113 | """ 114 | Unsupported, as SQS does not include this functionality. 115 | """ 116 | raise NotImplementedError( 117 | "SQS does not support fetching a specific message off the queue." 118 | ) 119 | -------------------------------------------------------------------------------- /tests/test_workers.py: -------------------------------------------------------------------------------- 1 | import os 2 | import unittest 3 | from unittest import mock 4 | 5 | from alligator.gator import Gator 6 | from alligator.workers import Worker 7 | 8 | 9 | ALLOW_SLOW = bool(os.environ.get("ALLIGATOR_SLOW", False)) 10 | CONN_STRING = os.environ.get("ALLIGATOR_CONN") 11 | FILENAME = "/tmp/alligator_test_workers.txt" 12 | 13 | 14 | def touch_file(): 15 | with open(FILENAME, "w") as write_file: 16 | write_file.write("0") 17 | 18 | 19 | def read_file(): 20 | with open(FILENAME, "r") as read_file: 21 | return int(read_file.read().strip()) 22 | 23 | 24 | def incr_file(incr): 25 | value = read_file() 26 | 27 | with open(FILENAME, "w") as write_file: 28 | value += incr 29 | write_file.write(str(value)) 30 | 31 | 32 | def rm_file(): 33 | try: 34 | os.unlink(FILENAME) 35 | except OSError: 36 | pass 37 | 38 | 39 | def raise_error(val): 40 | raise ValueError("You've chosen... poorly.") 41 | 42 | 43 | @unittest.skipIf(not ALLOW_SLOW, "Skipping slow worker tests") 44 | class WorkerTestCase(unittest.TestCase): 45 | def setUp(self): 46 | super(WorkerTestCase, self).setUp() 47 | self.gator = Gator("locmem://") 48 | self.worker = Worker(self.gator, max_tasks=2, nap_time=1) 49 | 50 | self.gator.backend.drop_all("all") 51 | rm_file() 52 | touch_file() 53 | 54 | def test_init(self): 55 | self.assertEqual(self.worker.gator, self.gator) 56 | self.assertEqual(self.worker.max_tasks, 2) 57 | self.assertEqual(self.worker.to_consume, "all") 58 | self.assertEqual(self.worker.nap_time, 1) 59 | self.assertEqual(self.worker.tasks_complete, 0) 60 | 61 | def test_ident(self): 62 | ident = self.worker.ident() 63 | self.assertTrue(ident.startswith("Alligator Worker (#")) 64 | 65 | def test_check_and_run_task(self): 66 | self.assertEqual(read_file(), 0) 67 | 68 | self.gator.task(incr_file, 2) 69 | self.gator.task(incr_file, 3) 70 | 71 | self.assertEqual(self.gator.backend.len("all"), 2) 72 | self.assertEqual(self.worker.tasks_complete, 0) 73 | 74 | self.worker.check_and_run_task() 75 | 76 | self.assertEqual(self.gator.backend.len("all"), 1) 77 | self.assertEqual(self.worker.tasks_complete, 1) 78 | self.assertEqual(read_file(), 2) 79 | 80 | def test_check_and_run_task_trap_exception(self): 81 | self.assertEqual(read_file(), 0) 82 | 83 | self.gator.task(incr_file, 2) 84 | self.gator.task(incr_file, 3) 85 | self.gator.task(raise_error, 75) 86 | self.gator.task(incr_file, 4) 87 | 88 | self.assertEqual(self.gator.backend.len("all"), 4) 89 | self.assertEqual(self.worker.tasks_complete, 0) 90 | 91 | with mock.patch.object( 92 | self.worker.log, "exception" 93 | ) as mock_exception: 94 | self.assertTrue(self.worker.check_and_run_task()) 95 | self.assertTrue(self.worker.check_and_run_task()) 96 | 97 | # Here, it should hit the exception BUT not stop execution. 98 | self.assertFalse(self.worker.check_and_run_task()) 99 | 100 | self.assertTrue(self.worker.check_and_run_task()) 101 | 102 | # Make sure we tried to log that exception. 103 | mock_exception.called_once() 104 | 105 | self.assertEqual(self.gator.backend.len("all"), 0) 106 | self.assertEqual(self.worker.tasks_complete, 3) 107 | self.assertEqual(read_file(), 9) 108 | 109 | def test_run_forever(self): 110 | self.assertEqual(read_file(), 0) 111 | 112 | self.gator.task(incr_file, 2) 113 | self.gator.task(incr_file, 3) 114 | self.gator.task(incr_file, 4) 115 | 116 | self.assertEqual(self.gator.backend.len("all"), 3) 117 | 118 | # Should actually only run for two of the three tasks. 119 | self.worker.run_forever() 120 | 121 | self.assertEqual(self.gator.backend.len("all"), 1) 122 | self.assertEqual(read_file(), 5) 123 | -------------------------------------------------------------------------------- /alligator/backends/redis_backend.py: -------------------------------------------------------------------------------- 1 | import math 2 | import time 3 | from urllib.parse import urlparse 4 | 5 | import redis 6 | 7 | 8 | class Client(object): 9 | def __init__(self, conn_string): 10 | """ 11 | A Redis-based ``Client``. 12 | 13 | Args: 14 | conn_string (str): The DSN. The host/port/db are parsed out of it. 15 | Should be of the format ``redis://host:port/db`` 16 | """ 17 | self.conn_string = conn_string 18 | bits = urlparse(self.conn_string) 19 | self.conn = self.get_connection( 20 | host=bits.hostname, 21 | port=bits.port, 22 | db=bits.path.lstrip("/").split("/")[0], 23 | ) 24 | 25 | def get_connection(self, host, port, db): 26 | """ 27 | Returns a ``StrictRedis`` connection instance. 28 | """ 29 | return redis.StrictRedis( 30 | host=host, port=port, db=db, decode_responses=True 31 | ) 32 | 33 | def len(self, queue_name): 34 | """ 35 | Returns the length of the queue. 36 | 37 | Args: 38 | queue_name (str): The name of the queue. Usually handled by the 39 | `Gator`` instance. 40 | 41 | Returns: 42 | int: The length of the queue 43 | """ 44 | return self.conn.zcard(queue_name) 45 | 46 | def drop_all(self, queue_name): 47 | """ 48 | Drops all the task in the queue. 49 | 50 | Args: 51 | queue_name (str): The name of the queue. Usually handled by the 52 | ``Gator`` instance. 53 | """ 54 | task_ids = self.conn.zrange(queue_name, 0, -1) 55 | 56 | for task_id in task_ids: 57 | self.conn.delete(task_id) 58 | 59 | self.conn.delete(queue_name) 60 | 61 | def push(self, queue_name, task_id, data, delay_until=None): 62 | """ 63 | Pushes a task onto the queue. 64 | 65 | Args: 66 | queue_name (str): The name of the queue. Usually handled by the 67 | ``Gator`` instance. 68 | task_id (str): The identifier of the task. 69 | data (str): The relevant data for the task. 70 | delay_until (float): Optional. The Unix timestamp to delay 71 | processing of the task until. Default is `None`. 72 | 73 | Returns: 74 | str: The task ID. 75 | """ 76 | if delay_until is None: 77 | delay_until = math.ceil(time.time()) 78 | 79 | self.conn.zadd(queue_name, {task_id: delay_until}, nx=True) 80 | self.conn.set(task_id, data) 81 | return task_id 82 | 83 | def pop(self, queue_name): 84 | """ 85 | Pops a task off the queue. 86 | 87 | Args: 88 | queue_name (str): The name of the queue. Usually handled by the 89 | ``Gator`` instance. 90 | 91 | Returns: 92 | str: The data for the task. 93 | """ 94 | now = math.floor(time.time()) 95 | available_to_pop = self.conn.zrangebyscore( 96 | queue_name, 0, now, start=0, num=1 97 | ) 98 | 99 | if not len(available_to_pop): 100 | return None 101 | 102 | popped = self.conn.zpopmin(queue_name) 103 | task_id, delay_until = popped[0][0], popped[0][1] 104 | data = self.conn.get(task_id) 105 | self.conn.delete(task_id) 106 | return data 107 | 108 | def get(self, queue_name, task_id): 109 | """ 110 | Pops a specific task off the queue by identifier. 111 | 112 | Args: 113 | queue_name (str): The name of the queue. Usually handled by the 114 | ``Gator`` instance. 115 | task_id (str): The identifier of the task. 116 | 117 | Returns: 118 | str: The data for the task. 119 | """ 120 | self.conn.zrem(queue_name, task_id) 121 | data = self.conn.get(task_id) 122 | 123 | if data: 124 | self.conn.delete(task_id) 125 | return data 126 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Alligator 2 | ========= 3 | 4 | .. image:: https://travis-ci.org/toastdriven/alligator.png?branch=master 5 | :target: https://travis-ci.org/toastdriven/alligator 6 | 7 | Simple offline task queues. For Python. 8 | 9 | `"See you later, alligator."` 10 | 11 | Latest documentation at http://alligator.readthedocs.org/en/latest/. 12 | 13 | 14 | Requirements 15 | ------------ 16 | 17 | * Python 3.6+ 18 | * (Optional) ``redis`` for the Redis backend 19 | * (Optional) ``boto3>=1.12.0`` for the SQS backend 20 | 21 | 22 | Basic Usage 23 | ----------- 24 | 25 | This example uses Django, but there's nothing Django-specific about Alligator. 26 | 27 | I repeat, You can use it with **any** Python code that would benefit from 28 | background processing. 29 | 30 | .. code:: python 31 | 32 | from alligator import Gator 33 | 34 | from django.contrib.auth.models import User 35 | from django.shortcuts import send_email 36 | 37 | 38 | # Make a Gator instance. 39 | # Under most circumstances, you would configure this in one place & 40 | # import that instance instead. 41 | gator = Gator('redis://localhost:6379/0') 42 | 43 | 44 | # The task itself. 45 | # Nothing special, just a plain *undecorated* function. 46 | def follow_email(followee_username, follower_username): 47 | followee = User.objects.get(username=followee_username) 48 | follower = User.objects.get(username=follower_username) 49 | 50 | subject = 'You got followed!' 51 | message = 'Hey {}, you just got followed by {}! Whoohoo!'.format( 52 | followee.username, 53 | follower.username 54 | ) 55 | send_email(subject, message, 'server@example.com', [followee.email]) 56 | 57 | 58 | # An simple, previously expensive view. 59 | @login_required 60 | def follow(request, username): 61 | # You'd import the task function above. 62 | if request.method == 'POST': 63 | # Schedule the task. 64 | # Use args & kwargs as normal. 65 | gator.task(follow_email, request.user.username, username) 66 | return redirect('...') 67 | 68 | 69 | Running Tasks 70 | ------------- 71 | 72 | Rather than trying to do autodiscovery, fanout, etc., you control how your 73 | workers are configured & what they consume. 74 | 75 | If your needs are simple, run the included ``latergator.py`` worker: 76 | 77 | .. code:: bash 78 | 79 | $ python latergator.py redis://localhost:6379/0 80 | 81 | If you have more complex needs, you can create a new executable file 82 | (bin script, management command, whatever) & drop in the following code. 83 | 84 | .. code:: python 85 | 86 | from alligator import Gator, Worker 87 | 88 | # Bonus points if you import that one pre-configured ``Gator`` instead. 89 | gator = Gator('redis://localhost:6379/0') 90 | 91 | # Consume & handle all tasks. 92 | worker = Worker(gator) 93 | worker.run_forever() 94 | 95 | 96 | License 97 | ------- 98 | 99 | New BSD 100 | 101 | 102 | Running Tests 103 | ------------- 104 | 105 | Alligator has 95%+ test coverage & aims to be passing/stable at all times. 106 | 107 | If you'd like to run the tests, clone the repo, then run:: 108 | 109 | $ virtualenv -p python3 env 110 | $ . env/bin/activate 111 | $ pip install -r requirements-tests.txt 112 | $ python setup.py develop 113 | $ pytest -s -v --cov=alligator --cov-report=html tests 114 | 115 | The full test suite can be run via: 116 | 117 | $ export ALLIGATOR_TESTS_INCLUDE_SQS=true 118 | $ ./tests/run_all.sh 119 | 120 | This requires all backends/queues to be running, as well as valid AWS 121 | credentials if ``ALLIGATOR_TESTS_INCLUDE_SQS=true`` is set. 122 | 123 | 124 | WHY?!!1! 125 | -------- 126 | 127 | * Because I have NIH-syndrome. 128 | * Or because I longed for something simple (~375 loc). 129 | * Or because I wanted something with tests (90%+ coverage) & docs. 130 | * Or because I wanted pluggable backends. 131 | * Or because testing some other queuing system was a pain. 132 | * Or because I'm an idiot. 133 | 134 | 135 | Roadmap 136 | ------- 137 | 138 | Post-`1.0.0`: 139 | 140 | * Expand the supported backends 141 | * Kafka? 142 | * ActiveMQ support? 143 | * RabbitMQ support? 144 | * ??? 145 | -------------------------------------------------------------------------------- /alligator/backends/sqlite_backend.py: -------------------------------------------------------------------------------- 1 | import sqlite3 2 | import time 3 | 4 | 5 | class Client(object): 6 | def __init__(self, conn_string): 7 | """ 8 | A SQLite-based ``Client``. 9 | 10 | Args: 11 | conn_string (str): The DSN. The host/port/db are parsed out of it. 12 | Should be of the format ``sqlite:///path/to/db/file.db`` 13 | """ 14 | # This is actually the filepath to the DB file. 15 | self.conn_string = conn_string 16 | # Kill the 'sqlite://' portion. 17 | path = self.conn_string.split("://", 1)[1] 18 | self.conn = sqlite3.connect(path) 19 | 20 | def _run_query(self, query, args): 21 | cur = self.conn.cursor() 22 | 23 | if not args: 24 | cur.execute(query) 25 | else: 26 | cur.execute(query, args) 27 | 28 | self.conn.commit() 29 | return cur 30 | 31 | def setup_tables(self, queue_name="all"): 32 | """ 33 | Allows for manual creation of the needed tables. 34 | 35 | Args: 36 | queue_name (str): Optional. The name of the queue. Default is 37 | `all`. 38 | """ 39 | # For manually creating the tables... 40 | query = ( 41 | "CREATE TABLE `queue_{}` " 42 | "(task_id text, data text, delay_until integer)" 43 | ).format(queue_name) 44 | self._run_query(query, None) 45 | 46 | def len(self, queue_name): 47 | """ 48 | Returns the length of the queue. 49 | 50 | Args: 51 | queue_name (str): The name of the queue. Usually handled by the 52 | `Gator`` instance. 53 | 54 | Returns: 55 | int: The length of the queue 56 | """ 57 | query = "SELECT COUNT(task_id) FROM `queue_{}`".format(queue_name) 58 | cur = self._run_query(query, []) 59 | res = cur.fetchone() 60 | return res[0] 61 | 62 | def drop_all(self, queue_name): 63 | """ 64 | Drops all the task in the queue. 65 | 66 | Args: 67 | queue_name (str): The name of the queue. Usually handled by the 68 | ``Gator`` instance. 69 | """ 70 | query = "DELETE FROM `queue_{}`".format(queue_name) 71 | self._run_query(query, []) 72 | 73 | def push(self, queue_name, task_id, data, delay_until=None): 74 | """ 75 | Pushes a task onto the queue. 76 | 77 | Args: 78 | queue_name (str): The name of the queue. Usually handled by the 79 | ``Gator`` instance. 80 | task_id (str): The identifier of the task. 81 | data (str): The relevant data for the task. 82 | delay_until (float): Optional. The Unix timestamp to delay 83 | processing of the task until. Default is `None`. 84 | 85 | Returns: 86 | str: The task ID. 87 | """ 88 | if delay_until is None: 89 | delay_until = time.time() 90 | 91 | query = ( 92 | "INSERT INTO `queue_{}` " 93 | "(task_id, data, delay_until) " 94 | "VALUES (?, ?, ?)" 95 | ).format(queue_name) 96 | self._run_query(query, [task_id, data, int(delay_until)]) 97 | return task_id 98 | 99 | def pop(self, queue_name): 100 | """ 101 | Pops a task off the queue. 102 | 103 | Args: 104 | queue_name (str): The name of the queue. Usually handled by the 105 | ``Gator`` instance. 106 | 107 | Returns: 108 | str: The data for the task. 109 | """ 110 | now = int(time.time()) 111 | query = ( 112 | "SELECT task_id, data, delay_until " 113 | "FROM `queue_{}` " 114 | "WHERE delay_until <= ? " 115 | "LIMIT 1" 116 | ).format(queue_name) 117 | cur = self._run_query(query, [now]) 118 | res = cur.fetchone() 119 | 120 | if res: 121 | query = "DELETE FROM `queue_{}` WHERE task_id = ?".format( 122 | queue_name 123 | ) 124 | self._run_query(query, [res[0]]) 125 | 126 | return res[1] 127 | 128 | def get(self, queue_name, task_id): 129 | """ 130 | Pops a specific task off the queue by identifier. 131 | 132 | Args: 133 | queue_name (str): The name of the queue. Usually handled by the 134 | ``Gator`` instance. 135 | task_id (str): The identifier of the task. 136 | 137 | Returns: 138 | str: The data for the task. 139 | """ 140 | # fmt: off 141 | query = ( 142 | "SELECT task_id, data " 143 | "FROM `queue_{}` " 144 | "WHERE task_id = ?" 145 | ).format(queue_name) 146 | # fmt: on 147 | cur = self._run_query(query, [task_id]) 148 | res = cur.fetchone() 149 | 150 | query = "DELETE FROM `queue_{}` WHERE task_id = ?".format(queue_name) 151 | self._run_query(query, [task_id]) 152 | 153 | return res[1] 154 | -------------------------------------------------------------------------------- /alligator/workers.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import os 3 | import signal 4 | import time 5 | import traceback 6 | 7 | from alligator.constants import ALL 8 | 9 | 10 | class Worker(object): 11 | def __init__( 12 | self, 13 | gator, 14 | max_tasks=0, 15 | to_consume=ALL, 16 | nap_time=0.1, 17 | log_level=logging.INFO, 18 | ): 19 | """ 20 | An object for consuming the queue & running the tasks. 21 | 22 | Ex:: 23 | 24 | from alligator import Gator, Worker 25 | 26 | gator = Gator('locmem://') 27 | worker = Worker(gator) 28 | worker.run_forever() 29 | 30 | Args: 31 | gator (Gator): A configured `Gator` object 32 | max_tasks (int): Optional. The maximum number of tasks to consume. 33 | Useful if you're concerned about memory leaks or want 34 | short-lived workers. Defaults to `0` (unlimited tasks). 35 | to_consume (str): Optional. The queue name the worker should 36 | consume from. Defaults to `ALL`. 37 | nap_time (float): Optional. To prevent high CPU usage in the busy 38 | loop, you can specify a time delay (in seconds) between 39 | tasks. Set to `0` to disable sleep & consume as fast as 40 | possible. Defaults to `0.1` 41 | log_level (int): Optional. The logging level you'd like for 42 | output. Default is `logging.INFO`. 43 | """ 44 | self.gator = gator 45 | self.max_tasks = int(max_tasks) 46 | self.to_consume = to_consume 47 | self.nap_time = nap_time 48 | self.tasks_complete = 0 49 | self.keep_running = False 50 | self.log_level = log_level 51 | self.log = self.get_log(self.log_level) 52 | 53 | def get_log(self, log_level=logging.INFO): 54 | """ 55 | Sets up logging for the instance. 56 | 57 | Args: 58 | log_level (int): Optional. The logging level you'd like for 59 | output. Default is `logging.INFO`. 60 | 61 | Returns: 62 | logging.Logger: The log instance. 63 | """ 64 | log = logging.getLogger(__name__) 65 | default_format = logging.Formatter( 66 | "%(asctime)s %(name)s %(levelname)s %(message)s" 67 | ) 68 | stdout_handler = logging.StreamHandler() 69 | stdout_handler.setFormatter(default_format) 70 | log.addHandler(stdout_handler) 71 | log.setLevel(logging.INFO) 72 | return log 73 | 74 | def ident(self): 75 | """ 76 | Returns a string identifier for the worker. 77 | 78 | Used in the printed messages & includes the process ID. 79 | """ 80 | return "Alligator Worker (#{})".format(os.getpid()) 81 | 82 | def starting(self): 83 | """ 84 | Prints a startup message to stdout. 85 | """ 86 | self.keep_running = True 87 | ident = self.ident() 88 | self.log.info( 89 | '{} starting & consuming "{}".'.format(ident, self.to_consume) 90 | ) 91 | 92 | if self.max_tasks: 93 | self.log.info( 94 | "{} will die after {} tasks.".format(ident, self.max_tasks) 95 | ) 96 | else: 97 | self.log.info("{} will never die.".format(ident)) 98 | 99 | def interrupt(self): 100 | """ 101 | Prints an interrupt message to stdout. 102 | """ 103 | self.keep_running = False 104 | ident = self.ident() 105 | self.log.info( 106 | '{} for "{}" saw interrupt. Finishing in-progress task.'.format( 107 | ident, self.to_consume 108 | ) 109 | ) 110 | 111 | def stopping(self): 112 | """ 113 | Prints a shutdown message to stdout. 114 | """ 115 | self.keep_running = False 116 | ident = self.ident() 117 | self.log.info( 118 | '{} for "{}" shutting down. Consumed {} tasks.'.format( 119 | ident, self.to_consume, self.tasks_complete 120 | ) 121 | ) 122 | 123 | def result(self, result): 124 | """ 125 | Prints the received result from a task to stdout. 126 | 127 | :param result: The result of the task 128 | """ 129 | if result is not None: 130 | self.log.info(result) 131 | 132 | def check_and_run_task(self): 133 | """ 134 | Handles the logic of checking for & executing a task. 135 | 136 | `Worker.run_forever` uses this in a loop to actually handle the main 137 | logic, though you can call this on your own if you have different 138 | needs. 139 | 140 | Returns: 141 | bool: `True` if a task was run successfully, `False` if there was 142 | no task to process or executing the task failed. 143 | """ 144 | if self.gator.len(): 145 | try: 146 | task = self.gator.pop() 147 | except Exception as err: 148 | self.log.exception(err) 149 | return False 150 | 151 | if task is None: 152 | return False 153 | 154 | self.tasks_complete += 1 155 | self.result(task.result) 156 | return True 157 | 158 | return False 159 | 160 | def run_forever(self): 161 | """ 162 | Causes the worker to run either forever or until the 163 | `Worker.max_tasks` are reached. 164 | """ 165 | self.starting() 166 | 167 | def handle(signum, frame): 168 | self.interrupt() 169 | 170 | signal.signal(signal.SIGINT, handle) 171 | 172 | while self.keep_running: 173 | if self.max_tasks and self.tasks_complete >= self.max_tasks: 174 | self.stopping() 175 | break 176 | 177 | self.check_and_run_task() 178 | 179 | if self.nap_time >= 0: 180 | time.sleep(self.nap_time) 181 | 182 | return 0 183 | -------------------------------------------------------------------------------- /tests/test_gator.py: -------------------------------------------------------------------------------- 1 | import os 2 | import unittest 3 | 4 | from alligator.backends.locmem_backend import Client as LocmemClient 5 | from alligator.backends.redis_backend import Client as RedisClient 6 | from alligator.constants import ( 7 | ALL, 8 | WAITING, 9 | SUCCESS, 10 | FAILED, 11 | RETRYING, 12 | CANCELED, 13 | ) 14 | from alligator.gator import Gator 15 | from alligator.tasks import Task 16 | 17 | 18 | class CustomTask(Task): 19 | pass 20 | 21 | 22 | class CustomClient(LocmemClient): 23 | pass 24 | 25 | 26 | def so_computationally_expensive(initial, incr): 27 | return initial + incr 28 | 29 | 30 | def fail_task(initial, incr): 31 | raise IOError("Math is hard.") 32 | 33 | 34 | def eventual_success(): 35 | data = {"count": 0} 36 | 37 | def wrapped(initial, incr): 38 | data["count"] += 1 39 | 40 | if data["count"] < 3: 41 | raise IOError("Nope.") 42 | 43 | return initial + incr 44 | 45 | return wrapped 46 | 47 | 48 | class GatorTestCase(unittest.TestCase): 49 | def setUp(self): 50 | super(GatorTestCase, self).setUp() 51 | self.conn_string = os.environ.get("ALLIGATOR_CONN") 52 | self.gator = Gator(self.conn_string) 53 | 54 | # Just reach in & clear things out. 55 | self.gator.backend.drop_all(ALL) 56 | 57 | def test_init(self): 58 | gator = Gator("whatever://", backend_class=CustomClient) 59 | self.assertEqual(gator.conn_string, "whatever://") 60 | self.assertEqual(gator.queue_name, ALL) 61 | self.assertEqual(gator.task_class, Task) 62 | self.assertEqual(gator.backend_class, CustomClient) 63 | self.assertTrue(isinstance(gator.backend, CustomClient)) 64 | 65 | gator = Gator( 66 | "locmem://", queue_name="hard_things", task_class=CustomTask 67 | ) 68 | self.assertEqual(gator.conn_string, "locmem://") 69 | self.assertEqual(gator.queue_name, "hard_things") 70 | self.assertEqual(gator.task_class, CustomTask) 71 | self.assertEqual(gator.backend_class, None) 72 | self.assertTrue(isinstance(gator.backend, LocmemClient)) 73 | 74 | def test_build_backend(self): 75 | backend = self.gator.build_backend("locmem://") 76 | self.assertTrue(isinstance(backend, LocmemClient)) 77 | 78 | backend = self.gator.build_backend("redis://localhost:6379/0") 79 | self.assertTrue(isinstance(backend, RedisClient)) 80 | 81 | def test_push_async(self): 82 | self.assertEqual(self.gator.backend.len(ALL), 0) 83 | 84 | task = Task(is_async=True) 85 | self.gator.push(task, so_computationally_expensive, 1, 1) 86 | self.assertEqual(self.gator.backend.len(ALL), 1) 87 | 88 | def test_push_sync(self): 89 | self.assertEqual(self.gator.backend.len(ALL), 0) 90 | 91 | def success(t, result): 92 | t.result = result 93 | 94 | task = Task(is_async=False, on_success=success) 95 | res = self.gator.push(task, so_computationally_expensive, 1, 1) 96 | self.assertEqual(self.gator.backend.len(ALL), 0) 97 | self.assertEqual(res.result, 2) 98 | 99 | def test_pop(self): 100 | self.assertEqual(self.gator.backend.len(ALL), 0) 101 | 102 | task = Task(is_async=True) 103 | self.gator.push(task, so_computationally_expensive, 1, 1) 104 | self.assertEqual(self.gator.backend.len(ALL), 1) 105 | 106 | complete = self.gator.pop() 107 | self.assertEqual(complete.result, 2) 108 | 109 | def test_get(self): 110 | self.assertEqual(self.gator.backend.len(ALL), 0) 111 | 112 | task_1 = Task(is_async=True) 113 | task_2 = Task(task_id="hello", is_async=True) 114 | self.gator.push(task_1, so_computationally_expensive, 1, 1) 115 | self.gator.push(task_2, so_computationally_expensive, 3, 5) 116 | self.assertEqual(self.gator.backend.len(ALL), 2) 117 | 118 | complete = self.gator.get(task_2.task_id) 119 | self.assertEqual(complete.result, 8) 120 | 121 | complete = self.gator.get(task_1.task_id) 122 | self.assertEqual(complete.result, 2) 123 | 124 | def test_cancel(self): 125 | self.assertEqual(self.gator.backend.len(ALL), 0) 126 | 127 | task_1 = Task(is_async=True) 128 | task_2 = Task(task_id="hello", is_async=True) 129 | self.gator.push(task_1, so_computationally_expensive, 1, 1) 130 | self.gator.push(task_2, so_computationally_expensive, 3, 5) 131 | self.assertEqual(self.gator.backend.len(ALL), 2) 132 | 133 | task = self.gator.cancel(task_2.task_id) 134 | self.assertEqual(task.status, CANCELED) 135 | self.assertEqual(self.gator.backend.len(ALL), 1) 136 | 137 | def test_execute_success(self): 138 | task = Task(retries=3, is_async=True) 139 | task.to_call(so_computationally_expensive, 2, 7) 140 | 141 | complete = self.gator.execute(task) 142 | self.assertEqual(complete.result, 9) 143 | self.assertEqual(task.retries, 3) 144 | self.assertEqual(task.status, SUCCESS) 145 | 146 | def test_execute_failed(self): 147 | task = Task(retries=3, is_async=True) 148 | task.to_call(fail_task, 2, 7) 149 | self.assertEqual(task.status, WAITING) 150 | 151 | try: 152 | self.gator.execute(task) 153 | self.assertEqual(task.status, RETRYING) 154 | self.gator.execute(task) 155 | self.gator.execute(task) 156 | self.gator.execute(task) 157 | self.fail() 158 | except IOError: 159 | self.assertEqual(task.retries, 0) 160 | self.assertEqual(task.status, FAILED) 161 | 162 | def test_execute_retries(self): 163 | task = Task(retries=3, is_async=True) 164 | task.to_call(eventual_success(), 2, 7) 165 | 166 | try: 167 | self.gator.execute(task) 168 | except IOError: 169 | pass 170 | 171 | try: 172 | self.gator.execute(task) 173 | except IOError: 174 | pass 175 | 176 | complete = self.gator.execute(task) 177 | self.assertEqual(complete.result, 9) 178 | self.assertEqual(task.retries, 1) 179 | 180 | def test_task(self): 181 | self.assertEqual(self.gator.backend.len(ALL), 0) 182 | 183 | self.gator.task(so_computationally_expensive, 1, 1) 184 | self.assertEqual(self.gator.backend.len(ALL), 1) 185 | 186 | def test_options(self): 187 | self.assertEqual(self.gator.backend.len(ALL), 0) 188 | 189 | def success(t, result): 190 | t.result = result 191 | 192 | with self.gator.options( 193 | retries=4, is_async=False, on_success=success, delay_by=60 * 3 194 | ) as opts: 195 | res = opts.task(eventual_success(), 3, 9) 196 | 197 | self.assertEqual(res.retries, 2) 198 | self.assertEqual(res.result, 12) 199 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | PAPER = 8 | BUILDDIR = _build 9 | 10 | # User-friendly check for sphinx-build 11 | ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) 12 | $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) 13 | endif 14 | 15 | # Internal variables. 16 | PAPEROPT_a4 = -D latex_paper_size=a4 17 | PAPEROPT_letter = -D latex_paper_size=letter 18 | ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 19 | # the i18n builder cannot share the environment and doctrees with the others 20 | I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 21 | 22 | .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext 23 | 24 | help: 25 | @echo "Please use \`make ' where is one of" 26 | @echo " html to make standalone HTML files" 27 | @echo " dirhtml to make HTML files named index.html in directories" 28 | @echo " singlehtml to make a single large HTML file" 29 | @echo " pickle to make pickle files" 30 | @echo " json to make JSON files" 31 | @echo " htmlhelp to make HTML files and a HTML help project" 32 | @echo " qthelp to make HTML files and a qthelp project" 33 | @echo " devhelp to make HTML files and a Devhelp project" 34 | @echo " epub to make an epub" 35 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 36 | @echo " latexpdf to make LaTeX files and run them through pdflatex" 37 | @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" 38 | @echo " text to make text files" 39 | @echo " man to make manual pages" 40 | @echo " texinfo to make Texinfo files" 41 | @echo " info to make Texinfo files and run them through makeinfo" 42 | @echo " gettext to make PO message catalogs" 43 | @echo " changes to make an overview of all changed/added/deprecated items" 44 | @echo " xml to make Docutils-native XML files" 45 | @echo " pseudoxml to make pseudoxml-XML files for display purposes" 46 | @echo " linkcheck to check all external links for integrity" 47 | @echo " doctest to run all doctests embedded in the documentation (if enabled)" 48 | 49 | clean: 50 | rm -rf $(BUILDDIR)/* 51 | 52 | html: 53 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html 54 | @echo 55 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." 56 | 57 | dirhtml: 58 | $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml 59 | @echo 60 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." 61 | 62 | singlehtml: 63 | $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml 64 | @echo 65 | @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." 66 | 67 | pickle: 68 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle 69 | @echo 70 | @echo "Build finished; now you can process the pickle files." 71 | 72 | json: 73 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json 74 | @echo 75 | @echo "Build finished; now you can process the JSON files." 76 | 77 | htmlhelp: 78 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp 79 | @echo 80 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 81 | ".hhp project file in $(BUILDDIR)/htmlhelp." 82 | 83 | qthelp: 84 | $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp 85 | @echo 86 | @echo "Build finished; now you can run "qcollectiongenerator" with the" \ 87 | ".qhcp project file in $(BUILDDIR)/qthelp, like this:" 88 | @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Alligator.qhcp" 89 | @echo "To view the help file:" 90 | @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Alligator.qhc" 91 | 92 | devhelp: 93 | $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp 94 | @echo 95 | @echo "Build finished." 96 | @echo "To view the help file:" 97 | @echo "# mkdir -p $$HOME/.local/share/devhelp/Alligator" 98 | @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Alligator" 99 | @echo "# devhelp" 100 | 101 | epub: 102 | $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub 103 | @echo 104 | @echo "Build finished. The epub file is in $(BUILDDIR)/epub." 105 | 106 | latex: 107 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 108 | @echo 109 | @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." 110 | @echo "Run \`make' in that directory to run these through (pdf)latex" \ 111 | "(use \`make latexpdf' here to do that automatically)." 112 | 113 | latexpdf: 114 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 115 | @echo "Running LaTeX files through pdflatex..." 116 | $(MAKE) -C $(BUILDDIR)/latex all-pdf 117 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 118 | 119 | latexpdfja: 120 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 121 | @echo "Running LaTeX files through platex and dvipdfmx..." 122 | $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja 123 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 124 | 125 | text: 126 | $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text 127 | @echo 128 | @echo "Build finished. The text files are in $(BUILDDIR)/text." 129 | 130 | man: 131 | $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man 132 | @echo 133 | @echo "Build finished. The manual pages are in $(BUILDDIR)/man." 134 | 135 | texinfo: 136 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 137 | @echo 138 | @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." 139 | @echo "Run \`make' in that directory to run these through makeinfo" \ 140 | "(use \`make info' here to do that automatically)." 141 | 142 | info: 143 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 144 | @echo "Running Texinfo files through makeinfo..." 145 | make -C $(BUILDDIR)/texinfo info 146 | @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." 147 | 148 | gettext: 149 | $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale 150 | @echo 151 | @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." 152 | 153 | changes: 154 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes 155 | @echo 156 | @echo "The overview file is in $(BUILDDIR)/changes." 157 | 158 | linkcheck: 159 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck 160 | @echo 161 | @echo "Link check complete; look for any errors in the above output " \ 162 | "or in $(BUILDDIR)/linkcheck/output.txt." 163 | 164 | doctest: 165 | $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest 166 | @echo "Testing of doctests in the sources finished, look at the " \ 167 | "results in $(BUILDDIR)/doctest/output.txt." 168 | 169 | xml: 170 | $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml 171 | @echo 172 | @echo "Build finished. The XML files are in $(BUILDDIR)/xml." 173 | 174 | pseudoxml: 175 | $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml 176 | @echo 177 | @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." 178 | -------------------------------------------------------------------------------- /docs/bestpractices.rst: -------------------------------------------------------------------------------- 1 | .. _bestpractices: 2 | 3 | ============== 4 | Best Practices 5 | ============== 6 | 7 | Moving to offlining tasks requires some shifts in the way you develop your 8 | code. There are also some good tricks/ideas for integrating Alligator. 9 | 10 | If you have suggestions for other best practices, please submit a pull request 11 | at https://github.com/toastdriven/alligator/pulls! 12 | 13 | 14 | Configure One ``Gator`` 15 | ======================= 16 | 17 | This is alluded to in the :ref:`tutorial`, but unless you have advanced needs, 18 | you're probably best off configuring a **single** ``Gator`` instance in your 19 | code. Then you can import that instance wherever you need it. 20 | 21 | Generally speaking, you'll want to create a new file for just this, though if 22 | you have a ``utils.py`` or other common file, you can add it there. For 23 | example: 24 | 25 | .. code:: python 26 | 27 | # Create a new file, like ``myapp/gator.py`` 28 | from alligator import Gator 29 | 30 | gator = Gator('redis://localhost:6379/0') 31 | 32 | Then your code elsewhere imports it: 33 | 34 | .. code:: python 35 | 36 | # ``myapp/views.py`` 37 | from myapp.gator import gator 38 | 39 | # ...Later... 40 | def previously_slow_view(request): 41 | gator.task(expensive_cache_rebuild, user_id=request.user.pk) 42 | 43 | This helps :abbr:`DRY (Don't Repeat Yourself)` up your code. It also helps you 44 | avoid having to change many files if you change backends or configuration. 45 | 46 | 47 | Use Environment Variables or Settings for the ``Gator`` DSN 48 | =========================================================== 49 | 50 | Instead of hard-coding the :abbr:`DSN (Data Source Name)` for each ``Gator`` 51 | instance, you should rely on a configuration setting instead. 52 | 53 | If you're using plain old Python or subscribe to the `Twelve-Factor App`_, 54 | you might lean on environment variables set in the shell. For instance, the 55 | Alligator test suite does: 56 | 57 | .. code:: python 58 | 59 | import os 60 | 61 | from alligator import Gator 62 | 63 | 64 | # Lean on the ENV variable. 65 | gator = Gator(os.environ['ALLIGATOR_CONN']) 66 | 67 | Then when running your app, you could do the following in development, for 68 | ease of setup: 69 | 70 | .. code:: bash 71 | 72 | $ export ALLIGATOR_CONN=locmem:// 73 | $ python myapp.py 74 | 75 | But the following on production, for handling large loads: 76 | 77 | .. code:: bash 78 | 79 | $ export ALLIGATOR_CONN=redis://some.dns.name.com:6379/0 80 | $ python myapp.py 81 | 82 | If you're using something like `Django`_, you could lean on ``settings`` 83 | instead, like: 84 | 85 | .. code:: python 86 | 87 | from alligator import Gator 88 | 89 | from django.conf import settings 90 | 91 | 92 | # Lean on the settings variable. 93 | gator = Gator(settings.ALLIGATOR_CONN) 94 | 95 | And have differing settings files for development vs. production. 96 | 97 | .. _`Twelve-Factor App`: http://12factor.net/ 98 | .. _`Django`: http://djangoproject.com/ 99 | 100 | 101 | Use an Alternate Queue for Testing 102 | ================================== 103 | 104 | This is an **important** one. By default, Alligator doesn't make any assumptions 105 | about what environment (development, testing, production) it is in. So the same 106 | queue name will be used. 107 | 108 | Especially if you have a shared queue setup for running tests, you can 109 | **accidentally** add testing data to your queue! There are two possible 110 | resolutions to this: 111 | 112 | 1. Don't Share 113 | 114 | Set your testing environment up such that it has it's own queue stack. This 115 | will nicely isolate things & not require any code changes. 116 | 117 | 2. Prefix your ``queue_name`` 118 | 119 | If you must share setup (for instance, developing & testing on the same 120 | machine), use a similar approach to the "Env/Settings for Gator DSN" tip, 121 | providing a prefix for your queue name. For example: 122 | 123 | .. code:: python 124 | 125 | import os 126 | 127 | from alligator import Gator 128 | 129 | 130 | # Lean on the ENV variable for a queue prefix. 131 | gator = Gator( 132 | 'redis://localhost:6379/0', 133 | # If you ``export ALLIGATOR_PREFIX=test```, your queue name 134 | # becomes 'test_all'. If not set, it's just 'all'. 135 | queue_name='_'.join([os.environ.get('ALLIGATOR_PREFIX', ''), 'all']) 136 | ) 137 | 138 | 139 | Use Environment Variables or Settings for ``Task.is_async`` 140 | =========================================================== 141 | 142 | If you're just using ``gator.task`` & trying to write tests, you may have a 143 | hard time verifying behavior in an integration test (though you should be able 144 | to just unit test the task function). 145 | 146 | On the other hand, if you use the ``gator.options`` context manager & supply 147 | an ``is_async=False`` execution option, integration tests become easy, as the 148 | expense of possibly accidentally committing that & causing issues in production. 149 | 150 | The best approach is to use the ``gator.options`` context manager, but use 151 | an environment variable/setting to control if things run asynchronously. 152 | 153 | .. code:: python 154 | 155 | import os 156 | 157 | # Using the above tip of a single import... 158 | from myapp.gator import gator 159 | 160 | 161 | def some_view(request): 162 | with gator.options(is_async=os.environ['ALLIGATOR_ASYNC']) as opts: 163 | opts.task(expensive_thing) 164 | 165 | This allows you to set ``export ALLIGATOR_ASYNC=False`` in development/testing 166 | (so the task runs right away in-process) but queues appropriately in 167 | production. 168 | 169 | 170 | Simple Task Parameters 171 | ====================== 172 | 173 | When creating task functions, you want to simplify the arguments passed to it, 174 | as well as removing as many assumptions as possible. 175 | 176 | You may be tempted to try to save queries by passing full objects or large lists 177 | of things as a parameter. 178 | 179 | However, you must remember that the task may run at a very different time 180 | (perhaps hours in the future if you're overloaded) or on a completely different 181 | machine than the one scheduling the task. Data goes stale easily & few things 182 | are as frustrating to debug as stale data being re-written over the top of new 183 | data. 184 | 185 | Where possible, do the following things: 186 | 187 | * Pass primary keys or identifiers instead of rich objects 188 | * Persist large collections in the database or elsewhere, then pass a lookup 189 | identifier to the task 190 | * Use simple data types, as they serialize well & result in smaller queue 191 | payloads, meaning faster scheduling & consuming of tasks 192 | 193 | 194 | Re-use the ``Gator.options`` Context Manager 195 | ============================================ 196 | 197 | All the examples in the Alligator docs show creating a single task within 198 | a ``gator.options(...)`` context manager. So you might be tempted to write code 199 | like: 200 | 201 | .. code:: python 202 | 203 | with gator.options(retries=3) as opts: 204 | opts.task(send_mass_mail, list_id) 205 | 206 | with gator.options(retries=3) as opts: 207 | opts.task(update_follow_counts, request.user.pk) 208 | 209 | However, you can reuse that context manager to provide the same execution 210 | options to **all** tasks within the block. So we can clean up & shorten our 211 | code to: 212 | 213 | .. code:: python 214 | 215 | with gator.options(retries=3) as opts: 216 | opts.task(send_mass_mail, list_id) 217 | opts.task(update_follow_counts, request.user.pk) 218 | 219 | Two unique tasks will still be created, but both will have the ``retries=3`` 220 | provided to better ensure they succeeed. 221 | -------------------------------------------------------------------------------- /docs/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | REM Command file for Sphinx documentation 4 | 5 | if "%SPHINXBUILD%" == "" ( 6 | set SPHINXBUILD=sphinx-build 7 | ) 8 | set BUILDDIR=_build 9 | set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . 10 | set I18NSPHINXOPTS=%SPHINXOPTS% . 11 | if NOT "%PAPER%" == "" ( 12 | set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% 13 | set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS% 14 | ) 15 | 16 | if "%1" == "" goto help 17 | 18 | if "%1" == "help" ( 19 | :help 20 | echo.Please use `make ^` where ^ is one of 21 | echo. html to make standalone HTML files 22 | echo. dirhtml to make HTML files named index.html in directories 23 | echo. singlehtml to make a single large HTML file 24 | echo. pickle to make pickle files 25 | echo. json to make JSON files 26 | echo. htmlhelp to make HTML files and a HTML help project 27 | echo. qthelp to make HTML files and a qthelp project 28 | echo. devhelp to make HTML files and a Devhelp project 29 | echo. epub to make an epub 30 | echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter 31 | echo. text to make text files 32 | echo. man to make manual pages 33 | echo. texinfo to make Texinfo files 34 | echo. gettext to make PO message catalogs 35 | echo. changes to make an overview over all changed/added/deprecated items 36 | echo. xml to make Docutils-native XML files 37 | echo. pseudoxml to make pseudoxml-XML files for display purposes 38 | echo. linkcheck to check all external links for integrity 39 | echo. doctest to run all doctests embedded in the documentation if enabled 40 | goto end 41 | ) 42 | 43 | if "%1" == "clean" ( 44 | for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i 45 | del /q /s %BUILDDIR%\* 46 | goto end 47 | ) 48 | 49 | 50 | %SPHINXBUILD% 2> nul 51 | if errorlevel 9009 ( 52 | echo. 53 | echo.The 'sphinx-build' command was not found. Make sure you have Sphinx 54 | echo.installed, then set the SPHINXBUILD environment variable to point 55 | echo.to the full path of the 'sphinx-build' executable. Alternatively you 56 | echo.may add the Sphinx directory to PATH. 57 | echo. 58 | echo.If you don't have Sphinx installed, grab it from 59 | echo.http://sphinx-doc.org/ 60 | exit /b 1 61 | ) 62 | 63 | if "%1" == "html" ( 64 | %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html 65 | if errorlevel 1 exit /b 1 66 | echo. 67 | echo.Build finished. The HTML pages are in %BUILDDIR%/html. 68 | goto end 69 | ) 70 | 71 | if "%1" == "dirhtml" ( 72 | %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml 73 | if errorlevel 1 exit /b 1 74 | echo. 75 | echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. 76 | goto end 77 | ) 78 | 79 | if "%1" == "singlehtml" ( 80 | %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml 81 | if errorlevel 1 exit /b 1 82 | echo. 83 | echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. 84 | goto end 85 | ) 86 | 87 | if "%1" == "pickle" ( 88 | %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle 89 | if errorlevel 1 exit /b 1 90 | echo. 91 | echo.Build finished; now you can process the pickle files. 92 | goto end 93 | ) 94 | 95 | if "%1" == "json" ( 96 | %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json 97 | if errorlevel 1 exit /b 1 98 | echo. 99 | echo.Build finished; now you can process the JSON files. 100 | goto end 101 | ) 102 | 103 | if "%1" == "htmlhelp" ( 104 | %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp 105 | if errorlevel 1 exit /b 1 106 | echo. 107 | echo.Build finished; now you can run HTML Help Workshop with the ^ 108 | .hhp project file in %BUILDDIR%/htmlhelp. 109 | goto end 110 | ) 111 | 112 | if "%1" == "qthelp" ( 113 | %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp 114 | if errorlevel 1 exit /b 1 115 | echo. 116 | echo.Build finished; now you can run "qcollectiongenerator" with the ^ 117 | .qhcp project file in %BUILDDIR%/qthelp, like this: 118 | echo.^> qcollectiongenerator %BUILDDIR%\qthelp\Alligator.qhcp 119 | echo.To view the help file: 120 | echo.^> assistant -collectionFile %BUILDDIR%\qthelp\Alligator.ghc 121 | goto end 122 | ) 123 | 124 | if "%1" == "devhelp" ( 125 | %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp 126 | if errorlevel 1 exit /b 1 127 | echo. 128 | echo.Build finished. 129 | goto end 130 | ) 131 | 132 | if "%1" == "epub" ( 133 | %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub 134 | if errorlevel 1 exit /b 1 135 | echo. 136 | echo.Build finished. The epub file is in %BUILDDIR%/epub. 137 | goto end 138 | ) 139 | 140 | if "%1" == "latex" ( 141 | %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex 142 | if errorlevel 1 exit /b 1 143 | echo. 144 | echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. 145 | goto end 146 | ) 147 | 148 | if "%1" == "latexpdf" ( 149 | %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex 150 | cd %BUILDDIR%/latex 151 | make all-pdf 152 | cd %BUILDDIR%/.. 153 | echo. 154 | echo.Build finished; the PDF files are in %BUILDDIR%/latex. 155 | goto end 156 | ) 157 | 158 | if "%1" == "latexpdfja" ( 159 | %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex 160 | cd %BUILDDIR%/latex 161 | make all-pdf-ja 162 | cd %BUILDDIR%/.. 163 | echo. 164 | echo.Build finished; the PDF files are in %BUILDDIR%/latex. 165 | goto end 166 | ) 167 | 168 | if "%1" == "text" ( 169 | %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text 170 | if errorlevel 1 exit /b 1 171 | echo. 172 | echo.Build finished. The text files are in %BUILDDIR%/text. 173 | goto end 174 | ) 175 | 176 | if "%1" == "man" ( 177 | %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man 178 | if errorlevel 1 exit /b 1 179 | echo. 180 | echo.Build finished. The manual pages are in %BUILDDIR%/man. 181 | goto end 182 | ) 183 | 184 | if "%1" == "texinfo" ( 185 | %SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo 186 | if errorlevel 1 exit /b 1 187 | echo. 188 | echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo. 189 | goto end 190 | ) 191 | 192 | if "%1" == "gettext" ( 193 | %SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale 194 | if errorlevel 1 exit /b 1 195 | echo. 196 | echo.Build finished. The message catalogs are in %BUILDDIR%/locale. 197 | goto end 198 | ) 199 | 200 | if "%1" == "changes" ( 201 | %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes 202 | if errorlevel 1 exit /b 1 203 | echo. 204 | echo.The overview file is in %BUILDDIR%/changes. 205 | goto end 206 | ) 207 | 208 | if "%1" == "linkcheck" ( 209 | %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck 210 | if errorlevel 1 exit /b 1 211 | echo. 212 | echo.Link check complete; look for any errors in the above output ^ 213 | or in %BUILDDIR%/linkcheck/output.txt. 214 | goto end 215 | ) 216 | 217 | if "%1" == "doctest" ( 218 | %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest 219 | if errorlevel 1 exit /b 1 220 | echo. 221 | echo.Testing of doctests in the sources finished, look at the ^ 222 | results in %BUILDDIR%/doctest/output.txt. 223 | goto end 224 | ) 225 | 226 | if "%1" == "xml" ( 227 | %SPHINXBUILD% -b xml %ALLSPHINXOPTS% %BUILDDIR%/xml 228 | if errorlevel 1 exit /b 1 229 | echo. 230 | echo.Build finished. The XML files are in %BUILDDIR%/xml. 231 | goto end 232 | ) 233 | 234 | if "%1" == "pseudoxml" ( 235 | %SPHINXBUILD% -b pseudoxml %ALLSPHINXOPTS% %BUILDDIR%/pseudoxml 236 | if errorlevel 1 exit /b 1 237 | echo. 238 | echo.Build finished. The pseudo-XML files are in %BUILDDIR%/pseudoxml. 239 | goto end 240 | ) 241 | 242 | :end 243 | -------------------------------------------------------------------------------- /tests/test_tasks.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import json 3 | import unittest 4 | from unittest import mock 5 | 6 | from alligator.constants import WAITING, SUCCESS, FAILED, RETRYING, CANCELED 7 | from alligator.tasks import Task 8 | 9 | 10 | def run_me(x, y=None): 11 | pass 12 | 13 | 14 | def start(task): 15 | pass 16 | 17 | 18 | def error(task, err): 19 | pass 20 | 21 | 22 | def success(task, result): 23 | pass 24 | 25 | 26 | class TaskTestCase(unittest.TestCase): 27 | def setUp(self): 28 | super(TaskTestCase, self).setUp() 29 | self.task = Task() 30 | 31 | def test_default_init(self): 32 | task = Task() 33 | self.assertNotEqual(task.task_id, None) 34 | self.assertEqual(task.retries, 0) 35 | self.assertEqual(task.is_async, True) 36 | self.assertEqual(task.on_start, None) 37 | self.assertEqual(task.on_success, None) 38 | self.assertEqual(task.on_error, None) 39 | self.assertEqual(task.depends_on, None) 40 | self.assertEqual(task.delay_until, None) 41 | self.assertEqual(task.status, WAITING) 42 | self.assertEqual(task.func, None) 43 | self.assertEqual(task.func_args, []) 44 | self.assertEqual(task.func_kwargs, {}) 45 | 46 | @mock.patch("time.time") 47 | def test_custom_init(self, mock_time): 48 | mock_time.return_value = 12345678 49 | 50 | task = Task( 51 | task_id="hello", 52 | retries=3, 53 | is_async=False, 54 | on_start=start, 55 | on_success=success, 56 | on_error=error, 57 | depends_on=["id1", "id3", "id4"], 58 | delay_by=60 * 2, 59 | ) 60 | self.assertNotEqual(task.task_id, None) 61 | self.assertEqual(task.retries, 3) 62 | self.assertEqual(task.is_async, False) 63 | self.assertEqual(task.on_start, start) 64 | self.assertEqual(task.on_success, success) 65 | self.assertEqual(task.on_error, error) 66 | self.assertEqual(task.depends_on, ["id1", "id3", "id4"]) 67 | self.assertEqual(task.delay_until, 12345798) 68 | self.assertEqual(task.status, WAITING) 69 | self.assertEqual(task.func, None) 70 | self.assertEqual(task.func_args, []) 71 | self.assertEqual(task.func_kwargs, {}) 72 | 73 | def test_custom_init_delay_until(self): 74 | task = Task( 75 | task_id="hello", 76 | retries=3, 77 | is_async=False, 78 | on_start=start, 79 | on_success=success, 80 | on_error=error, 81 | depends_on=["id1", "id3", "id4"], 82 | delay_until=12345798, 83 | ) 84 | self.assertNotEqual(task.task_id, None) 85 | self.assertEqual(task.retries, 3) 86 | self.assertEqual(task.is_async, False) 87 | self.assertEqual(task.on_start, start) 88 | self.assertEqual(task.on_success, success) 89 | self.assertEqual(task.on_error, error) 90 | self.assertEqual(task.depends_on, ["id1", "id3", "id4"]) 91 | self.assertEqual(task.delay_until, 12345798) 92 | self.assertEqual(task.status, WAITING) 93 | self.assertEqual(task.func, None) 94 | self.assertEqual(task.func_args, []) 95 | self.assertEqual(task.func_kwargs, {}) 96 | 97 | def test_custom_init_delay_until_datetime(self): 98 | du = datetime.datetime(2020, 9, 5, 16, 45, 32) 99 | 100 | task = Task( 101 | task_id="hello", 102 | retries=3, 103 | is_async=False, 104 | on_start=start, 105 | on_success=success, 106 | on_error=error, 107 | depends_on=["id1", "id3", "id4"], 108 | delay_until=du, 109 | ) 110 | self.assertNotEqual(task.task_id, None) 111 | self.assertEqual(task.retries, 3) 112 | self.assertEqual(task.is_async, False) 113 | self.assertEqual(task.on_start, start) 114 | self.assertEqual(task.on_success, success) 115 | self.assertEqual(task.on_error, error) 116 | self.assertEqual(task.depends_on, ["id1", "id3", "id4"]) 117 | self.assertEqual(task.delay_until, 1599342332.0) 118 | self.assertEqual(task.status, WAITING) 119 | self.assertEqual(task.func, None) 120 | self.assertEqual(task.func_args, []) 121 | self.assertEqual(task.func_kwargs, {}) 122 | 123 | def test_to_call(self): 124 | self.assertEqual(self.task.func, None) 125 | self.assertEqual(self.task.func_args, []) 126 | self.assertEqual(self.task.func_kwargs, {}) 127 | 128 | self.task.to_call(run_me, 1, y=2) 129 | 130 | self.assertEqual(self.task.func, run_me) 131 | self.assertEqual(self.task.func_args, (1,)) 132 | self.assertEqual(self.task.func_kwargs, {"y": 2}) 133 | 134 | def test_to_waiting(self): 135 | # This shouldn't normally be done. Better to use `task.to_success`... 136 | self.task.status = SUCCESS 137 | self.assertEqual(self.task.status, SUCCESS) 138 | 139 | self.task.to_waiting() 140 | self.assertEqual(self.task.status, WAITING) 141 | 142 | def test_to_success(self): 143 | self.assertEqual(self.task.status, WAITING) 144 | 145 | self.task.to_success() 146 | self.assertEqual(self.task.status, SUCCESS) 147 | 148 | def test_to_failed(self): 149 | self.assertEqual(self.task.status, WAITING) 150 | 151 | self.task.to_failed() 152 | self.assertEqual(self.task.status, FAILED) 153 | 154 | def test_to_canceled(self): 155 | self.assertEqual(self.task.status, WAITING) 156 | 157 | self.task.to_canceled() 158 | self.assertEqual(self.task.status, CANCELED) 159 | 160 | def test_to_retrying(self): 161 | self.assertEqual(self.task.status, WAITING) 162 | 163 | self.task.to_retrying() 164 | self.assertEqual(self.task.status, RETRYING) 165 | 166 | def test_serialize(self): 167 | # Shenanigans. You'd normally use the kwargs at ``__init__``... 168 | self.task.task_id = "hello" 169 | self.task.on_success = success 170 | self.task.delay_until = 12345798 171 | 172 | self.task.to_call(run_me, 1, y=2) 173 | raw_json = self.task.serialize() 174 | 175 | data = json.loads(raw_json) 176 | self.assertEqual( 177 | data, 178 | { 179 | "task_id": "hello", 180 | "retries": 0, 181 | "is_async": True, 182 | "module": "tests.test_tasks", 183 | "callable": "run_me", 184 | "args": [1], 185 | "kwargs": {"y": 2}, 186 | "options": { 187 | "on_success": { 188 | "module": "tests.test_tasks", 189 | "callable": "success", 190 | }, 191 | "delay_until": 12345798, 192 | }, 193 | }, 194 | ) 195 | 196 | def test_deserialize(self): 197 | raw_json = json.dumps( 198 | { 199 | "task_id": "hello", 200 | "retries": 3, 201 | "is_async": False, 202 | "module": "tests.test_tasks", 203 | "callable": "run_me", 204 | "args": [1], 205 | "kwargs": {"y": 2}, 206 | "options": { 207 | "on_error": { 208 | "module": "tests.test_tasks", 209 | "callable": "error", 210 | }, 211 | "delay_until": 12345678, 212 | }, 213 | } 214 | ) 215 | task = Task.deserialize(raw_json) 216 | self.assertEqual(task.task_id, "hello") 217 | self.assertEqual(task.retries, 3) 218 | self.assertEqual(task.is_async, False) 219 | self.assertEqual(task.on_start, None) 220 | self.assertEqual(task.on_success, None) 221 | self.assertEqual(task.on_error, error) 222 | self.assertEqual(task.depends_on, None) 223 | self.assertEqual(task.delay_until, 12345678) 224 | self.assertEqual(task.status, WAITING) 225 | self.assertEqual(task.func, run_me) 226 | self.assertEqual(task.func_args, (1,)) 227 | self.assertEqual(task.func_kwargs, {"y": 2}) 228 | 229 | def test_run_failed(self): 230 | def start(t): 231 | t.started = True 232 | 233 | def error(t, err): 234 | t.err_msg = str(err) 235 | 236 | def success(t, result): 237 | t.success_result = result 238 | 239 | def fail_task(inital, incr_by=1): 240 | raise IOError("Math is hard.") 241 | 242 | task = Task(on_start=start, on_success=success, on_error=error) 243 | 244 | # Should fail. 245 | task.to_call(fail_task, 2) 246 | 247 | try: 248 | task.run() 249 | except IOError: 250 | pass 251 | 252 | self.assertTrue(task.started) 253 | self.assertEqual(task.err_msg, "Math is hard.") 254 | 255 | def test_run_success(self): 256 | def start(t): 257 | t.started = True 258 | 259 | def error(t, err): 260 | t.err_msg = str(err) 261 | 262 | def success(t, result): 263 | t.success_result = result 264 | 265 | def success_task(initial, incr_by=1): 266 | return initial + incr_by 267 | 268 | task = Task(on_start=start, on_success=success, on_error=error) 269 | 270 | # Should succeed. 271 | task.to_call(success_task, 12, 3) 272 | task.run() 273 | self.assertEqual(task.result, 15) 274 | self.assertTrue(task.started) 275 | self.assertEqual(task.success_result, 15) 276 | -------------------------------------------------------------------------------- /alligator/tasks.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import json 3 | import time 4 | import uuid 5 | 6 | from .constants import WAITING, SUCCESS, FAILED, RETRYING, CANCELED 7 | from .exceptions import MultipleDelayError 8 | from .utils import determine_module, determine_name, import_attr 9 | 10 | 11 | class Task(object): 12 | def __init__( 13 | self, 14 | task_id=None, 15 | retries=0, 16 | is_async=True, 17 | on_start=None, 18 | on_success=None, 19 | on_error=None, 20 | depends_on=None, 21 | delay_by=None, 22 | delay_until=None, 23 | ): 24 | """ 25 | A base class for managing the execution & serialization of tasks. 26 | 27 | Ex:: 28 | 29 | from alligator import Task 30 | 31 | # Create the task itself. 32 | task = Task() 33 | # ...or... 34 | task = Task(task_id='my-unique-id', retries=4, is_async=False) 35 | 36 | # Hook up what will be called when the task comes off the queue. 37 | task.to_call(email_followers, emails=user.followers.emails()) 38 | 39 | Args: 40 | task_id: (str): Optional. A unique identifier for the task. 41 | Defaults to `None` (create a `uuid4`). 42 | retries (int): Optional. The number of times to retry a task if it 43 | fails (throws an exception). Defaults to `0` (no retries). 44 | is_async (bool): Optional. If the task should be run 45 | asynchronously or not. Defaults to `True`. 46 | on_start (callable): Optional. A hook function to run when the 47 | task is first pulled off the queue. Defaults to `None`. 48 | on_success (callable): Optional. A hook function to run when the 49 | task completes successfully. Defaults to `None`. 50 | on_error (callable): Optional. A hook function to run when the 51 | task is fails. If a non-zero number of retries are provided, 52 | this will fire *each time* the task fails. Defaults to `None`. 53 | depends_on (list): Optional. A list of task_ids that must be 54 | complete before this task will fire. Defaults to `None`. 55 | delay_by (int): Optional. The number of seconds to delay before 56 | the task can be processed. *Mutually exclusive* with 57 | `delay_until`. 58 | delay_until (float|datetime|date): Optional. The Unix timestamp 59 | (or a UTC datetime/date object) to delay processing the task 60 | until. *Mutually exclusive* with `delay_by`. 61 | """ 62 | self.task_id = task_id 63 | self.retries = int(retries) 64 | self.is_async = is_async 65 | self.status = WAITING 66 | self.on_start = on_start 67 | self.on_success = on_success 68 | self.on_error = on_error 69 | self.depends_on = depends_on 70 | self.delay_until = delay_until 71 | self.result = None 72 | 73 | if self.delay_until is not None: 74 | if isinstance( 75 | self.delay_until, (datetime.datetime, datetime.date) 76 | ): 77 | self.delay_until = time.mktime(self.delay_until.timetuple()) 78 | 79 | # If the convenience option `delay_by` is seen, calculate the correct 80 | # `delay_until`. 81 | if delay_by is not None: 82 | # The delay options are exclusive. 83 | if delay_until is not None: 84 | raise MultipleDelayError( 85 | "Only one of 'delay_by' or 'delay_until' can be used at " 86 | "a time." 87 | ) 88 | 89 | # Compute the desired timestamp. 90 | self.delay_until = time.time() + delay_by 91 | 92 | self.func = None 93 | self.func_args = [] 94 | self.func_kwargs = {} 95 | 96 | if self.task_id is None: 97 | self.task_id = str(uuid.uuid4()) 98 | 99 | def to_call(self, func, *args, **kwargs): 100 | """ 101 | Sets the function & its arguments to be called when the task is 102 | processed. 103 | 104 | Ex:: 105 | 106 | task.to_call(my_function, 1, 'c', another=True) 107 | 108 | Args: 109 | func (callable): The callable with business logic to execute 110 | args (list): Positional arguments to pass to the callable task 111 | kwargs (dict): Keyword arguments to pass to the callable task 112 | """ 113 | self.func = func 114 | self.func_args = args 115 | self.func_kwargs = kwargs 116 | 117 | def to_waiting(self): 118 | """ 119 | Sets the task's status as "waiting". 120 | 121 | Useful for the `on_start/on_success/on_failed` hook methods for 122 | figuring out what the status of the task is. 123 | """ 124 | self.status = WAITING 125 | 126 | def to_success(self): 127 | """ 128 | Sets the task's status as "success". 129 | 130 | Useful for the `on_start/on_success/on_failed` hook methods for 131 | figuring out what the status of the task is. 132 | """ 133 | self.status = SUCCESS 134 | 135 | def to_failed(self): 136 | """ 137 | Sets the task's status as "failed". 138 | 139 | Useful for the `on_start/on_success/on_failed` hook methods for 140 | figuring out what the status of the task is. 141 | """ 142 | self.status = FAILED 143 | 144 | def to_canceled(self): 145 | """ 146 | Sets the task's status as "canceled". 147 | 148 | Useful for the `on_start/on_success/on_failed` hook methods for 149 | figuring out what the status of the task is. 150 | """ 151 | self.status = CANCELED 152 | 153 | def to_retrying(self): 154 | """ 155 | Sets the task's status as "retrying". 156 | 157 | Useful for the `on_start/on_success/on_failed` hook methods for 158 | figuring out what the status of the task is. 159 | """ 160 | self.status = RETRYING 161 | 162 | def serialize(self): 163 | """ 164 | Serializes the `Task` data for storing in the queue. 165 | 166 | All data must be JSON-serializable in order to be stored properly. 167 | 168 | Returns: 169 | str: A JSON string of the task data. 170 | """ 171 | data = { 172 | "task_id": self.task_id, 173 | "retries": self.retries, 174 | "is_async": self.is_async, 175 | "module": determine_module(self.func), 176 | "callable": determine_name(self.func), 177 | "args": self.func_args, 178 | "kwargs": self.func_kwargs, 179 | "options": {}, 180 | } 181 | 182 | if self.on_start: 183 | data["options"]["on_start"] = { 184 | "module": determine_module(self.on_start), 185 | "callable": determine_name(self.on_start), 186 | } 187 | 188 | if self.on_success: 189 | data["options"]["on_success"] = { 190 | "module": determine_module(self.on_success), 191 | "callable": determine_name(self.on_success), 192 | } 193 | 194 | if self.on_error: 195 | data["options"]["on_error"] = { 196 | "module": determine_module(self.on_error), 197 | "callable": determine_name(self.on_error), 198 | } 199 | 200 | if self.delay_until: 201 | data["options"]["delay_until"] = self.delay_until 202 | 203 | return json.dumps(data) 204 | 205 | @classmethod 206 | def deserialize(cls, data): 207 | """ 208 | Given some data from the queue, deserializes it into a `Task` 209 | instance. 210 | 211 | The data must be similar in format to what comes from 212 | `Task.serialize` (a JSON-serialized dictionary). Required keys are 213 | `task_id`, `retries` & `is_async`. 214 | 215 | Args: 216 | data (str): A JSON-serialized string of the task data 217 | 218 | Returns: 219 | Task: A populated task 220 | """ 221 | data = json.loads(data) 222 | options = data.get("options", {}) 223 | 224 | task = cls( 225 | task_id=data["task_id"], 226 | retries=data["retries"], 227 | is_async=data["is_async"], 228 | ) 229 | 230 | func = import_attr(data["module"], data["callable"]) 231 | task.to_call(func, *data.get("args", []), **data.get("kwargs", {})) 232 | 233 | if options.get("on_start"): 234 | task.on_start = import_attr( 235 | options["on_start"]["module"], options["on_start"]["callable"] 236 | ) 237 | 238 | if options.get("on_success"): 239 | task.on_success = import_attr( 240 | options["on_success"]["module"], 241 | options["on_success"]["callable"], 242 | ) 243 | 244 | if options.get("on_error"): 245 | task.on_error = import_attr( 246 | options["on_error"]["module"], options["on_error"]["callable"] 247 | ) 248 | 249 | if options.get("delay_until"): 250 | task.delay_until = options["delay_until"] 251 | 252 | return task 253 | 254 | def run(self): 255 | """ 256 | Runs the task. 257 | 258 | This fires the `on_start` hook function first (if present), passing 259 | the task itself. 260 | 261 | Then it runs the target function supplied via `Task.to_call` with 262 | its arguments & stores the result. 263 | 264 | If the target function succeeded, the `on_success` hook function is 265 | called, passing both the task & the result to it. 266 | 267 | If the target function failed (threw an exception), the `on_error` 268 | hook function is called, passing both the task & the exception to it. 269 | Then the exception is re-raised. 270 | 271 | Finally, it returns itself, with `Task.result` set to the result 272 | from the target function's execution. 273 | 274 | Returns: 275 | Task: The processed Task. 276 | """ 277 | if self.on_start: 278 | self.on_start(self) 279 | 280 | try: 281 | self.result = self.func(*self.func_args, **self.func_kwargs) 282 | except Exception as err: 283 | self.to_failed() 284 | 285 | if self.on_error: 286 | self.on_error(self, err) 287 | 288 | raise 289 | 290 | self.to_success() 291 | 292 | if self.on_success: 293 | self.on_success(self, self.result) 294 | 295 | return self 296 | -------------------------------------------------------------------------------- /alligator/gator.py: -------------------------------------------------------------------------------- 1 | from .constants import ALL 2 | from .tasks import Task 3 | from .utils import import_attr 4 | 5 | 6 | class Gator(object): 7 | def __init__( 8 | self, conn_string, queue_name=ALL, task_class=Task, backend_class=None 9 | ): 10 | """ 11 | A coordination for scheduling & processing tasks. 12 | 13 | Handles creating tasks (with options), using the backend to place tasks 14 | in the queue & pulling/processing tasks off the queue. 15 | 16 | Ex:: 17 | 18 | from alligator import Gator 19 | 20 | def add(a, b): 21 | return a + b 22 | 23 | gator = Gator('locmem://') 24 | 25 | gator.task(add, 3, 7) 26 | 27 | Args: 28 | conn_string (str): A DSN for connecting to the queue. Passed along 29 | to the backend. 30 | queue_name (str): Optional. The name of the queue the tasks 31 | should be placed in. Defaults to ``ALL``. 32 | task_class (class): Optional. The class to use for instantiating 33 | tasks. Defaults to ``Task``. 34 | backend_class (class): Optional. The class to use for 35 | instantiating the backend. Defaults to ``None`` (DSN 36 | detection). 37 | """ 38 | self.conn_string = conn_string 39 | self.queue_name = queue_name 40 | self.task_class = task_class 41 | self.backend_class = backend_class 42 | 43 | if not backend_class: 44 | self.backend = self.build_backend(self.conn_string) 45 | else: 46 | self.backend = backend_class(self.conn_string) 47 | 48 | def build_backend(self, conn_string): 49 | """ 50 | Given a DSN, returns an instantiated backend class. 51 | 52 | Ex:: 53 | 54 | backend = gator.build_backend('locmem://') 55 | # ...or... 56 | backend = gator.build_backend('redis://127.0.0.1:6379/0') 57 | 58 | Args: 59 | conn_string (str): A DSN for connecting to the queue. Passed along 60 | to the backend. 61 | 62 | Returns: 63 | Client: A backend ``Client`` instance 64 | """ 65 | backend_name, _ = conn_string.split(":", 1) 66 | backend_path = "alligator.backends.{}_backend".format(backend_name) 67 | client_class = import_attr(backend_path, "Client") 68 | return client_class(conn_string) 69 | 70 | def len(self): 71 | """ 72 | Returns the number of remaining queued tasks. 73 | 74 | Returns: 75 | int: A count of the remaining tasks 76 | """ 77 | return self.backend.len(self.queue_name) 78 | 79 | def push(self, task, func, *args, **kwargs): 80 | """ 81 | Pushes a configured task onto the queue. 82 | 83 | Typically, you'll favor using the ``Gator.task`` method or 84 | ``Gator.options`` context manager for creating a task. Call this 85 | only if you have specific needs or know what you're doing. 86 | 87 | If the ``Task`` has the ``is_async = False`` option, the task will be 88 | run immediately (in-process). This is useful for development and 89 | in testing. 90 | 91 | Ex:: 92 | 93 | task = Task(is_async=False, retries=3) 94 | finished = gator.push(task, increment, incr_by=2) 95 | 96 | Args: 97 | task (Task): A mostly-configured task 98 | func (callable): The callable with business logic to execute 99 | args (list): Positional arguments to pass to the callable task 100 | kwargs (dict): Keyword arguments to pass to the callable task 101 | 102 | Returns: 103 | Task: The fleshed-out ``Task`` instance 104 | """ 105 | task.to_call(func, *args, **kwargs) 106 | data = task.serialize() 107 | 108 | if task.is_async: 109 | task.task_id = self.backend.push( 110 | self.queue_name, 111 | task.task_id, 112 | data, 113 | delay_until=task.delay_until, 114 | ) 115 | else: 116 | self.execute(task) 117 | 118 | return task 119 | 120 | def pop(self): 121 | """ 122 | Pops a task off the front of the queue & runs it. 123 | 124 | Typically, you'll favor using a ``Worker`` to handle processing the 125 | queue (to constantly consume). However, if you need to custom-process 126 | the queue in-order, this method is useful. 127 | 128 | Ex:: 129 | 130 | # Tasks were previously added, maybe by a different process or 131 | # machine... 132 | finished_topmost_task = gator.pop() 133 | 134 | Returns: 135 | Task: The completed ``Task`` instance 136 | """ 137 | data = self.backend.pop(self.queue_name) 138 | 139 | if data: 140 | task = self.task_class.deserialize(data) 141 | return self.execute(task) 142 | 143 | def get(self, task_id): 144 | """ 145 | Gets a specific task, by ``task_id`` off the queue & runs it. 146 | 147 | Using this is not as performant (because it has to search the queue), 148 | but can be useful if you need to specifically handle a task 149 | *right now*. 150 | 151 | Ex:: 152 | 153 | # Tasks were previously added, maybe by a different process or 154 | # machine... 155 | finished_task = gator.get('a-specific-uuid-here') 156 | 157 | Args: 158 | task_id (str): The identifier of the task to process 159 | 160 | Returns: 161 | Task: The completed ``Task`` instance 162 | """ 163 | data = self.backend.get(self.queue_name, task_id) 164 | 165 | if data: 166 | task = self.task_class.deserialize(data) 167 | return self.execute(task) 168 | 169 | def cancel(self, task_id): 170 | """ 171 | Takes an existing task & cancels it before it is processed. 172 | 173 | Returns the canceled task, as that could be useful in creating a new 174 | task. 175 | 176 | Ex:: 177 | 178 | task = gator.task(add, 18, 9) 179 | 180 | # Whoops, didn't mean to do that. 181 | gator.cancel(task.task_id) 182 | 183 | Args: 184 | task_id (str): The identifier of the task to process 185 | 186 | Returns: 187 | Task: The canceled ``Task`` instance 188 | """ 189 | data = self.backend.get(self.queue_name, task_id) 190 | 191 | if data: 192 | task = self.task_class.deserialize(data) 193 | task.to_canceled() 194 | return task 195 | 196 | def execute(self, task): 197 | """ 198 | Given a task instance, this runs it. 199 | 200 | This includes handling retries & re-raising exceptions. 201 | 202 | Ex:: 203 | 204 | task = Task(is_async=False, retries=5) 205 | task.to_call(add, 101, 35) 206 | finished_task = gator.execute(task) 207 | 208 | Args: 209 | task_id (str): The identifier of the task to process 210 | 211 | Returns: 212 | Task: The completed ``Task`` instance 213 | """ 214 | try: 215 | return task.run() 216 | except Exception: 217 | if task.retries > 0: 218 | task.retries -= 1 219 | task.to_retrying() 220 | 221 | if task.is_async: 222 | # Place it back on the queue. 223 | data = task.serialize() 224 | task.task_id = self.backend.push( 225 | self.queue_name, task.task_id, data 226 | ) 227 | else: 228 | return self.execute(task) 229 | else: 230 | raise 231 | 232 | def task(self, func, *args, **kwargs): 233 | """ 234 | Pushes a task onto the queue. 235 | 236 | This will instantiate a ``Gator.task_class`` instance, configure 237 | the callable & its arguments, then push it onto the queue. 238 | 239 | You'll typically want to use either this method or the 240 | ``Gator.options`` context manager (if you need to configure the 241 | ``Task`` arguments, such as retries, is_async, task_id, etc.) 242 | 243 | Ex:: 244 | 245 | on_queue = gator.task(increment, incr_by=2) 246 | 247 | Args: 248 | func (callable): The callable with business logic to execute 249 | args (list): Positional arguments to pass to the callable task 250 | kwargs (dict): Keyword arguments to pass to the callable task 251 | 252 | Returns: 253 | Task: The ``Task`` instance 254 | """ 255 | task = self.task_class() 256 | return self.push(task, func, *args, **kwargs) 257 | 258 | def options(self, **kwargs): 259 | """ 260 | Allows specifying advanced ``Task`` options to control how the task 261 | runs. 262 | 263 | This returns a context manager which will create ``Task`` instances 264 | with the supplied options. See ``Task.__init__`` for the available 265 | arguments. 266 | 267 | Ex:: 268 | 269 | def party_time(task, result): 270 | # Throw a party in honor of this task completing. 271 | # ... 272 | 273 | with gator.options(retries=2, on_success=party_time) as opts: 274 | opts.task(increment, incr_by=2678) 275 | 276 | Args: 277 | kwargs (dict): Keyword arguments to control the task execution 278 | 279 | Returns: 280 | Options: An ``Options`` context manager instance 281 | """ 282 | return Options(self, **kwargs) 283 | 284 | 285 | class Options(object): 286 | def __init__(self, gator, **kwargs): 287 | """ 288 | A context manager for specifying task execution options. 289 | 290 | Typically, you'd use ``Gator.options``, which creates this context 291 | manager for you. You probably don't want to directly use this. 292 | 293 | Args: 294 | gator (Gator): A configured ``Gator`` instance. 295 | **kwargs (dict): Keyword arguments to control the task execution 296 | """ 297 | self.gator = gator 298 | self.task_kwargs = kwargs 299 | 300 | def __enter__(self): 301 | return self 302 | 303 | def __exit__(self, type, value, traceback): 304 | pass 305 | 306 | def task(self, func, *args, **kwargs): 307 | """ 308 | Pushes a task onto the queue (with the specified options). 309 | 310 | This will instantiate a ``Gator.task_class`` instance, configure the 311 | task execution options, configure the callable & its arguments, then 312 | push it onto the queue. 313 | 314 | You'll typically call this method when specifying advanced options. 315 | 316 | Args: 317 | func (callable): The callable with business logic to execute 318 | args (list): Positional arguments to pass to the callable task 319 | kwargs (dict): Keyword arguments to pass to the callable task 320 | 321 | Returns: 322 | Task: The ``Task`` instance 323 | """ 324 | task = self.gator.task_class(**self.task_kwargs) 325 | return self.gator.push(task, func, *args, **kwargs) 326 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # Alligator documentation build configuration file, created by 4 | # sphinx-quickstart on Wed Dec 31 14:13:41 2014. 5 | # 6 | # This file is execfile()d with the current directory set to its 7 | # containing dir. 8 | # 9 | # Note that not all possible configuration values are present in this 10 | # autogenerated file. 11 | # 12 | # All configuration values have a default; values that are commented out 13 | # serve to show the default. 14 | 15 | import sys 16 | import os 17 | 18 | # If extensions (or modules to document with autodoc) are in another directory, 19 | # add these directories to sys.path here. If the directory is relative to the 20 | # documentation root, use os.path.abspath to make it absolute, like shown here. 21 | # sys.path.insert(0, os.path.abspath('.')) 22 | 23 | # -- General configuration ------------------------------------------------ 24 | 25 | # If your documentation needs a minimal Sphinx version, state it here. 26 | # needs_sphinx = '1.0' 27 | 28 | # Add any Sphinx extension module names here, as strings. They can be 29 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 30 | # ones. 31 | extensions = [ 32 | "sphinx.ext.autodoc", 33 | "sphinxcontrib.napoleon", 34 | ] 35 | 36 | # Add any paths that contain templates here, relative to this directory. 37 | templates_path = ["_templates"] 38 | 39 | # The suffix of source filenames. 40 | source_suffix = ".rst" 41 | 42 | # The encoding of source files. 43 | # source_encoding = 'utf-8-sig' 44 | 45 | # The master toctree document. 46 | master_doc = "index" 47 | 48 | # General information about the project. 49 | project = u"Alligator" 50 | copyright = u"2014-2020, Daniel Lindsley" 51 | 52 | # The version info for the project you're documenting, acts as replacement for 53 | # |version| and |release|, also used in various other places throughout the 54 | # built documents. 55 | # 56 | # The short X.Y version. 57 | version = "1.0.0" 58 | # The full version, including alpha/beta/rc tags. 59 | release = "1.0.0" 60 | 61 | # The language for content autogenerated by Sphinx. Refer to documentation 62 | # for a list of supported languages. 63 | # language = None 64 | 65 | # There are two options for replacing |today|: either, you set today to some 66 | # non-false value, then it is used: 67 | # today = '' 68 | # Else, today_fmt is used as the format for a strftime call. 69 | # today_fmt = '%B %d, %Y' 70 | 71 | # List of patterns, relative to source directory, that match files and 72 | # directories to ignore when looking for source files. 73 | exclude_patterns = ["_build"] 74 | 75 | # The reST default role (used for this markup: `text`) to use for all 76 | # documents. 77 | # default_role = None 78 | 79 | # If true, '()' will be appended to :func: etc. cross-reference text. 80 | # add_function_parentheses = True 81 | 82 | # If true, the current module name will be prepended to all description 83 | # unit titles (such as .. function::). 84 | # add_module_names = True 85 | 86 | # If true, sectionauthor and moduleauthor directives will be shown in the 87 | # output. They are ignored by default. 88 | # show_authors = False 89 | 90 | # The name of the Pygments (syntax highlighting) style to use. 91 | pygments_style = "sphinx" 92 | 93 | # A list of ignored prefixes for module index sorting. 94 | # modindex_common_prefix = [] 95 | 96 | # If true, keep warnings as "system message" paragraphs in the built documents. 97 | # keep_warnings = False 98 | 99 | 100 | # -- Options for HTML output ---------------------------------------------- 101 | 102 | # The theme to use for HTML and HTML Help pages. See the documentation for 103 | # a list of builtin themes. 104 | html_theme = "default" 105 | 106 | # Theme options are theme-specific and customize the look and feel of a theme 107 | # further. For a list of options available for each theme, see the 108 | # documentation. 109 | # html_theme_options = {} 110 | 111 | # Add any paths that contain custom themes here, relative to this directory. 112 | # html_theme_path = [] 113 | 114 | # The name for this set of Sphinx documents. If None, it defaults to 115 | # " v documentation". 116 | # html_title = None 117 | 118 | # A shorter title for the navigation bar. Default is the same as html_title. 119 | # html_short_title = None 120 | 121 | # The name of an image file (relative to this directory) to place at the top 122 | # of the sidebar. 123 | # html_logo = None 124 | 125 | # The name of an image file (within the static path) to use as favicon of the 126 | # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 127 | # pixels large. 128 | # html_favicon = None 129 | 130 | # Add any paths that contain custom static files (such as style sheets) here, 131 | # relative to this directory. They are copied after the builtin static files, 132 | # so a file named "default.css" will overwrite the builtin "default.css". 133 | html_static_path = ["_static"] 134 | 135 | # Add any extra paths that contain custom files (such as robots.txt or 136 | # .htaccess) here, relative to this directory. These files are copied 137 | # directly to the root of the documentation. 138 | # html_extra_path = [] 139 | 140 | # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, 141 | # using the given strftime format. 142 | # html_last_updated_fmt = '%b %d, %Y' 143 | 144 | # If true, SmartyPants will be used to convert quotes and dashes to 145 | # typographically correct entities. 146 | # html_use_smartypants = True 147 | 148 | # Custom sidebar templates, maps document names to template names. 149 | # html_sidebars = {} 150 | 151 | # Additional templates that should be rendered to pages, maps page names to 152 | # template names. 153 | # html_additional_pages = {} 154 | 155 | # If false, no module index is generated. 156 | # html_domain_indices = True 157 | 158 | # If false, no index is generated. 159 | # html_use_index = True 160 | 161 | # If true, the index is split into individual pages for each letter. 162 | # html_split_index = False 163 | 164 | # If true, links to the reST sources are added to the pages. 165 | # html_show_sourcelink = True 166 | 167 | # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. 168 | # html_show_sphinx = True 169 | 170 | # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. 171 | # html_show_copyright = True 172 | 173 | # If true, an OpenSearch description file will be output, and all pages will 174 | # contain a tag referring to it. The value of this option must be the 175 | # base URL from which the finished HTML is served. 176 | # html_use_opensearch = '' 177 | 178 | # This is the file name suffix for HTML files (e.g. ".xhtml"). 179 | # html_file_suffix = None 180 | 181 | # Output file base name for HTML help builder. 182 | htmlhelp_basename = "Alligatordoc" 183 | 184 | 185 | # -- Options for LaTeX output --------------------------------------------- 186 | 187 | latex_elements = { 188 | # The paper size ('letterpaper' or 'a4paper'). 189 | #'papersize': 'letterpaper', 190 | # The font size ('10pt', '11pt' or '12pt'). 191 | #'pointsize': '10pt', 192 | # Additional stuff for the LaTeX preamble. 193 | #'preamble': '', 194 | } 195 | 196 | # Grouping the document tree into LaTeX files. List of tuples 197 | # (source start file, target name, title, 198 | # author, documentclass [howto, manual, or own class]). 199 | latex_documents = [ 200 | ( 201 | "index", 202 | "Alligator.tex", 203 | u"Alligator Documentation", 204 | u"Daniel Lindsley", 205 | "manual", 206 | ), 207 | ] 208 | 209 | # The name of an image file (relative to this directory) to place at the top of 210 | # the title page. 211 | # latex_logo = None 212 | 213 | # For "manual" documents, if this is true, then toplevel headings are parts, 214 | # not chapters. 215 | # latex_use_parts = False 216 | 217 | # If true, show page references after internal links. 218 | # latex_show_pagerefs = False 219 | 220 | # If true, show URL addresses after external links. 221 | # latex_show_urls = False 222 | 223 | # Documents to append as an appendix to all manuals. 224 | # latex_appendices = [] 225 | 226 | # If false, no module index is generated. 227 | # latex_domain_indices = True 228 | 229 | 230 | # -- Options for manual page output --------------------------------------- 231 | 232 | # One entry per manual page. List of tuples 233 | # (source start file, name, description, authors, manual section). 234 | man_pages = [ 235 | ( 236 | "index", 237 | "alligator", 238 | u"Alligator Documentation", 239 | [u"Daniel Lindsley"], 240 | 1, 241 | ) 242 | ] 243 | 244 | # If true, show URL addresses after external links. 245 | # man_show_urls = False 246 | 247 | 248 | # -- Options for Texinfo output ------------------------------------------- 249 | 250 | # Grouping the document tree into Texinfo files. List of tuples 251 | # (source start file, target name, title, author, 252 | # dir menu entry, description, category) 253 | texinfo_documents = [ 254 | ( 255 | "index", 256 | "Alligator", 257 | u"Alligator Documentation", 258 | u"Daniel Lindsley", 259 | "Alligator", 260 | "One line description of project.", 261 | "Miscellaneous", 262 | ), 263 | ] 264 | 265 | # Documents to append as an appendix to all manuals. 266 | # texinfo_appendices = [] 267 | 268 | # If false, no module index is generated. 269 | # texinfo_domain_indices = True 270 | 271 | # How to display URL addresses: 'footnote', 'no', or 'inline'. 272 | # texinfo_show_urls = 'footnote' 273 | 274 | # If true, do not generate a @detailmenu in the "Top" node's menu. 275 | # texinfo_no_detailmenu = False 276 | 277 | 278 | # -- Options for Epub output ---------------------------------------------- 279 | 280 | # Bibliographic Dublin Core info. 281 | epub_title = u"Alligator" 282 | epub_author = u"Daniel Lindsley" 283 | epub_publisher = u"Daniel Lindsley" 284 | epub_copyright = u"2014, Daniel Lindsley" 285 | 286 | # The basename for the epub file. It defaults to the project name. 287 | # epub_basename = u'Alligator' 288 | 289 | # The HTML theme for the epub output. Since the default themes are not optimized 290 | # for small screen space, using the same theme for HTML and epub output is 291 | # usually not wise. This defaults to 'epub', a theme designed to save visual 292 | # space. 293 | # epub_theme = 'epub' 294 | 295 | # The language of the text. It defaults to the language option 296 | # or en if the language is not set. 297 | # epub_language = '' 298 | 299 | # The scheme of the identifier. Typical schemes are ISBN or URL. 300 | # epub_scheme = '' 301 | 302 | # The unique identifier of the text. This can be a ISBN number 303 | # or the project homepage. 304 | # epub_identifier = '' 305 | 306 | # A unique identification for the text. 307 | # epub_uid = '' 308 | 309 | # A tuple containing the cover image and cover page html template filenames. 310 | # epub_cover = () 311 | 312 | # A sequence of (type, uri, title) tuples for the guide element of content.opf. 313 | # epub_guide = () 314 | 315 | # HTML files that should be inserted before the pages created by sphinx. 316 | # The format is a list of tuples containing the path and title. 317 | # epub_pre_files = [] 318 | 319 | # HTML files shat should be inserted after the pages created by sphinx. 320 | # The format is a list of tuples containing the path and title. 321 | # epub_post_files = [] 322 | 323 | # A list of files that should not be packed into the epub file. 324 | epub_exclude_files = ["search.html"] 325 | 326 | # The depth of the table of contents in toc.ncx. 327 | # epub_tocdepth = 3 328 | 329 | # Allow duplicate toc entries. 330 | # epub_tocdup = True 331 | 332 | # Choose between 'default' and 'includehidden'. 333 | # epub_tocscope = 'default' 334 | 335 | # Fix unsupported image types using the PIL. 336 | # epub_fix_images = False 337 | 338 | # Scale large images. 339 | # epub_max_image_width = 0 340 | 341 | # How to display URL addresses: 'footnote', 'no', or 'inline'. 342 | # epub_show_urls = 'inline' 343 | 344 | # If false, no index is generated. 345 | # epub_use_index = True 346 | 347 | -------------------------------------------------------------------------------- /docs/extending.rst: -------------------------------------------------------------------------------- 1 | .. _extending: 2 | 3 | =================== 4 | Extending Alligator 5 | =================== 6 | 7 | If you've read the :ref:`tutorial`, you've seen some basic usage of Alligator, 8 | which largely comes down to: 9 | 10 | * create a ``Gator`` instance, ... 11 | * use either ``gator.task(...)`` or ``gator.options(...)`` to enqueue the 12 | task, ... 13 | * and then using either ``latergator.py`` or a custom script with a ``Worker`` 14 | to process the queue. 15 | 16 | While this is great in the common/simple case, there may be times where you 17 | need more complex things. Alligator was built for extension, so this document 18 | will outline ways you can get more out of it. 19 | 20 | 21 | Hook Methods 22 | ============ 23 | 24 | In addition to the task function itself, every task supports three optional 25 | hook functions: 26 | 27 | ``on_start(task)`` 28 | A function called when the task is first pulled off the queue but 29 | processing hasn't started. 30 | 31 | ``on_success(task, result)`` 32 | A function called when the task completes successfully (no exceptions 33 | thrown). If the task function returns a value, it is passed to this 34 | function as well. 35 | 36 | ``on_error(task, err)`` 37 | A function called when the task fails during processing (an exception was 38 | raised). The exception is passed as well as the failed task. 39 | 40 | All together, this lets you do more complex things without muddying the task 41 | function itself. For instance, the following code would log the start/completion 42 | of a task, increment a success count & potentially extend the number of retries 43 | on error. 44 | 45 | .. code:: python 46 | 47 | import logging 48 | 49 | from alligator import Gator 50 | import requests 51 | 52 | from myapp.exceptions import FeedUnavailable, FeedNotFound 53 | from myapp.utils import cache 54 | 55 | 56 | log = logging.getLogger(__file__) 57 | 58 | 59 | # This is the main task function. 60 | def fetch_feeds(feeds): 61 | for feed_url in feeds: 62 | resp = requests.get(feed_url) 63 | 64 | if resp.status_code == 503: 65 | raise FeedUnavailable(feed_url) 66 | elif resp.status_code != 200: 67 | raise FeedNotFound(feed_url) 68 | 69 | # Some other processing of the feed data 70 | 71 | return len(feeds) 72 | 73 | 74 | # Hook functions 75 | def log_start(task): 76 | log.info('Starting to import feeds via task {}...'.format(task.task_id)) 77 | 78 | def log_success(task, result): 79 | log.info('Finished importing {} feeds via task {}.'.format( 80 | result, 81 | task.task_id 82 | )) 83 | cache.incr('feeds_imported', incr_by=result) 84 | 85 | def maybe_retry_error(task, err): 86 | if isinstance(err, FeedUnavailable): 87 | # Try again soon. 88 | task.retries += 1 89 | else: 90 | log.error('Importing feed url {} failed.'.format(str(err))) 91 | 92 | 93 | # Now we can use those hooks. 94 | with gator.options(on_start=log_start, on_success=log_success, on_error=maybe_retry_error) as opts: 95 | opts.task(fetch_feeds, [ 96 | 'http://daringfireball.net/feeds/main', 97 | 'http://xkcd.com/rss.xml', 98 | 'http://www.reddit.com/r/DotA2/.rss', 99 | ]) 100 | 101 | 102 | Custom Task Classes 103 | =================== 104 | 105 | Sometimes, just the built-in arguments for ``Task`` (like ``retries``, 106 | ``is_async``, ``on_start``/``on_success``/``on_error``) may not be enough. Or 107 | perhaps your hook methods will *always* be the same & you don't want to have to 108 | pass them all the time. Or perhaps you never need the hook methods, but are 109 | running into payload size restrictions by your preferred backend & need some 110 | extra space. 111 | 112 | For this, you can create custom ``Task`` classes for use in place of the 113 | built-in one. Since that last restriction can be especially pertinent, let's 114 | show how we'd handle getting more space in our payload. 115 | 116 | First, we need a ``Task`` subclass. You can create your own (as long as they 117 | follow the protocol), but subclassing is easier here. 118 | 119 | .. code:: python 120 | 121 | # myapp/skinnytask.py 122 | import bz2 123 | 124 | from alligator import Task 125 | 126 | 127 | class SkinnyTask(Task): 128 | # We're both going to ignore some keys (is_async, options) we don't 129 | # care about, as well as compress/decompress the payload. 130 | def serialize(self): 131 | data = { 132 | 'task_id': self.task_id, 133 | 'retries': self.retries, 134 | 'module': determine_module(self.func), 135 | 'callable': determine_name(self.func), 136 | 'args': self.func_args, 137 | 'kwargs': self.func_kwargs, 138 | } 139 | raw_json = json.dumps(data) 140 | return bz2.compress(raw_json) 141 | 142 | @classmethod 143 | def deserialize(cls, data): 144 | raw_json = bz2.decompress(data) 145 | data = json.loads(data) 146 | 147 | task = cls( 148 | task_id=data['task_id'], 149 | retries=data['retries'], 150 | is_async=data['is_async'] 151 | ) 152 | 153 | func = import_attr(data['module'], data['callable']) 154 | task.to_call(func, *data.get('args', []), **data.get('kwargs', {})) 155 | return task 156 | 157 | Now that we have our ``SkinnyTask``, all we need is to use it. Each ``Gator`` 158 | instance supports a ``task_class=...`` keyword argument to replace the class 159 | used. So we'd do: 160 | 161 | .. code:: python 162 | 163 | from alligator import Gator 164 | 165 | from myapp.skinnytask import SkinnyTask 166 | 167 | 168 | gator = Gator('redis://localhost:6379/0', task_class=SkinnyTask) 169 | 170 | Every call to ``gator.task(...)`` or ``gator.options(...)`` will now use our 171 | ``SkinnyTask``. 172 | 173 | The last bit is that you can no longer use the included ``latergator.py`` script 174 | to process your queue. Instead, you'll have to manually run a ``Worker``. 175 | 176 | .. code:: python 177 | 178 | # myapp/skinnylatergator.py 179 | from alligator import Gator, Worker 180 | 181 | from myapp.skinnytask import SkinnyTask 182 | 183 | 184 | gator = Gator('redis://localhost:6379/0', task_class=SkinnyTask) 185 | # Now the worker will pick up the class as well. 186 | worker = Worker(gator) 187 | worker.run_forever() 188 | 189 | 190 | Multiple Queues 191 | =============== 192 | 193 | If you have a high-volume site or the priority of tasks is important, the one 194 | main default queue (``alligator.constants.ALL``) may not work well. 195 | Fortunately, each ``Gator`` instance supports customizing the queue name it 196 | places tasks in. 197 | 198 | Let's say that sending a notification email is way more important to use than 199 | creating thumbnails of photo uploads. We'll create two ``Gator`` instances, one 200 | for each type of processing. 201 | 202 | .. code:: python 203 | 204 | from alligator import Gator 205 | 206 | redis_dsn = 'redis://localhost:' 207 | email_gator = Gator(redis_dsn, queue_name='notifications') 208 | image_gator = Gator(redis_dsn, queue_name='images') 209 | 210 | 211 | # Later... 212 | email_gator.task(send_welcome_email, request.user.pk) 213 | # And elsewhere... 214 | image_gator.task(create_thumbnail, photo_path) 215 | 216 | Now several large uploads won't block the sending of emails later in the queue. 217 | You will however now need to run more ``Workers``. Just like the "Custom Task 218 | Classes" section, your ``Worker`` instances will need either ``email_gator`` or 219 | ``image_gator`` passed to them. 220 | 221 | You could also fire up many ``email_gator`` workers (say 4) and just 1-2 222 | ``image_gator`` workers if the number of tasks justifies it. 223 | 224 | 225 | Custom Backend Clients 226 | ====================== 227 | 228 | As of the time of writing, Alligator supports the following clients: 229 | 230 | * Locmem 231 | * Redis 232 | * SQS 233 | * SQLite 234 | 235 | However, if you have a different datastore or queue you'd like to use, you can 236 | write a custom backend ``Client`` to talk to that store. For example, let's 237 | write a naive version based on SQLite using the ``sqlite3`` module included 238 | with Python. 239 | 240 | .. warning:: 241 | 242 | This code is simplistic for purposes of illustration. It's not thread-safe 243 | nor particularly suited to large loads. It's a demonstration of how you 244 | might approach things. Your Mileage May Vary.™ 245 | 246 | First, we need to create our custom ``Client``. Where you put it doesn't matter 247 | much, as long as it is importable. 248 | 249 | Each ``Client`` must have the following methods: 250 | 251 | * ``len`` 252 | * ``drop_all`` 253 | * ``push`` 254 | * ``pop`` 255 | * ``get`` 256 | 257 | .. code:: python 258 | 259 | # myapp/sqlite_backend.py 260 | import sqlite3 261 | 262 | 263 | class Client(object): 264 | def __init__(self, conn_string): 265 | # This is actually the filepath to the DB file. 266 | self.conn_string = conn_string 267 | # Kill the 'sqlite://' portion. 268 | path = self.conn_string.split('://', 1)[1] 269 | self.conn = sqlite3.connect(path) 270 | 271 | def _run_query(self, query, args): 272 | cur = self.conn.cursor() 273 | 274 | if not args: 275 | cur.execute(query) 276 | else: 277 | cur.execute(query, args) 278 | 279 | return cur 280 | 281 | def _setup_tables(self, queue_name="all"): 282 | # For manually creating the tables... 283 | query = ( 284 | "CREATE TABLE `{}` " 285 | "(task_id text, data text, delay_until integer)" 286 | ).format(queue_name) 287 | self._run_query(query, None) 288 | 289 | def len(self, queue_name): 290 | query = 'SELECT COUNT(task_id) FROM `{}`'.format(queue_name) 291 | cur = self._run_query(query, []) 292 | res = cur.fetchone() 293 | return res[0] 294 | 295 | def drop_all(self, queue_name): 296 | query = 'DELETE FROM `{}`'.format(queue_name) 297 | self._run_query(query, []) 298 | 299 | def push(self, queue_name, task_id, data, delay_until=None): 300 | if delay_until is None: 301 | delay_until = int(time.time()) 302 | 303 | query = ( 304 | "INSERT INTO `{}` (task_id, data, delay_until) VALUES (?, ?, ?)" 305 | ).format(queue_name) 306 | self._run_query(query, [task_id, data, delay_until]) 307 | return task_id 308 | 309 | def pop(self, queue_name): 310 | now = int(time.time()) 311 | query = ( 312 | "SELECT task_id, data " 313 | "FROM `{}` " 314 | "WHERE delay_until <= ?" 315 | "LIMIT 1" 316 | ).format(queue_name) 317 | cur = self._run_query(query, [now]) 318 | res = cur.fetchone() 319 | 320 | query = "DELETE FROM `{}` WHERE task_id = ?".format(queue_name) 321 | self._run_query(query, [res[0]]) 322 | 323 | return res[1] 324 | 325 | def get(self, queue_name, task_id): 326 | query = 'SELECT task_id, data FROM `{}` WHERE task_id = ?'.format( 327 | queue_name 328 | ) 329 | cur = self._run_query(query, [task_id]) 330 | res = cur.fetchone() 331 | 332 | query = 'DELETE FROM `{}` WHERE task_id = ?'.format(queue_name) 333 | self._run_query(query, [task_id]) 334 | 335 | return res[1] 336 | 337 | Now using it is simple. We'll make a ``Gator`` instance, passing our new class 338 | via the ``backend_class=...`` keyword argument. 339 | 340 | .. code:: python 341 | 342 | from alligator import Gator 343 | 344 | from myapp.sqlite_backend import Client as SQLiteClient 345 | 346 | 347 | gator = Gator('sqlite:///tmp/myapp_queue.db', backend_class=SQLiteClient) 348 | 349 | And use that ``Gator`` instance as normal! 350 | 351 | 352 | Different Workers 353 | ================= 354 | 355 | The ``Worker`` class that ships with Alligator is somewhat opinionated & 356 | simple-minded. It assumes it will be used from a command-line & can print 357 | informational messages to ``STDOUT``. 358 | 359 | However, this may not work for your purposes. To work around this, you can 360 | subclass ``Worker`` (or make your own entirely new one). 361 | 362 | For instance, let's make ``Worker`` use ``logging`` instead of ``STDOUT``. 363 | We'll swap out all the methods that ``print(...)`` for methods that log instead. 364 | 365 | .. code:: python 366 | 367 | # myapp/logworkers.py 368 | import logging 369 | 370 | from alligator import Worker 371 | 372 | 373 | log = logging.getLogger('alligator.worker') 374 | 375 | 376 | class LoggingWorker(Worker): 377 | def starting(self): 378 | ident = self.ident() 379 | log.info('{} starting & consuming "{}".'.format(ident, self.to_consume)) 380 | 381 | if self.max_tasks: 382 | log.info('{} will die after {} tasks.'.format(ident, self.max_tasks)) 383 | else: 384 | log.info('{} will never die.'.format(ident)) 385 | 386 | def stopping(self): 387 | ident = self.ident() 388 | log.info('{} for "{}" shutting down. Consumed {} tasks.'.format( 389 | ident, 390 | self.to_consume, 391 | self.tasks_complete 392 | )) 393 | 394 | def result(self, result): 395 | # Because we don't usually care about the return values. 396 | log.debug(result) 397 | 398 | As with previous ``Worker`` customizations, you won't be able to use 399 | ``latergator.py`` anymore. Instead, we'll make a script. 400 | 401 | .. code:: python 402 | 403 | # myapp/logginggator.py 404 | from alligator import Gator 405 | 406 | from myapp.logworkers import LoggingWorker 407 | 408 | 409 | gator = Gator('redis://localhost:6379/0') 410 | worker = LoggingWorker(gator) 411 | worker.run_forever() 412 | 413 | And now there's no more nasty ``STDOUT`` messages! 414 | -------------------------------------------------------------------------------- /docs/tutorial.rst: -------------------------------------------------------------------------------- 1 | .. _tutorial: 2 | 3 | ================== 4 | Alligator Tutorial 5 | ================== 6 | 7 | Alligator is a simple offline task queuing system. It enables you to take 8 | expensive operations & move them offline, either to a different process or 9 | even a whole different server. 10 | 11 | This is extremely useful in the world of web development, where request-response 12 | cycles should be kept as quick as possible. Scheduling tasks helps remove 13 | expensive operations & keeps end-users happy. 14 | 15 | Some example good use-cases for offline tasks include: 16 | 17 | * Sending emails 18 | * Resizing images/creating thumbnails 19 | * Notifying social networks 20 | * Fetching data from other data sources 21 | 22 | You should check out the instructions on :ref:`installing` to install Alligator. 23 | 24 | Alligator is written in pure Python & can work with all frameworks. For this 25 | tutorial, we'll assume integration with a `Django`_-based web application, but 26 | it could just as easily be used with `Flask`_, `Pyramid`_, pure WSGI 27 | applications, etc. 28 | 29 | .. _`Django`: http://djangoproject.com/ 30 | .. _`Flask`: http://flask.pocoo.org/ 31 | .. _`Pyramid`: http://www.pylonsproject.org/ 32 | 33 | 34 | Philosophy 35 | ========== 36 | 37 | Alligator is a bit different in approach from other offline task systems. Let's 38 | highlight some ways & the why's. 39 | 40 | **Tasks Are Any Plain Old Function** 41 | No decorators, no special logic/behavior needed inside, no inheritance. 42 | **ANY** importable Python function can become a task with no modifications 43 | required. 44 | 45 | Importantly, it must be importable. So instance methods on a class aren't 46 | processable. 47 | 48 | **Plain Old Python** 49 | Nothing specific to any framework or architecture here. Plug it in to 50 | whatever code you want. 51 | 52 | **Simplicity** 53 | The code for Alligator should be small & fast. No complex gymnastics, no 54 | premature optimizations or specialized code to suit a specific backend. 55 | 56 | **You're In Control** 57 | Your code calls the tasks & can setup all the execution options needed. 58 | There are hook functions for special processing, or you can use your 59 | own ``Task`` or ``Client`` classes. 60 | 61 | Additionally, you control the consuming of the queue, so it can be 62 | processed your way (or fanned out, or prioritized, or whatever). 63 | 64 | 65 | Figure Out What To Offline 66 | ========================== 67 | 68 | The very first thing to do is figure out where the pain points in your 69 | application are. Doing this analysis differs wildly (though things like 70 | `django-debug-toolbar`_, `profile`_ or `snakeviz`_ can be helpful). Broadly 71 | speaking, you should look for things that: 72 | 73 | * access the network 74 | * do an expensive operation 75 | * may fail & require retrying 76 | * things that aren't immediately required for success 77 | 78 | If you have a web application, just navigating around & timing pageloads can 79 | be a cheap/easy way of finding pain points. 80 | 81 | For the purposes of this tutorial, we'll assume a user of our hot new 82 | Web 3.0 social network made a new post & all their followers need to see it. 83 | 84 | So our existing view code might look like: 85 | 86 | .. code:: python 87 | 88 | from django.conf import settings 89 | from django.http import Http404 90 | from django.shortcuts import redirect, send_email 91 | 92 | from sosocial.models import Post 93 | 94 | 95 | def new_post(request): 96 | if not request.method == 'POST': 97 | raise Http404('Gotta use POST.') 98 | 99 | # Don't write code like this. Sanitize your data, kids. 100 | post = Post.objects.create( 101 | message=request.POST['message'] 102 | ) 103 | 104 | # Ugh. We're sending an email to everyone who follows the user, which 105 | # could mean hundreds or thousands of emails. This could timeout! 106 | subject = "A new post by {}".format(request.user.username) 107 | to_emails = [follow.email for follow in request.user.followers.all()] 108 | send_email( 109 | subject, 110 | post.message, 111 | settings.SERVER_EMAIL, 112 | recipient_list=to_emails 113 | ) 114 | 115 | # Redirect like a good webapp should. 116 | return redirect('activity_feed') 117 | 118 | .. _`django-debug-toolbar`: https://django-debug-toolbar.readthedocs.org/ 119 | .. _`profile`: https://docs.python.org/3.3/library/profile.html 120 | .. _`snakeviz`: https://jiffyclub.github.io/snakeviz/ 121 | 122 | 123 | Creating a Task 124 | =============== 125 | 126 | The next step won't involve Alligator at all. We'll extract that slow code into 127 | an importable function, then call it from where the code used to be. 128 | So we can convert our existing code into: 129 | 130 | .. code:: python 131 | 132 | from django.contrib.auth.models import User 133 | from django.conf import settings 134 | from django.http import Http404 135 | from django.shortcuts import redirect, send_email 136 | 137 | from sosocial.models import Post 138 | 139 | 140 | def send_post_email(user_id, post_id): 141 | post = Post.objects.get(pk=post_id) 142 | user = User.objects.get(pk=user_id) 143 | 144 | subject = "A new post by {}".format(user.username) 145 | to_emails = [follow.email for follow in user.followers.all()] 146 | send_email( 147 | subject, 148 | post.message, 149 | settings.SERVER_EMAIL, 150 | recipient_list=to_emails 151 | ) 152 | 153 | 154 | def new_post(request): 155 | if not request.method == 'POST': 156 | raise Http404('Gotta use POST.') 157 | 158 | # Don't write code like this. Sanitize your data, kids. 159 | post = Post.objects.create( 160 | message=request.POST['message'] 161 | ) 162 | 163 | # The code was here. Now we'll call the function, just to make sure 164 | # things still work. 165 | send_post_email(request.user.pk, post.pk) 166 | 167 | # Redirect like a good webapp should. 168 | return redirect('activity_feed') 169 | 170 | Now go run your tests or hand-test things to ensure they still work. This is 171 | important because it helps guard against regressions in your code. 172 | 173 | You'll note we're not directly passing the ``User`` or ``Post`` instances, 174 | instead passing the primary identifiers, even as it stands it's causing two 175 | extra queries. While this is sub-optimal as things stands, it neatly prepares 176 | us for offlining the task. 177 | 178 | .. note:: 179 | 180 | **Why not pass the instances themselves?** 181 | 182 | While it's possible to create instances that nicely serialize, the problem 183 | with this approach is stale data & unnecessarily large payloads. 184 | 185 | While the ideal situation is tasks that are processed within seconds of 186 | being added to the queue, in the real world, queues can get backed up & 187 | users may further change data. By fetching the data fresh when processing 188 | the task, you ensure you're not working with old data. 189 | 190 | Further, most queues are optimized for small payloads. The more data to 191 | send over the wire, the slower things go. Given that's the opposite reason 192 | for adding a task queue, it doesn't make sense. 193 | 194 | 195 | Create a Gator Instance 196 | ======================= 197 | 198 | While it's great we got better encapsulation by pulling out the logic into 199 | its own function, we're still doing the sending of email in-process, which means 200 | our view is still slow. 201 | 202 | This is where Alligator comes in. We'll start off by importing the ``Gator`` 203 | class at the top of the file & making an instance. 204 | 205 | .. note:: 206 | 207 | Unless you're only using Alligator in **one** file, a best practice would 208 | be to put that import & initialization into it's own file, then import that 209 | configured ``gator`` object into your other files. Configuring it in one 210 | place is better than many instantiations (but also allows for setting 211 | up a different instance elsewhere). 212 | 213 | When creating a ``Gator`` instance, you'll need to choose a queue backend. 214 | Alligator ships with support for local-memory, Redis & SQS. See the 215 | :ref:`installing` docs for setup info. 216 | 217 | Local Memory 218 | ------------ 219 | 220 | Primarily only for development or in testing, this has no dependencies, but 221 | keeps everything in-process. 222 | 223 | .. code:: python 224 | 225 | from alligator import Gator 226 | 227 | # Creates an in-memory/in-process queue. 228 | # The same process must consume from the queue, or things will be thrown 229 | # away when the process exits. 230 | gator = Gator('locmem://') 231 | 232 | 233 | Redis 234 | ----- 235 | 236 | Redis is a good option for production and small-large installations. 237 | 238 | .. code:: python 239 | 240 | from alligator import Gator 241 | 242 | # Connect to a locally-running Redis server & use DB 0. 243 | gator = Gator('redis://localhost:6379/0') 244 | 245 | 246 | SQS 247 | --- 248 | 249 | `Amazon SQS`_ is specifically a queue service & works well in large-scale 250 | environments. 251 | 252 | .. code:: python 253 | 254 | from alligator import Gator 255 | 256 | # Connect to the globally available SQS service. 257 | gator = Gator('sqs://us-west-2/') 258 | 259 | 260 | **For the duration of the tutorial, we'll assume you chose Redis.** 261 | 262 | .. _`Amazon SQS`: http://aws.amazon.com/sqs/ 263 | 264 | 265 | SQLite 266 | ------ 267 | 268 | SQLite excels in small/light loads & simple setups (or development). 269 | 270 | .. code:: python 271 | 272 | from alligator import Gator 273 | 274 | # Setup the SQLite database & the `all` queue table. 275 | gator = Gator('sqlite:///var/data/sqlite/my_queue.db') 276 | 277 | # This only needs to be run *once* per-queue. 278 | gator.backend.setup_tables("all") 279 | 280 | 281 | Put the Task on the Queue 282 | ========================= 283 | 284 | After we make a ``Gator`` instance, the only other change is to how we call 285 | ``send_post_email``. Instead of calling it directly, we'll need to enqueue 286 | a task. 287 | 288 | There are two common ways of creating a task in Alligator: 289 | 290 | ``gator.task()`` 291 | A typical function call. You pass in the callable & the 292 | ``*args``/``**kwargs`` to provide to the callable. It gets put on the 293 | queue with the default task execution options. 294 | 295 | ``gator.options()`` 296 | Creates a context manager that has a ``.task()`` method that works 297 | like the above. This is useful for controlling the task execution options, 298 | such as retries or if the task should be asynchronous. See the "Working 299 | Around Failsome Tasks" section below. 300 | 301 | Since we're just starting out with Alligator & looking to replicate the 302 | existing behavior, we'll use ``gator.task(...)`` to create & enqueue the task. 303 | 304 | .. code:: python 305 | 306 | # Old code 307 | send_post_email(request.user.pk, post.pk) 308 | 309 | # New code 310 | gator.task(send_post_email, request.user.pk, post.pk) 311 | 312 | Hardly changed in code, but a world of difference in execution speed. Rather 313 | than blasting out hundreds of emails & possibly timing out, a task is placed on 314 | the queue & execution continues quickly. The complete code looks like: 315 | 316 | .. code:: python 317 | 318 | from alligator import Gator 319 | 320 | from django.contrib.auth.models import User 321 | from django.conf import settings 322 | from django.http import Http404 323 | from django.shortcuts import redirect, send_email 324 | 325 | from sosocial.models import Post 326 | 327 | 328 | # Please configure this once & import it elsewhere. 329 | # Bonus points if you use a settings (e.g. ``settings.ALLIGATOR_DSN``) 330 | # instead of a hard-coded string. 331 | gator = Gator('redis://localhost:6379/0') 332 | 333 | def send_post_email(user_id, post_id): 334 | post = Post.objects.get(pk=post_id) 335 | user = User.objects.get(pk=user_id) 336 | 337 | subject = "A new post by {}".format(user.username) 338 | to_emails = [follow.email for follow in user.followers.all()] 339 | send_email( 340 | subject, 341 | post.message, 342 | settings.SERVER_EMAIL, 343 | recipient_list=to_emails 344 | ) 345 | 346 | 347 | def new_post(request): 348 | if not request.method == 'POST': 349 | raise Http404('Gotta use POST.') 350 | 351 | # Don't write code like this. Sanitize your data, kids. 352 | post = Post.objects.create( 353 | message=request.POST['message'] 354 | ) 355 | 356 | # The function call was here. Now we'll create a task then carry on. 357 | gator.task(send_post_email, request.user.pk, post.pk) 358 | 359 | # Redirect like a good webapp should. 360 | return redirect('activity_feed') 361 | 362 | 363 | Running a Worker 364 | ================ 365 | 366 | Time to kick back, relax & enjoy your speedy new site, right? 367 | 368 | Unfortunately, not quite. Now we're successfully queuing up tasks for later 369 | processing & things are completing quickly, but *nothing is processing those 370 | tasks*. So we need to run a ``Worker`` to consume the queued tasks. 371 | 372 | We have two options here. We can either use the included ``latergator.py`` 373 | script or we can create our own. The following are identical in function: 374 | 375 | .. code:: bash 376 | 377 | $ latergator.py redis://localhost:6379/0 378 | 379 | Or... 380 | 381 | .. code:: python 382 | 383 | # Within something like ``run_tasks.py``... 384 | from alligator import Gator, Worker 385 | 386 | # Again, bonus points for an import and/or settings usage. 387 | gator = Gator('redis://localhost:6379/0') 388 | 389 | worker = Worker(gator) 390 | worker.run_forever() 391 | 392 | Both of these will create a long-running process, which will consume tasks off 393 | the queue as fast as they can. 394 | 395 | While this is fine to start off, if you have a heavily trafficked site, you'll 396 | likely need many workers. Simply start more processes (using a tool like 397 | `Supervisor`_ works best). 398 | 399 | You can also make things like management commands, build other custom tooling 400 | around processing or even launch workers on their own dedicated servers. 401 | 402 | .. _`Supervisor`: http://supervisord.org/ 403 | 404 | 405 | Working Around Failsome Tasks 406 | ============================= 407 | 408 | Sometimes tasks don't always succeed on the first try. Maybe the database is 409 | down, the mail server isn't working or a remote resource can't be loaded. As it 410 | stands, our task will try once then fail loudly. 411 | 412 | Alligator also supports retrying tasks, as well as having an ``on_error`` hook. 413 | To specify we want retries, we'll have to use the other important bit of 414 | Alligator, ``Gator.options``. 415 | 416 | ``Gator.options`` gives you a context manager & allows you to configure task 417 | execution options that then apply to all tasks within the manager. Using that 418 | looks like: 419 | 420 | .. code:: python 421 | 422 | # Old code 423 | # gator.task(send_post_email, request.user.pk, post.pk) 424 | 425 | # New code 426 | with gator.options(retries=3) as opts: 427 | # Be careful to use ``opts.task``, not ``gator.task`` here! 428 | opts.task(send_post_email, request.user.pk, post.pk) 429 | 430 | Now that task will get three retries when it's processed, making network 431 | failures much more tolerable. 432 | 433 | 434 | Delaying/Scheduling Tasks 435 | ========================= 436 | 437 | You can also choose to either delay the execution of a task by a set number 438 | of seconds **OR** schedule when the task can first run. 439 | 440 | To delay a task, just add the ``delay_by`` parameter: 441 | 442 | .. code:: python 443 | 444 | # Don't execute this task for at least 5 minutes. 445 | with gator.options(delay_by=60 * 5): 446 | opts.task(send_post_email, request.user.pk, post.pk) 447 | 448 | To schedule a task in the future, you'll need to provide the ``delay_until`` 449 | parameter, set to a future Unix timestamp: 450 | 451 | .. code:: python 452 | 453 | import time 454 | 455 | # There are lots of ways to compute a future timestamp. 456 | # For our purposes here, let's just do some simple math. 457 | tomorrow = time.time() + (60 * 60 * 24) 458 | 459 | with gator.options(delay_until=tomorrow): 460 | opts.task(send_post_email, request.user.pk, post.pk) 461 | 462 | 463 | Testing Tasks 464 | ============= 465 | 466 | All of this is great, but if you can't test the task, you might as well not 467 | have code. 468 | 469 | Alligator supports an ``is_async=False`` option, which means that 470 | rather than being put on the queue, your task runs right away (acting like you 471 | just called the function, but with all the retries & hooks included). 472 | 473 | .. code:: python 474 | 475 | # Bonus points for using ``settings.DEBUG`` (or similar) instead of a 476 | # hard-coded ``False``. 477 | with gator.options(is_async=False) as opts: 478 | opts.task(send_post_email, request.user.pk, post.pk) 479 | 480 | Now your existing integration tests (from before converting to offline tasks) 481 | should work as expected. 482 | 483 | .. warning:: 484 | 485 | Make sure you don't accidently commit this & deploy to production. If 486 | so, why have an offline task system at all? 487 | 488 | Additionally, you get naturally improved ability to test, because now your 489 | tasks are just plain old functions. This means you can typically just import 490 | the function & write tests against it (rather than the whole view), which 491 | makes for better unit tests & fewer integration tests to ensure things work 492 | right. 493 | 494 | 495 | Going Beyond 496 | ============ 497 | 498 | This is 90%+ of the day-to-day usage of Alligator, but there's plenty more 499 | you can do with it. 500 | 501 | You may wish to peruse the :ref:`bestpractices` docs for ideas on how to keep 502 | your Alligator clean & flexible. 503 | 504 | If you need more custom functionality, the :ref:`extending` docs have 505 | examples on: 506 | 507 | * Customizing task behavior using the ``on_start/on_success/on_error`` hook 508 | functions. 509 | * Custom ``Task`` classes. 510 | * Multiple queues & ``Workers`` for scalability. 511 | * Custom backends. 512 | * ``Worker`` subclasses. 513 | 514 | Happy queuing! 515 | --------------------------------------------------------------------------------