├── ci ├── content │ ├── README.md │ ├── styles │ │ └── errata │ │ │ └── Spelling.yml │ ├── .vale.ini │ ├── vocab.txt │ └── cmd.sh ├── structure │ ├── README.md │ ├── .remarkrc.json │ ├── schema.json │ ├── frontmatter.js │ └── cmd.sh ├── accessibility │ ├── README.md │ └── cmd.sh ├── util.sh └── errata.sh ├── .gitignore ├── requirements.txt ├── accepted ├── index.rst └── README.rst ├── rejected ├── index.rst └── README.rst ├── superseded ├── index.rst └── README.rst ├── withdrawn ├── index.rst └── README.rst ├── draft ├── celery-5-architecture-figure01.png ├── celery-nextgen-architecture-figure01.png ├── README.rst ├── index.rst ├── celery-5-architecture ├── celery-nextgen-architecture ├── router.rst ├── controller.rst ├── publisher.rst ├── configuration.rst ├── observability.rst ├── execution-platform.rst ├── features-release-schedule.rst ├── cli-refactor.rst ├── celery-kubernetes-operator.rst ├── high-level-architecture.rst └── celery-5-high-level-architecture.rst ├── final ├── index.rst ├── README.rst └── 0001-cep-process.rst ├── README.rst ├── .travis.yml ├── index.rst ├── Makefile ├── conf.py ├── template.rst └── glossary.rst /ci/content/README.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /ci/structure/README.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /ci/accessibility/README.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | _build/ 2 | .ropeproject/ 3 | .idea/ -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | Sphinx>=2.2.0 2 | sphinx_celery==2.0.0 3 | -------------------------------------------------------------------------------- /accepted/index.rst: -------------------------------------------------------------------------------- 1 | .. include:: README.rst 2 | 3 | .. toctree:: 4 | :maxdepth: 2 5 | :caption: Contents: 6 | -------------------------------------------------------------------------------- /rejected/index.rst: -------------------------------------------------------------------------------- 1 | .. include:: README.rst 2 | 3 | .. toctree:: 4 | :maxdepth: 2 5 | :caption: Contents: 6 | -------------------------------------------------------------------------------- /superseded/index.rst: -------------------------------------------------------------------------------- 1 | .. include:: README.rst 2 | 3 | .. toctree:: 4 | :maxdepth: 2 5 | :caption: Contents: 6 | -------------------------------------------------------------------------------- /withdrawn/index.rst: -------------------------------------------------------------------------------- 1 | .. include:: README.rst 2 | 3 | .. toctree:: 4 | :maxdepth: 2 5 | :caption: Contents: 6 | -------------------------------------------------------------------------------- /draft/celery-5-architecture-figure01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/celery/ceps/master/draft/celery-5-architecture-figure01.png -------------------------------------------------------------------------------- /draft/celery-nextgen-architecture-figure01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/celery/ceps/master/draft/celery-nextgen-architecture-figure01.png -------------------------------------------------------------------------------- /final/index.rst: -------------------------------------------------------------------------------- 1 | .. include:: README.rst 2 | 3 | .. toctree:: 4 | :maxdepth: 2 5 | :caption: Contents: 6 | 7 | 0001-cep-process.rst 8 | -------------------------------------------------------------------------------- /ci/accessibility/cmd.sh: -------------------------------------------------------------------------------- 1 | set -e 2 | 3 | errata_print "TEST 5: Installing and running a11y ..." 4 | npm install --silent -g a11y && a11y $2'/**/*.html' -------------------------------------------------------------------------------- /ci/content/styles/errata/Spelling.yml: -------------------------------------------------------------------------------- 1 | extends: spelling 2 | message: "Did you really mean '%s'?" 3 | level: error 4 | ignore: ci/content/vocab.txt 5 | -------------------------------------------------------------------------------- /ci/structure/.remarkrc.json: -------------------------------------------------------------------------------- 1 | { 2 | "plugins": [ 3 | "remark-preset-lint-markdown-style-guide", 4 | "remark-lint-no-dead-urls" 5 | ] 6 | } 7 | -------------------------------------------------------------------------------- /final/README.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | Final CEPs 4 | ========== 5 | 6 | CEPs that have been approved and fully implemented. 7 | See `CEP 1 <../final/0001-cep-process.rst>`_ for details. 8 | -------------------------------------------------------------------------------- /withdrawn/README.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | Withdrawn CEPs 4 | ============== 5 | 6 | CEPs that have been withdrawn by their authors. 7 | See `CEP 1 <../final/0001-cep-process.rst>`_ for details. 8 | -------------------------------------------------------------------------------- /rejected/README.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | Rejected CEPs 4 | ============= 5 | 6 | CEPs that have been rejected by the Technical Board. 7 | See `CEP 1 <../final/0001-cep-process.rst>`_ for details. 8 | -------------------------------------------------------------------------------- /superseded/README.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | Superseded CEPs 4 | =============== 5 | 6 | CEPs that have been replaced/superseded by newer CEPs. 7 | See `CEP 1 <../final/0001-cep-process.rst>`_ for details. 8 | -------------------------------------------------------------------------------- /draft/README.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | Draft CEPs 4 | ========== 5 | 6 | CEPs that are still in the process of being drafted and have not yet been 7 | approved or denied. See `CEP 1 <../final/0001-cep-process.rst>`_ for details. 8 | -------------------------------------------------------------------------------- /accepted/README.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | Approved CEPs 4 | ============= 5 | 6 | CEPs that have been approved by the Technical Board and are in the 7 | process of being implemented. See `CEP 1 <../final/0001-cep-process.rst>`_ for 8 | details. 9 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Celery Enhancement Proposals (CEPs) 2 | =================================== 3 | 4 | .. image:: https://travis-ci.com/celery/ceps.svg?branch=master 5 | :target: https://travis-ci.com/celery/ceps 6 | 7 | Celery Enhancement Proposals are a formal way of proposing large feature additions to the Celery Project (http://www.celeryproject.org/). 8 | 9 | See `CEP 1 `_ for details. 10 | -------------------------------------------------------------------------------- /draft/index.rst: -------------------------------------------------------------------------------- 1 | .. include:: README.rst 2 | 3 | .. toctree:: 4 | :maxdepth: 2 5 | :caption: Contents: 6 | 7 | celery-5-high-level-architecture.rst 8 | features-release-schedule.rst 9 | high-level-architecture.rst 10 | controller.rst 11 | publisher.rst 12 | router.rst 13 | execution-platform.rst 14 | configuration.rst 15 | observability.rst 16 | celery-kubernetes-operator.rst 17 | cli-refactor.rst 18 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: node_js 2 | dist: xenial 3 | node_js: 4 | - stable 5 | 6 | cache: 7 | npm: true 8 | directories: 9 | - $HOME/.cache/pip 10 | 11 | install: 12 | - pyenv global 3.8 13 | - pip3 install --user -r requirements.txt 14 | - npm install prettyjson 15 | 16 | script: 17 | - bash ci/errata.sh pre ./ content/ 18 | - rm -rf content/ 19 | - mkdir _static/ 20 | - make html linkcheck -j2 21 | - bash ci/errata.sh post _build/html 22 | -------------------------------------------------------------------------------- /index.rst: -------------------------------------------------------------------------------- 1 | ================================ 2 | Welcome to CEPs's documentation! 3 | ================================ 4 | 5 | .. include:: README.rst 6 | 7 | .. toctree:: 8 | :maxdepth: 2 9 | :caption: Contents: 10 | 11 | glossary.rst 12 | final/index.rst 13 | accepted/index.rst 14 | draft/index.rst 15 | rejected/index.rst 16 | superseded/index.rst 17 | withdrawn/index.rst 18 | template.rst 19 | 20 | 21 | 22 | Indices and tables 23 | ================== 24 | 25 | * :ref:`genindex` 26 | * :ref:`search` 27 | -------------------------------------------------------------------------------- /ci/content/.vale.ini: -------------------------------------------------------------------------------- 1 | StylesPath = styles 2 | MinAlertLevel = warning 3 | SkippedScopes = script, style, pre, figure, blockquote 4 | 5 | [*.rst] 6 | # `errata` represents your custom house style (feel free to re-name it). 7 | # 8 | # You can add your own rules at `/styles/errata`. 9 | BasedOnStyles = proselint, write-good, Joblint, errata 10 | 11 | # Catch flag of repeated words (e.g., 'the the') as warnings. 12 | # 13 | # NOTE: We use 'warning' here because there are some edge cases that we don't 14 | # want to be marked as errors (e.g., 'that that'). 15 | vale.Repetition = warning 16 | -------------------------------------------------------------------------------- /ci/structure/schema.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "object", 3 | "properties": { 4 | "tags": { 5 | "type": "array", 6 | "items": {"type": "string"} 7 | }, 8 | "categories": { 9 | "type": "array", 10 | "items": {"type": "string"} 11 | }, 12 | "title": {"type": "string"}, 13 | "date": {"type": "string"}, 14 | 15 | "highlight": {"type": "boolean"}, 16 | "math": {"type": "boolean"}, 17 | "draft": {"type": "boolean"}, 18 | "disable_comments": {"type": "boolean"}, 19 | "preview": {"type": "boolean"} 20 | 21 | }, 22 | "required": ["title"] 23 | } 24 | -------------------------------------------------------------------------------- /ci/content/vocab.txt: -------------------------------------------------------------------------------- 1 | amqp 2 | AMQP 3 | argparse 4 | Armin 5 | async 6 | asyncio 7 | autoscale 8 | Autoscaler 9 | autoscaling 10 | awaitable 11 | Backends 12 | barebones 13 | BigQuery 14 | ceps 15 | Config 16 | cooldown 17 | coroutine 18 | coroutines 19 | cryptographically 20 | deduplication 21 | Deduplication 22 | deserialization 23 | dev 24 | devs 25 | Docopt 26 | dsl 27 | enqueues 28 | entrypoints 29 | Etcd 30 | Github 31 | http 32 | https 33 | journald 34 | Katz 35 | Mesos 36 | Omer 37 | png 38 | Pre 39 | prefetching 40 | Preload 41 | repl 42 | repo 43 | Rigo 44 | runtimes 45 | Seemann 46 | Starmaps 47 | StatsD 48 | stdout 49 | systemd 50 | SystemD 51 | trustme 52 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = -W 6 | SPHINXBUILD = sphinx-build 7 | SOURCEDIR = . 8 | BUILDDIR = _build 9 | 10 | # Put it first so that "make" without argument is like "make help". 11 | help: 12 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 13 | 14 | linkcheck: 15 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" -b linkcheck $(O) 16 | 17 | .PHONY: help Makefile 18 | 19 | # Catch-all target: route all unknown targets to Sphinx using the new 20 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 21 | %: Makefile 22 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 23 | -------------------------------------------------------------------------------- /ci/structure/frontmatter.js: -------------------------------------------------------------------------------- 1 | var glob = require('glob') 2 | var matter = require('gray-matter') 3 | var toml = require('toml') 4 | var Ajv = require('ajv') 5 | var prettyjson = require('prettyjson') 6 | 7 | var schema = require('./schema.json') 8 | if (process.argv[3] === 'TOML') { 9 | options = { 10 | engines: { 11 | toml: toml.parse.bind(toml) 12 | }, 13 | language: 'toml', 14 | delims: [process.argv[4], process.argv[4]] 15 | } 16 | } else { 17 | options = { 18 | language: process.argv[3].toLowerCase(), 19 | delims: [process.argv[4], process.argv[4]] 20 | } 21 | } 22 | 23 | glob(process.argv[2] + '/*.md', {}, function (er, files) { 24 | var ajv = new Ajv() 25 | files.forEach(function (file, index) { 26 | var fm = matter.read(file, options) 27 | var valid = ajv.validate(schema, fm.data) 28 | if (!valid) { 29 | console.log(prettyjson.render({'src': file, 'errors': ajv.errors})) 30 | return process.exit(1) 31 | } 32 | }) 33 | }) 34 | -------------------------------------------------------------------------------- /ci/util.sh: -------------------------------------------------------------------------------- 1 | ############################## UTILITY FUNCTIONS ############################## 2 | 3 | # Print to given argument to stdout, followed by a newline. 4 | errata_print () { 5 | printf "\\n" && printf %"s\\n" "$1" 6 | } 7 | 8 | # Run the given command with its output suppressed. 9 | errata_quiet_cmd () { 10 | "$1" &> /dev/null 11 | } 12 | 13 | errata_banner_print () { 14 | printf '#%.0s' {1..100} 15 | } 16 | 17 | ################################ DEPENDENCIES ################################# 18 | 19 | if which node > /dev/null && which npm > /dev/null 20 | then 21 | echo "Found Node.js ($(node -v)) and npm ($(npm -v))." 22 | else 23 | export NVM_DIR="$HOME/.nvm" 24 | # `NVM` is our version of `nvm` (https://github.com/creationix/nvm). 25 | NVM="0.33.11" 26 | 27 | NVM_SH="https://raw.githubusercontent.com/creationix/nvm/v$NVM/install.sh" 28 | errata_quiet_cmd "curl -o- $NVM_INSTALL | bash" 29 | 30 | # shellcheck source=/dev/null 31 | [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" 32 | nvm install 10.11.0 33 | fi -------------------------------------------------------------------------------- /ci/content/cmd.sh: -------------------------------------------------------------------------------- 1 | set -e 2 | 3 | # Section 1: blocktest (https://github.com/errata-ai/blocktest) 4 | # 5 | # These tests ensure that all of our code examples are up-to-date and 6 | # functional. 7 | ba="https://github.com/errata-ai/blocktest/releases/download/" 8 | 9 | if [ "${OS}" == "Linux" ] 10 | then 11 | bb="v${BLOCKTEST}/trust-v${BLOCKTEST}-x86_64-unknown-linux-gnu.tar.gz" 12 | else 13 | bb="v${BLOCKTEST}/trust-v${BLOCKTEST}-x86_64-apple-darwin.tar.gz" 14 | fi 15 | 16 | bc=$ba$bb 17 | 18 | errata_print 'TEST 1: Installing & running blocktest ...' 19 | curl -sL $bc | tar xz && ./blocktest $2 .rst $3 20 | 21 | # Section 2: Vale (https://github.com/errata-ai/vale) 22 | # 23 | # These tests relate to our actual written content. We use Vale to check our 24 | # spelling and, optionally, our adherence to an external style guide 25 | # (see the `.vale.ini` file for more details). 26 | 27 | # Install Vale using the version defined in `errata.sh`: 28 | va="https://github.com/errata-ai/vale/releases/download/" 29 | vb="v${VALE}/vale_${VALE}_${OS}_64-bit.tar.gz" 30 | vc=$va$vb 31 | 32 | errata_print 'TEST 2: Installing & running Vale ...' 33 | curl -sL $vc | tar xz && ./vale --config='ci/content/.vale.ini' $3 34 | -------------------------------------------------------------------------------- /ci/structure/cmd.sh: -------------------------------------------------------------------------------- 1 | set -e 2 | 3 | # Section 1: Front Matter 4 | # 5 | # The first structure test we run is for our front matter. 6 | # 7 | # We use the Node.js script `frontmatter.js` to define a JSON Schema that we then 8 | # check our front matter against. 9 | errata_print "TEST 3: Front Matter ..." 10 | 11 | npm install --silent gray-matter # Extract the front matter. 12 | npm install --silent glob # Find the relevant files. 13 | npm install --silent toml # Used by `gray-matter`. 14 | npm install --silent ajv # Validate our schema. 15 | 16 | node ci/structure/frontmatter.js $3 $FM_STYLE $FM_DELIM 17 | 18 | 19 | # Section 2: Markup Style 20 | # 21 | # These tests relate to the structure (i.e., not the actual written content) of 22 | # our markup (Markdown only, for now). 23 | # 24 | # See the `.markdownlint.json` file for more details. 25 | # 26 | # See https://github.com/igorshubovych/markdownlint-cli. 27 | errata_print "TEST 4: Installing & running remark-lint ..." 28 | 29 | npm install -g remark-cli &> /dev/null 30 | 31 | # Install plugins: 32 | 33 | # See https://github.com/cirosantilli/markdown-style-guide/ 34 | npm install -g remark-preset-lint-markdown-style-guide &> /dev/null 35 | # Ensure external links are working 36 | npm install -g remark-lint-no-dead-urls &> /dev/null 37 | 38 | remark --quiet --rc-path="ci/structure/.remarkrc.json" $3 39 | 40 | -------------------------------------------------------------------------------- /conf.py: -------------------------------------------------------------------------------- 1 | # Configuration file for the Sphinx documentation builder. 2 | # 3 | # This file only contains a selection of the most common options. For a full 4 | # list see the documentation: 5 | # http://www.sphinx-doc.org/en/master/config 6 | 7 | # -- Path setup -------------------------------------------------------------- 8 | 9 | # If extensions (or modules to document with autodoc) are in another directory, 10 | # add these directories to sys.path here. If the directory is relative to the 11 | # documentation root, use os.path.abspath to make it absolute, like shown here. 12 | # 13 | # import os 14 | # import sys 15 | # sys.path.insert(0, os.path.abspath('.')) 16 | 17 | 18 | # -- Project information ----------------------------------------------------- 19 | 20 | project = 'CEPs' 21 | copyright = "2019, Celery's Techincal Board" 22 | author = "Celery's Techincal Board" 23 | 24 | 25 | # -- General configuration --------------------------------------------------- 26 | 27 | # Add any Sphinx extension module names here, as strings. They can be 28 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 29 | # ones. 30 | extensions = [ 31 | 'sphinx.ext.intersphinx', 32 | 'sphinx.ext.todo', 33 | 'sphinx.ext.coverage', 34 | 'sphinx.ext.imgmath', 35 | 'sphinx.ext.ifconfig', 36 | 'sphinx.ext.viewcode', 37 | 'sphinx.ext.extlinks', 38 | 'sphinx.ext.autosectionlabel', 39 | ] 40 | 41 | autosectionlabel_prefix_document = True 42 | 43 | # Add any paths that contain templates here, relative to this directory. 44 | templates_path = ['_templates'] 45 | 46 | # List of patterns, relative to source directory, that match files and 47 | # directories to ignore when looking for source files. 48 | # This pattern also affects html_static_path and html_extra_path. 49 | exclude_patterns = [ 50 | '_build', 51 | 'Thumbs.db', 52 | '.DS_Store', 53 | 'README.rst', 54 | '**/README.rst' 55 | ] 56 | 57 | 58 | # -- Options for HTML output ------------------------------------------------- 59 | 60 | # The theme to use for HTML and HTML Help pages. See the documentation for 61 | # a list of builtin themes. 62 | # 63 | html_theme = 'sphinx_celery' 64 | 65 | # Add any paths that contain custom static files (such as style sheets) here, 66 | # relative to this directory. They are copied after the builtin static files, 67 | # so a file named "default.css" will overwrite the builtin "default.css". 68 | html_static_path = ['_static'] 69 | 70 | 71 | # -- Extension configuration ------------------------------------------------- 72 | 73 | # -- Options for intersphinx extension --------------------------------------- 74 | 75 | # Example configuration for intersphinx: refer to the Python standard library. 76 | intersphinx_mapping = { 77 | 'python': ('https://docs.python.org/', None), 78 | 'celery4': ('http://docs.celeryproject.org/en/v4.3.0', None) 79 | } 80 | -------------------------------------------------------------------------------- /ci/errata.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # 3 | # errata.sh - A full-featured, self-contained script for documentation QA. 4 | # 5 | # https://github.com/errata-ai/errata.sh 6 | # 7 | # The MIT License (MIT) 8 | # 9 | # Copyright (c) 2018 Joseph Kato 10 | # 11 | # Permission is hereby granted, free of charge, to any person obtaining a copy 12 | # of this software and associated documentation files (the "Software"), to deal 13 | # in the Software without restriction, including without limitation the rights 14 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 15 | # copies of the Software, and to permit persons to whom the Software is 16 | # furnished to do so, subject to the following conditions: 17 | # 18 | # The above copyright notice and this permission notice shall be included in all 19 | # copies or substantial portions of the Software. 20 | # 21 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 22 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 23 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 24 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 25 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 26 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 27 | # SOFTWARE. 28 | 29 | ############################## GLOBAL VARIABLES ############################### 30 | 31 | # [TODO]: Specify your OS 32 | # 33 | # "Linux" or "macOS" 34 | OS="Linux" 35 | 36 | # [TODO]: Specify the version of `vale` and `blocktest` to use. 37 | # 38 | # See `/content/` for configuration details. 39 | VALE="2.2.0" 40 | BLOCKTEST="0.1.1" 41 | 42 | # [TODO]: Specify the type of front matter you're using. 43 | # 44 | # See `/content/` for configuration details. 45 | FM_STYLE="YAML" # YAML, TOML, or JSON 46 | FM_DELIM="---" # What delimiter are you using (e.g., "---")? 47 | 48 | #################################### STEPS #################################### 49 | 50 | # Inlude our utility functions. 51 | source "ci/util.sh" 52 | 53 | if [ "$1" == "pre" ] 54 | then 55 | # Step 1: Prose & Code 56 | # 57 | # In this step, we test three aspects of our documentation: 58 | # 59 | # 1. Spelling: We check our spelling via Vale using a custom; 60 | # 2. Style: we check that our docs adhere to our style (via Vale); and 61 | # 3. Code: we check that our code examples are working (via blocktest). 62 | # 63 | # See `/content` for more information. 64 | source "ci/content/cmd.sh" 65 | 66 | # Step 2: Markup Style 67 | # 68 | # In this step, we ensure that our markup style is consistent and 69 | # readable. See `/structure` for more information. 70 | source "ci/structure/cmd.sh" 71 | else 72 | # Step 3: Accessibility 73 | # 74 | source "ci/accessibility/cmd.sh" 75 | fi 76 | -------------------------------------------------------------------------------- /draft/celery-5-architecture: -------------------------------------------------------------------------------- 1 | 5Vvbdps6EP0ar9U+JIs7+DG2kzar7Ukb95w05w2DsIkxcoVI7H59JRAgEDgktgNu89CiYbjN7Nkz0sgDdbzafED2evEFuiAYKJK7GaiTgaLIqmKS/6hkm0pM00oFc+S7TKkQTP1fgAklJo19F0QlRQxhgP11WejAMAQOLslshOBTWc2DQfmpa3sOBMHUsQNReue7eMGksiQVJz4Cf75gj7Z0dmJmO8s5gnHInjdQVC/5S0+v7OxeTD9a2C584kTq5UAdIwhxerTajEFAbZuZLb3uquFs/t4IhLjNBd+ub7b3Fx8efnpP32If/rw92y7P2F0e7SBm9iBXAbQlsjEMMYIBGQ0UIyCPGM3o0ZwesS/C28yKjwBhnxj1IvDnIRFhuCZqNhsFwKNXRWvb8cP5d3puYhWCz8npiVJIbpm95US2sNf0MU48A3SYQojgRx25PiKY8CF9SARjavGRR957yt5MI+MFXgVUP39t3mTZ95PXBxtOxEz4AcAVwIk5srMacyeDu26w8RMHHovJFhxuFJMJbQbYeX7vwmnkgPktGf4bAXQze6CwV6TAnoEggxoXJqnoH4h9L73Xbo9rEvd9DCKcI4FLooMNIcILOIehHVwW0lECeUAfQT1Q6HyGiV+pnR8AxlsW6naMYdkLYOPjH+xyenzPHU+omRUrH1IrD3U2+gqQT0xGAMkuIADd/uAHya3O1WyY3C0/mdxM2gUDAiHkgB3GyyxvoznAu4yc6lFj7gQVAoGN/ccyFdUBJLn0AiF7yymsoR/iiLvzVyoosKpJegmr2rDCEBV9S9pH3ZCtCpTT9y2AnX94HdbJkIN7PYU5n/6//m+haUv36frT1fZybGnOmawIbPQ3gDihufTtFIuNxzCAKDGBenU1GRpGDvYKEg+G/lqPDPcEewMbtstoSnNGuyUZgmazvzV3qUo5YvMa59ncZb08d7XzltrsrTuIln+ztzS17C11KHpLkWq8pWnH8pbWH6o9N3myLVFtN6WB1rY0kHtRG+TIybI3821Tshf0h3tl+5Z4E9hh6iyAGwc1vJDHLlytYZi4r4EVGAmoxmHiVDYrlqljVaUmTo1jhakumO3yESTOvA7nIEpYTDFIkbC5qJvLdWZJtcJ4+VyLt6T2lpZUzf4QnnoUwnNYyTijWHk1+VktyU8/NPft512rP97tWzpr69FsGa8nLrUE6pvY2C4zX1/pLl8S7Yzushfgy3G4Wtmhy+WLU7Ck1bUl1R5Vyno7apHL1KK3WH/g1hsk8jceU7kfBJzctYHlOdTzGMEl4M4YjgVm3l4klbn91PKO0Sk4ckDcc2cODI4+gqBnqUoVyfYLiCLarlKkEbUTehe9F5By2EUGo7okpOkicw5rmFM/GnNqp8KcVZDrwHK1OpBbykw1dqa650GutgS51S+Mi+t7X+MZudWU/lvtKr67IYfSQxzRqWqiJ4Hz+bmoeAfouY8QLt+36E0eNmJMQ65UbTURI79lxGhqlxEjvyBiGqYxpWRC6zQ+nchWPq62TioB6Hme4tRmGdeYGfp+AZgt2z0bgGpXAVjbihFX0MYI2BhEAmQI/HHZwwhE/i97lihQc7MlRaKtjwb6hCvuHWJO6pVRdXF+5btuArekeT3Kd0yUvUb3TLy0oSUEbL5DhL3xIN94wbt7RwA1RvyZdK6UG6nZtomXLdEWa6qZCvS8iICp6uRDrJyKGzuuIAJkKic4vj8Lp3WztmMtnNa3kiXBOn1oJZd41Hq2kfyKidzxGsn1dm7LpVI9gN6GO3fsjSLFSeBHixNvT9ZwqODNxujN98Fl7Um9JnoPtRGqpcN6NHU4znpuB7FqiLFaq2d2GaqGEKrX4SzZmlmsW363oyUpe6RvMchaHQfKgnvFkVbJgrpsdp0Fu12e+jPDyGwZRulssqs4MoU4olHDBdHlBjhxpYE8BejRJx9/pJbyYaNrqHQdXZ3O1v/Q6Bq2jC6jy+AaivUkDKN4lWywF4KprxGkaZ1H0B/ftu9iSqa3TVCdFnqyuLfpJsbVUu8WRHFAd7/lElb0nUCGMvTO6z+xChANys2AT7IQMOumq29qZqULFnuGWfbo4r+GHw/JX60L7D3p61W7bbXKYonJytCm3bZVfav6276K/lDbqb/37txaS4q9vZwmTqy3kMO+kWja9Baao7yRlUgpYBpmyXWZWbvoLZBh8XvUVL340a96+Rs=xVfLcpswFP0alskAgthe2thJ20knmZJOlhkZZFAsEBHCxvn6SiDxTptk6tYbW0dXr3POvZIN4CXlDYNZ/J2GiBi2GZYGWBu2bQF7Jr4kcqqR2WxeAxHDoQpqAR+/IgWaCi1wiPJeIKeUcJz1wYCmKQp4D4OM0WM/bEdJf9UMRmgE+AEkY/QRhzyu0blrtvgXhKNYr2yZqieBOlgBeQxDeuxAYGMAj1HK619J6SEiydO83Nz4njdjX7Pn4OlbXK7pE0wu6smuPzKkOQJDKf/81C/J/O7n/V3hXuSxf/3yfPDmaoh5gKRQfIlRiJ0E9kjZHjF1dH7SfB4Q41jQuyQ4SgXEaWaAFVQtgnZih6s8gwFOowfZt563wG3VvbZb5Iei3qqwGGZymaDYItmszSScBFYhZsIdmMpFclpIaVY7mnJf7cwR7ZgnRMbrrnq4LddXxxR7R+XAF38g1WqUFimCaIJ4xU7Zd7lKDm2VY8dpQGFx12XmTKFQ2Ttqpm4lFD+Uih9Q1BopOpKQ0SINUaiogtuckoKjJQsUY1afy5wzum+SR+oE21CzT7blnJHsqz7Z1gTbjjPBtuuei2x7RLaP2AEHouANSf8todr4SRnJKnyJ2CVMg5iyCXaPMebIF9kjkaMIfzfjb9plzHiH0sbU/4ZS8P/9O8VwRnHKq7O6K8NdN9N6lEiV1jiNEcOy9u0wIWP0rytUDtTRFWh26Y6ywlrMxxI6C/dMEjojCR9gvt+UKCiqCj7UMz/ihMBUlvwgxiS8hSdR4WWh5zDY69ZKpAN+FVLBVljItHDA7EX4cqSAzYEghg1Mcx7sdkr2Dn5dfQTet9fQHN1Lx5x0F0Py4rrXupoNdAtzrreu3xLV3QbzuFrP7NykgRiN2DmdAxY95wB3XE6tq6nLyz5X7rsTub9acs7wViS4OM6V0KlcVqS0uP0GDkZGG75Sehmc0sqBXbcoqP94cQamkJJPP4oq1eG2sQqjHPJOWwio/fB5kZ1pkbWo79R08XFJRbN9+1Z9nX8QYPML -------------------------------------------------------------------------------- /draft/celery-nextgen-architecture: -------------------------------------------------------------------------------- 1 | 7Vxbc6O4Ev41qdp9CMX98pjrmaqd2U1ltmp2nqZkkG02gDxcEmd+/UogAULCxheMkxNPVcY0ksDdn/rrbgkujJt4/b8UrJZfUACjC10N1hfG7YWue46K/xLBayXQHSZZpGFQybRG8DX8BamQNSvCAGZcwxyhKA9XvNBHSQL9nJOBNEUvfLM5ivirrsACCoKvPohE6bcwyJeV1LXURv4Jhoslu7Km0jMxYI2pIFuCAL20RMbdhXGTIpRX3+L1DYyI8pheqn73PWfrG0thkg/pED36a3/9bbb4tE69P+M/fj6G/146ZjXMM4gK+otvUJIVMaQ3nb8yTcAAK4YeojRfogVKQHTXSK9TVCQBJJdT8VHT5jNCKyzUsPBfmOev1MqgyBEWLfM4omfhOsz/Id0V02PH38k5RdXY8e2ajl8evLYOHmAaxjCHKZMlefr6T/vge/ugHEh3m2MyllcfdkfLcpDmVwRRWJCgBDLZfRhFtM0cJTn9cbpLj29QhNJSg4ZXfrC80i1RaK8tqShDRerDDQa0LTopQLqA+YaGeg05PFchwj8tfcX9UhiBPHzmbwTQSbOo29Gu+PeD11aDFQqTPGuN/EAEuAGd/5qB9VAOSee/bXIgxV+qIdlR694aUQnkXUDtTAreFnDVNmg1i4OtZr894M5xl5YcaoEFnXK8FD3Bdg/bMYB9XKjrA5FuawdCvYXO3cHn6oJH/RtkT5kUkp/BDPMlByMQhYsEf/exfogNr59hmoeYkK7oiTgMggqxMAt/gVk5HjEinYx4cOv6wrrd2R+RC8E1px3KqvQiDZe1zbdhCoo2oMNfqgoeS+dcw2VNnbv5pMbpsCZoPs8wQrqWPIJnwdrqGhdPtyzMcrl9p3Y51Eswl+Md4nF4f+Mc5G9gErAWfgSyLPQrIW2i7e+S5vO57vsylxTYM9uayCV57iTsSyiGY1+7EyJ2OxC8HNTB6Qah3Q6murHDOAGB6JIvdDvKKaq4aWv/LBA7cZmV+LvCDXR3tW5O4m8L+n85yowJHlFBJgIV43uddZtiWXVNJu54DZwlrMhXv5iRifOyDHP4dQVKFL7gFIt3ALPKg3ye1QLgPy1Kv/JXkUchmXylPADp0194mDAvI2ZFtXihXkpJyzpNUcXJhdneDUzZ5HL1mWHb+1CMMO16ScPtYM1iSdhLk5jVeFy2kzI2DY5O9owPW8j6ArOM5JS6ek10lAoWZnxecv8DysI8RITXZyjPUczbt8v9OSEGMUTgzUF9bsd2qnp9d69XBi4RFq8XJGVXwK8ihcrPAhbwxwIm2Jv7LJiA6d0zrGIKERrEP9NfJfPXIxjcMOxhBnfdkextGJPSvGK1ib4+2iclVlSXo/pqsB6y7+X6CRJe3RzIubo3Cee6DodZFtT2EuLm9iPxoVj1ufJzJPoq5iuKOKoaNC6p131F5MR1zUN8bIY/vW6thXhUkddNXdRTpe6njbwOUwHozqVhoO27cDY/joPS1I7tBPdU5zQcH7lj8ZF9npUPzjc5fc6JSxyatKCdO+ySGBzT5wwtslUENVXpwdHO0vq6zdl/1yR0P2Y603LtcCRNCSR2l5KEKVuB5IgJ00Mxi8JsuTVnqi77f5YztUNslmgcQleGpiusPMnW4zxJDmWfNIdiC4LnEFOPs8ik6W/caXlDQ25rUq8lFmcFZIFsVQWU83BN4LM9zw0Adk8MaC0A9QeWUScyFir5vaHzwHS/N9btZPZx5gOolL1WaZhBxY9CfPQDrFYRvkx19TESd93TFFPjHI0nuhm38UZ8dDwWQFj5QEJrNefQ8k1Wn0jZGZLSRGghJ6oOzKT5y+7EwGHvWOhpYVjmSxrCwX0Hzha+nvSSmYoP/CX8kaAAdueTj68BsG6Yzzt+GqY7kjxMWhccLQ9zDQFpj3jgkOAKj6jeYPrKsU4SevwVaysoIqxjXe1fLZw6kNeM41GiyVMiXQx/Q6Ro2qcixb3qUKZlcJPCMk5QV3LFutIXkFTu9BzxPHRDxpuH4KE7IvaDoMlD0PZOAEGzn+OPvNaHyR0TdxR9rPeNtN5Xpatq87GkHm3K5T9NANvdGvpFGZnp6gOeXXOUxqLRX8I4AokQG+EAKiARenclr+NaStdDBdaRYij86dYGHEkcZZoSBRvaaLUBWSGqo0wC/1WvBvbewbRFX6aq2F7rwzbSMt2JqjMMV7G8FpwNUZP17R1dk6YYkY7kF7+h9OnDJ+7tEzfMAxGM07k+NvAHnt4TnhgZYE/ltp0bz7zTE68tEu8H+t4L+qzN6LMnR58jBiEnTGfbD8YctgeIK8+8q2VWGuxsT4xVORJPs2DRH5IJTuYwr1bvSezxWAMq2ue5azGmP2zAOs4RciRb5wvNhoQI5YVmdTQi7F+pbxcqyAN9pLYsAqHf4hKD9hpfsO3em8AqULBHLXUZM7Ux0bNO10VLiDJHCXEumCkweYYRWsEfv1Hw/C5zUTwK78vPcUCk6WYHRayusRVF1miENum21g6hHbIGr6iu0WG16sHRd8JrzAFtfwZukkUHTdc7mQItKY37uKdY8n2Eq4jc+G9X/hMe80/siH6fEuLHW1JTTNvkEe5Z7yly099E5OaK+ecNimOQcCQrLVSKxNeEUCgNf5Hl8XqPCUnlksXfJdrcRnBNe9xe6htDmyHFz21L653FdVeVxDzSTc6jxTzOgH0+YhjBxQfilpoG6W4rfsB3E64ySVzqquRfXxC7e+C5AWUbq36mrriWo5k6/WuLhjDHsoMrlgDJ3oWxbJGWW1HE0Iwaovs80sg2YLUKR2mUT/7a3GyR1CqmtJjjbp85u7KilOJoMSHo8MvgvXSd6dXGzoGctJWA3KEhFnNDe0PoRFRlHW5i2dPag21ZP3ctt+U9iMOITJdPMHqGhBh3MLKkptCJYHqeDuefNR8ev5zmNStCYG133pXgdXeuVbCmvRrE7BrAC9fZ8vy3093luWN7VlEZNT/wdEl+kBVRzzsb3u07OdzNZXdVcTyVPj92xm/gELeY3SK/iGF5/b1D75MH1o5+dnG1J1tUkyry6NCsrLox0j21MmTr2ydSxvaw/8TKcAdkXGMpoyfPn1AZznZlvI/0s+cB+jNJZlwxmanST3wTZEeacV2x/FjGkeejN86VWi4VjJWP9hjlTeajrmyz7kc+KoX51syD+aW9IXSafNST1SBG2bpzC3KAW38Nk6eP/Tuj7NAW3ubFXu854eYcTRXD2IciW+LBHhDp3n2ur4LJgV5nurdm1EtD9Trn9/bB3ktC/S/gGO/lfYPf2nKUpaY6RDzTd+fqncnlaVZ7dmzv4GgnqK7UWjydQ68M++HST+LSXVPcnnI0l44PmxegV4hsXiNv3P0H -------------------------------------------------------------------------------- /draft/router.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | ================ 4 | CEP XXXX: Router 5 | ================ 6 | 7 | :CEP: XXXX 8 | :Author: Omer Katz 9 | :Implementation Team: Omer Katz 10 | :Shepherd: Omer Katz 11 | :Status: Draft 12 | :Type: Feature 13 | :Created: 2019-04-03 14 | :Last-Modified: 2019-04-03 15 | 16 | .. contents:: Table of Contents 17 | :depth: 3 18 | :local: 19 | 20 | This CEP provides a sample template for creating your own CEPs. In conjunction 21 | with the content guidelines in :doc:`/final/0001-cep-process`, 22 | this should make it easy for you to conform your own CEPs to the format 23 | outlined below. 24 | 25 | Note: if you are reading this CEP via the web, you should first grab `the source 26 | of this CEP `_ in 27 | order to complete the steps below. **DO NOT USE THE HTML FILE AS YOUR 28 | TEMPLATE!** 29 | 30 | To get the source this (or any) CEP, look at the top of the Github page 31 | and click "raw". 32 | 33 | If you're unfamiliar with reStructuredText (the format required of CEPs), 34 | see these resources: 35 | 36 | * `A ReStructuredText Primer`__, a gentle introduction. 37 | * `Quick reStructuredText`__, a users' quick reference. 38 | * `reStructuredText Markup Specification`__, the final authority. 39 | 40 | __ http://docutils.sourceforge.net/docs/user/rst/quickstart.html 41 | __ http://docutils.sourceforge.net/docs/user/rst/quickref.html 42 | __ http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html 43 | 44 | Once you've made a copy of this template, remove this abstract, fill out the 45 | metadata above and the sections below, then submit the CEP. Follow the 46 | guidelines in :doc:`/final/0001-cep-process`. 47 | 48 | Abstract 49 | ======== 50 | 51 | This should be a short (~200 word) description of the technical issue being 52 | addressed. 53 | 54 | This (and the above metadata) is the only section strictly required to submit a 55 | draft CEP; the following sections can be barebones and fleshed out as you work 56 | through the CEP process. 57 | 58 | Specification 59 | ============= 60 | 61 | This section should contain a complete, detailed technical specification should 62 | describe the syntax and semantics of any new feature. The specification should 63 | be detailed enough to allow implementation -- that is, developers other than the 64 | author should (given the right experience) be able to independently implement 65 | the feature, given only the CEP. 66 | 67 | Motivation 68 | ========== 69 | 70 | This section should explain *why* this CEP is needed. The motivation is critical 71 | for CEPs that want to add substantial new features or materially refactor 72 | existing ones. It should clearly explain why the existing solutions are 73 | inadequate to address the problem that the CEP solves. CEP submissions without 74 | sufficient motivation may be rejected outright. 75 | 76 | Rationale 77 | ========= 78 | 79 | This section should flesh out the specification by describing what motivated 80 | the specific design design and why particular design decisions were made. It 81 | should describe alternate designs that were considered and related work. 82 | 83 | The rationale should provide evidence of consensus within the community and 84 | discuss important objections or concerns raised during discussion. 85 | 86 | Backwards Compatibility 87 | ======================= 88 | 89 | If this CEP introduces backwards incompatibilities, you must must include this 90 | section. It should describe these incompatibilities and their severity, and what 91 | mitigation you plan to take to deal with these incompatibilities. 92 | 93 | Reference Implementation 94 | ======================== 95 | 96 | If there's an implementation of the feature under discussion in this CEP, 97 | this section should include or link to that implementation and provide any 98 | notes about installing/using/trying out the implementation. 99 | 100 | Copyright 101 | ========= 102 | 103 | This document has been placed in the public domain per the Creative Commons 104 | CC0 1.0 Universal license (https://creativecommons.org/publicdomain/zero/1.0/deed). 105 | 106 | (All CEPs must include this exact copyright statement.) 107 | -------------------------------------------------------------------------------- /draft/controller.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | ==================== 4 | CEP XXXX: Controller 5 | ==================== 6 | 7 | :CEP: XXXX 8 | :Author: Omer Katz 9 | :Implementation Team: Omer Katz 10 | :Shepherd: Omer Katz 11 | :Status: Draft 12 | :Type: Feature 13 | :Created: 2019-04-03 14 | :Last-Modified: 2019-04-03 15 | 16 | .. contents:: Table of Contents 17 | :depth: 3 18 | :local: 19 | 20 | This CEP provides a sample template for creating your own CEPs. In conjunction 21 | with the content guidelines in :doc:`/final/0001-cep-process`, 22 | this should make it easy for you to conform your own CEPs to the format 23 | outlined below. 24 | 25 | Note: if you are reading this CEP via the web, you should first grab `the source 26 | of this CEP `_ in 27 | order to complete the steps below. **DO NOT USE THE HTML FILE AS YOUR 28 | TEMPLATE!** 29 | 30 | To get the source this (or any) CEP, look at the top of the Github page 31 | and click "raw". 32 | 33 | If you're unfamiliar with reStructuredText (the format required of CEPs), 34 | see these resources: 35 | 36 | * `A ReStructuredText Primer`__, a gentle introduction. 37 | * `Quick reStructuredText`__, a users' quick reference. 38 | * `reStructuredText Markup Specification`__, the final authority. 39 | 40 | __ http://docutils.sourceforge.net/docs/user/rst/quickstart.html 41 | __ http://docutils.sourceforge.net/docs/user/rst/quickref.html 42 | __ http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html 43 | 44 | Once you've made a copy of this template, remove this abstract, fill out the 45 | metadata above and the sections below, then submit the CEP. Follow the 46 | guidelines in :doc:`/final/0001-cep-process`. 47 | 48 | Abstract 49 | ======== 50 | 51 | This should be a short (~200 word) description of the technical issue being 52 | addressed. 53 | 54 | This (and the above metadata) is the only section strictly required to submit a 55 | draft CEP; the following sections can be barebones and fleshed out as you work 56 | through the CEP process. 57 | 58 | Specification 59 | ============= 60 | 61 | This section should contain a complete, detailed technical specification should 62 | describe the syntax and semantics of any new feature. The specification should 63 | be detailed enough to allow implementation -- that is, developers other than the 64 | author should (given the right experience) be able to independently implement 65 | the feature, given only the CEP. 66 | 67 | Motivation 68 | ========== 69 | 70 | This section should explain *why* this CEP is needed. The motivation is critical 71 | for CEPs that want to add substantial new features or materially refactor 72 | existing ones. It should clearly explain why the existing solutions are 73 | inadequate to address the problem that the CEP solves. CEP submissions without 74 | sufficient motivation may be rejected outright. 75 | 76 | Rationale 77 | ========= 78 | 79 | This section should flesh out the specification by describing what motivated 80 | the specific design design and why particular design decisions were made. It 81 | should describe alternate designs that were considered and related work. 82 | 83 | The rationale should provide evidence of consensus within the community and 84 | discuss important objections or concerns raised during discussion. 85 | 86 | Backwards Compatibility 87 | ======================= 88 | 89 | If this CEP introduces backwards incompatibilities, you must must include this 90 | section. It should describe these incompatibilities and their severity, and what 91 | mitigation you plan to take to deal with these incompatibilities. 92 | 93 | Reference Implementation 94 | ======================== 95 | 96 | If there's an implementation of the feature under discussion in this CEP, 97 | this section should include or link to that implementation and provide any 98 | notes about installing/using/trying out the implementation. 99 | 100 | Copyright 101 | ========= 102 | 103 | This document has been placed in the public domain per the Creative Commons 104 | CC0 1.0 Universal license (https://creativecommons.org/publicdomain/zero/1.0/deed). 105 | 106 | (All CEPs must include this exact copyright statement.) 107 | -------------------------------------------------------------------------------- /draft/publisher.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | =================== 4 | CEP XXXX: Publisher 5 | =================== 6 | 7 | :CEP: XXXX 8 | :Author: Omer Katz 9 | :Implementation Team: Omer Katz 10 | :Shepherd: Omer Katz 11 | :Status: Draft 12 | :Type: Feature 13 | :Created: 2019-04-03 14 | :Last-Modified: 2019-04-03 15 | 16 | .. contents:: Table of Contents 17 | :depth: 3 18 | :local: 19 | 20 | This CEP provides a sample template for creating your own CEPs. In conjunction 21 | with the content guidelines in :doc:`/final/0001-cep-process`, 22 | this should make it easy for you to conform your own CEPs to the format 23 | outlined below. 24 | 25 | Note: if you are reading this CEP via the web, you should first grab `the source 26 | of this CEP `_ in 27 | order to complete the steps below. **DO NOT USE THE HTML FILE AS YOUR 28 | TEMPLATE!** 29 | 30 | To get the source this (or any) CEP, look at the top of the Github page 31 | and click "raw". 32 | 33 | If you're unfamiliar with reStructuredText (the format required of CEPs), 34 | see these resources: 35 | 36 | * `A ReStructuredText Primer`__, a gentle introduction. 37 | * `Quick reStructuredText`__, a users' quick reference. 38 | * `reStructuredText Markup Specification`__, the final authority. 39 | 40 | __ http://docutils.sourceforge.net/docs/user/rst/quickstart.html 41 | __ http://docutils.sourceforge.net/docs/user/rst/quickref.html 42 | __ http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html 43 | 44 | Once you've made a copy of this template, remove this abstract, fill out the 45 | metadata above and the sections below, then submit the CEP. Follow the 46 | guidelines in :doc:`/final/0001-cep-process`. 47 | 48 | Abstract 49 | ======== 50 | 51 | This should be a short (~200 word) description of the technical issue being 52 | addressed. 53 | 54 | This (and the above metadata) is the only section strictly required to submit a 55 | draft CEP; the following sections can be barebones and fleshed out as you work 56 | through the CEP process. 57 | 58 | Specification 59 | ============= 60 | 61 | This section should contain a complete, detailed technical specification should 62 | describe the syntax and semantics of any new feature. The specification should 63 | be detailed enough to allow implementation -- that is, developers other than the 64 | author should (given the right experience) be able to independently implement 65 | the feature, given only the CEP. 66 | 67 | Motivation 68 | ========== 69 | 70 | This section should explain *why* this CEP is needed. The motivation is critical 71 | for CEPs that want to add substantial new features or materially refactor 72 | existing ones. It should clearly explain why the existing solutions are 73 | inadequate to address the problem that the CEP solves. CEP submissions without 74 | sufficient motivation may be rejected outright. 75 | 76 | Rationale 77 | ========= 78 | 79 | This section should flesh out the specification by describing what motivated 80 | the specific design design and why particular design decisions were made. It 81 | should describe alternate designs that were considered and related work. 82 | 83 | The rationale should provide evidence of consensus within the community and 84 | discuss important objections or concerns raised during discussion. 85 | 86 | Backwards Compatibility 87 | ======================= 88 | 89 | If this CEP introduces backwards incompatibilities, you must must include this 90 | section. It should describe these incompatibilities and their severity, and what 91 | mitigation you plan to take to deal with these incompatibilities. 92 | 93 | Reference Implementation 94 | ======================== 95 | 96 | If there's an implementation of the feature under discussion in this CEP, 97 | this section should include or link to that implementation and provide any 98 | notes about installing/using/trying out the implementation. 99 | 100 | Copyright 101 | ========= 102 | 103 | This document has been placed in the public domain per the Creative Commons 104 | CC0 1.0 Universal license (https://creativecommons.org/publicdomain/zero/1.0/deed). 105 | 106 | (All CEPs must include this exact copyright statement.) 107 | -------------------------------------------------------------------------------- /template.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | ====================== 4 | CEP XXXX: CEP template 5 | ====================== 6 | 7 | :CEP: XXXX 8 | :Author: Omer Katz 9 | :Implementation Team: Omer Katz 10 | :Shepherd: Omer Katz 11 | :Status: Draft 12 | :Type: Feature 13 | :Created: 2019-04-03 14 | :Last-Modified: 2019-04-03 15 | 16 | .. contents:: Table of Contents 17 | :depth: 3 18 | :local: 19 | 20 | This CEP provides a sample template for creating your own CEPs. In conjunction 21 | with the content guidelines in :doc:`/final/0001-cep-process`, 22 | this should make it easy for you to conform your own CEPs to the format 23 | outlined below. 24 | 25 | Note: if you are reading this CEP via the web, you should first grab `the source 26 | of this CEP `_ in 27 | order to complete the steps below. **DO NOT USE THE HTML FILE AS YOUR 28 | TEMPLATE!** 29 | 30 | To get the source this (or any) CEP, look at the top of the Github page 31 | and click "raw". 32 | 33 | If you're unfamiliar with reStructuredText (the format required of CEPs), 34 | see these resources: 35 | 36 | * `A ReStructuredText Primer`__, a gentle introduction. 37 | * `Quick reStructuredText`__, a users' quick reference. 38 | * `reStructuredText Markup Specification`__, the final authority. 39 | 40 | __ http://docutils.sourceforge.net/docs/user/rst/quickstart.html 41 | __ http://docutils.sourceforge.net/docs/user/rst/quickref.html 42 | __ http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html 43 | 44 | Once you've made a copy of this template, remove this abstract, fill out the 45 | metadata above and the sections below, then submit the CEP. Follow the 46 | guidelines in :doc:`/final/0001-cep-process`. 47 | 48 | Abstract 49 | ======== 50 | 51 | This should be a short (~200 word) description of the technical issue being 52 | addressed. 53 | 54 | This (and the above metadata) is the only section strictly required to submit a 55 | draft CEP; the following sections can be barebones and fleshed out as you work 56 | through the CEP process. 57 | 58 | Specification 59 | ============= 60 | 61 | This section should contain a complete, detailed technical specification should 62 | describe the syntax and semantics of any new feature. The specification should 63 | be detailed enough to allow implementation -- that is, developers other than the 64 | author should (given the right experience) be able to independently implement 65 | the feature, given only the CEP. 66 | 67 | Motivation 68 | ========== 69 | 70 | This section should explain *why* this CEP is needed. The motivation is critical 71 | for CEPs that want to add substantial new features or materially refactor 72 | existing ones. It should clearly explain why the existing solutions are 73 | inadequate to address the problem that the CEP solves. CEP submissions without 74 | sufficient motivation may be rejected outright. 75 | 76 | Rationale 77 | ========= 78 | 79 | This section should flesh out the specification by describing what motivated 80 | the specific design design and why particular design decisions were made. It 81 | should describe alternate designs that were considered and related work. 82 | 83 | The rationale should provide evidence of consensus within the community and 84 | discuss important objections or concerns raised during discussion. 85 | 86 | Backwards Compatibility 87 | ======================= 88 | 89 | If this CEP introduces backwards incompatibilities, you must must include this 90 | section. It should describe these incompatibilities and their severity, and what 91 | mitigation you plan to take to deal with these incompatibilities. 92 | 93 | Reference Implementation 94 | ======================== 95 | 96 | If there's an implementation of the feature under discussion in this CEP, 97 | this section should include or link to that implementation and provide any 98 | notes about installing/using/trying out the implementation. 99 | 100 | Copyright 101 | ========= 102 | 103 | This document has been placed in the public domain per the Creative Commons 104 | CC0 1.0 Universal license (https://creativecommons.org/publicdomain/zero/1.0/deed). 105 | 106 | (All CEPs must include this exact copyright statement.) 107 | -------------------------------------------------------------------------------- /draft/configuration.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | ======================= 4 | CEP XXXX: Configuration 5 | ======================= 6 | 7 | :CEP: XXXX 8 | :Author: Omer Katz 9 | :Implementation Team: Omer Katz 10 | :Shepherd: Omer Katz 11 | :Status: Draft 12 | :Type: Feature 13 | :Created: 2019-04-03 14 | :Last-Modified: 2019-04-03 15 | 16 | .. contents:: Table of Contents 17 | :depth: 3 18 | :local: 19 | 20 | This CEP provides a sample template for creating your own CEPs. In conjunction 21 | with the content guidelines in :doc:`/final/0001-cep-process`, 22 | this should make it easy for you to conform your own CEPs to the format 23 | outlined below. 24 | 25 | Note: if you are reading this CEP via the web, you should first grab `the source 26 | of this CEP `_ in 27 | order to complete the steps below. **DO NOT USE THE HTML FILE AS YOUR 28 | TEMPLATE!** 29 | 30 | To get the source this (or any) CEP, look at the top of the Github page 31 | and click "raw". 32 | 33 | If you're unfamiliar with reStructuredText (the format required of CEPs), 34 | see these resources: 35 | 36 | * `A ReStructuredText Primer`__, a gentle introduction. 37 | * `Quick reStructuredText`__, a users' quick reference. 38 | * `reStructuredText Markup Specification`__, the final authority. 39 | 40 | __ http://docutils.sourceforge.net/docs/user/rst/quickstart.html 41 | __ http://docutils.sourceforge.net/docs/user/rst/quickref.html 42 | __ http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html 43 | 44 | Once you've made a copy of this template, remove this abstract, fill out the 45 | metadata above and the sections below, then submit the CEP. Follow the 46 | guidelines in :doc:`/final/0001-cep-process`. 47 | 48 | Abstract 49 | ======== 50 | 51 | This should be a short (~200 word) description of the technical issue being 52 | addressed. 53 | 54 | This (and the above metadata) is the only section strictly required to submit a 55 | draft CEP; the following sections can be barebones and fleshed out as you work 56 | through the CEP process. 57 | 58 | Specification 59 | ============= 60 | 61 | This section should contain a complete, detailed technical specification should 62 | describe the syntax and semantics of any new feature. The specification should 63 | be detailed enough to allow implementation -- that is, developers other than the 64 | author should (given the right experience) be able to independently implement 65 | the feature, given only the CEP. 66 | 67 | Motivation 68 | ========== 69 | 70 | This section should explain *why* this CEP is needed. The motivation is critical 71 | for CEPs that want to add substantial new features or materially refactor 72 | existing ones. It should clearly explain why the existing solutions are 73 | inadequate to address the problem that the CEP solves. CEP submissions without 74 | sufficient motivation may be rejected outright. 75 | 76 | Rationale 77 | ========= 78 | 79 | This section should flesh out the specification by describing what motivated 80 | the specific design design and why particular design decisions were made. It 81 | should describe alternate designs that were considered and related work. 82 | 83 | The rationale should provide evidence of consensus within the community and 84 | discuss important objections or concerns raised during discussion. 85 | 86 | Backwards Compatibility 87 | ======================= 88 | 89 | If this CEP introduces backwards incompatibilities, you must must include this 90 | section. It should describe these incompatibilities and their severity, and what 91 | mitigation you plan to take to deal with these incompatibilities. 92 | 93 | Reference Implementation 94 | ======================== 95 | 96 | If there's an implementation of the feature under discussion in this CEP, 97 | this section should include or link to that implementation and provide any 98 | notes about installing/using/trying out the implementation. 99 | 100 | Copyright 101 | ========= 102 | 103 | This document has been placed in the public domain per the Creative Commons 104 | CC0 1.0 Universal license (https://creativecommons.org/publicdomain/zero/1.0/deed). 105 | 106 | (All CEPs must include this exact copyright statement.) 107 | -------------------------------------------------------------------------------- /draft/observability.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | ======================= 4 | CEP XXXX: Observability 5 | ======================= 6 | 7 | :CEP: XXXX 8 | :Author: Omer Katz 9 | :Implementation Team: Omer Katz 10 | :Shepherd: Omer Katz 11 | :Status: Draft 12 | :Type: Feature 13 | :Created: 2019-04-03 14 | :Last-Modified: 2019-04-03 15 | 16 | .. contents:: Table of Contents 17 | :depth: 3 18 | :local: 19 | 20 | This CEP provides a sample template for creating your own CEPs. In conjunction 21 | with the content guidelines in :doc:`/final/0001-cep-process`, 22 | this should make it easy for you to conform your own CEPs to the format 23 | outlined below. 24 | 25 | Note: if you are reading this CEP via the web, you should first grab `the source 26 | of this CEP `_ in 27 | order to complete the steps below. **DO NOT USE THE HTML FILE AS YOUR 28 | TEMPLATE!** 29 | 30 | To get the source this (or any) CEP, look at the top of the Github page 31 | and click "raw". 32 | 33 | If you're unfamiliar with reStructuredText (the format required of CEPs), 34 | see these resources: 35 | 36 | * `A ReStructuredText Primer`__, a gentle introduction. 37 | * `Quick reStructuredText`__, a users' quick reference. 38 | * `reStructuredText Markup Specification`__, the final authority. 39 | 40 | __ http://docutils.sourceforge.net/docs/user/rst/quickstart.html 41 | __ http://docutils.sourceforge.net/docs/user/rst/quickref.html 42 | __ http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html 43 | 44 | Once you've made a copy of this template, remove this abstract, fill out the 45 | metadata above and the sections below, then submit the CEP. Follow the 46 | guidelines in :doc:`/final/0001-cep-process`. 47 | 48 | Abstract 49 | ======== 50 | 51 | This should be a short (~200 word) description of the technical issue being 52 | addressed. 53 | 54 | This (and the above metadata) is the only section strictly required to submit a 55 | draft CEP; the following sections can be barebones and fleshed out as you work 56 | through the CEP process. 57 | 58 | Specification 59 | ============= 60 | 61 | This section should contain a complete, detailed technical specification should 62 | describe the syntax and semantics of any new feature. The specification should 63 | be detailed enough to allow implementation -- that is, developers other than the 64 | author should (given the right experience) be able to independently implement 65 | the feature, given only the CEP. 66 | 67 | Motivation 68 | ========== 69 | 70 | This section should explain *why* this CEP is needed. The motivation is critical 71 | for CEPs that want to add substantial new features or materially refactor 72 | existing ones. It should clearly explain why the existing solutions are 73 | inadequate to address the problem that the CEP solves. CEP submissions without 74 | sufficient motivation may be rejected outright. 75 | 76 | Rationale 77 | ========= 78 | 79 | This section should flesh out the specification by describing what motivated 80 | the specific design design and why particular design decisions were made. It 81 | should describe alternate designs that were considered and related work. 82 | 83 | The rationale should provide evidence of consensus within the community and 84 | discuss important objections or concerns raised during discussion. 85 | 86 | Backwards Compatibility 87 | ======================= 88 | 89 | If this CEP introduces backwards incompatibilities, you must must include this 90 | section. It should describe these incompatibilities and their severity, and what 91 | mitigation you plan to take to deal with these incompatibilities. 92 | 93 | Reference Implementation 94 | ======================== 95 | 96 | If there's an implementation of the feature under discussion in this CEP, 97 | this section should include or link to that implementation and provide any 98 | notes about installing/using/trying out the implementation. 99 | 100 | Copyright 101 | ========= 102 | 103 | This document has been placed in the public domain per the Creative Commons 104 | CC0 1.0 Universal license (https://creativecommons.org/publicdomain/zero/1.0/deed). 105 | 106 | (All CEPs must include this exact copyright statement.) 107 | -------------------------------------------------------------------------------- /draft/execution-platform.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | ============================ 4 | CEP XXXX: Execution Platform 5 | ============================ 6 | 7 | :CEP: XXXX 8 | :Author: Omer Katz 9 | :Implementation Team: Omer Katz 10 | :Shepherd: Omer Katz 11 | :Status: Draft 12 | :Type: Feature 13 | :Created: 2019-04-03 14 | :Last-Modified: 2019-04-03 15 | 16 | .. contents:: Table of Contents 17 | :depth: 3 18 | :local: 19 | 20 | This CEP provides a sample template for creating your own CEPs. In conjunction 21 | with the content guidelines in :doc:`/final/0001-cep-process`, 22 | this should make it easy for you to conform your own CEPs to the format 23 | outlined below. 24 | 25 | Note: if you are reading this CEP via the web, you should first grab `the source 26 | of this CEP `_ in 27 | order to complete the steps below. **DO NOT USE THE HTML FILE AS YOUR 28 | TEMPLATE!** 29 | 30 | To get the source this (or any) CEP, look at the top of the Github page 31 | and click "raw". 32 | 33 | If you're unfamiliar with reStructuredText (the format required of CEPs), 34 | see these resources: 35 | 36 | * `A ReStructuredText Primer`__, a gentle introduction. 37 | * `Quick reStructuredText`__, a users' quick reference. 38 | * `reStructuredText Markup Specification`__, the final authority. 39 | 40 | __ http://docutils.sourceforge.net/docs/user/rst/quickstart.html 41 | __ http://docutils.sourceforge.net/docs/user/rst/quickref.html 42 | __ http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html 43 | 44 | Once you've made a copy of this template, remove this abstract, fill out the 45 | metadata above and the sections below, then submit the CEP. Follow the 46 | guidelines in :doc:`/final/0001-cep-process`. 47 | 48 | Abstract 49 | ======== 50 | 51 | This should be a short (~200 word) description of the technical issue being 52 | addressed. 53 | 54 | This (and the above metadata) is the only section strictly required to submit a 55 | draft CEP; the following sections can be barebones and fleshed out as you work 56 | through the CEP process. 57 | 58 | Specification 59 | ============= 60 | 61 | This section should contain a complete, detailed technical specification should 62 | describe the syntax and semantics of any new feature. The specification should 63 | be detailed enough to allow implementation -- that is, developers other than the 64 | author should (given the right experience) be able to independently implement 65 | the feature, given only the CEP. 66 | 67 | Motivation 68 | ========== 69 | 70 | This section should explain *why* this CEP is needed. The motivation is critical 71 | for CEPs that want to add substantial new features or materially refactor 72 | existing ones. It should clearly explain why the existing solutions are 73 | inadequate to address the problem that the CEP solves. CEP submissions without 74 | sufficient motivation may be rejected outright. 75 | 76 | Rationale 77 | ========= 78 | 79 | This section should flesh out the specification by describing what motivated 80 | the specific design design and why particular design decisions were made. It 81 | should describe alternate designs that were considered and related work. 82 | 83 | The rationale should provide evidence of consensus within the community and 84 | discuss important objections or concerns raised during discussion. 85 | 86 | Backwards Compatibility 87 | ======================= 88 | 89 | If this CEP introduces backwards incompatibilities, you must must include this 90 | section. It should describe these incompatibilities and their severity, and what 91 | mitigation you plan to take to deal with these incompatibilities. 92 | 93 | Reference Implementation 94 | ======================== 95 | 96 | If there's an implementation of the feature under discussion in this CEP, 97 | this section should include or link to that implementation and provide any 98 | notes about installing/using/trying out the implementation. 99 | 100 | Copyright 101 | ========= 102 | 103 | This document has been placed in the public domain per the Creative Commons 104 | CC0 1.0 Universal license (https://creativecommons.org/publicdomain/zero/1.0/deed). 105 | 106 | (All CEPs must include this exact copyright statement.) 107 | -------------------------------------------------------------------------------- /draft/features-release-schedule.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | ================================== 4 | CEP XXXX: Feature Release Schedule 5 | ================================== 6 | 7 | :CEP: XXXX 8 | :Author: Omer Katz 9 | :Implementation Team: Omer Katz 10 | :Shepherd: Omer Katz 11 | :Status: Draft 12 | :Type: Feature 13 | :Created: 2019-04-03 14 | :Last-Modified: 2019-04-03 15 | 16 | .. contents:: Table of Contents 17 | :depth: 3 18 | :local: 19 | 20 | This CEP provides a sample template for creating your own CEPs. In conjunction 21 | with the content guidelines in :doc:`/final/0001-cep-process`, 22 | this should make it easy for you to conform your own CEPs to the format 23 | outlined below. 24 | 25 | Note: if you are reading this CEP via the web, you should first grab `the source 26 | of this CEP `_ in 27 | order to complete the steps below. **DO NOT USE THE HTML FILE AS YOUR 28 | TEMPLATE!** 29 | 30 | To get the source this (or any) CEP, look at the top of the Github page 31 | and click "raw". 32 | 33 | If you're unfamiliar with reStructuredText (the format required of CEPs), 34 | see these resources: 35 | 36 | * `A ReStructuredText Primer`__, a gentle introduction. 37 | * `Quick reStructuredText`__, a users' quick reference. 38 | * `reStructuredText Markup Specification`__, the final authority. 39 | 40 | __ http://docutils.sourceforge.net/docs/user/rst/quickstart.html 41 | __ http://docutils.sourceforge.net/docs/user/rst/quickref.html 42 | __ http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html 43 | 44 | Once you've made a copy of this template, remove this abstract, fill out the 45 | metadata above and the sections below, then submit the CEP. Follow the 46 | guidelines in :doc:`/final/0001-cep-process`. 47 | 48 | Abstract 49 | ======== 50 | 51 | This should be a short (~200 word) description of the technical issue being 52 | addressed. 53 | 54 | This (and the above metadata) is the only section strictly required to submit a 55 | draft CEP; the following sections can be barebones and fleshed out as you work 56 | through the CEP process. 57 | 58 | Specification 59 | ============= 60 | 61 | This section should contain a complete, detailed technical specification should 62 | describe the syntax and semantics of any new feature. The specification should 63 | be detailed enough to allow implementation -- that is, developers other than the 64 | author should (given the right experience) be able to independently implement 65 | the feature, given only the CEP. 66 | 67 | Motivation 68 | ========== 69 | 70 | This section should explain *why* this CEP is needed. The motivation is critical 71 | for CEPs that want to add substantial new features or materially refactor 72 | existing ones. It should clearly explain why the existing solutions are 73 | inadequate to address the problem that the CEP solves. CEP submissions without 74 | sufficient motivation may be rejected outright. 75 | 76 | Rationale 77 | ========= 78 | 79 | This section should flesh out the specification by describing what motivated 80 | the specific design design and why particular design decisions were made. It 81 | should describe alternate designs that were considered and related work. 82 | 83 | The rationale should provide evidence of consensus within the community and 84 | discuss important objections or concerns raised during discussion. 85 | 86 | Backwards Compatibility 87 | ======================= 88 | 89 | If this CEP introduces backwards incompatibilities, you must must include this 90 | section. It should describe these incompatibilities and their severity, and what 91 | mitigation you plan to take to deal with these incompatibilities. 92 | 93 | Reference Implementation 94 | ======================== 95 | 96 | If there's an implementation of the feature under discussion in this CEP, 97 | this section should include or link to that implementation and provide any 98 | notes about installing/using/trying out the implementation. 99 | 100 | Copyright 101 | ========= 102 | 103 | This document has been placed in the public domain per the Creative Commons 104 | CC0 1.0 Universal license (https://creativecommons.org/publicdomain/zero/1.0/deed). 105 | 106 | (All CEPs must include this exact copyright statement.) 107 | -------------------------------------------------------------------------------- /draft/cli-refactor.rst: -------------------------------------------------------------------------------- 1 | ====================== 2 | CEP XXXX: CLI Refactor 3 | ====================== 4 | 5 | :CEP: XXXX 6 | :Author: Omer Katz 7 | :Implementation Team: Omer Katz 8 | :Shepherd: Omer Katz 9 | :Status: Draft 10 | :Type: Feature 11 | :Created: 2019-11-10 12 | :Last-Modified: 2019-11-10 13 | 14 | .. contents:: Table of Contents 15 | :depth: 3 16 | :local: 17 | 18 | Abstract 19 | ======== 20 | 21 | Celery's CLI infrastructure is based on a custom framework built using the :mod:`argparse` built-in module. 22 | This implementation has multiple design defects and a few bugs and as a result a developer cannot easily read the code 23 | or extend it. 24 | 25 | As we're moving towards Celery 5, we are likely to add new sub-commands and/or enhance existing ones. 26 | Therefore, refactoring this part of the codebase will increase our future productivity and flexibility. 27 | 28 | This CEP proposes to refactor our CLI implementation in order to make that part of the code developer friendly 29 | and user extensible. 30 | 31 | Specification 32 | ============= 33 | 34 | The refactor replaces our custom :mod:`argparse` framework which can be found `here `_ 35 | with an implementation using Click_. 36 | 37 | CLI Context 38 | ----------- 39 | 40 | Instead of sharing common functionality in the `Command` base class we introduce a class called :class:`CLIContext` 41 | which provides access the Celery application, helper methods for printing informational messages or errors 42 | and other common functionality. 43 | 44 | Parameter Types 45 | --------------- 46 | 47 | The current implementation extracted common parsing and validation code of parameter types into 48 | `custom types `_ which can be reused 49 | across our CLI implementation. 50 | 51 | Plugins 52 | ------- 53 | 54 | Click allows extending existing CLIs using setuptools's entrypoints using the `click-plugins `_ 55 | extension. 56 | 57 | Frameworks which base themselves over Celery will now be able to extend or customize the CLI for their needs. 58 | 59 | Acceptance Test Suite 60 | --------------------- 61 | 62 | The former implementation was only covered by unit tests which did not cover the entire surface of the implementation. 63 | 64 | The new implementation will be covered by both unit tests and BDD-style acceptance tests. 65 | 66 | Motivation 67 | ========== 68 | 69 | The main purpose of this refactor is to allow us to use an event loop using Python 3's `async/await` syntax 70 | without investing further in our custom CLI framework. 71 | Instead we opt to use a battle-tested solution which allows us to remove the entire custom framework entirely. 72 | 73 | This allows us to delegate the maintenance overhead to others and reduce the surface of potential bugs introduced in 74 | Celery 5. 75 | 76 | Argparse which our previous implementation was based on wasn't a good fit. 77 | We needed to create a framework around it to support sub-commands such as `celery worker` and nested sub-commands 78 | were not possible at all which means that each command that had them figured out how they should be implemented for 79 | itself. 80 | In addition, We had to implement a REPL nearly from scratch for `celery amqp` while Click has a plugin which uses 81 | the `python-prompt-toolkit `_ library called `click-repl `_. 82 | 83 | This resulted in an implementation of ~3k LOC (without spaces or comments). 84 | 85 | By using Click, our new implementation has only ~2.2K LOC (without spaces or comments). 86 | This 27% reduction makes the code easier to reason about and is simpler due to Click's API. 87 | 88 | The Click ecosystem provides us with many features that argparse lacks such as `"Did you mean" `_ messages, 89 | `automatic documentation `_ using Sphinx and other user experience 90 | enhancing features which :mod:`argparse` lacks. 91 | 92 | We use these extensions for enriching our CLI implementation. 93 | 94 | Rationale 95 | ========= 96 | 97 | Alternative CLI Frameworks 98 | -------------------------- 99 | 100 | Docopt was considered as part of this effort but was found insufficient for our needs. 101 | 102 | While Doctopt does support sub-commands, it does not dispatching them to functions which requires us to write 103 | the same type of framework we wanted to avoid. 104 | 105 | Furthermore, Docopt does not parse parameter types and leaves that to the implementor. 106 | 107 | Docopt however does allow us to customize our help page better. 108 | 109 | The aforementioned disadvantages outweigh the only advantage. 110 | 111 | Parameter Types 112 | --------------- 113 | 114 | Our previous implementation used to parse and validate some of the arguments during the actual execution of the command. 115 | No infrastructure was provided to share the implementation of parsing and validating such special arguments such as 116 | ISO-8601 date time strings or comma separated lists. 117 | 118 | This resulted in violation of the `DRY `_ principle and 119 | more importantly the `Single Responsibility Principle (SRP) `_. 120 | 121 | Violating SRP makes unit testing harder as there are more code paths to take care of. 122 | This violation also increases the difficulty of reasoning about the code in question for the same reason. 123 | 124 | The current implementation separates the responsibility of parsing and validating arguments from the command invocation 125 | itself to small classes which are very easy to unit test and reason about. 126 | 127 | Backwards Compatibility 128 | ======================= 129 | 130 | This CEP is almost completely backwards compatible with our previous implementation. 131 | 132 | The only changes in our API are around the CLI's customization. 133 | 134 | User Options 135 | ------------ 136 | 137 | User Options now pass the relevant Click Command object to the callbacks. 138 | 139 | If you are using this feature you have to migrate your code from Argparse to Click. 140 | 141 | In addition the API changed. Previously the following code was required: 142 | 143 | .. code-block:: python 144 | 145 | def add_worker_arguments(parser): 146 | parser.add_argument( 147 | '--enable-my-option', action='store_true', default=False, 148 | help='Enable custom option.', 149 | ), 150 | app.user_options['worker'].add(add_worker_arguments) 151 | 152 | With this refactor you either need to set the relevant ``user_options`` key with a list of 153 | ``click.Option``s or ``click.Argument``'s or provide a callback which will return those. 154 | 155 | .. code-block:: python 156 | 157 | import click 158 | app.user_options['worker'] = [click.Option('--enable-my-option', is_flag=True, help='Enable custom option')] 159 | 160 | 161 | Preload Options 162 | --------------- 163 | 164 | Preload options are User Options and are subject to the same breaking change. 165 | 166 | In addition the signal's sender is now changed to the ``click.Context`` of the invoked command. 167 | 168 | Reference Implementation 169 | ======================== 170 | 171 | The reference implementation can be found at `celery/celery#5718 `_. 172 | 173 | Copyright 174 | ========= 175 | 176 | This document has been placed in the public domain per the Creative Commons 177 | CC0 1.0 Universal license (https://creativecommons.org/publicdomain/zero/1.0/deed). 178 | 179 | (All CEPs must include this exact copyright statement.) 180 | 181 | .. _Click: https://click.palletsprojects.com/ -------------------------------------------------------------------------------- /draft/celery-kubernetes-operator.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | ============================================================ 4 | CEP XXXX: Celery Kubernetes Operator - Architecture Document 5 | ============================================================ 6 | 7 | :CEP: XXXX 8 | :Author: Gautam Prajapati 9 | :Implementation Team: Gautam Prajapati 10 | :Shepherd: Omer Katz, Asif Saif Uddin 11 | :Status: Draft 12 | :Type: Feature 13 | :Created: 2020-08-30 14 | :Last-Modified: 2020-08-30 15 | 16 | .. contents:: Table of Contents 17 | :depth: 3 18 | :local: 19 | 20 | Abstract 21 | ======== 22 | 23 | Kubernetes is a popular deployment target nowadays. To run Celery applications on Kubernetes, there are manual steps involved like - 24 | 25 | * Writing deployment spec for workers 26 | * Setting up monitoring using `Flower `_ 27 | * Setting up Autoscaling 28 | 29 | Apart from that, there’s no standard or consistent way to set up multiple clusters, everyone configures their own way which could create problems for infrastructure teams to manage and audit later. 30 | 31 | This document proposes writing a Kubernetes `Operator `_ for automating management of Celery clusters. This CEP is currently written keeping Celery 4.X in mind. Controller implementation will differ for Celery 5 and is under discussion. 32 | 33 | Specification 34 | ============= 35 | 36 | Scope 37 | ----- 38 | 39 | 1. Provide a Custom Resource Definition(CRD) to spec out a Celery and 40 | Flower deployment having all the configuration options that they 41 | support. 42 | 2. A custom controller implementation that registers and manages 43 | self-healing capabilities of custom Celery resource for these 44 | operations - 45 | 46 | * CREATE - Creates the worker and flower deployments along with 47 | exposing a native Service object for Flower 48 | * UPDATE - Reads the CRD modifications and updates the running 49 | deployments using specified strategy 50 | * DELETE - Deletes the custom resource and all the child deployments 51 | 52 | 3. Support worker autoscaling/downscaling based on resource 53 | constraints(cpu, memory) and task queue length automatically. 54 | 55 | Discussions involving other things that this operator should do based on 56 | your production use-case are welcome. 57 | 58 | Diagram 59 | ------- 60 | 61 | .. figure:: https://i.imgur.com/dTBuG58.png 62 | :alt: CKO Arch Diagram 63 | 64 | Workflow 65 | -------- 66 | 67 | End user starts by writing and creating a YAML spec for the desired celery cluster. Creation event is listened by the Creation Handler(KOPF based) which creates deployment for workers, flower and a Service object to expose flower UI to external users. 68 | 69 | Assuming we have broker in place, any user facing application can start pushing messages to broker now and celery workers will start processing them. 70 | 71 | User can update the custom resource, when that happens, updation handler listening to the event will patch the relevant deployments for change. Rollout strategy can be default or to be specified by user in the spec. 72 | 73 | Both creation and updation handlers will return their statuses to be stored in parent resource's status field. Status field will contain the latest status of the cluster children at all times. 74 | 75 | User can choose to setup autoscaling of workers by resource constraints(CPU, Memory) or broker queue length. Operator will automatically take care of creating an HPA or use KEDA based autoscaling(See `Autoscaling <#Autoscaling>`_ section below) to make that happen. 76 | 77 | Components 78 | ---------- 79 | 80 | Worker Deployment 81 | +++++++++++++++++ 82 | 83 | A Kubernetes `Deployment `_ to manage celery worker pods/replicaset. 84 | These workers consume the tasks from broker and process them. 85 | 86 | Flower Deployment 87 | +++++++++++++++++ 88 | 89 | A Kubernetes `Deployment `_ to manage flower pods/replicaset. Flower is 90 | de-facto standard to monitor and remote control celery. 91 | 92 | Flower Service 93 | ++++++++++++++ 94 | 95 | Expose flower UI to an external IP through a Kubernetes `Service `_ 96 | object. We should additionally explore `Ingress `_ as well. 97 | 98 | Celery CRD(Custom Resource Definition) 99 | ++++++++++++++++++++++++++++++++++++++ 100 | 101 | CRDs are a native way to extend Kubernetes APIs to recognize custom 102 | applications/objects. Celery CRD will contain the schema for celery 103 | cluster to be setup. 104 | 105 | We plan to have following objects in place with their high level description - 106 | 107 | * ``common`` - common configuration parameters for Celery cluster 108 | - ``image`` - Celery application image to be run 109 | - ``imagePullPolicy`` - [Always, Never, IfNotPresent] 110 | - ``imagePullSecrets`` - to pull the image from a private registry 111 | - ``volumeMounts`` - describes mounting of a volume within container. 112 | - ``volumes`` - describes a volume to be used for storage 113 | - ``celeryVersion`` - Celery version 114 | - ``appName`` - App name for worker and flower deployments 115 | - ``celeryApp`` - celery app instance to use (e.g. module.celery_app_attr_name) 116 | * ``workerSpec`` - worker deployment specific parameters 117 | - ``numOfWorkers`` - Number of workers to launch initially 118 | - ``args`` - array of arguments(all celery supported options) to pass to worker process in container (TODO: Entrypoint vs args vs individual params) 119 | - ``rolloutStrategy`` - Rollout strategy to spawn new worker pods 120 | - ``resources`` - optional argument to specify cpu, mem constraints for worker deployment 121 | * ``flowerSpec`` - flower deployment and service specific parameters 122 | - ``replicas`` - Number of replicas for flower deployment 123 | - ``args`` - array of arguments(all flower supported options) to pass to flower process in the container 124 | - ``servicePort`` - Port to expose flower UI in the container 125 | - ``serviceType`` - [Default, NodePort, LoadBalancer] 126 | - ``resources`` - optional argument to specify cpu, mem constraints for flower deployment 127 | * ``scaleTargetRef`` - array of items describing auto scaling targets 128 | - ``kind`` - which application kind to scale (worker, flower) 129 | - ``minReplicas`` - min num of replicas 130 | - ``maxReplicas`` - max num of replicas 131 | - ``metrics`` - list of metrics to monitor 132 | - ``name`` - Enum type (memory, cpu, task_queue_length) 133 | - ``target`` - target values 134 | - ``type`` - [Utilization, Average Value] 135 | - ``averageValue/averageUtilization`` - Average values to maintain 136 | 137 | A more detailed version/documentation for CRD spec is underway. 138 | 139 | Celery CR(Custom Resource) 140 | ++++++++++++++++++++++++++ 141 | 142 | Custom Resource Object for a Celery application. Multiple clusters will 143 | have multiple custom resource objects. 144 | 145 | Custom Controller 146 | +++++++++++++++++ 147 | 148 | `Custom controller `__ implementation to manage Celery applications(CRs). Contains the code for creation, updation, deletion and scaling handlers of the cluster. We plan to use open-source `KOPF `_ (Kubernetes Operator Pythonic Framework) to write this implementation. 149 | 150 | 151 | Async KOPF Handlers(Controller Implementation) 152 | ---------------------------------------------- 153 | 154 | This section contains brief overview of creation and updation handlers 155 | which are going to react on celery resource creation and updation 156 | respectively and return their status to be stored back as resource's 157 | status. 158 | 159 | Creation Handler 160 | ++++++++++++++++ 161 | 162 | Generates deployment spec for worker and flower deployments dynamically 163 | based on incoming parameters specified in custom celery resource. Also 164 | creates the flower service to expose flower UI. Status of each children 165 | is sent back to be stored under parent resource status field. 166 | 167 | Additionally, it might handle the HPA object creation too if the scaling 168 | is to be done on native metrics(CPU and Memory utilization). 169 | 170 | Updation Handler 171 | ++++++++++++++++ 172 | 173 | Updates deployment spec for worker and flower deployments(and service - 174 | HPA) dynamically and patch them. Status of each children is sent back to 175 | be stored under parent resource status field. 176 | 177 | 178 | Autoscaling 179 | ----------- 180 | 181 | This section covers how operator is going to handle autoscaling. We plan 182 | to supporting scaling based on following two metrics. 183 | 184 | Native Metrics(CPU, Memory Utilization) 185 | +++++++++++++++++++++++++++++++++++++++ 186 | 187 | If workers need to be scaled only on CPU/Memory constraints, we can 188 | simply create an HPA object in creation/updation handlers and it'll take 189 | care of scaling relevant worker deployment automatically. HPA supports 190 | these two metrics out of the box. For custom metrics, we need to do 191 | additional work. 192 | 193 | Broker Queue Length(KEDA based autoscaling) 194 | +++++++++++++++++++++++++++++++++++++++++++ 195 | 196 | Queue Length based scaling needs custom metric server for an HPA to 197 | work. `KEDA `__ is a wonderful 198 | option because it is built for the same. It provides the 199 | `scalers `__ for all the popular 200 | brokers(RabbitMQ, Redis, Amazon SQS) supported in Celery. 201 | 202 | KEDA provides multiple ways to be deployed on a Kubernetes cluster - 203 | Helm, Operator Hub and Yaml. Celery Operator can package KEDA along with 204 | itself for distribution. 205 | 206 | 207 | Deployment Strategy 208 | ------------------- 209 | 210 | Probably the best way would be distribute a Helm Chart which packages 211 | CRD, controller and KEDA together(More to be explored here). We'll also 212 | support YAML apply based deployments. 213 | 214 | Additionally, Helm approach is extensible in the sense that we can 215 | package additional components like preferred broker(Redis, RMQ, SQS) as 216 | well to start with Celery on Kubernetes out of the box without much 217 | efforts. 218 | 219 | 220 | Motivation 221 | ========== 222 | 223 | Celery is one of the most popular distributed task queue system and Kubernetes 224 | is the de-facto standard for container-orchestration nowadays. 225 | Running and managing Celery applications on Kubernetes is a largely manual process. 226 | Having a Kubernetes operator to help setup and manage celery 227 | in a consistent and graceful way would benefit users immensely. 228 | We'd also be able to bridge the gap between application engineers and 229 | infrastructure operators who manually manage the celery clusters. 230 | 231 | 232 | Rationale 233 | ========= 234 | 235 | Having this operator will automate the setup and management of a Celery cluster. It'll 236 | also enable Celery project which is written in Python to spearhead the development of 237 | production ready Kubernetes extensions which are generally written in Golang. 238 | 239 | The rationale behind using KOPF and Python to building this operator is that there is a 240 | good learning curve to understand internals and write(also maintain) an operator with Go. 241 | It'll also motivate community to overcome the learning barrier and create useful 242 | libraries, tools and other operators while staying in Python ecosystem. 243 | 244 | 245 | Reference Implementation 246 | ======================== 247 | 248 | Github Repo - https://github.com/brainbreaker/Celery-Kubernetes-Operator 249 | Steps to try out are available in the Readme. 250 | 251 | Also a talk in EuroPython 2020 describing this POC implementation and demo - `Youtube `__ 252 | 253 | Want to Help? 254 | ------------- 255 | 256 | If you're running celery on a Kubernetes cluster, your inputs to how you 257 | manage applications will be valuable. You could contribute to the 258 | discussion `here `__. 259 | 260 | 261 | Future Work 262 | =========== 263 | 264 | `This `__ issue lists open points for the operator 265 | 266 | 267 | Copyright 268 | ========= 269 | 270 | This document has been placed in the public domain per the Creative Commons 271 | CC0 1.0 Universal license (https://creativecommons.org/publicdomain/zero/1.0/deed). -------------------------------------------------------------------------------- /draft/high-level-architecture.rst: -------------------------------------------------------------------------------- 1 | ================================================ 2 | CEP XXXX: Celery NextGen High Level Architecture 3 | ================================================ 4 | 5 | :CEP: XXXX 6 | :Author: Omer Katz 7 | :Implementation Team: Omer Katz 8 | :Shepherd: Omer Katz 9 | :Status: Draft 10 | :Type: Informational 11 | :Created: 2020-05-27 12 | :Last-Modified: 2020-05-27 13 | 14 | .. contents:: Table of Contents 15 | :depth: 4 16 | :local: 17 | 18 | Abstract 19 | ======== 20 | 21 | When Celery was conceived, production environments were radically different from today. 22 | 23 | Nowadays most applications are (or often should be): 24 | 25 | * Deployed to a cloud provider's computing resources. 26 | * Distributed (sometimes between data centers). 27 | * :term:`Available ` or :term:`Consistent ` (If we store state, we must pick one according to :term:`CAP Theorem`). 28 | * :term:`Network Partition Tolerant `. 29 | * :term:`Observable `. 30 | * Built with scalability in mind. 31 | * Cloud Native - The application's lifecycle is managed using Kubernetes, Swarm, or any other scheduler. 32 | * Heterogeneous - Microservices may be written in different languages. 33 | 34 | Also, Celery lacks proper support for large scale deployments and some useful messaging architectural patterns. 35 | 36 | Celery 5 is the next major version of Celery, and so we can break backward compatibility, even in significant ways. 37 | 38 | As such, our next major version should represent the beginning of a paradigm shift 39 | in the way we implement our task execution platform. 40 | Future major versions will drastically change how Celery works. 41 | 42 | This document provides a high-level overview of the new architecture for the next generation of Celery 43 | major versions. 44 | 45 | Specification 46 | ============= 47 | 48 | .. note:: 49 | From now on when we write Celery we refer to Celery NextGen. 50 | 51 | Whenever we refer to a previous major version of Celery we will specify the version number. 52 | 53 | Diagram 54 | ------- 55 | 56 | .. figure:: celery-nextgen-architecture-figure01.png 57 | :figwidth: 100% 58 | :height: 580px 59 | :align: center 60 | 61 | High Level Architecture Diagram 62 | 63 | Preface 64 | ------- 65 | 66 | In Celery 4.x we had the following architectural building blocks: 67 | 68 | - A :term:`Python` Runtime 69 | - :term:`Message Broker` 70 | - :term:`Result Backend` 71 | - :term:`Celery Master` 72 | - :term:`Celery Worker` 73 | - :term:`Celery Beat` 74 | 75 | In addition we had a few optional architectural building blocks (some of them maintained by the community): 76 | 77 | - :term:`Celery Multi` 78 | - :term:`Flower` 79 | - :term:`Cell` 80 | 81 | The only architectural building blocks that remain in Celery are the :term:`Python` Runtime and the :term:`Message Broker`. 82 | The rest are replaced by new ones which provide more functionality and flexibility for our users. 83 | 84 | In the rest of this specification we will describe the architectural building blocks of Celery. 85 | 86 | Python Runtime 87 | -------------- 88 | 89 | Python 2 no longer receives support and security patches from the community as of January 1st, 2020. 90 | Therefore, Celery will drop support for Python 2 and will run on Python 3 and above. 91 | 92 | Celery supports the :term:`CPython` and :term:`PyPy` Python runtimes. 93 | Other runtimes will be considered in different CEPs on a case-by-case basis. 94 | 95 | As a general guideline, we will attempt to keep support for the latest Python version :term:`PyPy` supports. 96 | However, if the need arises we will opt to use new Python language features and hope :term:`PyPy` can catch up. 97 | 98 | Message Types 99 | ------------- 100 | 101 | In Celery 4.x we only have tasks which are serialized to :term:`Command Messages ` 102 | that we publish to the :term:`Message Broker` whenever we want to 103 | execute a :term:`Task`. 104 | 105 | :term:`Document messages ` are the messages we got as a result. 106 | Those message were stored in the :term:`Result Backend`. 107 | They had a specific format which only the Celery 4.x library knew how to parse. 108 | 109 | In Celery, we now have new types of messages we can use. 110 | 111 | :term:`Document Messages ` may now also be produced whenever 112 | we publish a serialized representation of a :term:`Domain Model` to the :term:`Message Broker`. 113 | These messages may be received from a `Data Source `_ or published directly 114 | from the application. 115 | 116 | :term:`Event Messages ` are a new concept for Celery. 117 | They describe that a :term:`Domain Event` occurred. 118 | Multiple tasks can subscribe to an event. 119 | Whenever we receive an :term:`Event Message` we publish those tasks as 120 | :term:`Command Messages ` to the :term:`Message Broker`. 121 | 122 | These fundamental architectural building blocks will aid us in creating a better messaging 123 | system. To encourage :term:`Ubiquitous Language`, we will be using them in this 124 | document when applicable and in Celery's codebase as well. 125 | 126 | Message Broker 127 | -------------- 128 | 129 | In Celery 4.x each :term:`Celery Master` connected to only one :term:`Message Broker` cluster. 130 | 131 | This is no longer the case. 132 | Celery now allows connecting to multiple :term:`Message Brokers ` 133 | even if they are of clusters that use different implementations of a message broker. 134 | 135 | Users can consume messages from a Redis cluster, a RabbitMQ cluster, and an ActiveMQ cluster if they so desire. 136 | 137 | This feature is useful when, for example: 138 | 139 | - The user migrates from a legacy system that uses other implementation of a :term:`Message Broker`, but the new system uses a more modern one. 140 | - The user wants to split the load between clusters. 141 | - There's a security reason to publish some messages to a specific cluster. 142 | 143 | On some :term:`Message Broker` implementations the `Controller`_ will assist in managing the cluster. 144 | 145 | Data Sources & Sinks 146 | -------------------- 147 | 148 | In Celery 4.x we had a :term:`Result Backend` which was used to store task results and coordinate 149 | the execution of chords. 150 | 151 | We extend the :term:`Result Backend` concept further to allow new use cases such: 152 | 153 | - :term:`ETL`. 154 | - :term:`Data Integration`. 155 | - Reporting. 156 | - Taking action when data is inserted or updated. 157 | 158 | In addition, like we did for the `Message Broker`_, we now allow multiple data sources and sinks 159 | instead of one cluster of a :term:`Result Backend`. 160 | 161 | The responsibility for coordination of the execution of chords has moved to the `Execution Platform`_. 162 | 163 | Data Sinks 164 | ~~~~~~~~~~ 165 | 166 | A data sink is where task results are saved. 167 | 168 | A task result may be saved in more than one data sink (e.g. a Kafka Topic and S3). 169 | 170 | The `Router`_ is responsible for routing task results to the correct data sink(s) and properly 171 | serializing them. 172 | 173 | Data Sources 174 | ~~~~~~~~~~~~ 175 | 176 | A data source is a anything that stores data. 177 | It can be a Kafka topic, a S3 bucket, a RDBMS or even your local filesystem. 178 | 179 | Some data sources can notify Celery of incoming data. 180 | Others, Celery needs to poll periodically using the `Task Scheduler`_. 181 | 182 | The `Router`_ is responsible for listening to incoming data from the various data sources 183 | connected to it. 184 | 185 | Whenever the Router receives incoming data it sends a :term:`Document Message` to the 186 | `Publisher`_ which in turn will publish it to the :term:`Message Broker`. 187 | 188 | Controller 189 | ---------- 190 | 191 | In Celery 4.x we provided a basic tool for controlling Celery instances called :term:`Celery Multi`. 192 | We also provided :term:`Celery Beat` for periodically scheduling tasks. 193 | 194 | The Controller replaces those two components and extends their functionality in many ways. 195 | 196 | The Controller is responsible for managing the lifecycle of all other Celery 197 | components. 198 | 199 | Celery is a complex system with multiple components and will often be 200 | deployed in high throughput, highly available, cloud-native production systems. 201 | 202 | The introduction of multiple components require us to have another component 203 | that manages the entire Celery cluster. 204 | 205 | Instead of controlling Celery instances on one machine in a way that is agnostic to the production environment we're 206 | operating in the Controller now provides a `Platform Manager`_ which manages Celery instances on one or many 207 | machines. 208 | 209 | The Controller also manages and optimizes the 210 | execution of tasks to ensure we maximize the utilization of all our resources 211 | and to prevent expected errors. 212 | 213 | That is why the Controller is now responsible for auto-scaling Celery instances, rate-limiting tasks, 214 | task concurrency limitations, task execution prioritization and all other management operations a 215 | user needs to operate a Celery cluster. 216 | 217 | Platform Manager 218 | ~~~~~~~~~~~~~~~~ 219 | 220 | The Platform Manager is responsible for interacting with the production environment. 221 | 222 | The platform itself can be anything Celery can run on e.g.: Pre-fork, SystemD, OpenRC, Docker, Swarm, Kubernetes, Nomad. 223 | 224 | Each implementation of the Platform Manager will be provided in a different package. 225 | Some of them will be maintained by the community. 226 | 227 | Foreman 228 | ~~~~~~~ 229 | 230 | The Foreman is responsible for deploying & managing the lifecycle of all Celery instances and ensuring 231 | they stay up and running. 232 | 233 | It can spawn new instances of Celery processes, stop or restart them either on demand or based on policies 234 | the user has specified for auto-scaling. 235 | 236 | It interacts with the `Platform Manager`_ to do so on the platform the Controller manages. 237 | 238 | On some platforms, the Foreman can instruct the `Platform Manager` to deploy and manage the 239 | lifecycle of :term:`Message Brokers `, `Data Sources `_ & `Data Sinks `_. 240 | 241 | Task Scheduler 242 | ~~~~~~~~~~~~~~ 243 | 244 | The Task Scheduler is responsible for managing the scheduling of tasks for execution 245 | on a cluster of workers. 246 | 247 | The scheduler calculates the amount of tasks to be executed in any given time 248 | in order to make cluster wide decisions when autoscaling workers or increasing 249 | concurrency for an existing worker. 250 | 251 | The scheduler is aware when tasks should no longer be executed for any reason. 252 | To do so, it commands the router to avoid consuming the task or rejecting it. 253 | 254 | Publisher 255 | --------- 256 | 257 | The Publisher is responsible for publishing :term:`Messages ` 258 | to a :term:`Message Broker`. 259 | 260 | It is responsible for publishing a :term:`Message` to the appropriate :term:`Message Broker` cluster 261 | according to the configuration provided to the publisher. 262 | 263 | The publisher must be able to run in-process inside a long-running thread 264 | or a long running co-routine. 265 | 266 | It can also be run using a separate daemon which can serve all the processes 267 | publishing to the message brokers. 268 | 269 | Whenever the :term:`Message Broker` cluster is unavailable or unresponsive, the Publisher stores 270 | the :term:`Messages ` in the Messages Backlog. 271 | The Publisher will later retry publishing the message. 272 | 273 | Router 274 | ------ 275 | 276 | In Celery 4.x there was a :term:`Celery Master` which was responsible for consuming :term:`Messages `, storing 277 | results and executing canvases. 278 | 279 | In Celery we now decouple the execution of the tasks completely from the routing of :term:`Messages ` 280 | to the `Execution Platform`_. 281 | 282 | The Router is a combination of the following architectural patterns: 283 | - :term:`Event Driven Consumer` 284 | - :term:`Message Dispatcher` 285 | - :term:`Service Activator` 286 | - :term:`Process Manager`. 287 | 288 | The Router consumes :term:`Messages ` from a :term:`Message Broker` cluster or multiple clusters, 289 | dispatches the consumed :term:`Messages ` to the `Execution Platform`_ for processing and awaits 290 | for the results of the task and stores them in the appropriate `Data Sink(s) `_. 291 | 292 | When the execution of a task requires coordination in case of a workflow, an :term:`Event Message` or an incoming 293 | :term:`Document Message` from a `Data Source `_ the Router is responsible for the order of the execution 294 | of the task(s). 295 | 296 | Execution Platform 297 | ------------------ 298 | 299 | In Celery 4.x we had :term:`Celery Worker` which was responsible for executing tasks. 300 | 301 | In Celery we now have an execution platform which will execute tasks and report their results back to the `Router`_. 302 | 303 | Celery will provide a default implementation but other implementations may be provided as well by the community or 304 | the maintainers. 305 | 306 | Alternative implementations may provide task execution on :term:`Serverless Computing` platforms, as Kubernetes Jobs 307 | for example or provide task execution for tasks written in a different language. 308 | 309 | Motivation 310 | ========== 311 | 312 | We want to modernize Celery for the Cloud Native age. 313 | We need to keep Celery relevant for our users and help them in new ways. Therefore, we must adjust and evolve to meet the unique challenges of the Cloud Native age. 314 | 315 | Also, we want to modernize the code to support Python 3+, which will allow us to remove workarounds, backports, and compatibility shims. 316 | Refactoring the codebase to support Python 3+ allows us to keep a slimmer, more maintainable codebase. 317 | 318 | Furthermore, we'd like to resolve long-standing design bugs in our implementation. 319 | 320 | Gradually evolving our codebase is currently not possible due to the many changes 321 | in technology since Celery was conceived. 322 | We need to move fast and break things until we match all our goals. 323 | 324 | Rationale 325 | ========= 326 | 327 | This architecture moves Celery from the past to the present. 328 | 329 | We have the following goals: 330 | 331 | - Observability 332 | - Cloud Nativity 333 | - Flexibility 334 | - Increased Task Completion Throughput 335 | 336 | The proposed architecture will guarantee that those goals will be met. 337 | 338 | Backwards Compatibility 339 | ======================= 340 | 341 | As evident from this document, Celery NextGen is not backwards compatible with previous versions of Celery. 342 | 343 | The syntax for defining tasks will remain the same so you can reuse your code with little to no 344 | adjustments. 345 | 346 | Reference Implementation 347 | ======================== 348 | 349 | Copyright 350 | ========= 351 | 352 | This document has been placed in the public domain per the Creative Commons 353 | CC0 1.0 Universal license (https://creativecommons.org/publicdomain/zero/1.0/deed). 354 | -------------------------------------------------------------------------------- /glossary.rst: -------------------------------------------------------------------------------- 1 | ======== 2 | Glossary 3 | ======== 4 | 5 | .. glossary:: 6 | 7 | Message Broker 8 | `Enterprise Integration Patterns`_ defines a `Message Broker`_ as an architectural 9 | building block that can receive :term:`messages ` from 10 | multiple destinations, determine the correct destination and route the message 11 | to the correct channel. 12 | 13 | Message 14 | `Enterprise Integration Patterns`_ defines a `Message`_ as data record that 15 | the messaging system can transmit through a message channel. 16 | 17 | Command Message 18 | `Enterprise Integration Patterns`_ defines a `Command Message`_ as a 19 | :term:`Message` which instructs a worker to execute a task. 20 | 21 | Event Message 22 | `Enterprise Integration Patterns`_ defines an `Event Message`_ as a 23 | :term:`Message` which indicates that an event has occurred. 24 | 25 | Document Message 26 | `Enterprise Integration Patterns`_ defines a `Document Message`_ as a 27 | :term:`Message` containing data from a data source. 28 | 29 | Service Activator 30 | `Enterprise Integration Patterns`_ defines a `Service Activator`_ as a 31 | one-way (request only) or two-way (request-reply) adapter between the 32 | :term:`Message` and the service it invokes. 33 | The service can be a simple as a method call. 34 | The activator handles all of the messaging details and invokes the service 35 | like any other client, such that the service doesn’t even know it’s being 36 | invoked through messaging. 37 | 38 | Idempotent Receiver 39 | `Enterprise Integration Patterns`_ defines an `Idempotent Receiver`_ as a 40 | component that can safely receive the same message multiple times 41 | but will produce the same side effects when facing duplicated messages. 42 | 43 | Message Dispatcher 44 | `Enterprise Integration Patterns`_ defines a `Message Dispatcher`_ as a 45 | component that will consume messages from a channel and distribute them to 46 | performers. 47 | 48 | Process Manager 49 | `Enterprise Integration Patterns`_ defines a `Process Manager`_ as a component 50 | that maintains the state of the sequence and determines the next processing step 51 | based on intermediate results. 52 | 53 | Event Driven Consumer 54 | `Enterprise Integration Patterns`_ defines an `Event Driven Consumer`_ as a component 55 | that consumes a message as soon as it is delivered. 56 | 57 | Circuit Breaker 58 | Martin Fowler defines a `Circuit Breaker`_ in the following fashion: 59 | 60 | The basic idea behind the circuit breaker is very simple. 61 | You wrap a protected function call in a circuit breaker object, which monitors 62 | for failures. 63 | Once the failures reach a certain threshold, the circuit breaker trips, 64 | and all further calls to the circuit breaker return with an error, 65 | without the protected call being made at all. 66 | Usually you'll also want some kind of monitor alert if the circuit breaker 67 | trips. 68 | 69 | CAP Theorem 70 | The `CAP theorem`_ categorizes systems into three categories: 71 | 72 | * CP (:term:`Consistent ` and :term:`Partition Tolerant`) — At first glance, the CP category is confusing, i.e., a system that is consistent and partition tolerant but never available. CP is referring to a category of systems where availability is sacrificed only in the case of a network partition. 73 | * CA (:term:`Consistent ` and :term:`Available `) — CA systems are consistent and available systems in the absence of any network partition. Often a single node's DB servers are categorized as CA systems. Single node DB servers do not need to deal with partition tolerance and are thus considered CA systems. The only hole in this theory is that single node DB systems are not a network of shared data systems and thus do not fall under the preview of CAP. 74 | * AP (:term:`Available ` and :term:`Partition Tolerant`) — These are systems that are available and partition tolerant but cannot guarantee consistency. 75 | 76 | Consistency 77 | A guarantee that every node in a distributed cluster returns the same, most recent, successful write. 78 | Consistency refers to every client having the same view of the data. 79 | There are various types of consistency models. 80 | Consistency in CAP (used to prove the theorem) refers to linearizability or sequential consistency, a very strong form of consistency. 81 | 82 | Availability 83 | Every non-failing node returns a response for all read and write requests in a reasonable amount of time. 84 | The key word here is every. 85 | To be available, every node on (either side of a network partition) must be able to respond in a reasonable 86 | amount of time. 87 | 88 | Partition Tolerant 89 | The system continues to function and upholds its consistency guarantees in spite of network partitions. Network partitions are a fact of life. 90 | Distributed systems guaranteeing partition tolerance can gracefully recover from partitions once the partition heals. 91 | 92 | Fault Tolerance 93 | TODO 94 | 95 | Network Resilience 96 | According to Wikipedia `Network Resilience`_ is: 97 | 98 | In computer networking: resilience is the ability to provide and maintain 99 | an acceptable level of service in the face of faults and challenges to 100 | normal operation.” 101 | Threats and challenges for services can range from simple misconfiguration 102 | over large scale natural disasters to targeted attacks. 103 | As such, network resilience touches a very wide range of topics. 104 | In order to increase the resilience of a given communication network, 105 | the probable challenges and risks have to be identified 106 | and appropriate resilience metrics have to be defined 107 | for the service to be protected. 108 | 109 | Monitoring 110 | According to `fastly`_ monitoring is: 111 | 112 | The activity of observing the state of a system over time. 113 | It uses instrumentation for problem detection, resolution, 114 | and continuous improvement. 115 | Monitoring alerts are reactive–they tell you when a known issue has 116 | already occurred 117 | (i.e. maybe your available memory is too low or you need more compute). 118 | Monitoring provides automated checks that you can execute against a 119 | distributed system to make sure that none of the things you predicted 120 | signify any trouble. 121 | While monitoring these known quantities is important, 122 | the practice also has limitations, including the fact that you are only 123 | looking for known issues. Which begs an important question, 124 | “what about the problems that you didn’t predict?” 125 | 126 | Observability 127 | According to Wikipedia in the context of control theory `Observability`_ is: 128 | 129 | In control theory, observability is a measure of how well internal states 130 | of a system can be inferred from knowledge of its external outputs. 131 | 132 | In the context of distributed systems observability is a super-set of 133 | :term:`Monitoring`. 134 | 135 | According to `fastly`_ the three pillars of observability are: 136 | 137 | Logs: Logs are a verbose representation of events that have happened. 138 | Logs tell a linear story about an event using string processing 139 | and regular expressions. 140 | A common challenge with logs is that if you haven’t properly indexed 141 | something, it will be difficult to find due to the sheer volume of 142 | log data. 143 | Traces: A trace captures a user’s journey through your application. 144 | Traces provide end-to-end visibility and are useful when you need to 145 | identify which components cause system errors, find performance 146 | bottlenecks, or monitor flow through modules. 147 | Metrics: Metrics can be either a point in time or monitored over 148 | intervals. 149 | These data points could be counters, gauges, etc. 150 | They typically represent data over intervals, but sometimes sacrifice 151 | details of an event in order to present data that is easier to assimilate. 152 | 153 | Structured Logging 154 | Structured Logging is a method to make log messages easy to process 155 | by machines. 156 | A usual log message is a timestamp, level and a message string. 157 | The context describing the logged event is embedded inside the message 158 | string. 159 | A structured log message store their context in a predetermined message 160 | format which allows machines to parse them more easily. 161 | 162 | JSON 163 | JSON stands for JavaScript Object Notation, which is a way to format data so 164 | that it can be transmitted from one place to another, most commonly between 165 | a server and a Web application. 166 | 167 | stdout 168 | Stdout, also known as standard output, is the default file descriptor 169 | where a process can write output. 170 | 171 | Service Locator 172 | Martin Fowler defines a `Service Locator`_ in the following fashion: 173 | 174 | The basic idea behind a service locator is to have an object that knows 175 | how to get hold of all of the services that an application might need. 176 | So a service locator for this application would have a method that returns 177 | a movie finder when one is needed. 178 | 179 | GIL 180 | The Global Interpreter Lock, abbreviated as the `GIL`_ is a mutex which 181 | prevents executing threads in parallel if both are about to execute a python 182 | bytecode. 183 | 184 | This is by design since Python has many atomic operations and maintaining 185 | individual locks on each object results in slower execution. 186 | 187 | Depending on the implementation, a thread may be forced to release the `GIL`_ 188 | when a condition is met. In CPython's implementation of Python 3, 189 | a thread is forced to release the `GIL`_ after a it executes for a period of 190 | time. 191 | 192 | A thread may also release the `GIL`_ voluntarily when it uses a system call 193 | or when a C extension instructs to do so. 194 | 195 | IPC 196 | According to Wikipedia `Inter-process Communication`_: 197 | 198 | refers specifically to the mechanisms an operating system provides to allow 199 | the processes to manage shared data. 200 | Typically, applications can use IPC, categorized as clients and servers, 201 | where the client requests data and the server responds to client requests. 202 | Many applications are both clients and servers, as commonly seen in 203 | distributed computing. 204 | 205 | There are many `approaches `_ 206 | to IPC. Some of them are available in all operating systems, some are 207 | only available in specific operating systems. 208 | 209 | Task 210 | A task is a unit of business logic that is completely independent and can be 211 | executed regardless of the execution platform. 212 | 213 | Domain Model 214 | Martin Fowler defines a `Domain Model`_ in the following fashion: 215 | 216 | An object model of the domain that incorporates both behavior and data. 217 | 218 | Domain Event 219 | Martin Fowler defines a `Domain Event`_ in the following fashion: 220 | 221 | I go to Babur's for a meal on Tuesday, and pay by credit card. 222 | This might be modeled as an event, whose event type is 'make purchase', 223 | whose subject is my credit card, and whose occurred date is Tuesday. 224 | If Babur's uses an old manual system and doesn't transmit the transaction 225 | until Friday, the noticed date would be Friday. 226 | 227 | Things happen. Not all of them are interesting, some may be worth 228 | recording but don't provoke a reaction. 229 | The most interesting ones cause a reaction. 230 | Many systems need to react to interesting events. 231 | Often you need to know why a system reacts in the way it did. 232 | 233 | By funneling inputs to a system into streams of Domain Event you can keep 234 | a record of all the inputs to a system. 235 | This helps you organize your processing logic, and also allows you to keep 236 | an audit log of the inputs to the system. 237 | 238 | Serverless Computing 239 | TODO 240 | 241 | Ubiquitous Language 242 | TODO 243 | 244 | Result Backend 245 | TODO 246 | 247 | Celery Master 248 | TODO 249 | 250 | Celery Worker 251 | TODO 252 | 253 | Celery Multi 254 | TODO 255 | 256 | Celery Beat 257 | TODO 258 | 259 | Flower 260 | TODO 261 | 262 | Cell 263 | TODO 264 | 265 | ETL 266 | TODO 267 | 268 | Data Integration 269 | TODO 270 | 271 | Python 272 | Python is an easy to learn, powerful programming language. 273 | It has efficient high-level data structures and a simple but effective approach to object-oriented programming. 274 | Python’s elegant syntax and dynamic typing, together with its interpreted nature, 275 | make it an ideal language for scripting and rapid application development in many areas on most platforms. 276 | 277 | CPython 278 | CPython is the reference implementation of the Python programming language. 279 | Written in C and Python, CPython is the default and most widely used implementation of the language. 280 | 281 | PyPy 282 | PyPy is a replacement for CPython. 283 | It is built using the RPython language that was co-developed with it. 284 | The main reason to use it instead of CPython is speed: it runs generally faster. 285 | 286 | Copyright 287 | ========= 288 | 289 | This document has been placed in the public domain per the Creative Commons 290 | CC0 1.0 Universal license (https://creativecommons.org/publicdomain/zero/1.0/deed). 291 | 292 | .. _CAP theorem: https://dzone.com/articles/understanding-the-cap-theorem 293 | .. _Enterprise Integration Patterns: https://www.enterpriseintegrationpatterns.com 294 | .. _Message: https://www.enterpriseintegrationpatterns.com/patterns/messaging/Message.html 295 | .. _Command Message: https://www.enterpriseintegrationpatterns.com/patterns/messaging/CommandMessage.html 296 | .. _Event Message: https://www.enterpriseintegrationpatterns.com/patterns/messaging/EventMessage.html 297 | .. _Document Message: https://www.enterpriseintegrationpatterns.com/patterns/messaging/DocumentMessage.html 298 | .. _Message Dispatcher: https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageDispatcher.html 299 | .. _ubiquitous language: https://martinfowler.com/bliki/UbiquitousLanguage.html 300 | .. _Message Broker: https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageBroker.html 301 | .. _Circuit Breaker: https://martinfowler.com/bliki/CircuitBreaker.html 302 | .. _Network Resilience: https://en.wikipedia.org/wiki/Resilience_(network) 303 | .. _Observability: https://en.wikipedia.org/wiki/Observability 304 | .. _fastly: https://www.fastly.com/blog/monitoring-vs-observability 305 | .. _Service Locator: https://martinfowler.com/articles/injection.html#UsingAServiceLocator 306 | .. _Service Activator: https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessagingAdapter.html 307 | .. _Idempotent Receiver: https://www.enterpriseintegrationpatterns.com/patterns/messaging/IdempotentReceiver.html 308 | .. _Inter-process Communication: https://en.wikipedia.org/wiki/Inter-process_communication 309 | .. _Domain Event: https://martinfowler.com/eaaDev/DomainEvent.html 310 | .. _Domain Model: https://martinfowler.com/eaaCatalog/domainModel.html 311 | .. _GIL: https://realpython.com/python-gil/ 312 | .. _Process Manager: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ProcessManager.html 313 | .. _Event Driven Consumer: https://www.enterpriseintegrationpatterns.com/patterns/messaging/EventDrivenConsumer.html -------------------------------------------------------------------------------- /final/0001-cep-process.rst: -------------------------------------------------------------------------------- 1 | .. vale off 2 | 3 | ================================= 4 | CEP 1: CEP Purpose and Guidelines 5 | ================================= 6 | 7 | :CEP: 1 8 | :Author: Omer Katz 9 | :Status: Final 10 | :Type: Process 11 | :Created: 2019-04-03 12 | :Last-Modified: 2019-04-03 13 | 14 | .. contents:: Table of Contents 15 | :depth: 3 16 | :local: 17 | 18 | What is a CEP? 19 | ============== 20 | 21 | CEP stands for Celery Enhancement Proposal. A CEP is a design document 22 | providing information to the Celery community, or describing a new feature 23 | or process for Celery. CEPs provide concise technical specifications of 24 | features, along with rationales. 25 | 26 | We intend CEPs to be the primary mechanisms for proposing major new features, 27 | for collecting community input on issues, and for documenting design 28 | decisions that have gone into Celery. 29 | 30 | The concept and implementation of CEPs (and this document itself!) is a nearly 31 | direct copy of `Python's PEP process `_ . If 32 | you're already familiar with PEPs, you should be able to quickly grok CEPs by 33 | reading the `differences between CEPs and PEPs`_. 34 | 35 | CEP Types 36 | ========= 37 | 38 | There are three kinds of CEPs: 39 | 40 | 1. A **Feature** CEP describes a new feature or implementation 41 | for Celery. Most CEPs will be Feature CEPs. 42 | 43 | 2. An **Informational** CEP describes a Celery design issue, or 44 | provides general guidelines or information to the Celery community 45 | but does not propose a new feature. Informational CEPs do not 46 | necessarily represent a community consensus or 47 | recommendation, so users and implementers are free to ignore 48 | Informational CEPs or follow their advice. 49 | 50 | 3. A **Process** CEP describes a process surrounding Celery, or 51 | proposes a change to (or an event in) a process. Process CEPs are 52 | like Feature CEPs but apply to areas other than the Celery 53 | framework itself. They may propose an implementation, but not to 54 | Celery's codebase; they often require community consensus; unlike 55 | Informational CEPs, they are more than recommendations, and users 56 | are typically not free to ignore them. Examples include 57 | procedures, guidelines, changes to the decision-making process, and 58 | changes to the tools or environment used in Celery development. 59 | Any meta-CEP is also considered a Process CEP. (So this document 60 | is a Process CEP). 61 | 62 | CEP submission workflow 63 | ======================= 64 | 65 | So, you'd like to submit a CEP? Here's how it works, and what to expect. 66 | 67 | There are a couple of terms you should be familiar with before reading the 68 | rest of this document: 69 | 70 | The Technical Board 71 | There are several reference in this CEP to the **Technical Board** 72 | (sometimes just "the Board"). This refers to Celery's Technical Board, 73 | the group of experienced and active committers who steer technical 74 | choices. 75 | 76 | Core Developers 77 | Similarly, there are several references to **Core Developers** (sometimes 78 | "core devs"). This refers to the members of Celery's core team, 79 | and specifically those with commit access. 80 | 81 | At a very high level, the CEP submission process looks like this: 82 | 83 | 1. `Pre-proposal`_ — someone has an idea and starts collecting early input and 84 | feedback to see if it's worth writing a CEP. 85 | 86 | 2. `Forming the team`_ — the CEP author rounds up the help they'll need to get 87 | the CEP considered. 88 | 89 | 3. `Submitting the draft`_ — the CEP author writes a rough draft of the CEP and 90 | submits it via pull request. 91 | 92 | 4. `Discussion, development, and updates`_ — the CEP and reference 93 | implementation are discussed, improved, and updated as feedback comes in. 94 | 95 | 5. `Review & Resolution`_ — the CEP is reviewed by the Technical Board and 96 | either accepted or rejected. 97 | 98 | 6. `Implementation`_ — the implementation of the proposed feature is completed 99 | by the CEP team. 100 | 101 | For details on each step, read on. 102 | 103 | Pre-proposal 104 | ------------ 105 | 106 | The CEP process begins with a new idea for Celery. It is highly recommended 107 | that a single CEP contain a single key proposal or new idea. Small enhancements 108 | or patches usually don't need a CEP and follow Celery's normal `contribution 109 | process `_. 110 | 111 | The more focused the CEP, the more successful it tends to be. The Core 112 | Developers reserve the right to reject CEP proposals if they appear too 113 | unfocused or too broad. If in doubt, split your CEP into several well-focused 114 | ones. 115 | 116 | The CEP Author (see below for the formal definition of an Author) 117 | should first attempt to ascertain whether the idea is CEP-able. 118 | Opening an issue on the `celery/ceps `_ 119 | repository is the best way to do so. 120 | 121 | Vetting an idea publicly before going as far as writing a CEP is meant to save 122 | the potential author time. Many ideas have been brought forward for changing 123 | Celery that have been rejected for various reasons. Asking the Celery community 124 | first if an idea is original helps prevent too much time being spent on 125 | something that is guaranteed to be rejected based on prior discussions 126 | (searching the Internet does not always do the trick). It also helps to make 127 | sure the idea is applicable to the entire community and not just the author. 128 | Just because an idea sounds good to the author does not mean it will work for 129 | most people in most areas where Celery is used. 130 | 131 | Forming the team 132 | ---------------- 133 | 134 | Once a CEP has been roughly validated, the author needs to fill out three vital 135 | roles. These roles will be required to get a CEP read, approved, and the code 136 | developed, so you need to identify up-front who will do what. These roles are: 137 | 138 | Author 139 | The **Author** writes the CEP using the style and format described below 140 | (see `CEP format`_), shepherds the discussions in the appropriate forums, 141 | and attempts to build community consensus around the idea. 142 | 143 | Implementation Team 144 | The **Implementation Team** are the people (or single person) who will 145 | actually implement the thing being proposed. A CEP may have multiple 146 | implementers (and the best CEPs probably will). 147 | 148 | Feature CEPs must have an implementation team to be submitted. Informational 149 | CEPs generally don't have implementers, and Process CEPs sometimes will. 150 | 151 | Shepherd 152 | The **Shepherd** is the Core Developer who will be the primary reviewer 153 | of the CEP on behalf of the Celery team, will be the main point person 154 | who will help the Author assess the fitness of their proposal, and 155 | is the person who will finally submit the CEP for pronouncement by the 156 | Technical Board. When the implementation team doesn't contain someone 157 | who can commit to Celery, the Shepherd will be the one who actually merges 158 | the code into the project. 159 | 160 | It's normal for a single person to fulfill multiple roles -- in most cases the 161 | Author will be an/the Implementer, and it's not uncommon for the implementation 162 | team to include the Shepherd as well. It's unusual but acceptable for a single 163 | person to fulfill all roles, though this generally only happens when that person 164 | is a long-time committer. 165 | 166 | Submitting the draft 167 | -------------------- 168 | 169 | Once the idea's been vetted and the roles are filled, a draft CEP should be 170 | presented to Celery-developers. This gives the author a chance to flesh out the 171 | draft CEP to make sure it's properly formatted, of high quality, and to address 172 | initial concerns about the proposal. 173 | 174 | Following the discussion on Celery-developers, the proposal should be sent as a 175 | GitHub pull request to the `celery/ceps `_ 176 | repository. 177 | This PR should add a CEP to the ``drafts/`` directory, written in the style 178 | described below. The draft must be written in CEP style; if it isn't the pull 179 | request may be rejected until proper formatting rules are followed. 180 | 181 | At this point, a core dev will review the pull request. In most cases the 182 | reviewer will be the Shepherd of the CEP, but if that's not possible for some 183 | reason the author may want to ask on Celery-developers to ensure that this 184 | review happens quickly. The reviewer will do the following: 185 | 186 | * Read the CEP to check if it is ready: sound and complete. The ideas 187 | must make technical sense, even if they don't seem likely to be 188 | accepted. 189 | 190 | * Make sure the title accurately describes the content. 191 | 192 | * Check the CEP for language (spelling, grammar, sentence structure, 193 | etc.), markup, and code style (examples should match PEP 8). 194 | 195 | If the CEP isn't ready, the reviewer can leave comments on the pull request, 196 | asking for further revisions. If the CEP's really in bad form, the reviewer 197 | may reject the pull request outright and ask the author to submit a new one 198 | once the problems have been fixed. 199 | 200 | The reviewer doesn't pass judgment on CEPs. They merely do the administrative & 201 | editorial part (which is generally a low volume task). 202 | 203 | Once the CEP is ready for the repository, the reviewer will: 204 | 205 | * Merge the pull request. 206 | 207 | * Assign a CEP number (almost always just the next available number), and rename 208 | the CEP file with the new number (e.g. rename ``dep-process.rst`` to 209 | ``0001-dep-process.rst``) 210 | 211 | Developers with commit access to the CEPs repo may create drafts directly by 212 | committing and pushing a new CEP. However, when doing so they need to take on 213 | the tasks normally handled by the reviewer described above. This includes 214 | ensuring the initial version meets the expected standards for submitting a CEP. 215 | Of course, committers may still choose to submit CEPs as a pull request to 216 | benefit from peer review. 217 | 218 | Discussion, development, and updates 219 | ------------------------------------ 220 | 221 | At this point there will generally be more discussion, modifications to the 222 | reference implementation, and of course updates to the CEP. It's rare for 223 | a CEP to be judged on the first draft; far more common is several rounds 224 | of feedback and updates. 225 | 226 | Updates to a CEP can be submitted as pull requests; once again, 227 | a core developer will merge those pull requests (typically they don't 228 | require much if any review). In cases where the Author has commit access 229 | (fairly common), the Author should just update the draft CEP directly. 230 | 231 | Feature CEPs generally consist of two parts, a design document and a 232 | reference implementation. It is generally recommended that at least a 233 | prototype implementation be co-developed with the CEP, as ideas that sound 234 | good in principle sometimes turn out to be impractical when subjected to the 235 | test of implementation. 236 | 237 | CEP authors are responsible for collecting community feedback on a CEP 238 | before submitting it for review. However, wherever possible, long 239 | open-ended discussions on the relevant issue should be avoided. 240 | Strategies to keep the discussions efficient include: setting up a 241 | separate communication channel for the topic, having the CEP author accept 242 | private comments in the early design phases, setting up a wiki page, etc. 243 | CEP authors should use their discretion here. 244 | 245 | Review & Resolution 246 | ------------------- 247 | 248 | Once the author has completed a CEP, the shepherd will ask the Technical Board 249 | for review and pronouncement. The final authority for deciding on a CEP rests 250 | with the Technical Board. They may choose to rule on a CEP as a team, or they 251 | may designate one or more board members to review and decide. 252 | 253 | Having the shepherd (i.e. a core dev) rather than the author ask helps ensure 254 | that the CEP meets the basic technical bar before it's called for review. It 255 | also provides a fairly strong fitness test before the board is asked to rule on 256 | it, making board rulings fairly easy. If the core developer shepherd is happy, 257 | the board will likely be as well. 258 | 259 | For a CEP to be accepted it must meet certain minimum criteria. It must be a 260 | clear and complete description of the proposed enhancement. The enhancement must 261 | represent a net improvement. The proposed implementation, if applicable, must be 262 | solid and must not complicate Celery unduly. Finally, a proposed enhancement 263 | must "fit" with Celery's general philosophy and architecture. This last category 264 | is the most imprecise and takes the most judgment, so if the Board rejects a 265 | CEP for lack of "fit" they should provide a clear explanation for why. 266 | 267 | At this point, the CEP will be considered "Accepted" and moved to the 268 | ``accepted`` directory in the CEPs repo. 269 | 270 | A CEP can also be "Withdrawn". The CEP author or a core developer can assign 271 | the CEP this status when the author is no longer interested in the CEP, or if no 272 | progress is being made on the CEP. Once a CEP is withdrawn, it's moved 273 | to the ``withdrawn`` directory for reference. Later, another author may 274 | resurrect the CEP by opening a pull request, updating (at least) the author, 275 | and moving it back to ``draft``. 276 | 277 | Finally, a CEP can also be "Rejected". Perhaps after all is said and done it 278 | was not a good idea. It is still important to have a record of this 279 | fact. Rejected CEPs will be moved to the ``rejected`` directory, and 280 | generally should be updated with a rationale for rejection. 281 | 282 | CEPs can also be superseded by a different CEP, rendering the original 283 | obsolete. This is intended for Informational CEPs, where version 2 of 284 | an API can replace version 1. 285 | 286 | Implementation 287 | -------------- 288 | 289 | Finally, once a CEP has been accepted, the implementation must be completed. In 290 | many cases some (or all) implementation will actually happen during the CEP 291 | process: Feature CEPs will often have fairly complete implementations before 292 | being reviewed by the board. When the implementation is complete and 293 | incorporated into the main source code repository, the status will be changed to 294 | "Final" and the CEP moved to the ``final`` directory. 295 | 296 | CEP format 297 | ========== 298 | 299 | To save everyone time reading CEPs, they need to follow a common format 300 | and outline; this section describes that format. In most cases, it's probably 301 | easiest to start with copying the provided `CEP template <../template.rst>`_, 302 | and filling it in as you go. 303 | 304 | CEPs must be written in `reStructuredText `_ 305 | (the same format as Celery's documentation). 306 | 307 | Each CEP should have the following parts: 308 | 309 | #. A short descriptive title (e.g. "canvas-dsl"), which is also reflected 310 | in the CEP's filename (e.g. ``0181-canvas-dsl.rst``). 311 | 312 | #. A preamble -- a rST `field list `_ 313 | containing metadata about the CEP, including the CEP number, the names of the 314 | various members of the `CEP team <#forming- the-team>`_, and so forth. See 315 | `CEP Metadata`_ below for specific details. 316 | 317 | #. Abstract -- a short (~200 word) description of the technical issue 318 | being addressed. 319 | 320 | #. Specification -- The technical specification should describe the syntax and 321 | semantics of any new feature. The specification should be detailed enough to 322 | allow implementation -- that is, developers other than the author should 323 | (given the right experience) be able to independently implement the feature, 324 | given only the CEP. 325 | 326 | #. Motivation -- The motivation is critical for CEPs that want to add 327 | substantial new features or materially refactor existing ones. It should 328 | clearly explain why the existing solutions are inadequate to address the 329 | problem that the CEP solves. CEP submissions without sufficient motivation 330 | may be rejected outright. 331 | 332 | #. Rationale -- The rationale fleshes out the specification by describing what 333 | motivated the design and why particular design decisions were made. It 334 | should describe alternate designs that were considered and related work. 335 | 336 | The rationale should provide evidence of consensus within the community and 337 | discuss important objections or concerns raised during discussion. 338 | 339 | #. Backwards Compatibility -- All CEPs that introduce backwards 340 | incompatibilities must include a section describing these incompatibilities 341 | and their severity. The CEP must explain how the author proposes to deal 342 | with these incompatibilities. CEP submissions without a sufficient backwards 343 | compatibility treatise may be rejected outright. 344 | 345 | #. Reference Implementation -- The reference implementation must be completed 346 | before any CEP is given status "Final", but it need not be completed before 347 | the CEP is accepted. While there is merit to the approach of reaching 348 | consensus on the specification and rationale before writing code, the 349 | principle of "rough consensus and running code" is still useful when it comes 350 | to resolving many discussions of API details. 351 | 352 | The final implementation must include tests and documentation, per Celery's 353 | `contribution guidelines `_. 354 | 355 | #. Copyright/public domain -- Each CEP must be explicitly licensed 356 | as `CC0 `_. 357 | 358 | CEP Metadata 359 | ------------ 360 | 361 | Each CEP must begin with some metadata given as an rST 362 | `field list `_. 363 | The headers must contain the following fields: 364 | 365 | ``CEP`` 366 | The CEP number. In an initial pull request, this can be left out or given 367 | as ``XXXX``; the reviewer who merges the pull request will assign the CEP 368 | number. 369 | ``Type`` 370 | ``Feature``, ``Informational``, or ``Process`` 371 | ``Status`` 372 | ``Draft``, ``Accepted``, ``Rejected``, ``Withdrawn``, ``Final``, or ``Superseded`` 373 | ``Created`` 374 | Original creation date of the CEP (in ``yyyy-mm-dd`` format) 375 | ``Last-Modified`` 376 | Date the CEP was last modified (in ``yyyy-mm-dd`` format) 377 | ``Author`` 378 | The CEP's author(s). 379 | ``Implementation-Team`` 380 | The person/people who have committed to implementing this CEP 381 | ``Shepherd`` 382 | The core developer "on point" for the CEP 383 | ``Requires`` 384 | If this CEP depends on another CEP being implemented first, 385 | this should be a link to the required CEP. 386 | ``Celery-Version`` (optional) 387 | For Feature CEPs, the version of Celery (e.g. ``5.0``) that this 388 | feature will be released in. 389 | ``Replaces`` and ``Superseded-By`` (optional) 390 | These fields indicate that a CEP has been rendered obsolete. The newer CEP 391 | must have a ``Replaces`` header containing the number of the CEP that it 392 | rendered obsolete; the older CEP has a ``Superseded-By`` header pointing to 393 | the newer CEP. 394 | ``Resolution`` (optional) 395 | For CEPs that have been decided upon, this can be a link to the final 396 | rationale for acceptance/rejection. It's also reasonable to simply update 397 | the CEP with a "Resolution" section, in which case this header can be left 398 | out. 399 | 400 | Auxiliary Files 401 | --------------- 402 | 403 | CEPs may include auxiliary files such as diagrams. Such files must be named 404 | ``XXXX-descriptive-title.ext``, where "XXXX" is the CEP number, 405 | "descriptive-title" is a short slug indicating what the file contains, and 406 | "ext" is replaced by the actual file extension (e.g. "png"). 407 | 408 | Reporting CEP Bugs, or Submitting CEP Updates 409 | ============================================= 410 | 411 | How you report a bug, or submit a CEP update depends on several factors, such as 412 | the maturity of the CEP, the preferences of the CEP author, and the nature of 413 | your comments. For the early draft stages of the CEP, it's probably best to 414 | send your comments and changes directly to the CEP author. For more mature, or 415 | finished CEPs you can submit corrections as GitHub issues or pull requests 416 | against the CEP repository. 417 | 418 | When in doubt about where to send your changes, please check first with the CEP 419 | author and/or a core developer. 420 | 421 | CEP authors with git push privileges for the CEP repository can update the CEPs 422 | themselves. 423 | 424 | Transferring CEP Ownership 425 | ========================== 426 | 427 | It occasionally becomes necessary to transfer ownership of CEPs to a new author. 428 | In general, it is preferable to retain the original author as a co-author of the 429 | transferred CEP, but that's really up to the original author. A good reason to 430 | transfer ownership is because the original author no longer has the time or 431 | interest in updating it or following through with the CEP process, or has fallen 432 | off the face of the 'net (i.e. is unreachable or not responding to email). A 433 | bad reason to transfer ownership is because the new author doesn't agree with 434 | the direction of the CEP. One aim of the CEP process is to try to build 435 | consensus around a CEP, but if that's not possible, an author can always submit 436 | a competing CEP. 437 | 438 | If you are interested in assuming ownership of a CEP, first try to contact the 439 | original author and ask for permission. If they approve, ask them to open a pull 440 | request transferring the CEP to you. If the original author doesn't respond to 441 | email within a few weeks, contact Celery-developers. 442 | 443 | 444 | Differences between CEPs and PEPs 445 | ================================= 446 | 447 | As stated in the preamble, the CEP process is more or less a direct copy of 448 | the PEP process (and this document is a modified version of 449 | `PEP 1 `_). 450 | 451 | Relative to the PEP process, we made the following changes in CEPs: 452 | 453 | - The workflow is GitHub based (rather than email-based as in PEP 1). 454 | 455 | This is a simple enough change, but has a number of ramifications for the 456 | details of how CEPs work, including: 457 | 458 | - CEPs use pull requests (and direct commits) as the workflow process. 459 | - CEPs use rST-style headers rather than RFC822 (because rST-style headers 460 | render properly on GitHub without additional tooling). 461 | - CEPs have document titles rather than title fields in the metadata 462 | (again, because of GitHub rendering). 463 | - CEP are organized into directories based on statuses (e.g. ``draft/``, 464 | ``accepted/``, ``final/``, etc) so that additional tooling to create an 465 | index by status isn't needed. 466 | - CEP file names are more descriptive (e.g. ``0181-canvas-dsl.rst``), 467 | again to avoid the need for additional tooling. 468 | - CEPs are "edited" (e.g. pull request approved) by any core developer, 469 | rather than an explicit "editor" role like the PEP editors. 470 | 471 | - CEPs are pronounced upon by the Technical Board, rather than a BDFL (because 472 | Celery has no BDFLs). 473 | 474 | - CEPs explicitly require identifying a few roles (Author, Implementation Team, 475 | and Shepherd) before submission and throughout the process. With PEPs, most 476 | are authored and implemented by the same person, but the same doesn't seem to 477 | be true of CEPs (so far), hence the "implementer" role. As for the "shepherd": 478 | the BDFL or BDFL-delegate tends to be much more hands-on than the Technical 479 | Board, so the role of commenting and critiquing will be fulfilled by the 480 | shepherd, rather than the board. Further, we've observed that features are 481 | tremendously unlikely to make it into Celery without a committer on board to 482 | do the detail-work of merging a patch. 483 | 484 | - CEPs simplify the metadata somewhat, removing a few fields ("Post-History", 485 | etc.) and dropping a couple of statuses ("Active" gets merged into "Final", 486 | and "Deferred" merged into "Withdrawn"). 487 | 488 | - CEPs have "Feature CEPs" rather than "Standards Track" CEPs. 489 | 490 | - CEPs may only be reStructuredText (there is no plain text option). 491 | 492 | Copyright 493 | ========= 494 | 495 | This document has been placed in the public domain per the Creative Commons 496 | CC0 1.0 Universal license (https://creativecommons.org/publicdomain/zero/1.0/deed). 497 | -------------------------------------------------------------------------------- /draft/celery-5-high-level-architecture.rst: -------------------------------------------------------------------------------- 1 | ========================================== 2 | CEP XXXX: Celery 5 High Level Architecture 3 | ========================================== 4 | 5 | :CEP: XXXX 6 | :Author: Omer Katz 7 | :Implementation Team: Omer Katz 8 | :Shepherd: Omer Katz 9 | :Status: Draft 10 | :Type: Informational 11 | :Created: 2019-04-08 12 | :Last-Modified: 2019-04-08 13 | 14 | .. contents:: Table of Contents 15 | :depth: 4 16 | :local: 17 | 18 | Abstract 19 | ======== 20 | 21 | When Celery was conceived, production environments were radically different from today. 22 | 23 | Nowadays most applications are (or should be): 24 | 25 | * Deployed to a cloud provider's computing resources. 26 | * Distributed (sometimes between data centers). 27 | * Available or Consistent (We must pick one according to :term:`CAP Theorem`). 28 | * Network Partition Tolerant. 29 | * Observable. 30 | * Built with scalability in mind. 31 | * Cloud Native - The application's lifecycle is managed using Kubernetes, Swarm or any other scheduler. 32 | 33 | In addition, Celery lacks proper support for large scale deployments and some useful messaging architectural patterns. 34 | 35 | Celery 5 is the next major version of Celery and so we are able to break backwards compatibility, even in major ways. 36 | 37 | As such, our next major version should represent a paradigm shift in the way we implement our task execution platform. 38 | 39 | Specification 40 | ============= 41 | 42 | .. note:: 43 | 44 | The code examples below are for illustration purposes only. 45 | 46 | Unless explicitly specified, The API will be determined in other CEPs. 47 | 48 | .. figure:: celery-5-architecture-figure01.png 49 | 50 | High Level Architecture Diagram 51 | 52 | Message Types 53 | ------------- 54 | 55 | In relation to Celery :term:`Command messages ` 56 | are the messages we publish to the :term:`Message Broker` whenever we want to 57 | execute a :term:`Task`. 58 | 59 | :term:`Document messages ` are the messages we get as a result. 60 | 61 | :term:`Document messages ` may also be produced whenever 62 | we publish a serialized representation of a :term:`Domain Model`. 63 | 64 | .. code-block:: pycon 65 | 66 | >>> from celery import task 67 | >>> @task 68 | ... def add(a, b): 69 | ... return a + b 70 | >>> result = add.delay(1, 2) # Publish a command message 71 | >>> result.get() # Consume a Document message 72 | 3 73 | 74 | :term:`Event messages ` are a new concept for Celery. 75 | They describe that a :term:`Domain Event` occurred. 76 | Multiple tasks can be subscribed to an event. 77 | 78 | .. code-block:: pycon 79 | 80 | >>> from uuid import UUID 81 | >>> from celery import task, event 82 | >>> from myapp.models import User, AccountManager 83 | >>> @task 84 | ... def send_welcome_email(user_id, email): 85 | ... send_email(email=email, contents="hello, welcome", subject="welcome") # Send a welcome email to the user... 86 | ... User.objects.filter(pk=user_id).update(welcome_email_sent=True) 87 | >>> @task 88 | ... def notify_account_manager(user_id, email): 89 | ... account_manager = AccountManager.objects.assign_account_manager(user_id) 90 | ... send_email(email=account_manager.email, contents="you have a new user to attend to", subject="Alert") # Send an email to the account manager... 91 | >>> @event 92 | ... class UserRegistered: 93 | ... user_id: UUID 94 | ... email: str 95 | >>> UserRegistered.subscribe(send_welcome_email) 96 | >>> UserRegistered.subscribe(notify_account_manager) 97 | >>> UserRegistered.delay(user_id=1, email='foo@bar.com') # Calls both send_welcome_email and notify_account_manager with the provided arguments. 98 | 99 | These architectural building blocks will aid us in creating a better messaging 100 | system. To encourage :term:`Ubiquitous Language`, we will be using them in this 101 | document and in Celery 5's codebase as well. 102 | 103 | Message Passing Protocol 104 | ------------------------ 105 | 106 | In previous versions of Celery the only protocol we defined was for the message 107 | format itself. 108 | 109 | Whenever a client published a message to the broker using `task.delay()` Celery 110 | serialized it to a known format which it can use to invoke the task remotely. 111 | 112 | This meant that you needed a special client library to publish tasks that Celery 113 | will later consume and execute. 114 | The architectural implication is a coupling between the publisher and the consumer 115 | which defeats the general use-case of message passing: decoupling the publisher 116 | from the consumer. 117 | 118 | In addition, the communication protocol between the master process and the workers 119 | was an internal implementation detail which was not well defined or understood 120 | by the maintainers or users. 121 | 122 | Introduction to AMQP 1.0 Terminology 123 | ++++++++++++++++++++++++++++++++++++ 124 | 125 | AMQP 1.0 is a flexible messaging protocol that is designed to pass messages between nodes. 126 | Nodes can be queues, exchanges, Kafka-like log stream or any other entity which 127 | which publishes or consumes a message from another node. 128 | 129 | AMQP 1.0, unlike AMQP 0.9.1, does not mandate the presence of a broker to 130 | facilitate message passing. 131 | 132 | This feature allows us to use it to communicate effectively between the different 133 | Celery components without overloading the broker with connections and messages. 134 | It also allows users to invoke tasks directly without the usage 135 | of a :term:`Message Broker`, thus treating a Worker as an actor. 136 | 137 | Connections 138 | ~~~~~~~~~~~ 139 | 140 | Sessions 141 | ~~~~~~~~ 142 | 143 | Channels 144 | ~~~~~~~~ 145 | 146 | Nodes (Components) 147 | ~~~~~~~~~~~~~~~~~~ 148 | 149 | Nodes (also called Components) are entities on either different processes which 150 | establish links. 151 | 152 | Links 153 | ~~~~~ 154 | 155 | Terminus 156 | ~~~~~~~~ 157 | 158 | Implementation 159 | ++++++++++++++ 160 | 161 | The python bindings to the `qpid-proton`_ library exposes all these features, 162 | can be used with trio and is supported on PyPy. 163 | 164 | We will use it as a fundamental building block for our entire architecture. 165 | 166 | Canvas 167 | ------ 168 | 169 | In Celery Canvas is the mechanism which users can use to define workflows 170 | dynamically. 171 | 172 | In previous versions of Celery there are issues with the protocol which can 173 | cause Celery to generate messages too large for storage in :term:`Message Brokers `. 174 | There are also issues with the API and multiple implementation problems. 175 | 176 | In Celery 5 we're going to revamp the protocol, API and 177 | possibly the implementation itself to resolve these issues. 178 | 179 | Signatures 180 | ++++++++++ 181 | 182 | Primitives 183 | ++++++++++ 184 | 185 | Error Handling 186 | ~~~~~~~~~~~~~~ 187 | 188 | Error Recovery 189 | ~~~~~~~~~~~~~~ 190 | 191 | Chains 192 | ~~~~~~ 193 | 194 | Groups 195 | ~~~~~~ 196 | 197 | Chords 198 | ~~~~~~ 199 | 200 | Maps 201 | ~~~~ 202 | 203 | Starmaps 204 | ~~~~~~~~ 205 | 206 | Chunks 207 | ~~~~~~ 208 | 209 | Forks 210 | ~~~~~ 211 | 212 | Workflows 213 | --------- 214 | 215 | A Workflow is a declarative :ref:`draft/celery-5-high-level-architecture:Canvas`. 216 | 217 | Workflows provide an API for incrementally executing business logic by 218 | dividing it to small, self-contained tasks. 219 | 220 | Unlike :ref:`draft/celery-5-high-level-architecture:Canvas`, a Workflow is 221 | immutable, static and predicable. 222 | 223 | Observability 224 | ------------- 225 | 226 | One of Celery 5's goals is to be :term:`observable `. 227 | 228 | Each Celery component will record statistics, provide trace points for 229 | application monitoring tools and distributed tracing tools and emit log messages 230 | when appropriate. 231 | 232 | Metrics 233 | +++++++ 234 | 235 | Celery stores and publishes metrics which allows our users to debug their 236 | applications more easily and spot problems. 237 | 238 | By default each worker will publish the metrics to a dedicated queue. 239 | 240 | Other methods such as publishing them to StatsD is also possible using the 241 | provided extension point. 242 | 243 | Trace Points 244 | ++++++++++++ 245 | 246 | Celery provides trace points for application monitoring tools and distributed 247 | tracing tools. 248 | 249 | This allows our users to spot and debug performance issues. 250 | 251 | Logging 252 | +++++++ 253 | 254 | All log messages must be structured. 255 | :term:`Structured logs ` provide context for our users 256 | which allows them to debug problems more easily and aids the developers 257 | to resolve bugs in Celery. 258 | 259 | The structure of a log message is determined whenever a component 260 | is initialized. 261 | 262 | During initialization, an attempt will be made to detect how the component 263 | lifecycle is managed. 264 | If all attempts are unsuccessful, the logs will be formatted using 265 | :term:`JSON` and will be printed to stdout. 266 | 267 | Celery will provide an extension point for detection of different 268 | runtimes. 269 | 270 | .. admonition:: Example 271 | 272 | If a component's lifecycle is managed by a SystemD service, 273 | Celery will detect that the `JOURNAL_STREAM`_ environment variable 274 | is set when the process starts and use it's value to transmit structured 275 | data into `journald`_. 276 | 277 | Whenever Celery fails to log a message for any reason it publishes a command 278 | to the worker's :ref:`draft/celery-5-high-level-architecture:Inbox Queue` 279 | in order to log the message again. 280 | As usual messages which fail to be published are stored in the 281 | :ref:`draft/celery-5-high-level-architecture:messages backlog`. 282 | 283 | In past versions of Celery we've used the standard logging module. 284 | Unfortunately it does not meet the aforementioned requirements. 285 | 286 | `Eliot `_ is a logging library which provides 287 | structure and context to logs, even across coroutines, threads and processes. 288 | 289 | It is also able to emit logs to `journald`_ and has native trio integration. 290 | 291 | Network Resilience and Fault Tolerance 292 | -------------------------------------- 293 | 294 | Celery 5 aims to be network failure resilient and fault tolerant. 295 | As an architectural guideline Celery must retry operations **by default** 296 | and must avoid doing so **indefinitely and without proper limits**. 297 | 298 | Any operation which cannot be executed either momentarily or permanently 299 | as a result of a bug must not be retried beyond the configured limits. 300 | Instead, Celery must store the operation for further inspection 301 | and if required, manual intervention. 302 | 303 | Celery must track and automatically handle "poisonous messages" to ensure 304 | the recovery of the Celery cluster. 305 | 306 | Fault Tolerance 307 | +++++++++++++++ 308 | 309 | Distributed Systems suffer from an inherent property: 310 | 311 | Any distributed system is unreliable. 312 | 313 | * The network may be unavailable or slow. 314 | * Some or all of the servers might suffer from a hardware failure. 315 | * A node in the system may arbitrarily crash 316 | due to lack of memory or a bug. 317 | * Any number of unaccounted failure modes. 318 | 319 | Therefore, Celery must be fault tolerant and gracefully degrade its' operation 320 | when failures occur. 321 | 322 | Graceful Degradation 323 | ~~~~~~~~~~~~~~~~~~~~ 324 | 325 | Features which are less mission-critical may fail at any time, provided that 326 | a warning is logged. 327 | 328 | This document will highlight such features and describe what happens when 329 | they fail for any reason. 330 | 331 | Retries 332 | ~~~~~~~ 333 | 334 | In previous Celery versions tasks were not retried by default. 335 | 336 | This forces new adopters to carefully read our documentation to ensure 337 | the fault tolerance of their tasks. 338 | 339 | In addition, our retry policy was declared at the task level. 340 | When using :ref:`celery4:task-autoretry` Celery automatically retries tasks 341 | when specific exceptions are raised. 342 | 343 | However the same type of exception may hold a different meaning in different 344 | contexts. 345 | 346 | This created the following pattern: 347 | 348 | .. code-block:: python 349 | 350 | from celery import task 351 | from data_validation_lib import validate_data 352 | 353 | def _calculate(a, b): 354 | # Do something 355 | 356 | @task(autoretry_for=(ValueError,)) 357 | def complex_calculation(a, b): 358 | try: 359 | # Code that you don't control can raise a ValueError. 360 | validate_data(a, b) 361 | except ValueError: 362 | print("Complete failure!") 363 | return 364 | 365 | # May temporarily raise a ValueError due to some externally fetched 366 | # data which is currently incorrect but will be updated later. 367 | _calculate() 368 | 369 | An obvious way around this problem is to ensure that `_calculate()` 370 | raises a custom exception. 371 | 372 | But we shouldn't force the users to use workarounds. Our code should be 373 | ergonomic and idiomatic. 374 | 375 | Instead, we should allow users to declare sections as "poisonous" - tasks that 376 | if retried will surely fail if they fail at those sections. 377 | 378 | .. code-block:: python 379 | 380 | from celery import task, poisonous 381 | from data_validation_lib import validate_data 382 | 383 | def _calculate(a, b): 384 | # Do something 385 | 386 | @task(autoretry_for=(ValueError,)) 387 | def complex_calculation(a, b): 388 | with poisonous(): 389 | validate_data(a, b) 390 | 391 | # May temporarily raise a ValueError due to some externally fetched 392 | # data which is currently incorrect but will be updated later. 393 | _calculate() 394 | 395 | Not all operations are equal. Some may be retried more than others. 396 | Some may need to be retried less often. 397 | 398 | Currently there are multiple ways to achieve this: 399 | 400 | You can separate them to different tasks with a different retry policy: 401 | 402 | .. code-block:: python 403 | 404 | from celery import task 405 | 406 | @task(retry_policy={ 407 | 'max_retries': 3, 408 | 'interval_start': 0, 409 | 'interval_step': 0.2, 410 | 'interval_max': 0.2 411 | }) 412 | def foo(): 413 | second_operation() 414 | 415 | @task(retry_policy={ 416 | 'max_retries': 10, 417 | 'interval_start': 0, 418 | 'interval_step': 5, 419 | 'interval_max': 120 420 | }) 421 | def bar(): 422 | first_operation() 423 | foo.delay() 424 | 425 | Or you can wrap each code section in a try..except clause and call 426 | :py:meth:`celery.app.task.Task.retry`. 427 | 428 | .. code-block:: python 429 | 430 | @task(bind=True) 431 | def foo(self): 432 | try: 433 | # first operation 434 | except Exception: 435 | self.retry(retry_policy={ 436 | 'max_retries': 10, 437 | 'interval_start': 0, 438 | 'interval_step': 5, 439 | 'interval_max': 120 440 | }) 441 | 442 | try: 443 | first_operation() 444 | except Exception: 445 | self.retry(retry_policy={ 446 | 'max_retries': 10, 447 | 'interval_start': 0, 448 | 'interval_step': 5, 449 | 'interval_max': 120 450 | }) 451 | 452 | try: 453 | second_operation() 454 | except Exception: 455 | self.retry(retry_policy={ 456 | 'max_retries': 3, 457 | 'interval_start': 0, 458 | 'interval_step': 0.2, 459 | 'interval_max': 1 460 | }) 461 | 462 | Those solutions are unnecessarily verbose. Instead, we could use a with clause 463 | if all we want to do is retry. 464 | 465 | .. code-block:: python 466 | 467 | @task 468 | def foo(): 469 | with retry(max_retries=10, interval_start=0, interval_step=5, interval_max=120): 470 | first_operation() 471 | 472 | with retry(max_retries=3, interval_start=0, interval_step=0.2, interval_max=1): 473 | second_operation() 474 | 475 | By default messages which cannot be re-published will be stored 476 | in the :ref:`draft/celery-5-high-level-architecture:messages backlog`. 477 | 478 | Implementers may provide other fallbacks such as executing the retried task 479 | in the same worker or abandoning the task entirely. 480 | 481 | Some operations are not important enough to be retried if they fail. 482 | 483 | .. admonition:: Example 484 | 485 | We're implementing a BI system that records mouse 486 | interactions. 487 | 488 | The BI team has specified that it wants to store the raw data and 489 | the time span between interactions. 490 | Since we have a lot of data already, if the system failed to insert the raw data 491 | into the data store then we should not fail. Instead, we should emit a warning. 492 | However, the time span between mouse interactions is critical to the BI 493 | team's insight and if that fails to be inserted into the data store 494 | we must retry it. 495 | 496 | Such a task can be defined using the ``optional`` context manager. 497 | 498 | .. code-block:: python 499 | 500 | @task 501 | def foo(raw_data): 502 | # Using default retry policy 503 | with optional(): 504 | # ignore retry policy and proceed 505 | insert_raw_data(raw_data) 506 | 507 | with retry(max_retries=10, interval_start=0, interval_step=5, interval_max=120): 508 | calculation = time_span_calculation(raw_data) 509 | insert_time_spans(calculation) 510 | 511 | In case of a failure inside the optional context manager, a warning is logged. 512 | 513 | We can of course be more specific about the failures we allow: 514 | 515 | .. code-block:: python 516 | 517 | @task 518 | def foo(raw_data): 519 | # Using default retry policy 520 | with optional(ConnectionError, TimeoutError): 521 | # ignore retry policy and proceed 522 | insert_raw_data(raw_data) 523 | 524 | with retry(max_retries=10, interval_start=0, interval_step=5, interval_max=120): 525 | calculation = time_span_calculation(raw_data) 526 | insert_time_spans(calculation) 527 | 528 | Health Checks 529 | ~~~~~~~~~~~~~ 530 | 531 | Health Checks are used in Celery to verify that a worker is able to 532 | successfully execute a :ref:`task ` 533 | or a :ref:`service `. 534 | 535 | The :ref:`draft/celery-5-high-level-architecture:Scheduler` is responsible 536 | for scheduling the health checks for execution in each worker after 537 | each time the configured period of time lapses. 538 | 539 | Whenever a health check should be executed the 540 | :ref:`draft/celery-5-high-level-architecture:Scheduler` instructs the 541 | :ref:`draft/celery-5-high-level-architecture:Publisher` to send the 542 | `_expired` :term:`Event Message` to each worker's 543 | :ref:`draft/celery-5-high-level-architecture:Inbox Queue`. 544 | 545 | Workers which have tasks subscribed to the event will 546 | execute all the subscribed tasks in order to determine the state of the 547 | health check. 548 | 549 | Health Checks can handle :term:`Document Messages ` as input 550 | from :ref:`draft/celery-5-high-level-architecture:Data Sources`. 551 | 552 | This is useful when you want to respond to an alert from a monitoring system 553 | or when you want to verify that all incoming data from said source is 554 | valid at all times before executing the task. 555 | 556 | In addition to tasks, Health Checks can also use 557 | :ref:`draft/celery-5-high-level-architecture:Services` in order to track 558 | changes in the environment it is running on. 559 | 560 | .. admonition:: Example 561 | 562 | We have a task which requires 8GB of memory to complete. 563 | The worker runs a service which constantly monitors the system's available 564 | memory. 565 | If there is not enough memory it changes the task's health check to the 566 | **Unhealthy** state. 567 | 568 | If a task or a service that is part of a health check fails unexpectedly it 569 | is ignored and an error message is logged. 570 | 571 | Celery provides many types of health checks in order to verify that it can 572 | operate without any issues. 573 | 574 | Users may implement their own health checks in addition to the built-in health 575 | checks. 576 | 577 | Some health checks are specific to the worker they are executing on. 578 | Therefore, their state is stored in-memory in the worker. 579 | 580 | Other health checks are global to all or a group of workers. 581 | As such, their state is stored externally. 582 | 583 | If the state storage for health checks is not provided, these health checks 584 | are disabled. 585 | 586 | Health Checks can be associated with tasks in order to ensure that they are 587 | likely to succeed. Multiple Health Check failures may trigger 588 | a :term:`Circuit Breaker` which will prevent the task from running for a period 589 | of time or automatically mark it as failed. 590 | 591 | Each Health Check declares its possible states. 592 | Sometimes it makes sense to try to execute a task anyway even if the 593 | health check occasionally fails. 594 | 595 | .. admonition:: Example 596 | 597 | A health check that verifies whether we can send a HTTP request to an endpoint 598 | has multiple states. 599 | 600 | The health check performs an 601 | `OPTIONS `_ 602 | HTTP request to that endpoint and expects it to respond within the specified 603 | timeout. 604 | 605 | The health check is in a **Healthy** state if all the following conditions are 606 | met: 607 | 608 | * The DNS server is responding within the specified time limit and is 609 | resolving the address correctly. 610 | * The TLS certificates are valid and the connection is secure. 611 | * The Intrusion Detection System reports that the network is secure. 612 | * The HTTP method we're about to use is listed in the OPTIONS response's 613 | `ALLOW `_ 614 | header. 615 | * The content type we're about to format the request in is listed in the 616 | OPTIONS response's 617 | `ACCEPT `_ 618 | header. 619 | * The OPTIONS request responds within the specified time limits. 620 | * The OPTIONS request responds with 621 | `200 OK `_ 622 | status. 623 | 624 | In addition, the actual request performed in the task must also stand in the 625 | aforementioned conditions. Otherwise, the health check will change it's state. 626 | 627 | The health check can be in an **Insecure** state if one or more of the 628 | following conditions are met: 629 | 630 | * The TLS certificates are invalid for any reason. 631 | * The Intrusion Detection System has reported that the network is compromised 632 | for any reason. 633 | 634 | It is up for the user to configure the :term:`Circuit Breaker` to prevent 635 | insecure requests from being executed. 636 | 637 | The health check can be in an **Degraded** state if one or more of the 638 | following conditions are met: 639 | 640 | * The request does not reply with a 2xx HTTP status. 641 | * The request responds slowly and almost reaches it's time limits. 642 | 643 | It is up for the user to configure the :term:`Circuit Breaker` to prevent 644 | requests from being executed after multiple attempts or not all. 645 | 646 | The health check can be in an **Unhealthy** state if one or more of the 647 | following conditions are met: 648 | 649 | * The request responds with a 500 HTTP status. 650 | * The request's response has not been received within the specified time 651 | limits. 652 | 653 | It is up for the user to configure the :term:`Circuit Breaker` to prevent 654 | requests from being executed if there is an issue with the endpoint. 655 | 656 | The health check can be in an **Permanently Unavailable** state if one or more 657 | of the following conditions are met: 658 | 659 | * The request responds with a 660 | `404 Not Found `_ 661 | HTTP status. 662 | * The HTTP method we're about to use is not allowed. 663 | * The content type we're about to use is not allowed. 664 | 665 | Circuit Breaking 666 | ~~~~~~~~~~~~~~~~ 667 | 668 | Celery 5 introduces the concept of :term:`Circuit Breaker` into the framework. 669 | 670 | A Circuit Breaker prevents a :ref:`task ` 671 | or a :ref:`service ` 672 | from executing. 673 | 674 | Each task or a service has a Circuit Breaker which the user can associate 675 | health checks with. 676 | 677 | In addition, if the task or the service unexpectedly fails, the user 678 | can configure the Circuit Breaker to trip after a configured number of times. 679 | The default value is 3 times. 680 | 681 | Whenever a Circuit Breaker trips, the worker will emit a warning log message. 682 | 683 | After a configured period of time the circuit is opened again and tasks may 684 | execute. The default period of time is 30 seconds with no linear or exponential 685 | growth. 686 | 687 | The user will configure the following properties of the Circuit Breaker: 688 | 689 | * How many times the health checks may fail before 690 | the circuit breaker trips. 691 | * How many unexpected failures the task or service tolerates before tripping 692 | the Circuit Breaker. 693 | * The period of time after which the circuit is yet 694 | again closed. That time period may grow linearly or exponentially. 695 | * How many circuit breaker trips during a period of time should cause the worker 696 | to produce an error log message instead of a warning log message. 697 | * The period of time after which the circuit breaker downgrades 698 | it's log level back to warning. 699 | 700 | .. admonition:: Example 701 | 702 | We allow 2 **Unhealthy** health checks 703 | and/or 10 **Degraded** health checks in a period of 10 seconds. 704 | 705 | If we cross that threshold, the circuit breaker trips. 706 | 707 | The circuit will be closed again after 30 seconds. Afterwards, the task can 708 | be executed again. 709 | 710 | If 3 consequent circuit breaker trips occurred during a period of 5 minutes, 711 | all circuit breaker trips will emit an error log message instead of a warning. 712 | 713 | The circuit breaker will downgrade it's log level after 30 minutes. 714 | 715 | 716 | Network Resilience 717 | ++++++++++++++++++ 718 | 719 | Network Connections may fail at any time. 720 | In order to be network resilient we must use retries and circuit breakers on 721 | all outgoing and incoming network connections. 722 | 723 | In addition, proper timeouts must be set to avoid hanging when the connection 724 | is slow or unresponsive. 725 | 726 | Each network connection must be accompanied by a 727 | :ref:`health check `. 728 | 729 | Health check failures must eventually trip a 730 | :ref:`circuit breaker `. 731 | 732 | Command Line Interface 733 | ---------------------- 734 | 735 | Our command line interface is the user interface to all of Celery's 736 | functionality. It is crucial for us to provide an excellent user experience. 737 | 738 | Currently Celery uses :mod:`argparse` with a few custom hacks and workarounds for 739 | things which are not possible to do with :mod:`argparse`. 740 | This created some bugs in the past. 741 | 742 | Celery 5 will use `Click`_, a modern Python library for creating command line 743 | programs. 744 | 745 | Click's documentation `explains `_ 746 | why it is a good fit for us: 747 | 748 | There are so many libraries out there for writing command line utilities; 749 | why does Click exist? 750 | 751 | This question is easy to answer: because there is not a single command 752 | line utility for Python out there which ticks the following boxes: 753 | 754 | * is lazily composable without restrictions 755 | * supports implementation of Unix/POSIX command line conventions 756 | * supports loading values from environment variables out of the box 757 | * supports for prompting of custom values 758 | * is fully nestable and composable 759 | * works the same in Python 2 and 3 760 | * supports file handling out of the box 761 | * comes with useful common helpers (getting terminal dimensions, 762 | ANSI colors, fetching direct keyboard input, screen clearing, 763 | finding config paths, launching apps and editors, etc.) 764 | 765 | There are many alternatives to Click and you can have a look at them if 766 | you enjoy them better. The obvious ones are ``optparse`` and ``argparse`` 767 | from the standard library. 768 | 769 | Click actually implements its own parsing of arguments and does not use 770 | ``optparse`` or ``argparse`` following the ``optparse`` parsing behavior. 771 | The reason it's not based on ``argparse`` is that ``argparse`` does not 772 | allow proper nesting of commands by design and has some deficiencies when 773 | it comes to POSIX compliant argument handling. 774 | 775 | Click is designed to be fun to work with and at the same time not stand in 776 | your way. It's not overly flexible either. Currently, for instance, it 777 | does not allow you to customize the help pages too much. This is intentional 778 | because Click is designed to allow you to nest command line utilities. The 779 | idea is that you can have a system that works together with another system by 780 | tacking two Click instances together and they will continue working as they 781 | should. 782 | 783 | Too much customizability would break this promise. 784 | 785 | Click describes it's 786 | `advantages over argparse `_ 787 | in its documentation as well: 788 | 789 | Click is internally based on optparse instead of argparse. This however 790 | is an implementation detail that a user does not have to be concerned 791 | with. The reason however Click is not using argparse is that it has some 792 | problematic behaviors that make handling arbitrary command line interfaces 793 | hard: 794 | 795 | * argparse has built-in magic behavior to guess if something is an 796 | argument or an option. This becomes a problem when dealing with 797 | incomplete command lines as it's not possible to know without having a 798 | full understanding of the command line how the parser is going to 799 | behave. This goes against Click's ambitions of dispatching to 800 | subparsers. 801 | * argparse currently does not support disabling of interspersed 802 | arguments. Without this feature it's not possible to safely implement 803 | Click's nested parsing nature. 804 | 805 | In contrast to :mod:`argparse`, the `Click community `_ 806 | provides many extensions we can use to create a better user experience 807 | for our users. 808 | 809 | Click supports calling `async` methods and functions 810 | using the `asyncclick` `_ fork 811 | which is likely to be important for us in the future. 812 | 813 | Dependency Inversion 814 | -------------------- 815 | 816 | Currently Celery uses different singleton registries to customize the behavior 817 | of its' different components. 818 | This is known as the :term:`Service Locator` pattern. 819 | 820 | Mark Seemann criticized Service Locators as an anti-pattern for multiple reasons: 821 | 822 | * It has `API usage problems and maintenance issues `_. 823 | * It `violates encapsulation `_. 824 | * It `violates SOLID `_. 825 | 826 | Using constructor injection is a much better way to invert our dependencies. 827 | 828 | For that purpose we have selected the `dependencies`_ library. 829 | 830 | Worker 831 | ------ 832 | 833 | The Worker is the most fundamental architectural component in Celery. 834 | 835 | The role of the Worker is to be a :term:`Service Activator`. 836 | It executes :ref:`draft/celery-5-high-level-architecture:Tasks` in response 837 | to :term:`messages `. 838 | 839 | A Worker is also an :term:`Idempotent Receiver`. 840 | If the exact same :term:`Message` is received more than once, the duplicated 841 | messages are discarded. 842 | In this case, a warning log message is emitted. 843 | The Worker maintains a list of identifiers of recently received :term:`messages `. 844 | The number of :term:`messages ` is determined by a configuration 845 | value. 846 | By default that value is 100 :term:`messages `. 847 | 848 | Configuration 849 | +++++++++++++ 850 | 851 | In previous versions of Celery we had the option to load the configuration from 852 | a Python module. 853 | 854 | Cloud Native applications often use `Etcd `_, 855 | `Consul `_ 856 | or `Kubernetes Config Maps `_ (among others) 857 | to store configuration and adjust it when needed. 858 | 859 | Celery 5 introduces the concept of configuration backends. 860 | These backends allow you to load the Worker's configuration from any source. 861 | 862 | The default configuration backend loads the configuration from a Python module. 863 | 864 | Users may create their own configuration backends to load configuration from 865 | a `YAML `_ file, a `TOML `_ file 866 | or a database. 867 | 868 | Once the configuration has changed, the Worker stops consuming tasks, 869 | waits for all other tasks to finish and reloads the configuration. 870 | 871 | This behavior can be disabled using a CLI option. 872 | 873 | Event Loop 874 | ++++++++++ 875 | 876 | In Celery 4 we have implemented our own custom Event Loop. 877 | It is a cause for many bugs and issues in Celery. 878 | 879 | In addition, some I/O operations are still blocking the event loop since 880 | the clients we use do not allow non-blocking operations. 881 | 882 | The most important feature of Celery 5 is to replace the custom Event Loop 883 | with `Trio`_. 884 | 885 | We selected it because of its 886 | `design `_, 887 | `interoperability with asyncio `_ 888 | and its many features. 889 | 890 | Trio provides a context manager which limits the concurrency of coroutines 891 | and/or threads. This saves us from further bookkeeping when a Worker executes 892 | :ref:`draft/celery-5-high-level-architecture:Tasks`. 893 | 894 | Trio allows coroutines to report their status. This is especially useful 895 | when we want to block the execution of other coroutines until initialization 896 | of the coroutine completes. We require this feature for implementing 897 | :ref:`draft/celery-5-high-level-architecture:Boot Steps`. 898 | 899 | Trio also provides a feature called cancellation scopes which allows us to 900 | cancel a coroutine or multiple coroutines at once. 901 | This allows us to abort :ref:`draft/celery-5-high-level-architecture:Tasks` 902 | and handle the aborted tasks in an idiomatic fashion. 903 | 904 | All of those features save us from writing a lot of code. If we were to select 905 | asyncio as our Event Loop, we'd have to implement most of those features 906 | ourselves. 907 | 908 | Internal Tasks Queue 909 | ++++++++++++++++++++ 910 | 911 | The internal tasks queue is an in-memory queue which the worker uses 912 | to queue tasks for execution. 913 | 914 | Each task type has its own queue. 915 | 916 | The queue must be thread-safe and coroutine-safe. 917 | 918 | Internal Results Queue 919 | ++++++++++++++++++++++ 920 | 921 | The internal results queue is an in-memory queue which the worker uses 922 | to report the result of tasks back to the 923 | :ref:`draft/celery-5-high-level-architecture:Router`. 924 | 925 | The queue must be thread-safe and coroutine-safe. 926 | 927 | Services 928 | ++++++++ 929 | 930 | Services are stateful, long running tasks which are used by Celery to perform 931 | its internal operations. 932 | 933 | Some services publish :term:`messages ` to brokers, 934 | others consume :term:`messages ` from them. 935 | Other services are used to calculate optimal scheduling of tasks, routing, 936 | logging and even executing tasks. 937 | 938 | Users may create their own services as well. 939 | 940 | Internal Services 941 | +++++++++++++++++ 942 | 943 | The Worker defines internal services to ensure it's operation and to provide 944 | support for it's features. 945 | 946 | The exact API for each service will be determined in another CEP. 947 | 948 | This list of internal services is not final. 949 | Other internal services may be defined in other CEPs. 950 | 951 | Task Execution 952 | ~~~~~~~~~~~~~~ 953 | 954 | The ``Task Execution`` service is responsible for executing all Celery 955 | :ref:`tasks `. 956 | 957 | It consumes tasks from the 958 | :ref:`draft/celery-5-high-level-architecture:Internal Tasks Queue`, 959 | executes them and enqueues the results into the 960 | :ref:`draft/celery-5-high-level-architecture:Internal Results Queue`. 961 | 962 | The service supervises how many tasks are run concurrently and limits the 963 | number of concurrent tasks to the configured amount in accordance to the 964 | :ref:`draft/celery-5-high-level-architecture:Concurrency Budget`. 965 | 966 | The service also attempts to saturate all of the available resources by 967 | scheduling as many as :ref:`draft/celery-5-high-level-architecture:I/O Bound Tasks` 968 | and :ref:`draft/celery-5-high-level-architecture:CPU Bound Tasks` as possible. 969 | 970 | 971 | Consumer 972 | ~~~~~~~~ 973 | 974 | The ``Consumer`` service consumes :term:`messages ` from one or many 975 | :ref:`Routers ` or 976 | :term:`Message Brokers `. 977 | 978 | The service enqueues the consumed :term:`messages ` 979 | into the appropriate :ref:`draft/celery-5-high-level-architecture:Internal Tasks Queue` 980 | according to the task's type. 981 | 982 | Result Publisher 983 | ~~~~~~~~~~~~~~~~ 984 | 985 | The ``Result Publisher`` service consumes results from the 986 | :ref:`draft/celery-5-high-level-architecture:Internal Results Queue` and 987 | publishes them to the :ref:`draft/celery-5-high-level-architecture:Router`'s 988 | :ref:`draft/celery-5-high-level-architecture:Inbox Queue`. 989 | 990 | Maximal Concurrency Budget 991 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 992 | 993 | The ``Maximal Concurrency Budget`` service runs the user's concurrency budget 994 | strategies and notifies the :ref:`tasks ` 995 | service of changes in concurrency. 996 | 997 | Tasks 998 | +++++ 999 | 1000 | Tasks are short running, have a defined purpose and are triggered in response 1001 | to messages. 1002 | 1003 | Celery declares some tasks for internal usage. 1004 | 1005 | Users will create their own tasks for their own use. 1006 | 1007 | Deduplication 1008 | ~~~~~~~~~~~~~ 1009 | 1010 | Some Tasks are not idempotent and may not run more than once. 1011 | 1012 | Users may define a deduplication policy to help Celery discard duplicated 1013 | messages. 1014 | 1015 | .. admonition:: Example 1016 | 1017 | The ``send_welcome_email`` task is only allowed to send one welcome email per 1018 | user. 1019 | 1020 | The user defines a deduplication policy which checks with their 3rd party 1021 | email delivery provider if that email has already been sent. 1022 | If it did, the user instructs Celery to reject the task. 1023 | 1024 | I/O Bound Tasks 1025 | ~~~~~~~~~~~~~~~ 1026 | 1027 | I/O bound tasks are tasks which mainly perform a network operation or 1028 | a disk operation. 1029 | 1030 | I/O bound tasks are specifically marked as such using Python's `async def` 1031 | notation for defining awaitable functions. They will run in a Python coroutine. 1032 | 1033 | Due to that, any I/O operation in that task must be asynchronous in order to 1034 | avoid blocking the event loop. 1035 | 1036 | Some of the user's asynchronous tasks won't use trio as their event loop 1037 | but will use the more commonly used asyncio event loop which we do support. 1038 | 1039 | In that case, the user must specify the event loop they are going to use for the 1040 | task. 1041 | 1042 | CPU Bound Tasks 1043 | ~~~~~~~~~~~~~~~ 1044 | 1045 | CPU bound tasks are tasks which mainly perform a calculation of some sort such 1046 | as calculating an average, hashing, serialization or deserialization, 1047 | compression or decompression, encryption or decryption etc. 1048 | In some cases where no asynchronous code for the I/O operation is available 1049 | CPU bound tasks are also an appropriate choice as they will not block 1050 | the event loop for the duration of the task. 1051 | 1052 | Performing operations which release the :term:`GIL` is recommended to avoid 1053 | throttling the concurrency of the worker. 1054 | 1055 | CPU bound tasks are specifically marked as such using Python's 1056 | `def` notation for defining functions. They will run in a Python thread. 1057 | 1058 | Using threads instead of forking the main process has its upsides: 1059 | 1060 | - It simplifies the Worker's architecture and makes it less brittle. 1061 | 1062 | Processes require :term:`IPC` to communicate with each other. 1063 | This complicates implementation since multiple methods are required to support 1064 | :term:`IPC` reliably across all operating systems Celery supports. 1065 | Threads on the other hand require less complicated means of communication. 1066 | 1067 | In `trio`_, we simply use a memory channel which is a coroutine and thread safe 1068 | way to send and receive values. 1069 | 1070 | - PyPy's JIT warms up faster. 1071 | 1072 | When using PyPy, using threads means that we get to keep our previous JIT traces 1073 | and therefore JIT warmup will occur faster. 1074 | 1075 | If we'd use processes, each process has to warm up its own JIT which results 1076 | in tasks being executed slower for a longer period of time. 1077 | 1078 | Using threads for CPU bound tasks unfortunately has some downsides as well: 1079 | 1080 | - Pure Python CPU bound workloads cannot be executed in parallel. 1081 | 1082 | In both CPython and PyPy the :term:`GIL` prevents executing two Python bytecodes 1083 | in parallel by design. 1084 | 1085 | This results in slower execution of Python code when using threads. 1086 | 1087 | - The :term:`GIL`'s implementation in CPython 3.x has a defect in design. 1088 | 1089 | According to a `bug report `_ the new GIL 1090 | in Python 3 CPU bound threads may starve I/O threads (in our case the main thread). 1091 | 1092 | .. note :: 1093 | 1094 | This is not an issue with PyPy's implementation of the :term:`GIL` 1095 | `according to Armin Rigo `_, PyPy's creator. 1096 | 1097 | - Tasks are no longer isolated. 1098 | 1099 | Since we're mixing workloads to maximize our throughput a task which crashes 1100 | the worker or leaks memory can crash the entire worker. 1101 | 1102 | To mitigate these issues CPU Bound Tasks may be globally rate limited to allow 1103 | the main thread to complete executing :ref:`draft/celery-5-high-level-architecture:I/O Bound Tasks`. 1104 | 1105 | Boxed Tasks 1106 | ~~~~~~~~~~~ 1107 | 1108 | To minimize the disadvantages of using threads in Python and workaround the 1109 | shortcomings of the :term:`GIL`, Celery also provides a new type of tasks called 1110 | Boxed Tasks. 1111 | 1112 | Boxed Tasks are processes which execute tasks in an isolated manner. 1113 | 1114 | The processes' lifecycle is managed by the :ref:`draft/celery-5-high-level-architecture:Controller`. 1115 | 1116 | Since Boxed Tasks are run separately from Celery itself, the program the process 1117 | is running can be written in any language as long as it implements IPC in the 1118 | same way the :ref:`draft/celery-5-high-level-architecture:Controller` expects. 1119 | 1120 | Boxed tasks are a special kind of 1121 | :ref:`draft/celery-5-high-level-architecture:I/O Bound Tasks`. 1122 | They are executed the same way inside the worker but defined using a different 1123 | API. 1124 | 1125 | Concurrency Budget 1126 | ~~~~~~~~~~~~~~~~~~ 1127 | 1128 | Each worker has a concurrency budget for each type of task it can run. 1129 | 1130 | The budget for each type of task is defined by a minimal 1131 | and an optional maximal concurrency. 1132 | 1133 | The maximal concurrency budget can be dynamic or fixed. 1134 | Dynamic maximal concurrency strategies may be used to determine the maximum 1135 | concurrency based on the load factor of the server, available network bandwidth 1136 | or any other requirement the user may have. 1137 | 1138 | .. note:: 1139 | If a user specifies a concurrency of more than 10 for :ref:`draft/celery-5-high-level-architecture:CPU Bound Tasks` a warning log message is emitted. 1140 | 1141 | Too many threads can cause task execution to grind down to a halt. 1142 | 1143 | If there are more tasks in the :ref:`draft/celery-5-high-level-architecture:Internal Tasks Queue` 1144 | than what is currently the allowed maximum task concurrency we increase the 1145 | current maximum by that number of tasks. 1146 | After this increase, there will be a configurable cooldown period during which 1147 | the worker will execute the new tasks. 1148 | After the cooldown period, if there are still more tasks in the :ref:`draft/celery-5-high-level-architecture:Internal Tasks Queue` 1149 | than the current maximum capacity we increase the maximum concurrency exponentially 1150 | by a configurable exponent multiplied by the number of budget increases. 1151 | The result is rounded up. 1152 | 1153 | This process goes on until we either reach the maximum concurrency budget for 1154 | that type of tasks or if the number of tasks in 1155 | :ref:`draft/celery-5-high-level-architecture:Internal Tasks Queue` is lower than 1156 | the current maximum concurrency. 1157 | 1158 | If the current number of tasks is lower than the current maximal concurrency 1159 | we decrease it to the number of tasks that are currently executing. 1160 | 1161 | This algorithm can be replaced or customized by the user. 1162 | 1163 | Internal Tasks 1164 | ++++++++++++++ 1165 | 1166 | Celery defines internal tasks to ensure it's operation and to provide 1167 | support for it's features. 1168 | 1169 | The exact API for each task will be determined in another CEP. 1170 | 1171 | This list of internal tasks is not final. 1172 | Other internal tasks may be defined in other CEPs. 1173 | 1174 | SystemD Notify 1175 | ~~~~~~~~~~~~~~ 1176 | 1177 | This task reports the status of the worker to the SystemD service which is 1178 | running it. 1179 | 1180 | It uses the `sd_notify`_ protocol to do so. 1181 | 1182 | Retry Failed Boot Step 1183 | ~~~~~~~~~~~~~~~~~~~~~~ 1184 | 1185 | This task responds to a :term:`Command Message` which instructs the worker 1186 | to retry an optional 1187 | :ref:`Boot Step ` 1188 | which has failed during the worker's initialization procedure. 1189 | 1190 | The Boot Step's execution will be retried a configured amount of times 1191 | before giving up. 1192 | 1193 | By default this task's 1194 | :ref:`Circuit Breaker ` 1195 | is configured to never prevent or automatically fail the execution of this task. 1196 | 1197 | Boot Steps 1198 | ++++++++++ 1199 | 1200 | During the Worker's initialization procedure Boot Steps are executed to prepare 1201 | it for execution of tasks. 1202 | 1203 | Some Boot Steps are responsible for starting all the 1204 | :ref:`services ` required for 1205 | the worker to function correctly. 1206 | Others may publish a :ref:`task ` 1207 | for execution to the worker's 1208 | :ref:`draft/celery-5-high-level-architecture:Inbox Queue`. 1209 | 1210 | Some Boot Steps are mandatory and thus if they fail, 1211 | the worker refuses to start. 1212 | Others are optional and their execution will be deferred to the 1213 | :ref:`draft/celery-5-high-level-architecture:Retry Failed Boot Step` task. 1214 | 1215 | Users may create and use their own Boot Steps if they wish to do so. 1216 | 1217 | Worker Health Checks 1218 | ++++++++++++++++++++ 1219 | 1220 | Worker Circuit Breakers 1221 | +++++++++++++++++++++++ 1222 | 1223 | Inbox Queue 1224 | +++++++++++ 1225 | 1226 | Each worker declares an inbox queue in the :term:`Message Broker`. 1227 | 1228 | Publishers may publish :term:`messages ` to that queue in order to 1229 | execute tasks on a specific worker. 1230 | 1231 | Celery uses the Inbox Queue to schedule the execution of the worker's internal 1232 | tasks. 1233 | 1234 | :term:`Messages ` published to the inbox queue must be 1235 | cryptographically signed. 1236 | 1237 | This requirement can be disabled using a CLI option. 1238 | Whenever the user uses this CLI option a warning log message is emitted. 1239 | 1240 | While disabling the inbox queue is possible either through a configuration setting 1241 | or a CLI option, some functionality will be lost. 1242 | Whenever the user opts to disable the Inbox Queue a warning log message is emitted. 1243 | 1244 | Publisher 1245 | --------- 1246 | 1247 | The Publisher is responsible for publishing :term:`messages ` 1248 | to a :term:`Message Broker`. 1249 | 1250 | It is responsible for publishing the :term:`Message` to the appropriate broker cluster 1251 | according to the configuration provided to the publisher. 1252 | 1253 | The publisher must be able to run in-process inside a long-running thread 1254 | or a long running co-routine. 1255 | 1256 | It can also be run using a separate daemon which can serve all the processes 1257 | publishing to the message brokers. 1258 | 1259 | Messages Backlog 1260 | ++++++++++++++++ 1261 | 1262 | The messages backlog is a temporary queue of :term:`messages ` 1263 | yet to be published to the appropriate broker cluster. 1264 | 1265 | In the event where :term:`messages ` cannot be published 1266 | for any reason, the :term:`messages ` are kept inside the queue. 1267 | 1268 | By default, an in-memory queue will be used. The user may provide another 1269 | implementation which stores the :term:`messages ` on-disk 1270 | or in a central database. 1271 | 1272 | Implementers should take into account what happens whenever writing to the 1273 | messages backlog fails. 1274 | 1275 | The default fallback mechanism will append the :term:`messages ` into 1276 | an in-memory queue. 1277 | These :term:`messages ` will be published first in order to avoid 1278 | :term:`Message` loss in case the publisher goes down for any reason. 1279 | 1280 | Publisher Daemon 1281 | ++++++++++++++++ 1282 | 1283 | In sufficiently large deployments, one server runs multiple workloads which 1284 | may publish to a :term:`Message Broker`. 1285 | 1286 | Therefore, it is unnecessary to maintain a publisher for each process that 1287 | publishes to a :term:`Message Broker`. 1288 | 1289 | In such cases, a Publisher Daemon can be used. The publishing processes will 1290 | specify it as their target and communicate the :term:`messages ` 1291 | to be published via a socket. 1292 | 1293 | Publisher Internal Services 1294 | ++++++++++++++++++++++++++++ 1295 | 1296 | The Publisher defines internal services to ensure it's operation and to provide 1297 | support for it's features. 1298 | 1299 | The exact API for each service will be determined in another CEP. 1300 | 1301 | This list of internal services is not final. 1302 | Other internal services may be defined in other CEPs. 1303 | 1304 | Message Publisher 1305 | ~~~~~~~~~~~~~~~~~ 1306 | 1307 | The ``Message Publisher`` service is responsible for publishing 1308 | :term:`messages ` to a single :term:`Message Broker`. 1309 | 1310 | This service is run for each :term:`Message Broker` the user configured the 1311 | Publisher to publish messages to. 1312 | 1313 | During the service's initialization it initializes a 1314 | :ref:`draft/celery-5-high-level-architecture:Messages Backlog`. 1315 | This will be the backlog the service consumes :term:`messages ` from. 1316 | 1317 | The service maintains a connection pool to the :term:`Message Broker` and is 1318 | responsible for scaling the pool according to the pressure on the broker. 1319 | 1320 | The connection pool's limits are configurable by the user. 1321 | By default, we only maintain one connection to the :term:`Message Broker`. 1322 | 1323 | Listener 1324 | ~~~~~~~~ 1325 | 1326 | The ``Listener`` service is responsible for receiving messages and enqueuing 1327 | them in the appropriate :ref:`draft/celery-5-high-level-architecture:Messages Backlog`. 1328 | 1329 | During initialization the service starts listening to incoming TCP connections. 1330 | 1331 | The service is only run in case the user opts to run the Publisher in 1332 | :ref:`draft/celery-5-high-level-architecture:Publisher Daemon` mode. 1333 | 1334 | Publisher Health Checks 1335 | +++++++++++++++++++++++ 1336 | 1337 | The Publisher will perform health checks to ensure that 1338 | the :term:`Message Broker` the user is publishing to is available. 1339 | 1340 | If a health check fails a configured number of times, the relevant 1341 | :term:`Circuit Breaker` is tripped. 1342 | 1343 | Each :term:`Message Broker` Celery supports must provide an implementation for 1344 | the default health checks the Publisher will use for verifying its 1345 | availability for new :term:`messages `. 1346 | 1347 | Further health checks can be defined by the user. 1348 | These health checks allows the user to avoid publishing tasks if for example 1349 | a 3rd party API endpoint is not available or slow, if the database 1350 | the user stores the results in is available or any other check for that matter. 1351 | 1352 | Publisher Circuit Breakers 1353 | ++++++++++++++++++++++++++ 1354 | 1355 | Each :ref:`health check ` 1356 | has it's own Circuit Breaker. 1357 | Once a circuit breaker is tripped, the :term:`messages ` are stored 1358 | in the :ref:`draft/celery-5-high-level-architecture:messages backlog` until 1359 | the health check recovers and the circuit is once again closed. 1360 | 1361 | Router 1362 | ------ 1363 | 1364 | The Router is a :term:`Message Dispatcher`. 1365 | It is responsible for managing the connection to a :term:`Message Broker` 1366 | and consuming :term:`messages ` from the :term:`Message Broker`. 1367 | 1368 | The Router can maintain a connection to a cluster of 1369 | :term:`message brokers ` or even clusters of 1370 | :term:`message brokers `. 1371 | 1372 | Data Sources and Sinks 1373 | ++++++++++++++++++++++ 1374 | 1375 | Data Sources are a new concept in Celery. 1376 | Data Sinks are a concept which replaces Result Backends. 1377 | 1378 | Data Sinks consume :term:`Document Messages ` while Data Sources 1379 | produce them. 1380 | 1381 | Data Sources 1382 | ~~~~~~~~~~~~ 1383 | 1384 | Data Sources are :ref:`task ` 1385 | which either listen or poll for incoming data from a data source such as a 1386 | database, a file system or an HTTP(S) endpoint. 1387 | 1388 | These services produce :term:`Document Messages `. 1389 | 1390 | Tasks which are subscribed to Data Sources will receive the raw document 1391 | messages for further processing. 1392 | 1393 | .. admonition:: Example 1394 | 1395 | We'd like to design a feature which locks Github issues immediately after 1396 | they are closed. 1397 | 1398 | Github uses Webhooks to notify us when an issue is closed. 1399 | 1400 | We set up a Data Source which starts an HTTPS server and expects incoming 1401 | HTTP requests on an endpoint. 1402 | 1403 | Whenever a request arrives a :term:`Document Message` is published. 1404 | 1405 | Data Sinks 1406 | ~~~~~~~~~~ 1407 | 1408 | A result from a :ref:`task ` 1409 | produces a :term:`Document Message` which a Data Sink or multiple Data Sinks 1410 | consume. 1411 | 1412 | These :term:`Document Messages ` are then stored in the Sinks 1413 | the task is registered to. 1414 | 1415 | .. admonition:: Example 1416 | 1417 | We have a task which calculates the hourly average impressions of a user's 1418 | post over a period of time. 1419 | 1420 | The BI team requires the data to be inserted to `BigQuery `_ 1421 | because it uses it to research the effectiveness of users posts. 1422 | 1423 | However, the user-facing post analytics dashboard also requires this data 1424 | and the team that maintains it doesn't want to use BigQuery because it is not 1425 | a cost-effective solution and because they already use `MongoDB `_ 1426 | to store all user-facing analytics data. 1427 | 1428 | To resolve the issue we declare that the task routes it's results to two data 1429 | sinks. One for the BI team and the other for the analytics team. 1430 | 1431 | Each data sink is configured to insert the data to a specific table 1432 | or collection. 1433 | 1434 | Controller 1435 | ---------- 1436 | 1437 | The Controller is responsible for managing the lifecycle of all other Celery 1438 | components. 1439 | 1440 | Celery 5 is a more complex system with multiple components and will often be 1441 | deployed in high throughput, highly available production systems. 1442 | 1443 | The introduction of multiple components require us to have another component 1444 | that manages the entire Celery cluster. 1445 | 1446 | During the lifecycle of a worker the Controller also manages and optimizes the 1447 | execution of tasks to ensure we maximize the utilization of all our resources 1448 | and to prevent expected errors. 1449 | 1450 | .. note:: 1451 | 1452 | The Controller is meant to be run as a user service. 1453 | If the Controller is run with root privileges, a log message with 1454 | the warning level will be emitted. 1455 | 1456 | Foreman 1457 | +++++++ 1458 | 1459 | The Foreman service is responsible for spawning the :ref:`Workers `, 1460 | :ref:`Routers ` and 1461 | :ref:`Schedulers `. 1462 | 1463 | By default, the Foreman service creates sub-processes for 1464 | all the required components. This is suitable for small scale deployments. 1465 | 1466 | Development Mode 1467 | ~~~~~~~~~~~~~~~~ 1468 | 1469 | During development, if explicitly specified, the Foremen will start all of 1470 | Celery's services in the same process. 1471 | 1472 | Since some of the new features in Celery require cryptographically signed 1473 | messages Celery will generate self-signed certificates using the `trustme`_ 1474 | library unless certificates are already provided or the user has chosen to 1475 | disable this behavior through a CLI option. 1476 | 1477 | SystemD Integration 1478 | ~~~~~~~~~~~~~~~~~~~ 1479 | 1480 | Unless it is explicitly overridden by the configuration, whenever the Controller 1481 | is run as a SystemD service, it will use SystemD to spawn all other Celery 1482 | components. 1483 | 1484 | Celery will provide the required services for such a deployment. 1485 | 1486 | The Controller will use the `sd_notify`_ protocol to announce when the cluster 1487 | is fully operational. 1488 | 1489 | The user must configure the list of hosts the controller will manage and ensure 1490 | SSH communication between the Controller's host and the other hosts is possible. 1491 | 1492 | Other Integrations 1493 | ~~~~~~~~~~~~~~~~~~ 1494 | 1495 | Celery may be run in Kubernetes, Swarm, Mesos, Nomad or any other container 1496 | scheduler. 1497 | 1498 | Users may provide their own integrations with the Foreman which allows them to 1499 | create and manage the different Celery components in a way that is native to the 1500 | container scheduler. 1501 | 1502 | The Controller may also manage the lifecycle of the :term:`Message Broker` if 1503 | the user wishes to do so. 1504 | 1505 | Such an integration may be provided by the user as well. 1506 | 1507 | Scheduler 1508 | +++++++++ 1509 | 1510 | The scheduler is responsible for managing the scheduling of tasks for execution 1511 | on a cluster of workers. 1512 | 1513 | The scheduler calculates the amount of tasks to be executed in any given time 1514 | in order to make cluster wide decisions when autoscaling workers or increasing 1515 | concurrency for an existing worker. 1516 | 1517 | The scheduler is aware when tasks should no longer be executed due to manual 1518 | intervention or a circuit breaker trip. To do so, it commands the router to 1519 | avoid consuming the task or rejecting it. 1520 | 1521 | Concurrency Limitations 1522 | ~~~~~~~~~~~~~~~~~~~~~~~ 1523 | 1524 | Not all :ref:`draft/celery-5-high-level-architecture:Tasks` are born equal. 1525 | Some tasks require more resources than others, some may only be executed once 1526 | at a time due to a business requirement, 1527 | other tasks may be executed only once per user at a time to avoid data corruption. 1528 | At times, some tasks should not be executed at all. 1529 | 1530 | The Scheduler is responsible for limiting the concurrency of such tasks. 1531 | 1532 | A task's concurrency may be limited per worker or globally across all workers 1533 | depending on the requirements. 1534 | In case there are tasks which are limited globally, an external data store is required. 1535 | 1536 | If a :ref:`task ` is rate limited 1537 | any concurrency limitations are ignored. 1538 | 1539 | There are multiple types of limits the user can impose on a task's concurrency: 1540 | 1541 | * **Fixed Limit**: A task can only be run at a maximum concurrency of a fixed number. 1542 | This strategy is used when there is a predetermined limit on the number of 1543 | concurrent tasks of the same type either because of lack of computing resources 1544 | or due to business requirements. 1545 | * **Range**: A task can only be run at a maximum concurrency of a calculated limit 1546 | between a range of numbers. 1547 | This strategy is used to calculate the appropriate concurrency for a task based on some 1548 | external resource such as the number of available database connections or currently 1549 | available network bandwidth. 1550 | * **Concurrency Token**: A task can only be run at a maximum concurrency of either a **Fixed Limit** 1551 | or a **Range** if it has the same Concurrency Token. 1552 | A Concurrency Token is an identifier constructed from the task's :term:`Message` 1553 | by which we group a number of tasks for the purpose of limiting their concurrency. 1554 | This strategy is used when the user would like to run one concurrent task per 1555 | user or when a task may connect to multiple database instances in the cluster 1556 | and the user wishes to limit the concurrency of the task per the available 1557 | database connections in the selected instance. 1558 | 1559 | 1560 | A concurrency limitation of 0 implies that the task will be rejected and the queue 1561 | it is on will not be consumed if possible. 1562 | 1563 | The Scheduler may impose a concurrency limit if it deems fit at any time, these 1564 | limits take precedence over any user imposed limit. 1565 | 1566 | Suspend/Resume Tasks 1567 | ~~~~~~~~~~~~~~~~~~~~ 1568 | 1569 | Whenever a :term:`Circuit Breaker` trips, the :ref:`draft/celery-5-high-level-architecture:Router` 1570 | must issue an event to the Scheduler. 1571 | The exact payload of the suspension event will be determined in another CEP. 1572 | 1573 | This will notify the Scheduler that it no longer has to take this task into 1574 | account when calculating the Celery workers cluster capacity. 1575 | In addition this will set the task's :ref:`concurrency limitation ` 1576 | to 0. 1577 | 1578 | The user may elect to send this event directly to the Scheduler if suspension 1579 | of execution is required (E.g. The task interacts with a database which is 1580 | going under expected maintenance). 1581 | 1582 | Once scheduling can be resumed, the Scheduler sends another event to the :ref:`draft/celery-5-high-level-architecture:Router`. 1583 | The exact payload of the resumption event will be determined in another CEP. 1584 | 1585 | Task Prioritization 1586 | ~~~~~~~~~~~~~~~~~~~ 1587 | 1588 | The Scheduler may instruct workers to prioritize tasks and to prefer consuming 1589 | from specific queues first. 1590 | 1591 | Priority based queues are only a partial solution to prioritizing tasks. 1592 | Some :term:`Message Brokers ` don't support it. 1593 | Those who do support priority based queues do not prioritize messages between 1594 | queues. 1595 | 1596 | This feature can be used to prefer to execute tasks which can be quickly executed 1597 | first or to execute tasks which take a long time to complete first or to execute 1598 | tasks which are rarely seen first. 1599 | 1600 | Users may supply their own strategies for prioritizing tasks. 1601 | 1602 | Resource Saturation 1603 | ~~~~~~~~~~~~~~~~~~~ 1604 | 1605 | Celery provides the Resource Saturation :ref:`draft/celery-5-high-level-architecture:Task Prioritization` 1606 | strategy to ensure we can utilize the full capacity of all the workers in the cluster. 1607 | 1608 | The scheduler instructs each worker to prefer executing :ref:`draft/celery-5-high-level-architecture:I/O Bound Tasks` 1609 | if the capacity of the worker for executing :ref:`draft/celery-5-high-level-architecture:CPU Bound Tasks` 1610 | is nearing its maximum and vice versa. 1611 | 1612 | Rate Limiting 1613 | ~~~~~~~~~~~~~ 1614 | 1615 | A user may impose a rate limit on the execution of a :ref:`task `. 1616 | 1617 | For example, we only want to run 200 `send_welcome_email()` :ref:`draft/celery-5-high-level-architecture:Tasks` 1618 | per minute in order to avoid decreasing our email reputation. 1619 | 1620 | :ref:`draft/celery-5-high-level-architecture:Tasks` may define a global rate limit or a per worker rate limit. 1621 | 1622 | Whenever a :ref:`task ` reaches 1623 | it's rate limit, an event is published to the :ref:`draft/celery-5-high-level-architecture:Router`'s 1624 | :ref:`draft/celery-5-high-level-architecture:Inbox Queue`. 1625 | The event notifies the Router that it should not consume these tasks if possible. 1626 | The exact payload of the rate limiting event will be determined 1627 | in another CEP. 1628 | 1629 | In addition the task is :ref:`suspended ` until the rate 1630 | limiting period is over. 1631 | 1632 | Periodic Tasks 1633 | ~~~~~~~~~~~~~~ 1634 | 1635 | Previously, Celery used it's in-house periodic tasks scheduler which was the 1636 | source of many bugs. 1637 | 1638 | In Celery 5 we will use the `APScheduler `_. 1639 | 1640 | APScheduler has proved itself in production, is flexible and customizable and 1641 | will provide trio support in 4.0, it's next major version. 1642 | 1643 | In addition, APScheduler 4.0 will be highly available, a highly demanded feature 1644 | from our users. This means that two Controller instances may exist simultaneously 1645 | without duplicated :ref:`draft/celery-5-high-level-architecture:Tasks` being scheduled 1646 | for execution. 1647 | 1648 | The Scheduler only uses APScheduler to publish :ref:`draft/celery-5-high-level-architecture:Tasks` 1649 | at the appropriate time according to the schedule provided by the user. 1650 | Periodic tasks do not run inside the Scheduler. 1651 | 1652 | Autoscaler 1653 | ~~~~~~~~~~ 1654 | 1655 | The Scheduler contains all the data required for making autoscaling decisions. 1656 | 1657 | It is aware of how many tasks will be automatically rejected because 1658 | they are :ref:`suspended ` 1659 | for any reason. 1660 | 1661 | It is aware of how many :ref:`draft/celery-5-high-level-architecture:Periodic Tasks` 1662 | are going to be scheduled in the future. 1663 | 1664 | The Scheduler is aware for the maximum concurrency allowed for each worker and 1665 | the :ref:`draft/celery-5-high-level-architecture:Concurrency Limitations` of specific tasks. 1666 | 1667 | The Scheduler also periodically samples the queues' length. 1668 | 1669 | Unfortunately, modeling such a queuing system is beyond the scope of Celery 5 due 1670 | to the already large amount of new feature and changes in this version 1671 | and our lack of knowledge in the math involved in such a model. 1672 | 1673 | Instead we're going to provide the simple algorithm we use now in Celery 4 1674 | with some adjustments but allow room for extension. 1675 | 1676 | In Celery 4 each worker checks if it should autoscale every second. 1677 | This can cause a lot of thrashing as new processes are created and destroyed. 1678 | 1679 | In Celery 5 after each autoscale event, there will be a cooldown period. 1680 | The cooldown period increases exponentially until a configurable limit. 1681 | 1682 | If the number of tasks in all the queues is larger than the current concurrency 1683 | budget the Autoscaler publishes an event to all the routers. 1684 | The routers will increase their prefetching multiplier as a response to this event. 1685 | 1686 | The Scheduler will select which workers should increase their prefetching of tasks 1687 | in order to reach the maximal concurrency budget. 1688 | 1689 | Controller Internal Services 1690 | ++++++++++++++++++++++++++++ 1691 | 1692 | Motivation 1693 | ========== 1694 | 1695 | Rationale 1696 | ========= 1697 | 1698 | Backwards Compatibility 1699 | ======================= 1700 | 1701 | Reference Implementation 1702 | ======================== 1703 | 1704 | This document describes the high level architecture of Celery 5. 1705 | As such, it does not have an implementation at the time of writing. 1706 | 1707 | Copyright 1708 | ========= 1709 | 1710 | This document has been placed in the public domain per the Creative Commons 1711 | CC0 1.0 Universal license (https://creativecommons.org/publicdomain/zero/1.0/deed). 1712 | 1713 | .. _qpid-proton: http://qpid.apache.org/releases/qpid-proton-0.29.0/proton/python/book/tutorial.html 1714 | .. _JOURNAL_STREAM: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#%24JOURNAL_STREAM 1715 | .. _journald: https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html 1716 | .. _Click: https://click.palletsprojects.com/en/7.x/ 1717 | .. _Trio: https://trio.readthedocs.io/en/latest/ 1718 | .. _dependencies: https://github.com/dry-python/dependencies 1719 | .. _sd_notify: https://www.freedesktop.org/software/systemd/man/sd_notify.html 1720 | .. _trustme: https://github.com/python-trio/trustme 1721 | --------------------------------------------------------------------------------