├── .codecov.yml ├── .github └── workflows │ └── ci.yml ├── .gitignore ├── CHANGELOG.rst ├── CONTRIBUTORS.rst ├── LICENSE ├── MANIFEST.in ├── README.rst ├── docs ├── Makefile ├── make.bat └── source │ ├── api.rst │ ├── authors.rst │ ├── conf.py │ ├── index.rst │ ├── installation.rst │ ├── license.rst │ ├── security │ ├── CVE-2016-6580.rst │ └── index.rst │ └── using-priority.rst ├── setup.cfg ├── setup.py ├── src └── priority │ ├── __init__.py │ ├── priority.py │ └── py.typed ├── test └── test_priority.py └── tox.ini /.codecov.yml: -------------------------------------------------------------------------------- 1 | comment: off 2 | -------------------------------------------------------------------------------- /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | name: CI 2 | 3 | on: 4 | push: 5 | branches: ["master"] 6 | pull_request: 7 | branches: ["master"] 8 | 9 | jobs: 10 | tox: 11 | runs-on: ubuntu-latest 12 | strategy: 13 | max-parallel: 5 14 | matrix: 15 | python-version: 16 | - 3.6 17 | - 3.7 18 | - 3.8 19 | - 3.9 20 | - pypy3 21 | 22 | steps: 23 | - uses: actions/checkout@v2 24 | - uses: actions/setup-python@v2 25 | with: 26 | python-version: ${{ matrix.python-version }} 27 | - name: Install tox 28 | run: | 29 | python -m pip install --upgrade pip setuptools 30 | pip install --upgrade tox tox-gh-actions 31 | - name: Initialize tox envs 32 | run: | 33 | tox --parallel auto --notest 34 | - name: Test with tox 35 | run: | 36 | tox --parallel 0 37 | - uses: codecov/codecov-action@v1 38 | with: 39 | file: ./coverage.xml 40 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | build/ 2 | env/ 3 | dist/ 4 | *.egg-info/ 5 | *.pyc 6 | __pycache__ 7 | .coverage 8 | coverage.xml 9 | .tox/ 10 | .hypothesis/ 11 | .cache/ 12 | -------------------------------------------------------------------------------- /CHANGELOG.rst: -------------------------------------------------------------------------------- 1 | Changelog 2 | ========= 3 | 4 | 2.0.0 (2021-06-27) 5 | ------------------ 6 | 7 | **API Changes** 8 | 9 | - Python 3.6 is the minimal support Python version. 10 | - Support for Python 3.7 has been added. 11 | - Support for Python 3.8 has been added. 12 | - Support for Python 3.9 has been added. 13 | - Support for Python 2.7 has been removed. 14 | - Support for Python 3.3 has been removed. 15 | - Support for Python 3.4 has been removed. 16 | - Support for Python 3.5 has been removed. 17 | - Support for PyPy (Python 2.7 compatible) has been removed. 18 | - Add type hints throughout and support PEP 561 via a py.typed 19 | file. This should allow projects to type check their usage of this dependency. 20 | - Throw ``TypeError`` when creating a priority tree with a ``maximum_streams`` 21 | value that is not an integer. 22 | - Throw ``ValueError`` when creating a priority tree with a ``maximum_streams`` 23 | value that is not a positive integer. 24 | 25 | 1.3.0 (2017-01-27) 26 | ------------------ 27 | 28 | **API Changes** 29 | 30 | - Throw ``PriorityLoop`` when inserting or reprioritising a stream that 31 | depends on itself. 32 | - Throw ``BadWeightError`` when creating or reprioritising a stream with a 33 | weight that is not an integer between 1 and 256, inclusive. 34 | - Throw ``PseudoStreamError`` when trying to reprioritise, remove, block or 35 | unblock stream 0. 36 | - Add a new ``PriorityError`` parent class for the exceptions that can be 37 | thrown by priority. 38 | 39 | 1.2.2 (2016-11-11) 40 | ------------------ 41 | 42 | **Bugfixes** 43 | 44 | - Allow ``insert_stream`` to be called with ``exclusive=True`` but no explicit 45 | ``depends_on`` value. 46 | 47 | 1.2.1 (2016-10-26) 48 | ------------------ 49 | 50 | **Bugfixes** 51 | 52 | - Allow insertion of streams that have parents in the idle or closed states. 53 | This would previously raise a KeyError. 54 | 55 | 1.2.0 (2016-08-04) 56 | ------------------ 57 | 58 | **Security Fixes** 59 | 60 | - CVE-2016-6580: All versions of this library prior to 1.2.0 are vulnerable to 61 | a denial of service attack whereby a remote peer can cause a user to insert 62 | an unbounded number of streams into the priority tree, eventually consuming 63 | all available memory. 64 | 65 | This version adds a ``TooManyStreamsError`` exception that is raised when 66 | too many streams are inserted into the priority tree. It also adds a keyword 67 | argument to the priority tree, ``maximum_streams``, which limits how many 68 | streams may be inserted. By default, this number is set to 1000. 69 | Implementations should strongly consider whether they can set this value 70 | lower. 71 | 72 | 1.1.1 (2016-05-28) 73 | ------------------ 74 | 75 | **Bugfixes** 76 | 77 | - 2.5x performance improvement by swapping from ``queue.PriorityQueue`` to 78 | ``heapq``. 79 | 80 | 1.1.0 (2016-01-08) 81 | ------------------ 82 | 83 | **API Changes** 84 | 85 | - Throw ``DuplicateStreamError`` when inserting a stream that is already in the 86 | tree. 87 | - Throw ``MissingStreamError`` when reprioritising a stream that is not in the 88 | tree. 89 | 90 | 1.0.0 (2015-12-07) 91 | ------------------ 92 | 93 | - Initial release. 94 | -------------------------------------------------------------------------------- /CONTRIBUTORS.rst: -------------------------------------------------------------------------------- 1 | Priority is written and maintained by Cory Benfield and various contributors: 2 | 3 | Development Lead 4 | ```````````````` 5 | 6 | - Cory Benfield 7 | 8 | Contributors 9 | ```````````` 10 | 11 | In chronological order: 12 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015 Cory Benfield 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy 4 | of this software and associated documentation files (the "Software"), to deal 5 | in the Software without restriction, including without limitation the rights 6 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 7 | copies of the Software, and to permit persons to whom the Software is 8 | furnished to do so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in 11 | all copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 19 | THE SOFTWARE. -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | graft src/priority 2 | graft docs 3 | graft test 4 | graft visualizer 5 | graft examples 6 | prune docs/build 7 | include README.rst LICENSE CHANGELOG.rst CONTRIBUTORS.rst tox.ini 8 | global-exclude *.pyc *.pyo *.swo *.swp *.map *.yml *.DS_Store 9 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | ========================================== 2 | Priority: A HTTP/2 Priority Implementation 3 | ========================================== 4 | 5 | .. image:: https://github.com/python-hyper/priority/workflows/CI/badge.svg 6 | :target: https://github.com/python-hyper/priority/actions 7 | :alt: Build Status 8 | .. image:: https://codecov.io/gh/python-hyper/priority/branch/master/graph/badge.svg 9 | :target: https://codecov.io/gh/python-hyper/priority 10 | :alt: Code Coverage 11 | .. image:: https://readthedocs.org/projects/priority/badge/?version=latest 12 | :target: https://priority.readthedocs.io/en/latest/ 13 | :alt: Documentation Status 14 | .. image:: https://img.shields.io/badge/chat-join_now-brightgreen.svg 15 | :target: https://gitter.im/python-hyper/community 16 | :alt: Chat community 17 | 18 | .. image:: https://raw.github.com/python-hyper/documentation/master/source/logo/hyper-black-bg-white.png 19 | 20 | 21 | Priority is a pure-Python implementation of the priority logic for HTTP/2, set 22 | out in `RFC 7540 Section 5.3 (Stream Priority)`_. This logic allows for clients 23 | to express a preference for how the server allocates its (limited) resources to 24 | the many outstanding HTTP requests that may be running over a single HTTP/2 25 | connection. 26 | 27 | Specifically, this Python implementation uses a variant of the implementation 28 | used in the excellent `H2O`_ project. This original implementation is also the 29 | inspiration for `nghttp2's`_ priority implementation, and generally produces a 30 | very clean and even priority stream. The only notable changes from H2O's 31 | implementation are small modifications to allow the priority implementation to 32 | work cleanly as a separate implementation, rather than being embedded in a 33 | HTTP/2 stack directly. 34 | 35 | While priority information in HTTP/2 is only a suggestion, rather than an 36 | enforceable constraint, where possible servers should respect the priority 37 | requests of their clients. 38 | 39 | Using Priority 40 | -------------- 41 | 42 | Priority has a simple API. Streams are inserted into the tree: when they are 43 | inserted, they may optionally have a weight, depend on another stream, or 44 | become an exclusive dependent of another stream. 45 | 46 | .. code-block:: python 47 | 48 | >>> p = priority.PriorityTree() 49 | >>> p.insert_stream(stream_id=1) 50 | >>> p.insert_stream(stream_id=3) 51 | >>> p.insert_stream(stream_id=5, depends_on=1) 52 | >>> p.insert_stream(stream_id=7, weight=32) 53 | >>> p.insert_stream(stream_id=9, depends_on=7, weight=8) 54 | >>> p.insert_stream(stream_id=11, depends_on=7, exclusive=True) 55 | 56 | Once streams are inserted, the stream priorities can be requested. This allows 57 | the server to make decisions about how to allocate resources. 58 | 59 | Iterating The Tree 60 | ~~~~~~~~~~~~~~~~~~ 61 | 62 | The tree in this algorithm acts as a gate. Its goal is to allow one stream 63 | "through" at a time, in such a manner that all the active streams are served as 64 | evenly as possible in proportion to their weights. 65 | 66 | This is handled in Priority by iterating over the tree. The tree itself is an 67 | iterator, and each time it is advanced it will yield a stream ID. This is the 68 | ID of the stream that should next send data. 69 | 70 | This looks like this: 71 | 72 | .. code-block:: python 73 | 74 | >>> for stream_id in p: 75 | ... send_data(stream_id) 76 | 77 | If each stream only sends when it is 'ungated' by this mechanism, the server 78 | will automatically be emitting stream data in conformance to RFC 7540. 79 | 80 | Updating The Tree 81 | ~~~~~~~~~~~~~~~~~ 82 | 83 | If for any reason a stream is unable to proceed (for example, it is blocked on 84 | HTTP/2 flow control, or it is waiting for more data from another service), that 85 | stream is *blocked*. The ``PriorityTree`` should be informed that the stream is 86 | blocked so that other dependent streams get a chance to proceed. This can be 87 | done by calling the ``block`` method of the tree with the stream ID that is 88 | currently unable to proceed. This will automatically update the tree, and it 89 | will adjust on the fly to correctly allow any streams that were dependent on 90 | the blocked one to progress. 91 | 92 | For example: 93 | 94 | .. code-block:: python 95 | 96 | >>> for stream_id in p: 97 | ... send_data(stream_id) 98 | ... if blocked(stream_id): 99 | ... p.block(stream_id) 100 | 101 | When a stream goes from being blocked to being unblocked, call the ``unblock`` 102 | method to place it back into the sequence. Both the ``block`` and ``unblock`` 103 | methods are idempotent and safe to call repeatedly. 104 | 105 | Additionally, the priority of a stream may change. When it does, the 106 | ``reprioritize`` method can be used to update the tree in the wake of that 107 | change. ``reprioritize`` has the same signature as ``insert_stream``, but 108 | applies only to streams already in the tree. 109 | 110 | Removing Streams 111 | ~~~~~~~~~~~~~~~~ 112 | 113 | A stream can be entirely removed from the tree by calling ``remove_stream``. 114 | Note that this is not idempotent. Further, calling ``remove_stream`` and then 115 | re-adding it *may* cause a substantial change in the shape of the priority 116 | tree, and *will* cause the iteration order to change. 117 | 118 | License 119 | ------- 120 | 121 | Priority is made available under the MIT License. For more details, see the 122 | LICENSE file in the repository. 123 | 124 | Authors 125 | ------- 126 | 127 | Priority is maintained by Cory Benfield, with contributions from others. For 128 | more details about the contributors, please see CONTRIBUTORS.rst in the 129 | repository. 130 | 131 | 132 | .. _RFC 7540 Section 5.3 (Stream Priority): https://tools.ietf.org/html/rfc7540#section-5.3 133 | .. _nghttp2's: https://nghttp2.org/blog/2015/11/11/stream-scheduling-utilizing-http2-priority/ 134 | .. _H2O: https://h2o.examp1e.net/ 135 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | PAPER = 8 | BUILDDIR = build 9 | 10 | # User-friendly check for sphinx-build 11 | ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) 12 | $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) 13 | endif 14 | 15 | # Internal variables. 16 | PAPEROPT_a4 = -D latex_paper_size=a4 17 | PAPEROPT_letter = -D latex_paper_size=letter 18 | ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source 19 | # the i18n builder cannot share the environment and doctrees with the others 20 | I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source 21 | 22 | .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext 23 | 24 | help: 25 | @echo "Please use \`make ' where is one of" 26 | @echo " html to make standalone HTML files" 27 | @echo " dirhtml to make HTML files named index.html in directories" 28 | @echo " singlehtml to make a single large HTML file" 29 | @echo " pickle to make pickle files" 30 | @echo " json to make JSON files" 31 | @echo " htmlhelp to make HTML files and a HTML help project" 32 | @echo " qthelp to make HTML files and a qthelp project" 33 | @echo " devhelp to make HTML files and a Devhelp project" 34 | @echo " epub to make an epub" 35 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 36 | @echo " latexpdf to make LaTeX files and run them through pdflatex" 37 | @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" 38 | @echo " text to make text files" 39 | @echo " man to make manual pages" 40 | @echo " texinfo to make Texinfo files" 41 | @echo " info to make Texinfo files and run them through makeinfo" 42 | @echo " gettext to make PO message catalogs" 43 | @echo " changes to make an overview of all changed/added/deprecated items" 44 | @echo " xml to make Docutils-native XML files" 45 | @echo " pseudoxml to make pseudoxml-XML files for display purposes" 46 | @echo " linkcheck to check all external links for integrity" 47 | @echo " doctest to run all doctests embedded in the documentation (if enabled)" 48 | 49 | clean: 50 | rm -rf $(BUILDDIR)/* 51 | 52 | html: 53 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html 54 | @echo 55 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." 56 | 57 | dirhtml: 58 | $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml 59 | @echo 60 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." 61 | 62 | singlehtml: 63 | $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml 64 | @echo 65 | @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." 66 | 67 | pickle: 68 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle 69 | @echo 70 | @echo "Build finished; now you can process the pickle files." 71 | 72 | json: 73 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json 74 | @echo 75 | @echo "Build finished; now you can process the JSON files." 76 | 77 | htmlhelp: 78 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp 79 | @echo 80 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 81 | ".hhp project file in $(BUILDDIR)/htmlhelp." 82 | 83 | qthelp: 84 | $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp 85 | @echo 86 | @echo "Build finished; now you can run "qcollectiongenerator" with the" \ 87 | ".qhcp project file in $(BUILDDIR)/qthelp, like this:" 88 | @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/priority.qhcp" 89 | @echo "To view the help file:" 90 | @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/priority.qhc" 91 | 92 | devhelp: 93 | $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp 94 | @echo 95 | @echo "Build finished." 96 | @echo "To view the help file:" 97 | @echo "# mkdir -p $$HOME/.local/share/devhelp/priority" 98 | @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/priority" 99 | @echo "# devhelp" 100 | 101 | epub: 102 | $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub 103 | @echo 104 | @echo "Build finished. The epub file is in $(BUILDDIR)/epub." 105 | 106 | latex: 107 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 108 | @echo 109 | @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." 110 | @echo "Run \`make' in that directory to run these through (pdf)latex" \ 111 | "(use \`make latexpdf' here to do that automatically)." 112 | 113 | latexpdf: 114 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 115 | @echo "Running LaTeX files through pdflatex..." 116 | $(MAKE) -C $(BUILDDIR)/latex all-pdf 117 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 118 | 119 | latexpdfja: 120 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 121 | @echo "Running LaTeX files through platex and dvipdfmx..." 122 | $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja 123 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 124 | 125 | text: 126 | $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text 127 | @echo 128 | @echo "Build finished. The text files are in $(BUILDDIR)/text." 129 | 130 | man: 131 | $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man 132 | @echo 133 | @echo "Build finished. The manual pages are in $(BUILDDIR)/man." 134 | 135 | texinfo: 136 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 137 | @echo 138 | @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." 139 | @echo "Run \`make' in that directory to run these through makeinfo" \ 140 | "(use \`make info' here to do that automatically)." 141 | 142 | info: 143 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 144 | @echo "Running Texinfo files through makeinfo..." 145 | make -C $(BUILDDIR)/texinfo info 146 | @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." 147 | 148 | gettext: 149 | $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale 150 | @echo 151 | @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." 152 | 153 | changes: 154 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes 155 | @echo 156 | @echo "The overview file is in $(BUILDDIR)/changes." 157 | 158 | linkcheck: 159 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck 160 | @echo 161 | @echo "Link check complete; look for any errors in the above output " \ 162 | "or in $(BUILDDIR)/linkcheck/output.txt." 163 | 164 | doctest: 165 | $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest 166 | @echo "Testing of doctests in the sources finished, look at the " \ 167 | "results in $(BUILDDIR)/doctest/output.txt." 168 | 169 | xml: 170 | $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml 171 | @echo 172 | @echo "Build finished. The XML files are in $(BUILDDIR)/xml." 173 | 174 | pseudoxml: 175 | $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml 176 | @echo 177 | @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." 178 | -------------------------------------------------------------------------------- /docs/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | REM Command file for Sphinx documentation 4 | 5 | if "%SPHINXBUILD%" == "" ( 6 | set SPHINXBUILD=sphinx-build 7 | ) 8 | set BUILDDIR=build 9 | set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% source 10 | set I18NSPHINXOPTS=%SPHINXOPTS% source 11 | if NOT "%PAPER%" == "" ( 12 | set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% 13 | set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS% 14 | ) 15 | 16 | if "%1" == "" goto help 17 | 18 | if "%1" == "help" ( 19 | :help 20 | echo.Please use `make ^` where ^ is one of 21 | echo. html to make standalone HTML files 22 | echo. dirhtml to make HTML files named index.html in directories 23 | echo. singlehtml to make a single large HTML file 24 | echo. pickle to make pickle files 25 | echo. json to make JSON files 26 | echo. htmlhelp to make HTML files and a HTML help project 27 | echo. qthelp to make HTML files and a qthelp project 28 | echo. devhelp to make HTML files and a Devhelp project 29 | echo. epub to make an epub 30 | echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter 31 | echo. text to make text files 32 | echo. man to make manual pages 33 | echo. texinfo to make Texinfo files 34 | echo. gettext to make PO message catalogs 35 | echo. changes to make an overview over all changed/added/deprecated items 36 | echo. xml to make Docutils-native XML files 37 | echo. pseudoxml to make pseudoxml-XML files for display purposes 38 | echo. linkcheck to check all external links for integrity 39 | echo. doctest to run all doctests embedded in the documentation if enabled 40 | goto end 41 | ) 42 | 43 | if "%1" == "clean" ( 44 | for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i 45 | del /q /s %BUILDDIR%\* 46 | goto end 47 | ) 48 | 49 | 50 | %SPHINXBUILD% 2> nul 51 | if errorlevel 9009 ( 52 | echo. 53 | echo.The 'sphinx-build' command was not found. Make sure you have Sphinx 54 | echo.installed, then set the SPHINXBUILD environment variable to point 55 | echo.to the full path of the 'sphinx-build' executable. Alternatively you 56 | echo.may add the Sphinx directory to PATH. 57 | echo. 58 | echo.If you don't have Sphinx installed, grab it from 59 | echo.http://sphinx-doc.org/ 60 | exit /b 1 61 | ) 62 | 63 | if "%1" == "html" ( 64 | %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html 65 | if errorlevel 1 exit /b 1 66 | echo. 67 | echo.Build finished. The HTML pages are in %BUILDDIR%/html. 68 | goto end 69 | ) 70 | 71 | if "%1" == "dirhtml" ( 72 | %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml 73 | if errorlevel 1 exit /b 1 74 | echo. 75 | echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. 76 | goto end 77 | ) 78 | 79 | if "%1" == "singlehtml" ( 80 | %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml 81 | if errorlevel 1 exit /b 1 82 | echo. 83 | echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. 84 | goto end 85 | ) 86 | 87 | if "%1" == "pickle" ( 88 | %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle 89 | if errorlevel 1 exit /b 1 90 | echo. 91 | echo.Build finished; now you can process the pickle files. 92 | goto end 93 | ) 94 | 95 | if "%1" == "json" ( 96 | %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json 97 | if errorlevel 1 exit /b 1 98 | echo. 99 | echo.Build finished; now you can process the JSON files. 100 | goto end 101 | ) 102 | 103 | if "%1" == "htmlhelp" ( 104 | %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp 105 | if errorlevel 1 exit /b 1 106 | echo. 107 | echo.Build finished; now you can run HTML Help Workshop with the ^ 108 | .hhp project file in %BUILDDIR%/htmlhelp. 109 | goto end 110 | ) 111 | 112 | if "%1" == "qthelp" ( 113 | %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp 114 | if errorlevel 1 exit /b 1 115 | echo. 116 | echo.Build finished; now you can run "qcollectiongenerator" with the ^ 117 | .qhcp project file in %BUILDDIR%/qthelp, like this: 118 | echo.^> qcollectiongenerator %BUILDDIR%\qthelp\priority.qhcp 119 | echo.To view the help file: 120 | echo.^> assistant -collectionFile %BUILDDIR%\qthelp\priority.ghc 121 | goto end 122 | ) 123 | 124 | if "%1" == "devhelp" ( 125 | %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp 126 | if errorlevel 1 exit /b 1 127 | echo. 128 | echo.Build finished. 129 | goto end 130 | ) 131 | 132 | if "%1" == "epub" ( 133 | %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub 134 | if errorlevel 1 exit /b 1 135 | echo. 136 | echo.Build finished. The epub file is in %BUILDDIR%/epub. 137 | goto end 138 | ) 139 | 140 | if "%1" == "latex" ( 141 | %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex 142 | if errorlevel 1 exit /b 1 143 | echo. 144 | echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. 145 | goto end 146 | ) 147 | 148 | if "%1" == "latexpdf" ( 149 | %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex 150 | cd %BUILDDIR%/latex 151 | make all-pdf 152 | cd %BUILDDIR%/.. 153 | echo. 154 | echo.Build finished; the PDF files are in %BUILDDIR%/latex. 155 | goto end 156 | ) 157 | 158 | if "%1" == "latexpdfja" ( 159 | %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex 160 | cd %BUILDDIR%/latex 161 | make all-pdf-ja 162 | cd %BUILDDIR%/.. 163 | echo. 164 | echo.Build finished; the PDF files are in %BUILDDIR%/latex. 165 | goto end 166 | ) 167 | 168 | if "%1" == "text" ( 169 | %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text 170 | if errorlevel 1 exit /b 1 171 | echo. 172 | echo.Build finished. The text files are in %BUILDDIR%/text. 173 | goto end 174 | ) 175 | 176 | if "%1" == "man" ( 177 | %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man 178 | if errorlevel 1 exit /b 1 179 | echo. 180 | echo.Build finished. The manual pages are in %BUILDDIR%/man. 181 | goto end 182 | ) 183 | 184 | if "%1" == "texinfo" ( 185 | %SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo 186 | if errorlevel 1 exit /b 1 187 | echo. 188 | echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo. 189 | goto end 190 | ) 191 | 192 | if "%1" == "gettext" ( 193 | %SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale 194 | if errorlevel 1 exit /b 1 195 | echo. 196 | echo.Build finished. The message catalogs are in %BUILDDIR%/locale. 197 | goto end 198 | ) 199 | 200 | if "%1" == "changes" ( 201 | %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes 202 | if errorlevel 1 exit /b 1 203 | echo. 204 | echo.The overview file is in %BUILDDIR%/changes. 205 | goto end 206 | ) 207 | 208 | if "%1" == "linkcheck" ( 209 | %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck 210 | if errorlevel 1 exit /b 1 211 | echo. 212 | echo.Link check complete; look for any errors in the above output ^ 213 | or in %BUILDDIR%/linkcheck/output.txt. 214 | goto end 215 | ) 216 | 217 | if "%1" == "doctest" ( 218 | %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest 219 | if errorlevel 1 exit /b 1 220 | echo. 221 | echo.Testing of doctests in the sources finished, look at the ^ 222 | results in %BUILDDIR%/doctest/output.txt. 223 | goto end 224 | ) 225 | 226 | if "%1" == "xml" ( 227 | %SPHINXBUILD% -b xml %ALLSPHINXOPTS% %BUILDDIR%/xml 228 | if errorlevel 1 exit /b 1 229 | echo. 230 | echo.Build finished. The XML files are in %BUILDDIR%/xml. 231 | goto end 232 | ) 233 | 234 | if "%1" == "pseudoxml" ( 235 | %SPHINXBUILD% -b pseudoxml %ALLSPHINXOPTS% %BUILDDIR%/pseudoxml 236 | if errorlevel 1 exit /b 1 237 | echo. 238 | echo.Build finished. The pseudo-XML files are in %BUILDDIR%/pseudoxml. 239 | goto end 240 | ) 241 | 242 | :end 243 | -------------------------------------------------------------------------------- /docs/source/api.rst: -------------------------------------------------------------------------------- 1 | Priority API 2 | ============ 3 | 4 | Priority Tree 5 | ------------- 6 | 7 | .. autoclass:: priority.PriorityTree 8 | :members: 9 | 10 | Exceptions 11 | ---------- 12 | 13 | .. autoclass:: priority.PriorityError 14 | 15 | .. autoclass:: priority.DeadlockError 16 | 17 | .. autoclass:: priority.PriorityLoop 18 | 19 | .. autoclass:: priority.DuplicateStreamError 20 | 21 | .. autoclass:: priority.MissingStreamError 22 | 23 | .. autoclass:: priority.TooManyStreamsError 24 | 25 | .. autoclass:: priority.BadWeightError 26 | 27 | .. autoclass:: priority.PseudoStreamError 28 | -------------------------------------------------------------------------------- /docs/source/authors.rst: -------------------------------------------------------------------------------- 1 | Contributors 2 | ============ 3 | 4 | .. include:: ../../CONTRIBUTORS.rst -------------------------------------------------------------------------------- /docs/source/conf.py: -------------------------------------------------------------------------------- 1 | # Configuration file for the Sphinx documentation builder. 2 | # 3 | # This file only contains a selection of the most common options. For a full 4 | # list see the documentation: 5 | # https://www.sphinx-doc.org/en/master/usage/configuration.html 6 | 7 | # -- Path setup -------------------------------------------------------------- 8 | 9 | # If extensions (or modules to document with autodoc) are in another directory, 10 | # add these directories to sys.path here. If the directory is relative to the 11 | # documentation root, use os.path.abspath to make it absolute, like shown here. 12 | # 13 | import os 14 | import sys 15 | import re 16 | 17 | sys.path.insert(0, os.path.abspath('../..')) 18 | 19 | PROJECT_ROOT = os.path.dirname(__file__) 20 | # Get the version 21 | version_regex = r'__version__ = ["\']([^"\']*)["\']' 22 | with open(os.path.join(PROJECT_ROOT, '../../', 'src/priority/__init__.py')) as file_: 23 | text = file_.read() 24 | match = re.search(version_regex, text) 25 | version = match.group(1) 26 | 27 | 28 | # -- Project information ----------------------------------------------------- 29 | 30 | project = 'priority' 31 | copyright = '2021, Cory Benfield' 32 | author = 'Cory Benfield' 33 | release = version 34 | 35 | # -- General configuration ------------------------------------------------ 36 | 37 | # Add any Sphinx extension module names here, as strings. They can be 38 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 39 | # ones. 40 | extensions = [ 41 | 'sphinx.ext.autodoc', 42 | 'sphinx.ext.intersphinx', 43 | 'sphinx.ext.viewcode', 44 | ] 45 | 46 | # Add any paths that contain templates here, relative to this directory. 47 | templates_path = ['_templates'] 48 | 49 | # The master toctree document. 50 | master_doc = 'index' 51 | 52 | # List of patterns, relative to source directory, that match files and 53 | # directories to ignore when looking for source files. 54 | exclude_patterns = [] 55 | 56 | # Example configuration for intersphinx: refer to the Python standard library. 57 | intersphinx_mapping = {'http://docs.python.org/': None} 58 | 59 | # -- Options for HTML output ---------------------------------------------- 60 | 61 | # The theme to use for HTML and HTML Help pages. See the documentation for 62 | # a list of builtin themes. 63 | html_theme = 'default' 64 | 65 | # Add any paths that contain custom static files (such as style sheets) here, 66 | # relative to this directory. They are copied after the builtin static files, 67 | # so a file named "default.css" will overwrite the builtin "default.css". 68 | # html_static_path = ['_static'] 69 | -------------------------------------------------------------------------------- /docs/source/index.rst: -------------------------------------------------------------------------------- 1 | Priority: A pure-Python HTTP/2 Priority implementation 2 | ====================================================== 3 | 4 | Priority is a pure-Python implementation of the priority logic for HTTP/2, set 5 | out in `RFC 7540 Section 5.3 (Stream Priority)`_. This logic allows for clients 6 | to express a preference for how the server allocates its (limited) resources to 7 | the many outstanding HTTP requests that may be running over a single HTTP/2 8 | connection. 9 | 10 | Specifically, this Python implementation uses a variant of the implementation 11 | used in the excellent `H2O`_ project. This original implementation is also the 12 | inspiration for `nghttp2's`_ priority implementation, and generally produces a 13 | very clean and even priority stream. The only notable changes from H2O's 14 | implementation are small modifications to allow the priority implementation to 15 | work cleanly as a separate implementation, rather than being embedded in a 16 | HTTP/2 stack directly. 17 | 18 | Contents: 19 | 20 | .. toctree:: 21 | :maxdepth: 2 22 | 23 | installation 24 | using-priority 25 | api 26 | security/index 27 | license 28 | authors 29 | 30 | 31 | .. _RFC 7540 Section 5.3 (Stream Priority): https://tools.ietf.org/html/rfc7540#section-5.3 32 | .. _nghttp2's: https://nghttp2.org/blog/2015/11/11/stream-scheduling-utilizing-http2-priority/ 33 | .. _H2O: https://h2o.examp1e.net/ 34 | -------------------------------------------------------------------------------- /docs/source/installation.rst: -------------------------------------------------------------------------------- 1 | Installation 2 | ============ 3 | 4 | Priority is a pure-Python project. This means installing it is extremely 5 | simple. To get the latest release from PyPI, simply run: 6 | 7 | .. code-block:: console 8 | 9 | $ pip install priority 10 | -------------------------------------------------------------------------------- /docs/source/license.rst: -------------------------------------------------------------------------------- 1 | License 2 | ======= 3 | 4 | .. include:: ../../LICENSE 5 | -------------------------------------------------------------------------------- /docs/source/security/CVE-2016-6580.rst: -------------------------------------------------------------------------------- 1 | :orphan: 2 | 3 | DoS via Unlimited Stream Insertion 4 | ================================== 5 | 6 | Hyper Project security advisory, August 4th 2016. 7 | 8 | Vulnerability 9 | ------------- 10 | 11 | A HTTP/2 implementation built using the priority library could be targetted by 12 | a malicious peer by having that peer assign priority information for every 13 | possible HTTP/2 stream ID. The priority tree would happily continue to store 14 | the priority information for each stream, and would therefore allocate 15 | unbounded amounts of memory. Attempting to actually *use* a tree like this 16 | would also cause extremely high CPU usage to maintain the tree. 17 | 18 | We are not aware of any active exploits of this vulnerability, but as this 19 | class of attack was publicly described in `this report`_, users should assume 20 | that they are at imminent risk of this kind of attack. 21 | 22 | Info 23 | ---- 24 | 25 | This issue has been given the name CVE-2016-6580. 26 | 27 | Affected Versions 28 | ----------------- 29 | 30 | This issue affects all versions of the priority library prior to 1.2.0. 31 | 32 | The Solution 33 | ------------ 34 | 35 | In version 1.2.0, the priority library limits the maximum number of streams 36 | that can be inserted into the tree. By default this limit is 1000, but it is 37 | user-configurable. 38 | 39 | If it is necessary to backport a patch, the patch can be found in 40 | `this GitHub pull request`_. 41 | 42 | Recommendations 43 | --------------- 44 | 45 | We suggest you take the following actions immediately, in order of preference: 46 | 47 | 1. Update priority to 1.2.0 immediately, and consider revising the maximum 48 | number of streams downward to a suitable value for your application. 49 | 2. Backport the patch made available on GitHub. 50 | 3. Manually enforce a limit on the number of priority settings you'll allow at 51 | once. 52 | 53 | Timeline 54 | -------- 55 | 56 | This class of vulnerability was publicly reported in `this report`_ on the 57 | 3rd of August. We requested a CVE ID from Mitre the same day. 58 | 59 | Priority 1.2.0 was released on the 4th of August, at the same time as the 60 | publication of this advisory. 61 | 62 | 63 | .. _this report: http://www.imperva.com/docs/Imperva_HII_HTTP2.pdf 64 | .. _this GitHub pull request: https://github.com/python-hyper/priority/pull/23 65 | -------------------------------------------------------------------------------- /docs/source/security/index.rst: -------------------------------------------------------------------------------- 1 | Vulnerability Notifications 2 | =========================== 3 | 4 | This section of the page contains all known vulnerabilities in the priority 5 | library. These vulnerabilities have all been reported to us via our 6 | `vulnerability disclosure policy`_. 7 | 8 | Known Vulnerabilities 9 | --------------------- 10 | 11 | +----+---------------------------+----------------+---------------+--------------+---------------+ 12 | | \# | Vulnerability | Date Announced | First Version | Last Version | CVE | 13 | +====+===========================+================+===============+==============+===============+ 14 | | 1 | :doc:`DoS via unlimited | 2016-08-04 | 1.0.0 | 1.1.1 | CVE-2016-6580 | 15 | | | stream insertion. | | | | | 16 | | | ` | | | | | 17 | +----+---------------------------+----------------+---------------+--------------+---------------+ 18 | 19 | .. _vulnerability disclosure policy: http://python-hyper.org/en/latest/security.html#vulnerability-disclosure 20 | -------------------------------------------------------------------------------- /docs/source/using-priority.rst: -------------------------------------------------------------------------------- 1 | Using Priority 2 | ============== 3 | 4 | Priority has a simple API. Streams are inserted into the tree: when they are 5 | inserted, they may optionally have a weight, depend on another stream, or 6 | become an exclusive dependent of another stream. To manipulate the tree, we 7 | use a :class:`PriorityTree ` object. 8 | 9 | .. code-block:: python 10 | 11 | >>> p = priority.PriorityTree() 12 | >>> p.insert_stream(stream_id=1) 13 | >>> p.insert_stream(stream_id=3) 14 | >>> p.insert_stream(stream_id=5, depends_on=1) 15 | >>> p.insert_stream(stream_id=7, weight=32) 16 | >>> p.insert_stream(stream_id=9, depends_on=7, weight=8) 17 | >>> p.insert_stream(stream_id=11, depends_on=7, exclusive=True) 18 | 19 | Once streams are inserted, the stream priorities can be requested. This allows 20 | the server to make decisions about how to allocate resources. 21 | 22 | Iterating The Tree 23 | ------------------ 24 | 25 | The tree in this algorithm acts as a gate. Its goal is to allow one stream 26 | "through" at a time, in such a manner that all the active streams are served as 27 | evenly as possible in proportion to their weights. 28 | 29 | This is handled in Priority by iterating over the tree. The tree itself is an 30 | iterator, and each time it is advanced it will yield a stream ID. This is the 31 | ID of the stream that should next send data. 32 | 33 | This looks like this: 34 | 35 | .. code-block:: python 36 | 37 | >>> for stream_id in p: 38 | ... send_data(stream_id) 39 | 40 | If each stream only sends when it is 'ungated' by this mechanism, the server 41 | will automatically be emitting stream data in conformance to RFC 7540. 42 | 43 | Updating The Tree 44 | ----------------- 45 | 46 | If for any reason a stream is unable to proceed (for example, it is blocked on 47 | HTTP/2 flow control, or it is waiting for more data from another service), that 48 | stream is *blocked*. The :class:`PriorityTree ` should 49 | be informed that the stream is blocked so that other dependent streams get a 50 | chance to proceed. This can be done by calling the 51 | :meth:`block ` method of the tree with the stream 52 | ID that is currently unable to proceed. This will automatically update the 53 | tree, and it will adjust on the fly to correctly allow any streams that were 54 | dependent on the blocked one to progress. 55 | 56 | For example: 57 | 58 | .. code-block:: python 59 | 60 | >>> for stream_id in p: 61 | ... send_data(stream_id) 62 | ... if blocked(stream_id): 63 | ... p.block(stream_id) 64 | 65 | When a stream goes from being blocked to being unblocked, call the 66 | :meth:`unblock ` method to place it back into 67 | the sequence. Both the :meth:`block ` and 68 | :meth:`unblock ` methods are idempotent and safe 69 | to call repeatedly. 70 | 71 | Additionally, the priority of a stream may change. When it does, the 72 | :meth:`reprioritize ` method can be used to 73 | update the tree in the wake of that change. 74 | :meth:`reprioritize ` has the same 75 | signature as :meth:`insert_stream `, but 76 | applies only to streams already in the tree. 77 | 78 | Removing Streams 79 | ---------------- 80 | 81 | A stream can be entirely removed from the tree by calling 82 | :meth:`remove_stream `. Note that this is 83 | *not* idempotent. Further, calling 84 | :meth:`remove_stream ` and then re-adding 85 | it *may* cause a substantial change in the shape of the priority tree, and 86 | *will* cause the iteration order to change. 87 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [tool:pytest] 2 | testpaths = test 3 | 4 | [coverage:run] 5 | branch = True 6 | source = priority 7 | 8 | [coverage:report] 9 | fail_under = 100 10 | show_missing = True 11 | exclude_lines = 12 | pragma: no cover 13 | assert False, "Should not be reachable" 14 | .*:.* # Python \d.* 15 | .*:.* # Platform-specific: 16 | 17 | [coverage:paths] 18 | source = 19 | src 20 | .tox/*/site-packages 21 | 22 | [flake8] 23 | max-line-length = 120 24 | max-complexity = 10 25 | 26 | [mypy] 27 | strict = true 28 | warn_unused_configs = true 29 | show_error_codes = true 30 | 31 | [mypy-test_priority] 32 | allow_untyped_defs = true 33 | check_untyped_defs = false 34 | ignore_missing_imports = true 35 | disallow_subclassing_any = false 36 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import os 4 | import re 5 | 6 | from setuptools import setup, find_packages 7 | 8 | PROJECT_ROOT = os.path.dirname(__file__) 9 | 10 | with open(os.path.join(PROJECT_ROOT, 'README.rst')) as file_: 11 | long_description = file_.read() 12 | 13 | version_regex = r'__version__ = ["\']([^"\']*)["\']' 14 | with open(os.path.join(PROJECT_ROOT, 'src/priority/__init__.py')) as file_: 15 | text = file_.read() 16 | match = re.search(version_regex, text) 17 | if match: 18 | version = match.group(1) 19 | else: 20 | raise RuntimeError("No version number found!") 21 | 22 | setup( 23 | name='priority', 24 | version=version, 25 | description='A pure-Python implementation of the HTTP/2 priority tree', 26 | long_description=long_description, 27 | long_description_content_type='text/x-rst', 28 | author='Cory Benfield', 29 | author_email='cory@lukasa.co.uk', 30 | url='https://github.com/python-hyper/priority/', 31 | project_urls={ 32 | 'Documentation': 'https://python-hyper.org/projects/priority/', 33 | 'Source': 'https://github.com/python-hyper/priority/', 34 | 'Tracker': 'https://github.com/python-hyper/priority/issues', 35 | 'Changelog': 'https://github.com/python-hyper/priority/blob/master/HISTORY.rst', 36 | }, 37 | packages=find_packages(where='src'), 38 | package_data={'priority': ['py.typed']}, 39 | package_dir={'': 'src'}, 40 | python_requires='>=3.6.1', 41 | license='MIT License', 42 | classifiers=[ 43 | 'Development Status :: 5 - Production/Stable', 44 | 'Intended Audience :: Developers', 45 | 'License :: OSI Approved :: MIT License', 46 | 'Programming Language :: Python', 47 | 'Programming Language :: Python :: 3', 48 | 'Programming Language :: Python :: 3.6', 49 | 'Programming Language :: Python :: 3.7', 50 | 'Programming Language :: Python :: 3.8', 51 | 'Programming Language :: Python :: 3.9', 52 | 'Programming Language :: Python :: Implementation :: CPython', 53 | 'Programming Language :: Python :: Implementation :: PyPy', 54 | ], 55 | ) 56 | -------------------------------------------------------------------------------- /src/priority/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | priority: HTTP/2 priority implementation for Python 4 | """ 5 | from .priority import ( # noqa 6 | Stream, 7 | PriorityTree, 8 | DeadlockError, 9 | PriorityLoop, 10 | PriorityError, 11 | DuplicateStreamError, 12 | MissingStreamError, 13 | TooManyStreamsError, 14 | BadWeightError, 15 | PseudoStreamError, 16 | ) 17 | 18 | 19 | __version__ = "2.0.0" 20 | -------------------------------------------------------------------------------- /src/priority/priority.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | priority/tree 4 | ~~~~~~~~~~~~~ 5 | 6 | Implementation of the Priority tree data structure. 7 | """ 8 | 9 | import heapq 10 | 11 | from typing import List, Tuple, Optional 12 | 13 | 14 | class PriorityError(Exception): 15 | """ 16 | The base class for all ``priority`` exceptions. 17 | """ 18 | 19 | 20 | class DeadlockError(PriorityError): 21 | """ 22 | Raised when there are no streams that can make progress: all streams are 23 | blocked. 24 | """ 25 | 26 | pass 27 | 28 | 29 | class PriorityLoop(PriorityError): 30 | """ 31 | An unexpected priority loop has been detected. The tree is invalid. 32 | """ 33 | 34 | pass 35 | 36 | 37 | class DuplicateStreamError(PriorityError): 38 | """ 39 | An attempt was made to insert a stream that already exists. 40 | """ 41 | 42 | pass 43 | 44 | 45 | class MissingStreamError(KeyError, PriorityError): 46 | """ 47 | An operation was attempted on a stream that is not present in the tree. 48 | """ 49 | 50 | pass 51 | 52 | 53 | class TooManyStreamsError(PriorityError): 54 | """ 55 | An attempt was made to insert a dangerous number of streams into the 56 | priority tree at the same time. 57 | 58 | .. versionadded:: 1.2.0 59 | """ 60 | 61 | pass 62 | 63 | 64 | class BadWeightError(PriorityError): 65 | """ 66 | An attempt was made to create a stream with an invalid weight. 67 | 68 | .. versionadded:: 1.3.0 69 | """ 70 | 71 | pass 72 | 73 | 74 | class PseudoStreamError(PriorityError): 75 | """ 76 | An operation was attempted on stream 0. 77 | 78 | .. versionadded:: 1.3.0 79 | """ 80 | 81 | pass 82 | 83 | 84 | class Stream: 85 | """ 86 | Priority information for a given stream. 87 | 88 | :param stream_id: The stream ID for the new stream. 89 | :param weight: (optional) The stream weight. Defaults to 16. 90 | """ 91 | 92 | def __init__(self, stream_id: int, weight: int = 16) -> None: 93 | self.stream_id = stream_id 94 | self.weight = weight 95 | self.children: List[Stream] = [] 96 | self.parent: Optional[Stream] = None 97 | self.child_queue: List[Tuple[int, Stream]] = [] 98 | self.active = True 99 | self.last_weight = 0 100 | self._deficit = 0 101 | 102 | @property 103 | def weight(self) -> int: 104 | return self._weight 105 | 106 | @weight.setter 107 | def weight(self, value: int) -> None: 108 | # RFC 7540 § 5.3.2: "All dependent streams are allocated an integer 109 | # weight between 1 and 256 (inclusive)." 110 | if not isinstance(value, int): 111 | raise BadWeightError("Stream weight should be an integer") 112 | elif not (1 <= value <= 256): 113 | raise BadWeightError("Stream weight must be between 1 and 256 (inclusive)") 114 | self._weight = value 115 | 116 | def add_child(self, child: "Stream") -> None: 117 | """ 118 | Add a stream that depends on this one. 119 | 120 | :param child: A ``Stream`` object that depends on this one. 121 | """ 122 | child.parent = self 123 | self.children.append(child) 124 | heapq.heappush(self.child_queue, (self.last_weight, child)) 125 | 126 | def add_child_exclusive(self, child: "Stream") -> None: 127 | """ 128 | Add a stream that exclusively depends on this one. 129 | 130 | :param child: A ``Stream`` object that exclusively depends on this one. 131 | """ 132 | old_children = self.children 133 | self.children = [] 134 | self.child_queue = [] 135 | self.last_weight = 0 136 | self.add_child(child) 137 | 138 | for old_child in old_children: 139 | child.add_child(old_child) 140 | 141 | def remove_child( 142 | self, 143 | child: "Stream", 144 | strip_children: bool = True, 145 | ) -> None: 146 | """ 147 | Removes a child stream from this stream. This is a potentially somewhat 148 | expensive operation. 149 | 150 | :param child: The child stream to remove. 151 | :param strip_children: Whether children of the removed stream should 152 | become children of this stream. 153 | """ 154 | # To do this we do the following: 155 | # 156 | # - remove the child stream from the list of children 157 | # - build a new priority queue, filtering out the child when we find 158 | # it in the old one 159 | self.children.remove(child) 160 | 161 | new_queue: List[Tuple[int, Stream]] = [] 162 | 163 | while self.child_queue: 164 | level, stream = heapq.heappop(self.child_queue) 165 | if stream == child: 166 | continue 167 | 168 | heapq.heappush(new_queue, (level, stream)) 169 | 170 | self.child_queue = new_queue 171 | 172 | if strip_children: 173 | for new_child in child.children: 174 | self.add_child(new_child) 175 | 176 | def schedule(self) -> int: 177 | """ 178 | Returns the stream ID of the next child to schedule. Potentially 179 | recurses down the tree of priorities. 180 | """ 181 | # Cannot be called on active streams. 182 | assert not self.active 183 | 184 | next_stream = None 185 | popped_streams = [] 186 | 187 | # Spin looking for the next active stream. Everything we pop off has 188 | # to be rescheduled, even if it turns out none of them were active at 189 | # this time. 190 | try: 191 | while next_stream is None: 192 | # If the queue is empty, immediately fail. 193 | val = heapq.heappop(self.child_queue) 194 | popped_streams.append(val) 195 | level, child = val 196 | 197 | if child.active: 198 | next_stream = child.stream_id 199 | else: 200 | # Guard against the possibility that the child also has no 201 | # suitable children. 202 | try: 203 | next_stream = child.schedule() 204 | except IndexError: 205 | continue 206 | finally: 207 | for level, child in popped_streams: 208 | self.last_weight = level 209 | level += (256 + child._deficit) // child.weight 210 | child._deficit = (256 + child._deficit) % child.weight 211 | heapq.heappush(self.child_queue, (level, child)) 212 | 213 | return next_stream 214 | 215 | # Custom repr 216 | def __repr__(self) -> str: 217 | return "Stream" % (self.stream_id, self.weight) 218 | 219 | # Custom comparison 220 | def __eq__(self, other: object) -> bool: 221 | if not isinstance(other, Stream): # pragma: no cover 222 | return False 223 | 224 | return self.stream_id == other.stream_id 225 | 226 | def __ne__(self, other: object) -> bool: 227 | return not self.__eq__(other) 228 | 229 | def __lt__(self, other: "Stream") -> bool: 230 | if not isinstance(other, Stream): # pragma: no cover 231 | return NotImplemented 232 | 233 | return self.stream_id < other.stream_id 234 | 235 | def __le__(self, other: "Stream") -> bool: 236 | if not isinstance(other, Stream): # pragma: no cover 237 | return NotImplemented 238 | 239 | return self.stream_id <= other.stream_id 240 | 241 | def __gt__(self, other: "Stream") -> bool: 242 | if not isinstance(other, Stream): # pragma: no cover 243 | return NotImplemented 244 | 245 | return self.stream_id > other.stream_id 246 | 247 | def __ge__(self, other: "Stream") -> bool: 248 | if not isinstance(other, Stream): # pragma: no cover 249 | return NotImplemented 250 | 251 | return self.stream_id >= other.stream_id 252 | 253 | 254 | def _stream_cycle(new_parent: Stream, current: Stream) -> bool: 255 | """ 256 | Reports whether the new parent depends on the current stream. 257 | """ 258 | parent = new_parent 259 | 260 | # Don't iterate forever, but instead assume that the tree doesn't 261 | # get more than 100 streams deep. This should catch accidental 262 | # tree loops. This is the definition of defensive programming. 263 | for _ in range(100): 264 | parent = parent.parent # type: ignore[assignment] 265 | if parent.stream_id == current.stream_id: 266 | return True 267 | elif parent.stream_id == 0: 268 | return False 269 | 270 | raise PriorityLoop( 271 | "Stream %d is in a priority loop." % new_parent.stream_id 272 | ) # pragma: no cover 273 | 274 | 275 | class PriorityTree: 276 | """ 277 | A HTTP/2 Priority Tree. 278 | 279 | This tree stores HTTP/2 streams according to their HTTP/2 priorities. 280 | 281 | .. versionchanged:: 1.2.0 282 | Added ``maximum_streams`` keyword argument. 283 | 284 | :param maximum_streams: The maximum number of streams that may be active in 285 | the priority tree at any one time. If this number is exceeded, the 286 | priority tree will raise a :class:`TooManyStreamsError 287 | ` and will refuse to insert the stream. 288 | 289 | This parameter exists to defend against the possibility of DoS attack 290 | by attempting to overfill the priority tree. If any endpoint is 291 | attempting to manage the priority of this many streams at once it is 292 | probably trying to screw with you, so it is sensible to simply refuse 293 | to play ball at that point. 294 | 295 | While we allow the user to configure this, we don't really *expect* 296 | them too, unless they want to be even more conservative than we are by 297 | default. 298 | :type maximum_streams: ``int`` 299 | """ 300 | 301 | def __init__(self, maximum_streams: int = 1000) -> None: 302 | # This flat array keeps hold of all the streams that are logically 303 | # dependent on stream 0. 304 | self._root_stream = Stream(stream_id=0, weight=1) 305 | self._root_stream.active = False 306 | self._streams = {0: self._root_stream} 307 | 308 | if not isinstance(maximum_streams, int): 309 | raise TypeError("maximum_streams must be an int.") 310 | if maximum_streams <= 0: 311 | raise ValueError("maximum_streams must be a positive integer.") 312 | self._maximum_streams = maximum_streams 313 | 314 | def _get_or_insert_parent(self, parent_stream_id: int) -> Stream: 315 | """ 316 | When inserting or reprioritizing a stream it is possible to make it 317 | dependent on a stream that is no longer in the tree. In this situation, 318 | rather than bail out, we should insert the parent stream into the tree 319 | with default priority and mark it as blocked. 320 | """ 321 | try: 322 | return self._streams[parent_stream_id] 323 | except KeyError: 324 | self.insert_stream(parent_stream_id) 325 | self.block(parent_stream_id) 326 | return self._streams[parent_stream_id] 327 | 328 | def _exclusive_insert( 329 | self, 330 | parent_stream: Stream, 331 | inserted_stream: Stream, 332 | ) -> None: 333 | """ 334 | Insert ``inserted_stream`` beneath ``parent_stream``, obeying the 335 | semantics of exclusive insertion. 336 | """ 337 | parent_stream.add_child_exclusive(inserted_stream) 338 | 339 | def insert_stream( 340 | self, 341 | stream_id: int, 342 | depends_on: Optional[int] = None, 343 | weight: int = 16, 344 | exclusive: bool = False, 345 | ) -> None: 346 | """ 347 | Insert a stream into the tree. 348 | 349 | :param stream_id: The stream ID of the stream being inserted. 350 | :param depends_on: (optional) The ID of the stream that the new stream 351 | depends on, if any. 352 | :param weight: (optional) The weight to give the new stream. Defaults 353 | to 16. 354 | :param exclusive: (optional) Whether this new stream should be an 355 | exclusive dependency of the parent. 356 | """ 357 | if stream_id in self._streams: 358 | raise DuplicateStreamError("Stream %d already in tree" % stream_id) 359 | 360 | if (len(self._streams) + 1) > self._maximum_streams: 361 | raise TooManyStreamsError( 362 | "Refusing to insert %d streams into priority tree at once" 363 | % (self._maximum_streams + 1) 364 | ) 365 | 366 | stream = Stream(stream_id, weight) 367 | 368 | if not depends_on: 369 | depends_on = 0 370 | elif depends_on == stream_id: 371 | raise PriorityLoop("Stream %d must not depend on itself." % stream_id) 372 | 373 | if exclusive: 374 | parent_stream = self._get_or_insert_parent(depends_on) 375 | self._exclusive_insert(parent_stream, stream) 376 | self._streams[stream_id] = stream 377 | return 378 | 379 | parent = self._get_or_insert_parent(depends_on) 380 | parent.add_child(stream) 381 | self._streams[stream_id] = stream 382 | 383 | def reprioritize( 384 | self, 385 | stream_id: int, 386 | depends_on: Optional[int] = None, 387 | weight: int = 16, 388 | exclusive: bool = False, 389 | ) -> None: 390 | """ 391 | Update the priority status of a stream already in the tree. 392 | 393 | :param stream_id: The stream ID of the stream being updated. 394 | :param depends_on: (optional) The ID of the stream that the stream now 395 | depends on. If ``None``, will be moved to depend on stream 0. 396 | :param weight: (optional) The new weight to give the stream. Defaults 397 | to 16. 398 | :param exclusive: (optional) Whether this stream should now be an 399 | exclusive dependency of the new parent. 400 | """ 401 | if stream_id == 0: 402 | raise PseudoStreamError("Cannot reprioritize stream 0") 403 | 404 | try: 405 | current_stream = self._streams[stream_id] 406 | except KeyError: 407 | raise MissingStreamError("Stream %d not in tree" % stream_id) 408 | 409 | # Update things in a specific order to make sure the calculation 410 | # behaves properly. Specifically, we first update the weight. Then, 411 | # we check whether this stream is being made dependent on one of its 412 | # own dependents. Then, we remove this stream from its current parent 413 | # and move it to its new parent, taking its children with it. 414 | if depends_on: 415 | if depends_on == stream_id: 416 | raise PriorityLoop("Stream %d must not depend on itself" % stream_id) 417 | 418 | new_parent = self._get_or_insert_parent(depends_on) 419 | cycle = _stream_cycle(new_parent, current_stream) 420 | else: 421 | new_parent = self._streams[0] 422 | cycle = False 423 | 424 | current_stream.weight = weight 425 | 426 | # Our new parent is currently dependent on us. We should remove it from 427 | # its parent, and make it a child of our current parent, and then 428 | # continue. 429 | if cycle: 430 | new_parent.parent.remove_child( # type: ignore[union-attr] 431 | new_parent, 432 | ) 433 | current_stream.parent.add_child( # type: ignore[union-attr] 434 | new_parent, 435 | ) 436 | 437 | current_stream.parent.remove_child( # type: ignore[union-attr] 438 | current_stream, strip_children=False 439 | ) 440 | 441 | if exclusive: 442 | new_parent.add_child_exclusive(current_stream) 443 | else: 444 | new_parent.add_child(current_stream) 445 | 446 | def remove_stream(self, stream_id: int) -> None: 447 | """ 448 | Removes a stream from the priority tree. 449 | 450 | :param stream_id: The ID of the stream to remove. 451 | """ 452 | if stream_id == 0: 453 | raise PseudoStreamError("Cannot remove stream 0") 454 | 455 | try: 456 | child = self._streams.pop(stream_id) 457 | except KeyError: 458 | raise MissingStreamError("Stream %d not in tree" % stream_id) 459 | 460 | parent = child.parent 461 | parent.remove_child(child) # type: ignore[union-attr] 462 | 463 | def block(self, stream_id: int) -> None: 464 | """ 465 | Marks a given stream as blocked, with no data to send. 466 | 467 | :param stream_id: The ID of the stream to block. 468 | """ 469 | if stream_id == 0: 470 | raise PseudoStreamError("Cannot block stream 0") 471 | 472 | try: 473 | self._streams[stream_id].active = False 474 | except KeyError: 475 | raise MissingStreamError("Stream %d not in tree" % stream_id) 476 | 477 | def unblock(self, stream_id: int) -> None: 478 | """ 479 | Marks a given stream as unblocked, with more data to send. 480 | 481 | :param stream_id: The ID of the stream to unblock. 482 | """ 483 | if stream_id == 0: 484 | raise PseudoStreamError("Cannot unblock stream 0") 485 | 486 | try: 487 | self._streams[stream_id].active = True 488 | except KeyError: 489 | raise MissingStreamError("Stream %d not in tree" % stream_id) 490 | 491 | # The iterator protocol 492 | def __iter__(self) -> "PriorityTree": # pragma: no cover 493 | return self 494 | 495 | def __next__(self) -> int: # pragma: no cover 496 | try: 497 | return self._root_stream.schedule() 498 | except IndexError: 499 | raise DeadlockError("No unblocked streams to schedule.") 500 | 501 | def next(self) -> int: # pragma: no cover 502 | return self.__next__() 503 | -------------------------------------------------------------------------------- /src/priority/py.typed: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/python-hyper/priority/a9d56f011bc39d09607fe0510c8f38f849fecf3b/src/priority/py.typed -------------------------------------------------------------------------------- /test/test_priority.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | test_priority 4 | ~~~~~~~~~~~~~ 5 | 6 | Tests for the Priority trees 7 | """ 8 | 9 | import operator 10 | import collections 11 | import itertools 12 | 13 | import pytest 14 | 15 | from hypothesis import given, settings 16 | from hypothesis.stateful import invariant, RuleBasedStateMachine, rule 17 | from hypothesis.strategies import integers, lists, tuples, sampled_from 18 | 19 | import priority 20 | 21 | from typing import Iterable, List, Dict, Any 22 | 23 | 24 | STREAMS_AND_WEIGHTS = lists( 25 | elements=tuples(integers(min_value=1), integers(min_value=1, max_value=255)), 26 | unique_by=operator.itemgetter(0), 27 | ) 28 | 29 | BLOCKED_AND_ACTIVE = lists( 30 | elements=sampled_from([1, 3, 5, 7, 9, 11]), 31 | unique=True, 32 | ).map(lambda blocked: (blocked, active_readme_streams_from_filter(blocked))) 33 | 34 | UNBLOCKED_AND_ACTIVE = lists( 35 | elements=sampled_from([1, 3, 5, 7, 9, 11]), 36 | unique=True, 37 | ).map( 38 | lambda unblocked: ( 39 | unblocked, 40 | active_readme_streams_from_filter(unblocked, blocked=False), 41 | ) 42 | ) 43 | 44 | 45 | def readme_tree(): 46 | """ 47 | Provide a tree configured as the one in the readme. 48 | """ 49 | p = priority.PriorityTree() 50 | p.insert_stream(stream_id=1) 51 | p.insert_stream(stream_id=3) 52 | p.insert_stream(stream_id=5, depends_on=1) 53 | p.insert_stream(stream_id=7, weight=32) 54 | p.insert_stream(stream_id=9, depends_on=7, weight=8) 55 | p.insert_stream(stream_id=11, depends_on=7, exclusive=True) 56 | return p 57 | 58 | 59 | def active_readme_streams_from_filter( 60 | filtered: Iterable[int], 61 | blocked: bool = True, 62 | ) -> List[int]: 63 | """ 64 | Given a collection of filtered streams, determine which ones are active. 65 | This applies only to the readme tree at this time, though in future it 66 | should be possible to apply this to an arbitrary tree. 67 | 68 | If ``blocked`` is ``True``, the filter is a set of blocked streams. If 69 | ``False``, it's a collection of unblocked streams. 70 | """ 71 | tree = { 72 | 1: { 73 | 5: {}, 74 | }, 75 | 3: {}, 76 | 7: { 77 | 11: { 78 | 9: {}, 79 | }, 80 | }, 81 | } 82 | filtered = set(filtered) 83 | 84 | def get_expected(tree: Dict[Any, Any]) -> List[int]: 85 | expected = [] 86 | 87 | for stream_id in tree: 88 | if stream_id not in filtered and blocked: 89 | expected.append(stream_id) 90 | elif stream_id in filtered and not blocked: 91 | expected.append(stream_id) 92 | else: 93 | expected.extend(get_expected(tree[stream_id])) 94 | 95 | return expected 96 | 97 | return get_expected(tree) 98 | 99 | 100 | class TestStream: 101 | def test_stream_repr(self): 102 | """ 103 | The stream representation renders according to the README. 104 | """ 105 | s = priority.Stream(stream_id=80, weight=16) 106 | assert repr(s) == "Stream" 107 | 108 | @given(STREAMS_AND_WEIGHTS) 109 | def test_streams_are_well_ordered(self, streams_and_weights): 110 | """ 111 | Streams are ordered by their stream ID. 112 | """ 113 | stream_list = [ 114 | priority.Stream(stream_id=s, weight=w) for s, w in streams_and_weights 115 | ] 116 | stream_list = sorted(stream_list) 117 | streams_by_id = [stream.stream_id for stream in stream_list] 118 | assert sorted(streams_by_id) == streams_by_id 119 | 120 | @given( 121 | integers(min_value=1, max_value=2 ** 24), 122 | integers(min_value=1, max_value=2 ** 24), 123 | ) 124 | def test_stream_ordering(self, a, b): 125 | """ 126 | Two streams are well ordered based on their stream ID. 127 | """ 128 | s1 = priority.Stream(stream_id=a, weight=16) 129 | s2 = priority.Stream(stream_id=b, weight=32) 130 | 131 | assert (s1 < s2) == (a < b) 132 | assert (s1 <= s2) == (a <= b) 133 | assert (s1 > s2) == (a > b) 134 | assert (s1 >= s2) == (a >= b) 135 | assert (s1 == s2) == (a == b) 136 | assert (s1 != s2) == (a != b) 137 | 138 | 139 | class TestPriorityTreeManual: 140 | """ 141 | These tests manually confirm that the PriorityTree output is correct. They 142 | use the PriorityTree given in the README and confirm that it outputs data 143 | as expected. 144 | 145 | If possible, I'd like to eventually replace these tests with 146 | Hypothesis-based ones for the same data, but getting Hypothesis to generate 147 | useful data in this case is going to be quite tricky. 148 | """ 149 | 150 | @given(BLOCKED_AND_ACTIVE) 151 | def test_priority_tree_initially_outputs_all_stream_ids(self, blocked_expected): 152 | """ 153 | The first iterations of the priority tree initially output the active 154 | streams, in order of stream ID, regardless of weight. 155 | """ 156 | tree = readme_tree() 157 | blocked = blocked_expected[0] 158 | expected = blocked_expected[1] 159 | 160 | for stream_id in blocked: 161 | tree.block(stream_id) 162 | 163 | result = [next(tree) for _ in range(len(expected))] 164 | assert expected == result 165 | 166 | @given(UNBLOCKED_AND_ACTIVE) 167 | def test_priority_tree_blocking_is_isomorphic(self, allowed_expected): 168 | """ 169 | Blocking everything and then unblocking certain ones has the same 170 | effect as blocking specific streams. 171 | """ 172 | tree = readme_tree() 173 | allowed = allowed_expected[0] 174 | expected = allowed_expected[1] 175 | 176 | for stream_id in range(1, 12, 2): 177 | tree.block(stream_id) 178 | 179 | for stream_id in allowed: 180 | tree.unblock(stream_id) 181 | 182 | result = [next(tree) for _ in range(len(expected))] 183 | assert expected == result 184 | 185 | @given(BLOCKED_AND_ACTIVE) 186 | def test_removing_items_behaves_similarly_to_blocking(self, blocked_expected): 187 | """ 188 | From the perspective of iterating over items, removing streams should 189 | have the same effect as blocking them, except that the ordering 190 | changes. Because the ordering is not important, don't test for it. 191 | """ 192 | tree = readme_tree() 193 | blocked = blocked_expected[0] 194 | expected = set(blocked_expected[1]) 195 | 196 | for stream_id in blocked: 197 | tree.remove_stream(stream_id) 198 | 199 | result = set(next(tree) for _ in range(len(expected))) 200 | assert expected == result 201 | 202 | def test_priority_tree_raises_deadlock_error_if_all_blocked(self): 203 | """ 204 | Assuming all streams are blocked and none can progress, asking for the 205 | one with the next highest priority fires a DeadlockError. 206 | """ 207 | tree = readme_tree() 208 | for stream_id in range(1, 12, 2): 209 | tree.block(stream_id) 210 | 211 | with pytest.raises(priority.DeadlockError): 212 | next(tree) 213 | 214 | @pytest.mark.parametrize( 215 | "stream,new_parent,exclusive,weight,blocked,result", 216 | [ 217 | (1, 3, False, 16, [], [3, 7, 7, 3, 7, 7, 3, 7, 7]), 218 | (1, 5, False, 16, [], [3, 5, 7, 7, 3, 5, 7, 7, 3]), 219 | (1, 5, False, 16, [5], [3, 1, 7, 7, 3, 1, 7, 7, 3]), 220 | (5, 7, False, 16, [7, 1], [3, 5, 11, 3, 5, 11, 3, 5, 11]), 221 | (11, None, False, 16, [], [1, 3, 7, 11, 7, 1, 3, 7, 11]), 222 | (11, None, False, 16, [11], [1, 3, 7, 9, 7, 1, 3, 7, 9]), 223 | (7, 9, False, 16, [], [1, 3, 9, 1, 3, 1, 3, 9, 1]), 224 | (7, 1, True, 16, [], [1, 3, 1, 3, 1, 3, 1, 3, 1]), 225 | (7, 1, True, 16, [1], [7, 3, 7, 3, 7, 3, 7, 3, 7]), 226 | (7, 1, True, 16, [1, 7], [5, 3, 11, 3, 5, 3, 11, 3, 5]), 227 | (1, 0, False, 32, [], [1, 3, 7, 1, 7, 1, 3, 7, 1]), 228 | (1, 0, True, 32, [], [1, 1, 1, 1, 1, 1, 1, 1, 1]), 229 | (1, 0, True, 32, [1], [3, 5, 7, 7, 3, 5, 7, 7, 3]), 230 | (1, None, True, 32, [], [1, 1, 1, 1, 1, 1, 1, 1, 1]), 231 | (1, None, True, 32, [1], [3, 5, 7, 7, 3, 5, 7, 7, 3]), 232 | ], 233 | ) 234 | def test_can_reprioritize_a_stream( 235 | self, stream, new_parent, exclusive, weight, blocked, result 236 | ): 237 | """ 238 | Reprioritizing streams adjusts the outputs of the tree. 239 | """ 240 | t = readme_tree() 241 | 242 | for s in blocked: 243 | t.block(s) 244 | 245 | t.reprioritize( 246 | stream_id=stream, 247 | depends_on=new_parent, 248 | weight=weight, 249 | exclusive=exclusive, 250 | ) 251 | 252 | actual_result = [next(t) for _ in range(len(result))] 253 | assert actual_result == result 254 | 255 | def test_priority_tree_raises_error_inserting_duplicate(self): 256 | """ 257 | Attempting to insert a stream that is already in the tree raises a 258 | DuplicateStreamError 259 | """ 260 | p = priority.PriorityTree() 261 | p.insert_stream(1) 262 | 263 | with pytest.raises(priority.DuplicateStreamError): 264 | p.insert_stream(1) 265 | 266 | def test_priority_raises_good_errors_for_missing_streams(self): 267 | """ 268 | Attempting operations on absent streams raises a MissingStreamError. 269 | """ 270 | p = priority.PriorityTree() 271 | p.insert_stream(1) 272 | 273 | with pytest.raises(priority.MissingStreamError): 274 | p.reprioritize(3) 275 | 276 | with pytest.raises(priority.MissingStreamError): 277 | p.block(3) 278 | 279 | with pytest.raises(priority.MissingStreamError): 280 | p.unblock(3) 281 | 282 | with pytest.raises(priority.MissingStreamError): 283 | p.remove_stream(3) 284 | 285 | def test_priority_raises_good_errors_for_zero_stream(self): 286 | """ 287 | Attempting operations on stream 0 raises a PseudoStreamError. 288 | """ 289 | p = priority.PriorityTree() 290 | p.insert_stream(1) 291 | 292 | with pytest.raises(priority.PseudoStreamError): 293 | p.reprioritize(0) 294 | 295 | with pytest.raises(priority.PseudoStreamError): 296 | p.block(0) 297 | 298 | with pytest.raises(priority.PseudoStreamError): 299 | p.unblock(0) 300 | 301 | with pytest.raises(priority.PseudoStreamError): 302 | p.remove_stream(0) 303 | 304 | @pytest.mark.parametrize("exclusive", [True, False]) 305 | def test_priority_allows_inserting_stream_with_absent_parent(self, exclusive): 306 | """ 307 | Attemping to insert a stream that depends on a stream that is not in 308 | the tree automatically inserts the parent with default priority. 309 | """ 310 | p = priority.PriorityTree() 311 | p.insert_stream(stream_id=3, depends_on=1, exclusive=exclusive, weight=32) 312 | 313 | # Iterate 10 times to prove that the parent stream starts blocked. 314 | first_ten_ids = [next(p) for _ in range(0, 10)] 315 | assert first_ten_ids == [3] * 10 316 | 317 | # Unblock the parent. 318 | p.unblock(1) 319 | 320 | # Iterate 10 times, expecting only the parent. 321 | next_ten_ids = [next(p) for _ in range(0, 10)] 322 | assert next_ten_ids == [1] * 10 323 | 324 | # Insert a new stream into the tree with default priority. 325 | p.insert_stream(stream_id=5) 326 | 327 | # Iterate 10 more times. Expect the parent, and the new stream, in 328 | # equal amounts. 329 | next_ten_ids = [next(p) for _ in range(0, 10)] 330 | assert next_ten_ids == [5, 1] * 5 331 | 332 | @pytest.mark.parametrize("exclusive", [True, False]) 333 | def test_priority_reprioritizing_stream_with_absent_parent(self, exclusive): 334 | """ 335 | Attemping to reprioritize a stream to depend on a stream that is not in 336 | the tree automatically inserts the parent with default priority. 337 | """ 338 | p = priority.PriorityTree() 339 | p.insert_stream(stream_id=3) 340 | 341 | p.reprioritize(stream_id=3, depends_on=1, exclusive=exclusive, weight=32) 342 | 343 | # Iterate 10 times to prove that the parent stream starts blocked. 344 | first_ten_ids = [next(p) for _ in range(0, 10)] 345 | assert first_ten_ids == [3] * 10 346 | 347 | # Unblock the parent. 348 | p.unblock(1) 349 | 350 | # Iterate 10 times, expecting only the parent. 351 | next_ten_ids = [next(p) for _ in range(0, 10)] 352 | assert next_ten_ids == [1] * 10 353 | 354 | # Insert a new stream into the tree with default priority. 355 | p.insert_stream(stream_id=5) 356 | 357 | # Iterate 10 more times. Expect the parent, and the new stream, in 358 | # equal amounts. 359 | next_ten_ids = [next(p) for _ in range(0, 10)] 360 | assert next_ten_ids == [5, 1] * 5 361 | 362 | @pytest.mark.parametrize("count", range(2, 10000, 100)) 363 | def test_priority_refuses_to_allow_too_many_streams_in_tree(self, count): 364 | """ 365 | Attempting to insert more streams than maximum_streams into the tree 366 | fails. 367 | """ 368 | p = priority.PriorityTree(maximum_streams=count) 369 | 370 | # This isn't an off-by-one error: stream 0 is in the tree by default. 371 | for x in range(1, count): 372 | p.insert_stream(x) 373 | 374 | with pytest.raises(priority.TooManyStreamsError): 375 | p.insert_stream(x + 1) 376 | 377 | @pytest.mark.parametrize("depends_on", [0, None]) 378 | def test_can_insert_stream_with_exclusive_dependency_on_0(self, depends_on): 379 | """ 380 | It is acceptable to insert a stream with an exclusive dependency on 381 | stream 0, both explicitly and implicitly. 382 | """ 383 | p = priority.PriorityTree() 384 | p.insert_stream(stream_id=1) 385 | p.insert_stream(stream_id=3) 386 | 387 | p.insert_stream(stream_id=5, depends_on=depends_on, exclusive=True) 388 | 389 | next_ten_ids = [next(p) for _ in range(0, 10)] 390 | assert next_ten_ids == [5] * 10 391 | 392 | @pytest.mark.parametrize("weight", [None, 0.5, float("inf"), "priority", object]) 393 | def test_stream_with_non_integer_weight_is_error(self, weight): 394 | """ 395 | Giving a stream a non-integer weight is rejected. 396 | """ 397 | p = priority.PriorityTree() 398 | with pytest.raises(priority.BadWeightError) as err: 399 | p.insert_stream(stream_id=1, weight=weight) 400 | assert err.value.args[0] == "Stream weight should be an integer" 401 | 402 | p.insert_stream(stream_id=2) 403 | with pytest.raises(priority.BadWeightError) as err: 404 | p.reprioritize(stream_id=2, weight=weight) 405 | assert err.value.args[0] == "Stream weight should be an integer" 406 | 407 | @pytest.mark.parametrize( 408 | "weight", 409 | [ 410 | 0, 411 | 257, 412 | 1000, 413 | -42, 414 | ], 415 | ) 416 | def test_stream_with_out_of_bounds_weight_is_error(self, weight): 417 | """ 418 | Giving a stream an out-of-bounds integer weight is rejected. 419 | """ 420 | p = priority.PriorityTree() 421 | with pytest.raises(priority.BadWeightError) as err: 422 | p.insert_stream(stream_id=1, weight=weight) 423 | assert ( 424 | err.value.args[0] == "Stream weight must be between 1 and 256 (inclusive)" 425 | ) 426 | 427 | p.insert_stream(stream_id=2) 428 | with pytest.raises(priority.BadWeightError) as err: 429 | p.reprioritize(stream_id=2, weight=weight) 430 | assert ( 431 | err.value.args[0] == "Stream weight must be between 1 and 256 (inclusive)" 432 | ) 433 | 434 | @pytest.mark.parametrize("exclusive", (True, False)) 435 | @pytest.mark.parametrize("stream_id", (1, 5, 20, 32, 256)) 436 | def test_stream_depending_on_self_is_error(self, stream_id, exclusive): 437 | """ 438 | Inserting a stream that is dependent on itself is rejected. 439 | """ 440 | p = priority.PriorityTree() 441 | with pytest.raises(priority.PriorityLoop): 442 | p.insert_stream( 443 | stream_id=stream_id, depends_on=stream_id, exclusive=exclusive 444 | ) 445 | 446 | @pytest.mark.parametrize("exclusive", (True, False)) 447 | @pytest.mark.parametrize("stream_id", (1, 5, 20, 32, 256)) 448 | def test_reprioritize_depend_on_self_is_error(self, stream_id, exclusive): 449 | """ 450 | Reprioritizing a stream to make it dependent on itself is an error. 451 | """ 452 | p = priority.PriorityTree() 453 | p.insert_stream(stream_id=stream_id) 454 | with pytest.raises(priority.PriorityLoop): 455 | p.reprioritize( 456 | stream_id=stream_id, depends_on=stream_id, exclusive=exclusive 457 | ) 458 | 459 | @pytest.mark.parametrize("maximum_streams", (None, "foo", object(), 2.0)) 460 | def test_maximum_streams_with_non_int_is_error(self, maximum_streams): 461 | """ 462 | Creating a PriorityTree with a non-int argument for maximum_streams 463 | is an error. 464 | """ 465 | with pytest.raises(TypeError) as err: 466 | priority.PriorityTree(maximum_streams=maximum_streams) 467 | assert err.value.args[0] == "maximum_streams must be an int." 468 | 469 | @pytest.mark.parametrize("maximum_streams", (0, -1, -50)) 470 | def test_maximum_streams_with_bad_int_is_error(self, maximum_streams): 471 | """ 472 | Creating a PriorityTree with a non-positive integer for maximum_streams 473 | is an error. 474 | """ 475 | with pytest.raises(ValueError) as err: 476 | priority.PriorityTree(maximum_streams=maximum_streams) 477 | assert err.value.args[0] == "maximum_streams must be a positive integer." 478 | 479 | 480 | class TestPriorityTreeOutput: 481 | """ 482 | These tests use Hypothesis to attempt to bound the output of iterating over 483 | the priority tree. In particular, their goal is to ensure that the output 484 | of the tree is "good enough": that it meets certain requirements on 485 | fairness and equidistribution. 486 | """ 487 | 488 | @given(STREAMS_AND_WEIGHTS) 489 | @settings(deadline=None) 490 | def test_period_of_repetition(self, streams_and_weights): 491 | """ 492 | The period of repetition of a priority sequence is given by the sum of 493 | the weights of the streams. Once that many values have been pulled out 494 | the sequence repeats identically. 495 | """ 496 | p = priority.PriorityTree() 497 | weights = [] 498 | 499 | for stream, weight in streams_and_weights: 500 | p.insert_stream(stream_id=stream, weight=weight) 501 | weights.append(weight) 502 | 503 | period = sum(weights) 504 | 505 | # Pop off the first n elements, which will always be evenly 506 | # distributed. 507 | for _ in weights: 508 | next(p) 509 | 510 | pattern = [next(p) for _ in range(period)] 511 | pattern = itertools.cycle(pattern) 512 | 513 | for i in range(period * 20): 514 | assert next(p) == next(pattern), i 515 | 516 | @given(STREAMS_AND_WEIGHTS) 517 | def test_priority_tree_distribution(self, streams_and_weights): 518 | """ 519 | Once a full period of repetition has been observed, each stream has 520 | been emitted a number of times equal to its weight. 521 | """ 522 | p = priority.PriorityTree() 523 | weights = [] 524 | 525 | for stream, weight in streams_and_weights: 526 | p.insert_stream(stream_id=stream, weight=weight) 527 | weights.append(weight) 528 | 529 | period = sum(weights) 530 | 531 | # Pop off the first n elements, which will always be evenly 532 | # distributed. 533 | for _ in weights: 534 | next(p) 535 | 536 | count = collections.Counter(next(p) for _ in range(period)) 537 | 538 | assert len(count) == len(streams_and_weights) 539 | for stream, weight in streams_and_weights: 540 | count[stream] == weight 541 | 542 | 543 | class PriorityStateMachine(RuleBasedStateMachine): 544 | """ 545 | This test uses Hypothesis's stateful testing to exercise the priority tree. 546 | 547 | It randomly inserts, removes, blocks and unblocks streams in the tree, 548 | then checks that iterating over priority still selects a sensible stream. 549 | """ 550 | 551 | def __init__(self): 552 | super(PriorityStateMachine, self).__init__() 553 | self.tree = priority.PriorityTree() 554 | self.stream_ids = set([0]) 555 | self.blocked_stream_ids = set() 556 | 557 | @rule(stream_id=integers()) # type: ignore[no-untyped-call] 558 | def insert_stream(self, stream_id): 559 | try: 560 | self.tree.insert_stream(stream_id) 561 | except priority.DuplicateStreamError: 562 | assert stream_id in self.stream_ids 563 | else: 564 | assert stream_id not in self.stream_ids 565 | self.stream_ids.add(stream_id) 566 | 567 | def _run_action(self, action, stream_id): 568 | try: 569 | action(stream_id) 570 | except priority.MissingStreamError: 571 | assert stream_id not in self.stream_ids 572 | except priority.PseudoStreamError: 573 | assert stream_id == 0 574 | else: 575 | assert stream_id in self.stream_ids 576 | 577 | @rule(stream_id=integers()) # type: ignore[no-untyped-call] 578 | def remove_stream(self, stream_id): 579 | self._run_action(self.tree.remove_stream, stream_id) 580 | if stream_id != 0: 581 | self.stream_ids.discard(stream_id) 582 | 583 | @rule(stream_id=integers()) # type: ignore[no-untyped-call] 584 | def block_stream(self, stream_id): 585 | self._run_action(self.tree.block, stream_id) 586 | if (stream_id != 0) and (stream_id in self.stream_ids): 587 | self.blocked_stream_ids.add(stream_id) 588 | 589 | @rule(stream_id=integers()) # type: ignore[no-untyped-call] 590 | def unblock_stream(self, stream_id): 591 | self._run_action(self.tree.unblock, stream_id) 592 | self.blocked_stream_ids.discard(stream_id) 593 | 594 | @invariant() # type: ignore[no-untyped-call] 595 | def check_next_stream_consistent(self): 596 | """ 597 | If we ask priority for the next stream, it always returns a sensible 598 | result. 599 | """ 600 | try: 601 | next_stream_id = next(self.tree) 602 | except priority.DeadlockError: 603 | assert self.blocked_stream_ids ^ {0} == self.stream_ids 604 | else: 605 | stream = self.tree._streams[next_stream_id] 606 | 607 | # If a stream is selected, then it isn't blocked 608 | assert stream.active 609 | 610 | # If a stream is selected, then its parent is either the root 611 | # stream or blocked 612 | parent = stream.parent 613 | assert (parent.stream_id == 0) or (not parent.active) 614 | 615 | 616 | TestPriorityTreeStateful = PriorityStateMachine.TestCase 617 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py36, py37, py38, py39, pypy3, lint, docs, packaging 3 | 4 | [gh-actions] 5 | python = 6 | 3.6: py36 7 | 3.7: py37 8 | 3.8: py38 9 | 3.9: py39, lint, docs, packaging 10 | pypy3: pypy3 11 | 12 | [testenv] 13 | passenv = 14 | GITHUB_* 15 | deps = 16 | pytest>=6.2,<7 17 | pytest-cov>=2.10,<3 18 | pytest-xdist>=2.1,<3 19 | hypothesis>=6.9,<7 20 | commands = 21 | pytest --cov-report=xml --cov-report=term --cov=priority {posargs} 22 | 23 | [testenv:pypy3] 24 | # temporarily disable coverage testing on PyPy due to performance problems 25 | commands = pytest {posargs} 26 | 27 | [testenv:lint] 28 | deps = 29 | flake8>=3.8,<4 30 | black==21.6b0 31 | mypy==0.910 32 | {[testenv]deps} 33 | commands = 34 | flake8 src/ test/ 35 | black --check --diff src/ test/ 36 | mypy src/ test/ 37 | 38 | [testenv:docs] 39 | deps = 40 | sphinx>=3.5,<5 41 | whitelist_externals = make 42 | changedir = {toxinidir}/docs 43 | commands = 44 | make clean 45 | make html 46 | 47 | [testenv:packaging] 48 | basepython = python3.9 49 | deps = 50 | check-manifest==0.46 51 | readme-renderer==29.0 52 | twine>=3.4.1,<4 53 | whitelist_externals = rm 54 | commands = 55 | rm -rf dist/ 56 | check-manifest 57 | python setup.py sdist bdist_wheel 58 | twine check dist/* 59 | 60 | [testenv:publish] 61 | basepython = {[testenv:packaging]basepython} 62 | deps = 63 | {[testenv:packaging]deps} 64 | whitelist_externals = {[testenv:packaging]whitelist_externals} 65 | commands = 66 | {[testenv:packaging]commands} 67 | twine upload dist/* 68 | --------------------------------------------------------------------------------