├── LICENSE └── README.md /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2018, Eric Snow 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | * Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | * Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # multi-core-python 2 | Enabling CPython multi-core parallelism via subinterpreters. 3 | 4 | This repo is for tracking the effort and as a place to keep any tooling. 5 | 6 | [wiki](https://github.com/ericsnowcurrently/multi-core-python/wiki) | [projects](https://github.com/ericsnowcurrently/multi-core-python/projects) 7 | 8 | ## Project Summary 9 | 10 | From my [python-ideas post](https://mail.python.org/pipermail/python-ideas/2015-June/034177.html) (June 2015): 11 | 12 | Python's multi-core story is murky at best. Not only can we be more 13 | clear on the matter, we can improve Python's support. The result of any 14 | effort must make multi-core (and concurrency) support in Python obvious, 15 | unmistakable, and undeniable (and keep it Pythonic). 16 | 17 | The goal of this project is to do just that, by using the existing subinterpreter C-API. 18 | 19 | ### Solution 20 | 21 | The minimal multi-core solution will involve: 22 | 23 | * resolving existing bugs and blockers 24 | * exposing the existing support (from C-API) in a stdlib module 25 | * adding a mechanism to safely "share" basic immutable objects between interpreters 26 | * no longer sharing the GIL between interpreters 27 | 28 | Once a minimal solution is in place we can expand from there. 29 | 30 | At a high level, we're doing the following concrete tasks: 31 | 32 | * [PEP 554 (Multiple Interpreters in the Stdlib)](https://www.python.org/dev/peps/pep-0554/) 33 | * improve interpreter isolation (especially by moving fields from `PyRuntimeState` to `PyInterpreterState`) 34 | 35 | ### Constraints / Requirements 36 | 37 | 1. no significant impact on single-threaded performance 38 | 1. maintain backward compatibility (C-API, etc.) 39 | 1. (pseudo-)compatibility with multiprocessing/threading/concurrent.futures APIs 40 | 1. a multi-core concurrency model/approach that fits our brains 41 | 1. Python APIs 42 | 1. supportable on other Python implementations 43 | 44 | ### Status 45 | 46 | * slow progress 47 | * high level of community excitement 48 | * increasing number of collaborators 49 | * [phase 1](https://github.com/ericsnowcurrently/multi-core-python/projects/4) ("minimal solution") completion getting closer 50 | * still hopeful for Python 3.9 51 | * PEP 554 low-level implementation mostly complete 52 | * currently working on moving globals to runtime/interpreter state (or removing them) 53 | * will take a while (started with ~1500 global variables) 54 | * next up is making memory allocators per-interpreter (and then the GIL itself) 55 | 56 | ### Key Collaborators 57 | 58 | This project wouldn't be what it is without the help of these great people: 59 | 60 | * @ncoghlan - inspired the solution; PEP 432 61 | * @encukou - better subinterpreter support in extension modules 62 | * @vstinner - all sorts of runtime stuff 63 | * @emilyemorehouse - reviews & sanity checks 64 | * @nanjekyejoannah - C-API; PEP 554 high-level module 65 | * @eduardo-elizondo (Facebook/Instagram) - extension modules; globals 66 | * @vsajip - globals 67 | * many others 68 | 69 | ## Contributing 70 | 71 | There are many ways to contribute to this project. Aside from the main technical work in the CPython runtime, there are also a number of jobs that need to be done that are less expert-related and even (somewhat) non-technical. All of it is important and anyone interested in helping is welcome! 72 | 73 | Keep in mind that this project/repo is actually just a tool to organize the effort. The actual work is done on [bugs.python.org](https://bugs.python.org/), [github.com/python/cpython](https://github.com/python/cpython/), and the [python-dev](https://mail.python.org/archives/list/python-dev@python.org/) mailing list. 74 | 75 | Also note that contacting [@ericsnowcurrently](https://github.com/ericsnowcurrently) directly is fine, but you might get a faster response through those other channels or through the issue tracker here. :) 76 | 77 | For more information see the [wiki page](https://github.com/ericsnowcurrently/multi-core-python/wiki/9-%22How-Can-I-Help%3F%22). 78 | --------------------------------------------------------------------------------