2 | .-. 3 | |_:_| 4 | /(_Y_)\ 5 | ( \/M\/ ) 6 | '. _.'-/'-'\-'._ 7 | ': _/.--'[[[[]'--.\_ 8 | ': /_' : |::"| : '.\ 9 | ': // ./ |oUU| \.' :\ 10 | ': _:'..' \_|___|_/ : :| 11 | ':. .' |_[___]_| :.':\ 12 | [::\ | : | | : ; : \ 13 | '-' \/'.| |.' \ .;.' | 14 | |\_ \ '-' : | 15 | | \ \ .: : | | 16 | | \ | '. : \ | 17 | / \ :. .; | 18 | / | | :__/ : \\ 19 | | | | \: | \ | || 20 | / \ : : |: / |__| /| 21 | | : : :_/_| /'._\ '--|_\ 22 | /___.-/_|-' \ \ 23 | '-' 24 |-------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Sybren Jansen 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /.github/workflows/github-pages.yml: -------------------------------------------------------------------------------- 1 | name: Docs 2 | 3 | on: push 4 | 5 | jobs: 6 | build-n-publish: 7 | name: Build and publish documentation to Github 8 | runs-on: ubuntu-20.04 9 | 10 | steps: 11 | - uses: actions/checkout@v3 12 | - name: Set up Python 13 | uses: actions/setup-python@v4 14 | with: 15 | python-version: "3.6" 16 | - name: Install dependencies 17 | run: | 18 | python -m pip install --upgrade pip 19 | pip install setuptools wheel twine rich 20 | pip install .[dashboard] 21 | pip install .[dill] 22 | pip install .[docs] 23 | - name: Build documentation 24 | run: | 25 | sphinx-versioning build -r master ./docs/ ./docs/_build/html/ 26 | - name: Publish documentation to Github 27 | if: startsWith(github.ref, 'refs/tags') 28 | uses: peaceiris/actions-gh-pages@v3.8.0 29 | with: 30 | deploy_key: ${{ secrets.DEPLOY_GITHUB_PAGES_KEY }} 31 | external_repository: sybrenjansen/sybrenjansen.github.io 32 | publish_branch: main 33 | publish_dir: ./docs/_build/html/ 34 | destination_dir: mpire 35 | -------------------------------------------------------------------------------- /docs/usage/workerpool/dill.rst: -------------------------------------------------------------------------------- 1 | .. _use_dill: 2 | 3 | Dill 4 | ==== 5 | 6 | .. contents:: Contents 7 | :depth: 2 8 | :local: 9 | 10 | For some functions or tasks it can be useful to not rely on pickle, but on some more powerful serialization backends 11 | like dill_. ``dill`` isn't installed by default. See :ref:`dilldep` for more information on installing the dependencies. 12 | 13 | One specific example where ``dill`` shines is when using start method ``spawn`` (the default on Windows) in combination 14 | with iPython or Jupyter notebooks. ``dill`` enables parallelizing more exotic objects like lambdas and functions defined 15 | in iPython and Jupyter notebooks. For all benefits of ``dill``, please refer to the `dill documentation`_. 16 | 17 | Once the dependencies have been installed, you can enable it using the ``use_dill`` flag: 18 | 19 | .. code-block:: python 20 | 21 | with WorkerPool(n_jobs=4, use_dill=True) as pool: 22 | ... 23 | 24 | .. note:: 25 | 26 | When using ``dill`` it can potentially slow down processing. This is the cost of having a more reliable and 27 | powerful serialization backend. 28 | 29 | .. _dill: https://pypi.org/project/dill/ 30 | .. _dill documentation: https://github.com/uqfoundation/dill 31 | -------------------------------------------------------------------------------- /mpire/dashboard/connection_classes.py: -------------------------------------------------------------------------------- 1 | from dataclasses import dataclass 2 | from multiprocessing import Event 3 | from multiprocessing.managers import BaseManager 4 | from multiprocessing.synchronize import Event as EventType 5 | from typing import Optional 6 | 7 | 8 | class DashboardStartedEvent: 9 | 10 | def __init__(self) -> None: 11 | self.event: Optional[EventType] = None 12 | 13 | def init(self) -> None: 14 | self.event = Event() 15 | 16 | def reset(self) -> None: 17 | self.event = None 18 | 19 | def set(self) -> None: 20 | if self.event is None: 21 | self.init() 22 | self.event.set() 23 | 24 | def is_set(self) -> bool: 25 | return self.event.is_set() if self.event is not None else False 26 | 27 | def wait(self, timeout: Optional[float] = None) -> bool: 28 | return self.event.wait(timeout) if self.event is not None else False 29 | 30 | 31 | class DashboardManager(BaseManager): 32 | pass 33 | 34 | 35 | @dataclass 36 | class DashboardManagerConnectionDetails: 37 | host: Optional[str] = None 38 | port: Optional[int] = None 39 | 40 | def clear(self) -> None: 41 | self.host = None 42 | self.port = None 43 | -------------------------------------------------------------------------------- /docs/usage/workerpool/cpu_pinning.rst: -------------------------------------------------------------------------------- 1 | CPU pinning 2 | =========== 3 | 4 | You can pin the child processes of :obj:`mpire.WorkerPool` to specific CPUs by using the ``cpu_ids`` parameter in the 5 | constructor: 6 | 7 | .. code-block:: python 8 | 9 | # Pin the two child processes to CPUs 2 and 3 10 | with WorkerPool(n_jobs=2, cpu_ids=[2, 3]) as pool: 11 | ... 12 | 13 | # Pin the child processes to CPUs 40-59 14 | with WorkerPool(n_jobs=20, cpu_ids=list(range(40, 60))) as pool: 15 | ... 16 | 17 | # All child processes have to share a single core: 18 | with WorkerPool(n_jobs=4, cpu_ids=[0]) as pool: 19 | ... 20 | 21 | # All child processes have to share multiple cores, namely 4-7: 22 | with WorkerPool(n_jobs=4, cpu_ids=[[4, 5, 6, 7]]) as pool: 23 | ... 24 | 25 | # Each child process can use two distinctive cores: 26 | with WorkerPool(n_jobs=4, cpu_ids=[[0, 1], [2, 3], [4, 5], [6, 7]]) as pool: 27 | ... 28 | 29 | CPU IDs have to be positive integers, not exceeding the number of CPUs available (which can be retrieved by using 30 | :meth:`mpire.cpu_count`). Use ``None`` to disable CPU pinning (which is the default). 31 | 32 | .. note:: 33 | 34 | Pinning processes to CPU IDs doesn't work when using threading or when you're on macOS. -------------------------------------------------------------------------------- /docs/contributing.rst: -------------------------------------------------------------------------------- 1 | Contribution guidelines 2 | ======================= 3 | 4 | If you want to contribute to MPIRE, great! Please follow the steps below to ensure a smooth process: 5 | 6 | 1. Clone the project. 7 | 2. Create a new branch for your feature or bug fix. Give you branch a meaningful name. 8 | 3. Make your feature addition or bug fix. 9 | 4. Add tests for it and test it yourself. Make sure it both works for Unix and Windows based systems, or make sure to 10 | document why it doesn't work for one of the platforms. 11 | 5. Add documentation for it. Don't forget about the changelog: 12 | 13 | - Reference the issue number from GitHub in the changelog, if applicable (see current changelog for examples). 14 | - Don't mention a date or a version number here, but use ``Unreleased`` instead. 15 | 16 | 6. Commit with a meaningful commit message (e.g. the changelog). 17 | 7. Open a pull request. 18 | 8. Resolve any issues or comments by the reviewer. 19 | 9. Merge PR by squashing all your individual commits. 20 | 21 | Making a release 22 | ---------------- 23 | 24 | A release is only made by the project maintainer. The following steps are required: 25 | 26 | 1. Update the changelog with the release date and version number. Version numbers follow the `Semantic Versioning`_ 27 | guidelines 28 | 2. Update the version number in ``setup.py`` and ``docs/conf.py``. 29 | 3. Commit and push the changes. 30 | 4. Make sure the tests pass on GitHub Actions. 31 | 5. Create a tag for the release by using ``git tag -a vX.Y.Z -m "vX.Y.Z"``. 32 | 6. Push the tag to GitHub by using ``git push origin vX.Y.Z``. 33 | 34 | .. _Semantic Versioning: https://semver.org/ 35 | -------------------------------------------------------------------------------- /mpire/signal.py: -------------------------------------------------------------------------------- 1 | from inspect import Traceback 2 | from signal import getsignal, SIG_IGN, SIGINT, signal as signal_, Signals 3 | from threading import current_thread, main_thread 4 | from types import FrameType 5 | from typing import Type 6 | 7 | 8 | class DelayedKeyboardInterrupt: 9 | 10 | def __init__(self) -> None: 11 | self.signal_received = None 12 | 13 | def __enter__(self) -> None: 14 | # When we're in a thread we can't use signal handling 15 | if current_thread() == main_thread(): 16 | self.signal_received = False 17 | self.old_handler = signal_(SIGINT, self.handler) 18 | 19 | def handler(self, sig: Signals, frame: FrameType) -> None: 20 | self.signal_received = (sig, frame) 21 | 22 | def __exit__(self, exc_type: Type, exc_val: Exception, exc_tb: Traceback) -> None: 23 | if current_thread() == main_thread(): 24 | signal_(SIGINT, self.old_handler) 25 | if self.signal_received: 26 | self.old_handler(*self.signal_received) 27 | 28 | 29 | class DisableKeyboardInterruptSignal: 30 | 31 | def __enter__(self) -> None: 32 | if current_thread() == main_thread(): 33 | # Prevent signal from propagating to child process 34 | self._handler = getsignal(SIGINT) 35 | ignore_keyboard_interrupt() 36 | 37 | def __exit__(self, exc_type: Type, exc_val: Exception, exc_tb: Traceback) -> None: 38 | if current_thread() == main_thread(): 39 | # Restore signal 40 | signal_(SIGINT, self._handler) 41 | 42 | 43 | def ignore_keyboard_interrupt(): 44 | signal_(SIGINT, SIG_IGN) 45 | -------------------------------------------------------------------------------- /bin/mpire-dashboard: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import argparse 3 | import signal 4 | from typing import Sequence 5 | 6 | from mpire.dashboard import start_dashboard 7 | 8 | 9 | def get_port_range() -> Sequence: 10 | """ 11 | :return: port range 12 | """ 13 | def _port_range(range_str) -> Sequence: 14 | n1, n2 = map(int, range_str.split('-')) 15 | if len(range(n1, n2)) < 2: 16 | raise ValueError 17 | return range(n1, n2) 18 | 19 | parser = argparse.ArgumentParser(description='MPIRE Dashboard') 20 | parser.add_argument('--port-range', dest='port_range', required=False, default=range(8080, 8100), type=_port_range, 21 | help='Port range for starting a dashboard. The range should accommodate at least two ports: ' 22 | 'one for the webserver and one for the Python Manager server. Example: 6060-6080 will be ' 23 | 'converted to `range(6060, 6080)`. Default: `range(8080, 8100)`.') 24 | return parser.parse_args().port_range 25 | 26 | 27 | if __name__ == '__main__': 28 | # Obtain port range 29 | port_range = get_port_range() 30 | 31 | # Start a dashboard 32 | print("Starting MPIRE dashboard...") 33 | dashboard_details = start_dashboard(port_range) 34 | 35 | # Print some details on how to connect 36 | print() 37 | print("MPIRE dashboard started on http://localhost:{}".format(dashboard_details['dashboard_port_nr'])) 38 | print("Server is listening on {}:{}".format(dashboard_details['manager_host'], 39 | dashboard_details['manager_port_nr'])) 40 | print("-" * 50) 41 | signal.pause() 42 | -------------------------------------------------------------------------------- /docs/usage/workerpool/order_tasks.rst: -------------------------------------------------------------------------------- 1 | Order tasks 2 | =========== 3 | 4 | .. contents:: Contents 5 | :depth: 2 6 | :local: 7 | 8 | In some settings it can be useful to supply the tasks to workers in a round-robin fashion. This means worker 0 will get 9 | task 0, worker 1 will get task 1, etc. After each worker got a task, we start with worker 0 again instead of picking the 10 | worker that has most recently completed a task. 11 | 12 | When the chunk size is larger than 1, the tasks are distributed to the workers in order, but in chunks. I.e., when 13 | ``chunk_size=3`` tasks 0, 1, and 2 will be assigned to worker 0, tasks 3, 4, and 5 to worker 1, and so on. 14 | 15 | When ``keep_alive`` is set to ``True`` and the second ``map`` call is made, MPIRE resets the worker order and starts at 16 | worker 0 again. 17 | 18 | .. warning:: 19 | 20 | When tasks vary in execution time, the default task scheduler makes sure each worker is busy for approximately the 21 | same amount of time. This can mean that some workers execute more tasks than others. When using ``order_tasks`` this 22 | is no longer the case and therefore the total execution time is likely to be higher. 23 | 24 | You can enable/disable task ordering by setting the ``order_tasks`` flag: 25 | 26 | .. code-block:: python 27 | 28 | def task(x): 29 | pass 30 | 31 | with WorkerPool(n_jobs=4, order_tasks=True) as pool: 32 | pool.map(task, range(10)) 33 | 34 | Instead of passing the flag to the :obj:`mpire.WorkerPool` constructor you can also make use of 35 | :meth:`mpire.WorkerPool.set_order_tasks`: 36 | 37 | .. code-block:: python 38 | 39 | with WorkerPool(n_jobs=4) as pool: 40 | pool.set_order_tasks() 41 | pool.map(task, range(10)) 42 | -------------------------------------------------------------------------------- /.github/workflows/python-package.yml: -------------------------------------------------------------------------------- 1 | # This workflow will install Python dependencies, run tests and lint with a variety of Python versions 2 | # For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions 3 | 4 | name: Build 5 | 6 | on: 7 | push: 8 | branches: [ master ] 9 | pull_request: 10 | branches: [ master ] 11 | 12 | jobs: 13 | build: 14 | 15 | runs-on: ${{ matrix.os }} 16 | strategy: 17 | matrix: 18 | os: [ubuntu-20.04, windows-latest, macos-latest] 19 | python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"] 20 | 21 | steps: 22 | - uses: actions/checkout@v3 23 | - name: Set up Python ${{ matrix.python-version }} 24 | uses: actions/setup-python@v4 25 | with: 26 | python-version: ${{ matrix.python-version }} 27 | - name: Install dependencies 28 | run: | 29 | python -m pip install --upgrade pip 30 | pip install flake8 pytest 31 | pip install .[dashboard] 32 | pip install .[dill] 33 | pip install .[testing] 34 | - name: Set ulimit for macOS 35 | if: matrix.os == 'macos-latest' 36 | run: | 37 | ulimit -a 38 | ulimit -n 1024 39 | - name: Lint with flake8 40 | run: | 41 | # stop the build if there are Python syntax errors or undefined names 42 | flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics 43 | # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide 44 | flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics 45 | - name: Test with pytest 46 | timeout-minutes: 30 47 | run: | 48 | pytest -v -o log_cli=true -s 49 | -------------------------------------------------------------------------------- /mpire/context.py: -------------------------------------------------------------------------------- 1 | import multiprocessing as mp 2 | try: 3 | import multiprocess as mp_dill 4 | import multiprocess.managers # Needed in utils.py 5 | except ImportError: 6 | mp_dill = None 7 | import platform 8 | import threading 9 | 10 | # Check if fork is available as start method. It's not available on Windows machines 11 | try: 12 | mp.get_context('fork') 13 | FORK_AVAILABLE = True 14 | except ValueError: 15 | FORK_AVAILABLE = False 16 | 17 | # Check if we're running on Windows or MacOS 18 | RUNNING_WINDOWS = platform.system() == "Windows" 19 | RUNNING_MACOS = platform.system() == "Darwin" 20 | 21 | 22 | # Threading context so we can use threading as backend as well 23 | class ThreadingContext: 24 | 25 | Barrier = threading.Barrier 26 | Condition = threading.Condition 27 | Event = threading.Event 28 | Lock = threading.Lock 29 | RLock = threading.RLock 30 | Thread = threading.Thread 31 | 32 | # threading doesn't have Array and JoinableQueue, so we take it from multiprocessing. Both are thread-safe. We need 33 | # the Process class for the MPIRE insights SyncManager instance. 34 | Array = mp.Array 35 | JoinableQueue = mp.JoinableQueue 36 | Process = mp.Process 37 | Value = mp.Value 38 | 39 | 40 | MP_CONTEXTS = {'mp': {'fork': mp.get_context('fork') if FORK_AVAILABLE else None, 41 | 'forkserver': mp.get_context('forkserver') if FORK_AVAILABLE else None, 42 | 'spawn': mp.get_context('spawn')}, 43 | 'threading': ThreadingContext} 44 | if mp_dill is not None: 45 | MP_CONTEXTS['mp_dill'] = {'fork': mp_dill.get_context('fork') if FORK_AVAILABLE else None, 46 | 'forkserver': mp_dill.get_context('forkserver') if FORK_AVAILABLE else None, 47 | 'spawn': mp_dill.get_context('spawn')} 48 | 49 | DEFAULT_START_METHOD = 'fork' if FORK_AVAILABLE else 'spawn' 50 | -------------------------------------------------------------------------------- /docs/usage/map/max_tasks_active.rst: -------------------------------------------------------------------------------- 1 | .. _max_active_tasks: 2 | 3 | Maximum number of active tasks 4 | ============================== 5 | 6 | When you have tasks that take up a lot of memory you can do a few things: 7 | 8 | - Limit the number of jobs (i.e., the number of tasks currently being available to the workers, tasks that are in the 9 | queue ready to be processed). 10 | - Limit the number of active tasks 11 | 12 | The first option is the most obvious one to save memory when the processes themselves use up much memory. The second is 13 | convenient when the argument list takes up too much memory. For example, suppose you want to kick off an enormous amount 14 | of jobs (let's say a billion) of which the arguments take up 1 KB per task (e.g., large strings), then that task queue 15 | would take up ~1 TB of memory! 16 | 17 | In such cases, a good rule of thumb would be to have twice the amount of active chunks of tasks than there are jobs. 18 | This means that when all workers complete their task at the same time each would directly be able to continue with 19 | another task. When workers take on their new tasks the generator of tasks is iterated to the point that again there 20 | would be twice the amount of active chunks of tasks. 21 | 22 | In MPIRE, the maximum number of active tasks by default is set to ``n_jobs * chunk_size * 2``, so you don't have to 23 | tweak it for memory optimization. If, for whatever reason, you want to change this behavior, you can do so by setting 24 | the ``max_active_tasks`` parameter: 25 | 26 | .. code-block:: python 27 | 28 | with WorkerPool(n_jobs=4) as pool: 29 | results = pool.map(task, range(int(1e300)), iterable_len=int(1e300), 30 | chunk_size=int(1e5), max_tasks_active=4 * int(1e5)) 31 | 32 | .. note:: 33 | 34 | Setting the ``max_tasks_active`` parameter to a value lower than ``n_jobs * chunk_size`` can result in some workers 35 | not being able to do anything. 36 | -------------------------------------------------------------------------------- /mpire/dashboard/templates/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
| # | 36 |Tasks | 37 |Progress | 38 |Duration | 39 |Remaining | 40 |Started | 41 |Finished / ETA | 42 |
|---|
Function: {function_name}, on line {function_line_no}
22 | of {user}{function_filename}
Invoked on line {invoked_line_no} of {invoked_filename},
24 | through {invoked_code_context}
ⓘ Start up time denotes the time to spin up 37 | a worker. Init time is the time a worker spends on the initialization function, when 38 | provided. Waiting time is the time a worker needs to wait for new tasks to come in. 39 | Working time is the time a worker spends on the task at hand. Exit time is the time a 40 | worker spends on the exit function, when provided.
41 || 47 | | Total | Mean | Std | Ratio (%) | 48 |
|---|---|---|---|---|
| Start up time | 52 |53 | | 54 | | 55 | | 56 | |
| Init time | 59 |60 | | 61 | | 62 | | 63 | |
| Waiting time | 66 |67 | | 68 | | 69 | | 70 | |
| Working time | 73 |74 | | 75 | | 76 | | 77 | |
| Exit time | 80 |81 | | 82 | | 83 | | 84 | |
ⓘ This section shows the top 5 tasks 89 | based on duration and is updated every 2 seconds.
90 || Time | Arguments | 94 |
|---|
| Worker | Tasks completed | 132 |T. start up time | 133 |T. init time | 134 |T. waiting time | 135 |T. working time | 136 |T. exit time | 137 |
|---|