├── .coveragerc ├── .travis.yml ├── CHANGELOG.rst ├── LICENSE.txt ├── MANIFEST.in ├── Makefile ├── README.rst ├── capturer ├── __init__.py └── tests.py ├── docs ├── api.rst ├── changelog.rst ├── conf.py ├── index.rst └── readme.rst ├── requirements-checks.txt ├── requirements-tests.txt ├── requirements-travis.txt ├── requirements.txt ├── scripts └── travis.sh ├── setup.cfg ├── setup.py └── tox.ini /.coveragerc: -------------------------------------------------------------------------------- 1 | # .coveragerc: Configuration file for coverage.py. 2 | # http://nedbatchelder.com/code/coverage/ 3 | 4 | [run] 5 | omit = capturer/tests.py 6 | source = capturer 7 | 8 | # vim: ft=dosini 9 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | cache: pip 3 | matrix: 4 | include: 5 | - os: osx 6 | language: generic 7 | - python: pypy 8 | - python: 2.7 9 | - python: 3.5 10 | - python: 3.6 11 | - python: 3.7 12 | - python: 3.8 13 | - python: 3.9-dev 14 | install: 15 | - scripts/travis.sh pip install --upgrade --requirement=requirements-travis.txt 16 | - scripts/travis.sh LC_ALL=C pip install . 17 | script: 18 | - scripts/travis.sh make check 19 | - scripts/travis.sh make test 20 | after_success: 21 | - scripts/travis.sh coveralls 22 | branches: 23 | except: 24 | - /^[0-9]/ 25 | -------------------------------------------------------------------------------- /CHANGELOG.rst: -------------------------------------------------------------------------------- 1 | Changelog 2 | ========= 3 | 4 | The purpose of this document is to list all of the notable changes to this 5 | project. The format was inspired by `Keep a Changelog`_. This project adheres 6 | to `semantic versioning`_. 7 | 8 | .. contents:: 9 | :local: 10 | 11 | .. _Keep a Changelog: http://keepachangelog.com/ 12 | .. _semantic versioning: http://semver.org/ 13 | 14 | `Release 3.0`_ (2020-03-07) 15 | --------------------------- 16 | 17 | This is a maintenance release that updates the supported Python 18 | versions, adds a changelog and makes some minor internal changes: 19 | 20 | - Added support for Python 3.7 and 3.8. 21 | - Dropped support for Python 2.6 and 3.4. 22 | - Actively deprecate ``interpret_carriage_returns()``. 23 | - Moved test helpers to :mod:`humanfriendly.testing`. 24 | - Include documentation in source distributions. 25 | - Use Python 3 for local development (``Makefile``). 26 | - Restructured the online documentation. 27 | - Updated PyPI domain in documentation. 28 | - Added this changelog. 29 | 30 | .. _Release 3.0: https://github.com/xolox/python-capturer/compare/2.4...3.0 31 | 32 | `Release 2.4`_ (2017-05-17) 33 | --------------------------- 34 | 35 | - Allow capturing output without relaying it. 36 | - Make ``OutputBuffer.flush()`` more robust. 37 | - Add Python 3.6 to supported versions. 38 | 39 | .. _Release 2.4: https://github.com/xolox/python-capturer/compare/2.3...2.4 40 | 41 | `Release 2.3`_ (2016-11-12) 42 | --------------------------- 43 | 44 | - Clearly document supported operating systems (`#4`_). 45 | - Start testing Python 3.5 and Mac OS X on Travis CI. 46 | - Start publishing wheel distributions. 47 | - PEP-8 and PEP-257 checks. 48 | 49 | .. _Release 2.3: https://github.com/xolox/python-capturer/compare/2.2...2.3 50 | .. _#4: https://github.com/xolox/python-capturer/issues/4 51 | 52 | `Release 2.2`_ (2016-10-09) 53 | --------------------------- 54 | 55 | Switch to :func:`humanfriendly.terminal.clean_terminal_output()`. 56 | 57 | .. _Release 2.2: https://github.com/xolox/python-capturer/compare/2.1.1...2.2 58 | 59 | `Release 2.1.1`_ (2015-10-24) 60 | ----------------------------- 61 | 62 | Make it easier to run test suite from PyPI release (fixes `#3`_). 63 | 64 | .. _Release 2.1.1: https://github.com/xolox/python-capturer/compare/2.1...2.1.1 65 | .. _#3: https://github.com/xolox/python-capturer/issues/3 66 | 67 | `Release 2.1`_ (2015-06-21) 68 | --------------------------- 69 | 70 | Make "nested" output capturing work as expected (issue `#2`_). 71 | 72 | .. _Release 2.1: https://github.com/xolox/python-capturer/compare/2.0...2.1 73 | .. _#2: https://github.com/xolox/python-capturer/issues/2 74 | 75 | `Release 2.0`_ (2015-06-18) 76 | --------------------------- 77 | 78 | Experimental support for capturing stdout/stderr separately (issue `#2`_). 79 | 80 | .. _Release 2.0: https://github.com/xolox/python-capturer/compare/1.1...2.0 81 | .. _#2: https://github.com/xolox/python-capturer/issues/2 82 | 83 | `Release 1.1`_ (2015-06-16) 84 | --------------------------- 85 | 86 | - Expose captured output as file handle (wiht shortcuts for saving to files). 87 | - Improve documentation of ``interpret_carriage_returns()``. 88 | - Clearly document drawbacks of emulating a terminal. 89 | 90 | .. _Release 1.1: https://github.com/xolox/python-capturer/compare/1.0...1.1 91 | 92 | `Release 1.0`_ (2015-06-14) 93 | --------------------------- 94 | 95 | This was the initial release. 96 | 97 | .. _Release 1.0: https://github.com/xolox/python-capturer/tree/1.0 98 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2020 Peter Odding 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be 12 | included in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 15 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 17 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 18 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 19 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 20 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | graft docs 2 | include *.rst 3 | include *.txt 4 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for the 'capturer' package. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: March 7, 2020 5 | # URL: https://github.com/xolox/python-capturer 6 | 7 | PACKAGE_NAME = capturer 8 | WORKON_HOME ?= $(HOME)/.virtualenvs 9 | VIRTUAL_ENV ?= $(WORKON_HOME)/$(PACKAGE_NAME) 10 | PYTHON ?= python3 11 | PATH := $(VIRTUAL_ENV)/bin:$(PATH) 12 | MAKE := $(MAKE) --no-print-directory 13 | SHELL = bash 14 | 15 | default: 16 | @echo "Makefile for $(PACKAGE_NAME)" 17 | @echo 18 | @echo 'Usage:' 19 | @echo 20 | @echo ' make install install the package in a virtual environment' 21 | @echo ' make reset recreate the virtual environment' 22 | @echo ' make check check coding style (PEP-8, PEP-257)' 23 | @echo ' make test run the test suite, report coverage' 24 | @echo ' make tox run the tests on all Python versions' 25 | @echo ' make docs update documentation using Sphinx' 26 | @echo ' make publish publish changes to GitHub/PyPI' 27 | @echo ' make clean cleanup all temporary files' 28 | @echo 29 | 30 | install: 31 | @test -d "$(VIRTUAL_ENV)" || mkdir -p "$(VIRTUAL_ENV)" 32 | @test -x "$(VIRTUAL_ENV)/bin/python" || virtualenv --python=$(PYTHON) --quiet "$(VIRTUAL_ENV)" 33 | @pip install --quiet --requirement=requirements.txt 34 | @pip uninstall --yes $(PACKAGE_NAME) &>/dev/null || true 35 | @pip install --quiet --no-deps --ignore-installed . 36 | 37 | reset: 38 | @$(MAKE) clean 39 | @rm -Rf "$(VIRTUAL_ENV)" 40 | @$(MAKE) install 41 | 42 | check: install 43 | @pip install --upgrade --quiet --requirement=requirements-checks.txt 44 | @flake8 45 | 46 | test: install 47 | @pip install --quiet --requirement=requirements-tests.txt 48 | @py.test --cov --cov-report=html --no-cov-on-fail 49 | @coverage report --fail-under=90 50 | 51 | tox: install 52 | @pip install --quiet tox 53 | @tox 54 | 55 | docs: install 56 | @pip install --quiet sphinx 57 | @cd docs && sphinx-build -nb html -d build/doctrees . build/html 58 | 59 | publish: install 60 | @git push origin && git push --tags origin 61 | @$(MAKE) clean 62 | @pip install --quiet twine wheel 63 | @python setup.py sdist bdist_wheel 64 | @twine upload dist/* 65 | @$(MAKE) clean 66 | 67 | clean: 68 | @rm -Rf *.egg .cache .coverage .tox build dist docs/build htmlcov 69 | @find -depth -type d -name __pycache__ -exec rm -Rf {} \; 70 | @find -type f -name '*.pyc' -delete 71 | 72 | .PHONY: default install reset check test tox docs publish clean 73 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | capturer: Easily capture stdout/stderr of the current process and subprocesses 2 | ============================================================================== 3 | 4 | .. image:: https://travis-ci.org/xolox/python-capturer.svg?branch=master 5 | :target: https://travis-ci.org/xolox/python-capturer 6 | 7 | .. image:: https://coveralls.io/repos/xolox/python-capturer/badge.svg?branch=master 8 | :target: https://coveralls.io/r/xolox/python-capturer?branch=master 9 | 10 | The capturer package makes it easy to capture the stdout_ and stderr_ streams 11 | of the current process *and subprocesses*. Output can be relayed to the 12 | terminal in real time but is also available to the Python program for 13 | additional processing. It's currently tested on cPython 2.7, 3.5+ and PyPy 14 | (2.7). It's tested on Linux and Mac OS X and may work on other unixes but 15 | definitely won't work on Windows (due to the use of the platform dependent pty_ 16 | module). For usage instructions please refer to the documentation_. 17 | 18 | .. contents:: 19 | :local: 20 | 21 | Status 22 | ------ 23 | 24 | The `capturer` package was developed as a proof of concept over the course of a 25 | weekend, because I was curious to see if it could be done (reliably). After a 26 | weekend of extensive testing it seems to work fairly well so I'm publishing the 27 | initial release as version 1.0, however I still consider this a proof of 28 | concept because I don't have extensive "production" experience using it yet. 29 | Here's hoping it works as well in practice as it did during my testing :-). 30 | 31 | Installation 32 | ------------ 33 | 34 | The `capturer` package is available on PyPI_ which means installation should be 35 | as simple as: 36 | 37 | .. code-block:: console 38 | 39 | $ pip install capturer 40 | 41 | There's actually a multitude of ways to install Python packages (e.g. the `per 42 | user site-packages directory`_, `virtual environments`_ or just installing 43 | system wide) and I have no intention of getting into that discussion here, so 44 | if this intimidates you then read up on your options before returning to these 45 | instructions ;-). 46 | 47 | Getting started 48 | --------------- 49 | 50 | The easiest way to capture output is to use a context manager: 51 | 52 | .. code-block:: python 53 | 54 | import subprocess 55 | from capturer import CaptureOutput 56 | 57 | with CaptureOutput() as capturer: 58 | # Generate some output from Python. 59 | print "Output from Python" 60 | # Generate output from a subprocess. 61 | subprocess.call(["echo", "Output from a subprocess"]) 62 | # Get the output in each of the supported formats. 63 | assert capturer.get_bytes() == b'Output from Python\r\nOutput from a subprocess\r\n' 64 | assert capturer.get_lines() == [u'Output from Python', u'Output from a subprocess'] 65 | assert capturer.get_text() == u'Output from Python\nOutput from a subprocess' 66 | 67 | The use of a context manager (`the with statement`_) ensures that output 68 | capturing is enabled and disabled at the appropriate time, regardless of 69 | whether exceptions interrupt the normal flow of processing. 70 | 71 | Note that the first call to `get_bytes()`_, `get_lines()`_ or `get_text()`_ 72 | will stop the capturing of output by default. This is intended as a sane 73 | default to prevent partial reads (which can be confusing as hell when you don't 74 | have experience with them). So we could have simply used ``print`` to show 75 | the results without causing a recursive "captured output is printed and then 76 | captured again" loop. There's an optional ``partial=True`` keyword argument 77 | that can be used to disable this behavior (please refer to the documentation_ 78 | for details). 79 | 80 | Design choices 81 | -------------- 82 | 83 | There are existing solutions out there to capture the stdout_ and stderr_ 84 | streams of (Python) processes. The `capturer` package was created for a very 85 | specific use case that wasn't catered for by existing solutions (that I could 86 | find). This section documents the design choices that guided the development of 87 | the `capturer` package: 88 | 89 | .. contents:: 90 | :local: 91 | 92 | Intercepts writes to low level file descriptors 93 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 94 | 95 | Libraries like capture_ and iocapture_ change Python's sys.stdout_ and 96 | sys.stderr_ file objects to fake file objects (using StringIO_). This enables 97 | capturing of (most) output written to the stdout_ and stderr_ streams from the 98 | same Python process, however any output from subprocesses is unaffected by the 99 | redirection and not captured. 100 | 101 | The `capturer` package instead intercepts writes to low level file descriptors 102 | (similar to and inspired by `how pytest does it`_). This enables capturing of 103 | output written to the standard output and error streams from the same Python 104 | process as well as any subprocesses. 105 | 106 | Uses a pseudo terminal to emulate a real terminal 107 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 108 | 109 | The `capturer` package uses a pseudo terminal created using `pty.openpty()`_ to 110 | capture output. This means subprocesses will use ANSI escape sequences because 111 | they think they're connected to a terminal. In the current implementation you 112 | can't opt out of this, but feel free to submit a feature request to change this 113 | :-). This does have some drawbacks: 114 | 115 | - The use of `pty.openpty()`_ means you need to be running in a UNIX like 116 | environment for `capturer` to work (Windows definitely isn't supported). 117 | 118 | - All output captured is relayed on the stderr_ stream by default, so capturing 119 | changes the semantics of your programs. How much this matters obviously 120 | depends on your use case. For the use cases that triggered me to create 121 | `capturer` it doesn't matter, which explains why this is the default mode. 122 | 123 | There is experimental support for capturing stdout_ and stderr_ separately 124 | and relaying captured output to the appropriate original stream. Basically 125 | you call ``CaptureOutput(merged=False)`` and then you use the ``stdout`` and 126 | ``stderr`` attributes of the ``CaptureOutput`` object to get at the output 127 | captured on each stream. 128 | 129 | I say experimental because this method of capturing can unintentionally 130 | change the order in which captured output is emitted, in order to avoid 131 | interleaving output emitted on the stdout_ and stderr_ streams (which would 132 | most likely result in incomprehensible output). Basically output is relayed 133 | on each stream separately after each line break. This means interactive 134 | prompts that block on reading from standard input without emitting a line 135 | break won't show up (until it's too late ;-). 136 | 137 | Relays output to the terminal in real time 138 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 139 | 140 | The main use case of `capturer` is to capture all output of a snippet of Python 141 | code (including any output by subprocesses) but also relay the output to the 142 | terminal in real time. This has a couple of useful properties: 143 | 144 | - Long running operations can provide the operator with real time feedback by 145 | emitting output on the terminal. This sounds obvious (and it is!) but it is 146 | non-trivial to implement (an understatement :-) when you *also* want to 147 | capture the output. 148 | 149 | - Programs like gpg_ and ssh_ that use interactive password prompts will render 150 | their password prompt on the terminal in real time. This avoids the awkward 151 | interaction where a password prompt is silenced but the program still hangs, 152 | waiting for input on stdin_. 153 | 154 | Contact 155 | ------- 156 | 157 | The latest version of `capturer` is available on PyPI_ and GitHub_. The 158 | documentation is hosted on `Read the Docs`_ and includes a changelog_. For bug 159 | reports please create an issue on GitHub_. If you have questions, suggestions, 160 | etc. feel free to send me an e-mail at `peter@peterodding.com`_. 161 | 162 | License 163 | ------- 164 | 165 | This software is licensed under the `MIT license`_. 166 | 167 | © 2020 Peter Odding. 168 | 169 | A big thanks goes out to the pytest_ developers because pytest's mechanism for 170 | capturing the output of subprocesses provided inspiration for the `capturer` 171 | package. No code was copied, but both projects are MIT licensed anyway, so it's 172 | not like it's very relevant :-). 173 | 174 | .. External references: 175 | .. _capture: https://pypi.org/project/capture 176 | .. _changelog: https://capturer.readthedocs.io/en/latest/changelog.html 177 | .. _documentation: https://capturer.readthedocs.io 178 | .. _get_bytes(): https://capturer.readthedocs.io/en/latest/api.html#capturer.CaptureOutput.get_bytes 179 | .. _get_lines(): https://capturer.readthedocs.io/en/latest/api.html#capturer.CaptureOutput.get_lines 180 | .. _get_text(): https://capturer.readthedocs.io/en/latest/api.html#capturer.CaptureOutput.get_text 181 | .. _GitHub: https://github.com/xolox/python-capturer 182 | .. _gpg: https://en.wikipedia.org/wiki/GNU_Privacy_Guard 183 | .. _how pytest does it: https://pytest.org/latest/capture.html 184 | .. _iocapture: https://pypi.org/project/iocapture 185 | .. _MIT license: http://en.wikipedia.org/wiki/MIT_License 186 | .. _per user site-packages directory: https://www.python.org/dev/peps/pep-0370/ 187 | .. _peter@peterodding.com: peter@peterodding.com 188 | .. _pty.openpty(): https://docs.python.org/2/library/pty.html#pty.openpty 189 | .. _pty: https://docs.python.org/2/library/pty.html 190 | .. _PyPI: https://pypi.org/project/capturer 191 | .. _pytest: https://pypi.org/project/pytest 192 | .. _Read the Docs: https://capturer.readthedocs.io 193 | .. _ssh: https://en.wikipedia.org/wiki/Secure_Shell 194 | .. _stderr: https://en.wikipedia.org/wiki/Standard_streams#Standard_error_.28stderr.29 195 | .. _stdin: https://en.wikipedia.org/wiki/Standard_streams#Standard_input_.28stdin.29 196 | .. _stdout: https://en.wikipedia.org/wiki/Standard_streams#Standard_output_.28stdout.29 197 | .. _StringIO: https://docs.python.org/2/library/stringio.html 198 | .. _sys.stderr: https://docs.python.org/2/library/sys.html#sys.stderr 199 | .. _sys.stdout: https://docs.python.org/2/library/sys.html#sys.stdout 200 | .. _the with statement: https://docs.python.org/2/reference/compound_stmts.html#the-with-statement 201 | .. _virtual environments: http://docs.python-guide.org/en/latest/dev/virtualenvs/ 202 | -------------------------------------------------------------------------------- /capturer/__init__.py: -------------------------------------------------------------------------------- 1 | # Easily capture stdout/stderr of the current process and subprocesses. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: March 7, 2020 5 | # URL: https://capturer.readthedocs.io 6 | 7 | """Easily capture stdout/stderr of the current process and subprocesses.""" 8 | 9 | # Standard library modules. 10 | import multiprocessing 11 | import os 12 | import pty 13 | import shutil 14 | import signal 15 | import sys 16 | import tempfile 17 | import time 18 | 19 | # External dependencies. 20 | from humanfriendly.deprecation import define_aliases 21 | from humanfriendly.text import compact, dedent 22 | from humanfriendly.terminal import clean_terminal_output 23 | 24 | # Semi-standard module versioning. 25 | __version__ = '3.0' 26 | 27 | # Define aliases for backwards compatibility. 28 | define_aliases(module_name=__name__, interpret_carriage_returns='humanfriendly.terminal.clean_terminal_output') 29 | 30 | DEFAULT_TEXT_ENCODING = 'UTF-8' 31 | """ 32 | The name of the default character encoding used to convert captured output to 33 | Unicode text (a string). 34 | """ 35 | 36 | GRACEFUL_SHUTDOWN_SIGNAL = signal.SIGUSR1 37 | """ 38 | The number of the UNIX signal used to communicate graceful shutdown requests 39 | from the main process to the output relay process (an integer). See also 40 | :func:`~MultiProcessHelper.enable_graceful_shutdown()`. 41 | """ 42 | 43 | TERMINATION_DELAY = 0.01 44 | """ 45 | The number of seconds to wait before terminating the output relay process (a 46 | floating point number). 47 | """ 48 | 49 | PARTIAL_DEFAULT = False 50 | """Whether partial reads are enabled or disabled by default (a boolean).""" 51 | 52 | STDOUT_FD = 1 53 | """ 54 | The number of the file descriptor that refers to the standard output stream (an 55 | integer). 56 | """ 57 | 58 | STDERR_FD = 2 59 | """ 60 | The number of the file descriptor that refers to the standard error stream (an 61 | integer). 62 | """ 63 | 64 | 65 | def enable_old_api(): 66 | """ 67 | Enable backwards compatibility with the old API. 68 | 69 | This function is called when the :mod:`capturer` module is imported. It 70 | modifies the :class:`CaptureOutput` class to install method proxies for 71 | :func:`~PseudoTerminal.get_handle()`, :func:`~PseudoTerminal.get_bytes()`, 72 | :func:`~PseudoTerminal.get_lines()`, :func:`~PseudoTerminal.get_text()`, 73 | :func:`~PseudoTerminal.save_to_handle()` and 74 | :func:`~PseudoTerminal.save_to_path()`. 75 | """ 76 | for name in ('get_handle', 'get_bytes', 'get_lines', 'get_text', 'save_to_handle', 'save_to_path'): 77 | setattr(CaptureOutput, name, create_proxy_method(name)) 78 | 79 | 80 | def create_proxy_method(name): 81 | """ 82 | Create a proxy method for use by :func:`enable_old_api()`. 83 | 84 | :param name: The name of the :class:`PseudoTerminal` method to call when 85 | the proxy method is called. 86 | :returns: A proxy method (a callable) to be installed on the 87 | :class:`CaptureOutput` class. 88 | """ 89 | # Define the proxy method. 90 | def proxy_method(self, *args, **kw): 91 | if not hasattr(self, 'output'): 92 | raise TypeError(compact(""" 93 | The old calling interface is only supported when 94 | merged=True and start_capture() has been called! 95 | """)) 96 | real_method = getattr(self.output, name) 97 | return real_method(*args, **kw) 98 | # Get the docstring of the real method. 99 | docstring = getattr(PseudoTerminal, name).__doc__ 100 | # Change the docstring to explain that this concerns a proxy method, 101 | # but only when Sphinx is active (to avoid wasting time generating a 102 | # docstring that no one is going to look at). 103 | if 'sphinx' in sys.modules: 104 | # Remove the signature from the docstring to make it possible to 105 | # remove leading indentation from the remainder of the docstring. 106 | lines = docstring.splitlines() 107 | signature = lines.pop(0) 108 | # Recompose the docstring from the signature, the remainder of the 109 | # original docstring and the note about proxy methods. 110 | docstring = '\n\n'.join([ 111 | signature, 112 | dedent('\n'.join(lines)), 113 | dedent(""" 114 | .. note:: This method is a proxy for the :func:`~PseudoTerminal.{name}()` 115 | method of the :class:`PseudoTerminal` class. It requires 116 | `merged` to be :data:`True` and it expects that 117 | :func:`start_capture()` has been called. If this is not 118 | the case then :exc:`~exceptions.TypeError` is raised. 119 | """, name=name), 120 | ]) 121 | # Copy the (possible modified) docstring. 122 | proxy_method.__doc__ = docstring 123 | return proxy_method 124 | 125 | 126 | class MultiProcessHelper(object): 127 | 128 | """ 129 | Helper to spawn and manipulate child processes using :mod:`multiprocessing`. 130 | 131 | This class serves as a base class for :class:`CaptureOutput` and 132 | :class:`PseudoTerminal` because both classes need the same child process 133 | handling logic. 134 | """ 135 | 136 | def __init__(self): 137 | """Initialize a :class:`MultiProcessHelper` object.""" 138 | self.processes = [] 139 | 140 | def start_child(self, target): 141 | """ 142 | Start a child process using :class:`multiprocessing.Process`. 143 | 144 | :param target: The callable to run in the child process. Expected to 145 | take a single argument which is a 146 | :class:`multiprocessing.Event` to be set when the child 147 | process has finished initialization. 148 | """ 149 | started_event = multiprocessing.Event() 150 | child_process = multiprocessing.Process(target=target, args=(started_event,)) 151 | self.processes.append(child_process) 152 | child_process.daemon = True 153 | child_process.start() 154 | started_event.wait() 155 | 156 | def stop_children(self): 157 | """ 158 | Gracefully shut down all child processes. 159 | 160 | Child processes are expected to call :func:`enable_graceful_shutdown()` 161 | during initialization. 162 | """ 163 | while self.processes: 164 | child_process = self.processes.pop() 165 | if child_process.is_alive(): 166 | os.kill(child_process.pid, GRACEFUL_SHUTDOWN_SIGNAL) 167 | child_process.join() 168 | 169 | def wait_for_children(self): 170 | """Wait for all child processes to terminate.""" 171 | for child_process in self.processes: 172 | child_process.join() 173 | 174 | def enable_graceful_shutdown(self): 175 | """ 176 | Register a signal handler that converts :data:`GRACEFUL_SHUTDOWN_SIGNAL` to an exception. 177 | 178 | Used by :func:`~PseudoTerminal.capture_loop()` to gracefully interrupt 179 | the blocking :func:`os.read()` call when the capture loop needs to be 180 | terminated (this is required for coverage collection). 181 | """ 182 | signal.signal(GRACEFUL_SHUTDOWN_SIGNAL, self.raise_shutdown_request) 183 | 184 | def raise_shutdown_request(self, signum, frame): 185 | """Raise :exc:`ShutdownRequested` when :data:`GRACEFUL_SHUTDOWN_SIGNAL` is received.""" 186 | raise ShutdownRequested 187 | 188 | 189 | class CaptureOutput(MultiProcessHelper): 190 | 191 | """Context manager to capture the standard output and error streams.""" 192 | 193 | def __init__(self, merged=True, encoding=DEFAULT_TEXT_ENCODING, 194 | termination_delay=TERMINATION_DELAY, chunk_size=1024, 195 | relay=True): 196 | """ 197 | Initialize a :class:`CaptureOutput` object. 198 | 199 | :param merged: Whether to capture and relay the standard output and 200 | standard error streams as one stream (a boolean, 201 | defaults to :data:`True`). When this is :data:`False` 202 | the ``stdout`` and ``stderr`` attributes of the 203 | :class:`CaptureOutput` object are 204 | :class:`PseudoTerminal` objects that can be used to 205 | get at the output captured from each stream separately. 206 | :param encoding: The name of the character encoding used to decode the 207 | captured output (a string, defaults to 208 | :data:`DEFAULT_TEXT_ENCODING`). 209 | :param termination_delay: The number of seconds to wait before 210 | terminating the output relay process (a 211 | floating point number, defaults to 212 | :data:`TERMINATION_DELAY`). 213 | :param chunk_size: The maximum number of bytes to read from the 214 | captured streams on each call to :func:`os.read()` 215 | (an integer). 216 | :param relay: If this is :data:`True` (the default) then captured 217 | output is relayed to the terminal or parent process, 218 | if it's :data:`False` the captured output is hidden 219 | (swallowed). 220 | """ 221 | # Initialize the superclass. 222 | super(CaptureOutput, self).__init__() 223 | # Store constructor arguments. 224 | self.chunk_size = chunk_size 225 | self.encoding = encoding 226 | self.merged = merged 227 | self.relay = relay 228 | self.termination_delay = termination_delay 229 | # Initialize instance variables. 230 | self.pseudo_terminals = [] 231 | self.streams = [] 232 | # Initialize stdout/stderr stream containers. 233 | self.stdout_stream = self.initialize_stream(sys.stdout, STDOUT_FD) 234 | self.stderr_stream = self.initialize_stream(sys.stderr, STDERR_FD) 235 | 236 | def initialize_stream(self, file_obj, expected_fd): 237 | """ 238 | Initialize one or more :class:`Stream` objects to capture a standard stream. 239 | 240 | :param file_obj: A file-like object with a ``fileno()`` method. 241 | :param expected_fd: The expected file descriptor of the file-like object. 242 | :returns: The :class:`Stream` connected to the file descriptor of the 243 | file-like object. 244 | 245 | By default this method just initializes a :class:`Stream` object 246 | connected to the given file-like object and its underlying file 247 | descriptor (a simple one-liner). 248 | 249 | If however the file descriptor of the file-like object doesn't have the 250 | expected value (``expected_fd``) two :class:`Stream` objects will be 251 | created instead: One of the stream objects will be connected to the 252 | file descriptor of the file-like object and the other stream object 253 | will be connected to the file descriptor that was expected 254 | (``expected_fd``). 255 | 256 | This approach is intended to make sure that "nested" output capturing 257 | works as expected: Output from the current Python process is captured 258 | from the file descriptor of the file-like object while output from 259 | subprocesses is captured from the file descriptor given by 260 | ``expected_fd`` (because the operating system defines special semantics 261 | for the file descriptors with the numbers one and two that we can't 262 | just ignore). 263 | 264 | For more details refer to `issue 2 on GitHub 265 | `_. 266 | """ 267 | real_fd = file_obj.fileno() 268 | stream_obj = Stream(real_fd) 269 | self.streams.append((expected_fd, stream_obj)) 270 | if real_fd != expected_fd: 271 | self.streams.append((expected_fd, Stream(expected_fd))) 272 | return stream_obj 273 | 274 | def __enter__(self): 275 | """Automatically call :func:`start_capture()` when entering a :keyword:`with` block.""" 276 | self.start_capture() 277 | return self 278 | 279 | def __exit__(self, exc_type=None, exc_value=None, traceback=None): 280 | """Automatically call :func:`finish_capture()` when leaving a :keyword:`with` block.""" 281 | self.finish_capture() 282 | 283 | @property 284 | def is_capturing(self): 285 | """:data:`True` if output is being captured, :data:`False` otherwise.""" 286 | return any(stream.is_redirected for kind, stream in self.streams) 287 | 288 | def start_capture(self): 289 | """ 290 | Start capturing the standard output and error streams. 291 | 292 | :raises: :exc:`~exceptions.TypeError` when output is already being 293 | captured. 294 | 295 | This method is called automatically when using the capture object as a 296 | context manager. It's provided under a separate name in case someone 297 | wants to extend :class:`CaptureOutput` and build their own context 298 | manager on top of it. 299 | """ 300 | if self.is_capturing: 301 | raise TypeError("Output capturing is already enabled!") 302 | if self.merged: 303 | # Capture (and most likely relay) stdout/stderr as one stream. 304 | fd = self.stderr_stream.original_fd if self.relay else None 305 | self.output = self.allocate_pty(relay_fd=fd) 306 | for kind, stream in self.streams: 307 | self.output.attach(stream) 308 | else: 309 | # Capture (and most likely relay) stdout/stderr as separate streams. 310 | if self.relay: 311 | # Start the subprocess to relay output. 312 | self.output_queue = multiprocessing.Queue() 313 | self.start_child(self.merge_loop) 314 | else: 315 | # Disable relaying of output. 316 | self.output_queue = None 317 | self.stdout = self.allocate_pty(output_queue=self.output_queue, queue_token=STDOUT_FD) 318 | self.stderr = self.allocate_pty(output_queue=self.output_queue, queue_token=STDERR_FD) 319 | for kind, stream in self.streams: 320 | if kind == STDOUT_FD: 321 | self.stdout.attach(stream) 322 | elif kind == STDERR_FD: 323 | self.stderr.attach(stream) 324 | else: 325 | raise Exception("Programming error: Unrecognized stream type!") 326 | # Start capturing and relaying of output (in one or two subprocesses). 327 | for pseudo_terminal in self.pseudo_terminals: 328 | pseudo_terminal.start_capture() 329 | 330 | def finish_capture(self): 331 | """ 332 | Stop capturing the standard output and error streams. 333 | 334 | This method is called automatically when using the capture object as a 335 | context manager. It's provided under a separate name in case someone 336 | wants to extend :class:`CaptureOutput` and build their own context 337 | manager on top of it. 338 | """ 339 | for pseudo_terminal in self.pseudo_terminals: 340 | pseudo_terminal.finish_capture() 341 | self.wait_for_children() 342 | 343 | def allocate_pty(self, relay_fd=None, output_queue=None, queue_token=None): 344 | """ 345 | Allocate a pseudo terminal. 346 | 347 | Internal shortcut for :func:`start_capture()` to allocate multiple 348 | pseudo terminals without code duplication. 349 | """ 350 | obj = PseudoTerminal( 351 | self.encoding, self.termination_delay, self.chunk_size, 352 | relay_fd=relay_fd, output_queue=output_queue, 353 | queue_token=queue_token, 354 | ) 355 | self.pseudo_terminals.append(obj) 356 | return obj 357 | 358 | def merge_loop(self, started_event): 359 | """ 360 | Merge and relay output in a child process. 361 | 362 | This internal method is used when standard output and standard error 363 | are being captured separately. It's responsible for emitting each 364 | captured line on the appropriate stream without interleaving text 365 | within lines. 366 | """ 367 | buffers = { 368 | STDOUT_FD: OutputBuffer(self.stdout_stream.original_fd), 369 | STDERR_FD: OutputBuffer(self.stderr_stream.original_fd), 370 | } 371 | started_event.set() 372 | while buffers: 373 | captured_from, output = self.output_queue.get() 374 | if output: 375 | buffers[captured_from].add(output) 376 | else: 377 | buffers[captured_from].flush() 378 | buffers.pop(captured_from) 379 | 380 | 381 | class OutputBuffer(object): 382 | 383 | """ 384 | Helper for :func:`CaptureOutput.merge_loop()`. 385 | 386 | Buffers captured output and flushes to the appropriate stream after each 387 | line break. 388 | """ 389 | 390 | def __init__(self, fd): 391 | """ 392 | Initialize an :class:`OutputBuffer` object. 393 | 394 | :param fd: The number of the file descriptor where output should be 395 | flushed (an integer). 396 | """ 397 | self.fd = fd 398 | self.buffer = b'' 399 | 400 | def add(self, output): 401 | """ 402 | Add output to the buffer and flush appropriately. 403 | 404 | :param output: The output to add to the buffer (a string). 405 | """ 406 | self.buffer += output 407 | if b'\n' in self.buffer: 408 | before, _, self.buffer = self.buffer.rpartition(b'\n') 409 | os.write(self.fd, before + b'\n') 410 | 411 | def flush(self): 412 | """Flush any remaining buffered output to the stream.""" 413 | os.write(self.fd, self.buffer) 414 | self.buffer = b'' 415 | 416 | 417 | class PseudoTerminal(MultiProcessHelper): 418 | 419 | """ 420 | Helper for :class:`CaptureOutput`. 421 | 422 | Manages capturing of output and exposing the captured output. 423 | """ 424 | 425 | def __init__(self, encoding, termination_delay, chunk_size, relay_fd, output_queue, queue_token): 426 | """ 427 | Initialize a :class:`PseudoTerminal` object. 428 | 429 | :param encoding: The name of the character encoding used to decode the 430 | captured output (a string, defaults to 431 | :data:`DEFAULT_TEXT_ENCODING`). 432 | :param termination_delay: The number of seconds to wait before 433 | terminating the output relay process (a 434 | floating point number, defaults to 435 | :data:`TERMINATION_DELAY`). 436 | :param chunk_size: The maximum number of bytes to read from the 437 | captured stream(s) on each call to :func:`os.read()` 438 | (an integer). 439 | :param relay_fd: The number of the file descriptor where captured 440 | output should be relayed to (an integer or 441 | :data:`None` if ``output_queue`` and ``queue_token`` 442 | are given). 443 | :param output_queue: The multiprocessing queue where captured output 444 | chunks should be written to (a 445 | :class:`multiprocessing.Queue` object or 446 | :data:`None` if ``relay_fd`` is given). 447 | :param queue_token: A unique identifier added to each output chunk 448 | written to the queue (any value or :data:`None` if 449 | ``relay_fd`` is given). 450 | """ 451 | # Initialize the superclass. 452 | super(PseudoTerminal, self).__init__() 453 | # Store constructor arguments. 454 | self.encoding = encoding 455 | self.termination_delay = termination_delay 456 | self.chunk_size = chunk_size 457 | self.relay_fd = relay_fd 458 | self.output_queue = output_queue 459 | self.queue_token = queue_token 460 | # Initialize instance variables. 461 | self.streams = [] 462 | # Allocate a pseudo terminal so we can fake subprocesses into 463 | # thinking that they are connected to a real terminal (this will 464 | # trigger them to use e.g. ANSI escape sequences). 465 | self.master_fd, self.slave_fd = pty.openpty() 466 | # Create a temporary file in which we'll store the output received on 467 | # the master end of the pseudo terminal. 468 | self.output_fd, output_file = tempfile.mkstemp() 469 | self.output_handle = open(output_file, 'rb') 470 | # Unlink the temporary file because we have a readable file descriptor 471 | # and a writable file descriptor and that's all we need! If this 472 | # surprises you I suggest you investigate why unlink() was named the 473 | # way it was in UNIX :-). 474 | os.unlink(output_file) 475 | 476 | def attach(self, stream): 477 | """ 478 | Attach a stream to the pseudo terminal. 479 | 480 | :param stream: A :class:`Stream` object. 481 | """ 482 | stream.redirect(self.slave_fd) 483 | self.streams.append(stream) 484 | 485 | def start_capture(self): 486 | """Start the child process(es) responsible for capturing and relaying output.""" 487 | self.start_child(self.capture_loop) 488 | 489 | def finish_capture(self): 490 | """Stop the process of capturing output and destroy the pseudo terminal.""" 491 | time.sleep(self.termination_delay) 492 | self.stop_children() 493 | self.close_pseudo_terminal() 494 | self.restore_streams() 495 | 496 | def close_pseudo_terminal(self): 497 | """Close the pseudo terminal's master/slave file descriptors.""" 498 | for name in ('master_fd', 'slave_fd'): 499 | fd = getattr(self, name) 500 | if fd is not None: 501 | os.close(fd) 502 | setattr(self, name, None) 503 | 504 | def restore_streams(self): 505 | """Restore the stream(s) attached to the pseudo terminal.""" 506 | for stream in self.streams: 507 | stream.restore() 508 | 509 | # The CaptureOutput class contains proxy methods for the get_handle(), 510 | # get_bytes(), get_lines(), get_text(), save_to_handle() and save_to_path() 511 | # methods defined below. By default Sphinx generates method signatures of 512 | # the form f(proxy, *args, **kw) for these proxy methods, with the result 513 | # that the online documentation is rather confusing. As a workaround I've 514 | # included explicit method signatures in the first line of each of the 515 | # docstrings. This works because of the following Sphinx option: 516 | # http://www.sphinx-doc.org/en/latest/ext/autodoc.html#confval-autodoc_docstring_signature 517 | 518 | def get_handle(self, partial=PARTIAL_DEFAULT): 519 | """get_handle(partial=False) 520 | Get the captured output as a Python file object. 521 | 522 | :param partial: If :data:`True` (*not the default*) the partial output 523 | captured so far is returned, otherwise (*so by 524 | default*) the relay process is terminated and output 525 | capturing is disabled before returning the captured 526 | output (the default is intended to protect unsuspecting 527 | users against partial reads). 528 | :returns: The captured output as a Python file object. The file 529 | object's current position is reset to zero before this 530 | function returns. 531 | 532 | This method is useful when you're dealing with arbitrary amounts of 533 | captured data that you don't want to load into memory just so you can 534 | save it to a file again. In fact, in that case you might want to take a 535 | look at :func:`save_to_path()` and/or :func:`save_to_handle()` :-). 536 | 537 | .. warning:: Two caveats about the use of this method: 538 | 539 | 1. If partial is :data:`True` (not the default) the output 540 | can end in a partial line, possibly in the middle of an 541 | ANSI escape sequence or a multi byte character. 542 | 543 | 2. If you close this file handle you just lost your last 544 | chance to get at the captured output! (calling this 545 | method again will not give you a new file handle) 546 | """ 547 | if not partial: 548 | self.finish_capture() 549 | self.output_handle.seek(0) 550 | return self.output_handle 551 | 552 | def get_bytes(self, partial=PARTIAL_DEFAULT): 553 | """get_bytes(partial=False) 554 | Get the captured output as binary data. 555 | 556 | :param partial: Refer to :func:`get_handle()` for details. 557 | :returns: The captured output as a binary string. 558 | """ 559 | return self.get_handle(partial).read() 560 | 561 | def get_lines(self, interpreted=True, partial=PARTIAL_DEFAULT): 562 | """get_lines(interpreted=True, partial=False) 563 | Get the captured output split into lines. 564 | 565 | :param interpreted: If :data:`True` (the default) captured output is 566 | processed using :func:`.clean_terminal_output()`. 567 | :param partial: Refer to :func:`get_handle()` for details. 568 | :returns: The captured output as a list of Unicode strings. 569 | 570 | .. warning:: If partial is :data:`True` (not the default) the output 571 | can end in a partial line, possibly in the middle of a 572 | multi byte character (this may cause decoding errors). 573 | """ 574 | output = self.get_bytes(partial) 575 | output = output.decode(self.encoding) 576 | if interpreted: 577 | return clean_terminal_output(output) 578 | else: 579 | return output.splitlines() 580 | 581 | def get_text(self, interpreted=True, partial=PARTIAL_DEFAULT): 582 | """get_text(interpreted=True, partial=False) 583 | Get the captured output as a single string. 584 | 585 | :param interpreted: If :data:`True` (the default) captured output is 586 | processed using :func:`clean_terminal_output()`. 587 | :param partial: Refer to :func:`get_handle()` for details. 588 | :returns: The captured output as a Unicode string. 589 | 590 | .. warning:: If partial is :data:`True` (not the default) the output 591 | can end in a partial line, possibly in the middle of a 592 | multi byte character (this may cause decoding errors). 593 | """ 594 | output = self.get_bytes(partial) 595 | output = output.decode(self.encoding) 596 | if interpreted: 597 | output = u'\n'.join(clean_terminal_output(output)) 598 | return output 599 | 600 | def save_to_handle(self, handle, partial=PARTIAL_DEFAULT): 601 | """save_to_handle(handle, partial=False) 602 | Save the captured output to an open file handle. 603 | 604 | :param handle: A writable file-like object. 605 | :param partial: Refer to :func:`get_handle()` for details. 606 | """ 607 | shutil.copyfileobj(self.get_handle(partial), handle) 608 | 609 | def save_to_path(self, filename, partial=PARTIAL_DEFAULT): 610 | """save_to_path(filename, partial=False) 611 | Save the captured output to a file. 612 | 613 | :param filename: The pathname of the file where the captured output 614 | should be written to (a string). 615 | :param partial: Refer to :func:`get_handle()` for details. 616 | """ 617 | with open(filename, 'wb') as handle: 618 | self.save_to_handle(handle, partial) 619 | 620 | def capture_loop(self, started_event): 621 | """ 622 | Continuously read from the master end of the pseudo terminal and relay the output. 623 | 624 | This function is run in the background by :func:`start_capture()` 625 | using the :mod:`multiprocessing` module. It's role is to read output 626 | emitted on the master end of the pseudo terminal and relay this output 627 | to the real terminal (so the operator can see what's happening in real 628 | time) as well as a temporary file (for additional processing by the 629 | caller). 630 | """ 631 | self.enable_graceful_shutdown() 632 | started_event.set() 633 | try: 634 | while True: 635 | # Read from the master end of the pseudo terminal. 636 | output = os.read(self.master_fd, self.chunk_size) 637 | if output: 638 | # Store the output in the temporary file. 639 | os.write(self.output_fd, output) 640 | # Relay the output to the real terminal? 641 | if self.relay_fd is not None: 642 | os.write(self.relay_fd, output) 643 | # Relay the output to the master process? 644 | if self.output_queue is not None: 645 | self.output_queue.put((self.queue_token, output)) 646 | else: 647 | # Relinquish our time slice, or in other words: try to be 648 | # friendly to other processes when os.read() calls don't 649 | # block. Just for the record, all of my experiments have 650 | # shown that os.read() on the master file descriptor 651 | # returned by pty.openpty() does in fact block. 652 | time.sleep(0) 653 | except ShutdownRequested: 654 | # Let the master process know that we're shutting down. 655 | if self.output_queue is not None: 656 | self.output_queue.put((self.queue_token, '')) 657 | 658 | 659 | class Stream(object): 660 | 661 | """ 662 | Container for standard stream redirection logic. 663 | 664 | Used by :class:`CaptureOutput` to temporarily redirect the standard output 665 | and standard error streams. 666 | 667 | .. attribute:: is_redirected 668 | 669 | :data:`True` once :func:`redirect()` has been called, :data:`False` when 670 | :func:`redirect()` hasn't been called yet or :func:`restore()` has since 671 | been called. 672 | """ 673 | 674 | def __init__(self, fd): 675 | """ 676 | Initialize a :class:`Stream` object. 677 | 678 | :param fd: The file descriptor to be redirected (an integer). 679 | """ 680 | self.fd = fd 681 | self.original_fd = os.dup(self.fd) 682 | self.is_redirected = False 683 | 684 | def redirect(self, target_fd): 685 | """ 686 | Redirect output written to the file descriptor to another file descriptor. 687 | 688 | :param target_fd: The file descriptor that should receive the output 689 | written to the file descriptor given to the 690 | :class:`Stream` constructor (an integer). 691 | :raises: :exc:`~exceptions.TypeError` when the file descriptor is 692 | already being redirected. 693 | """ 694 | if self.is_redirected: 695 | msg = "File descriptor %s is already being redirected!" 696 | raise TypeError(msg % self.fd) 697 | os.dup2(target_fd, self.fd) 698 | self.is_redirected = True 699 | 700 | def restore(self): 701 | """Stop redirecting output written to the file descriptor.""" 702 | if self.is_redirected: 703 | os.dup2(self.original_fd, self.fd) 704 | self.is_redirected = False 705 | 706 | 707 | class ShutdownRequested(Exception): 708 | 709 | """ 710 | Raised by :func:`~MultiProcessHelper.raise_shutdown_request()` to signal 711 | graceful termination requests (in :func:`~PseudoTerminal.capture_loop()`). 712 | """ 713 | 714 | 715 | enable_old_api() 716 | -------------------------------------------------------------------------------- /capturer/tests.py: -------------------------------------------------------------------------------- 1 | # Easily capture stdout/stderr of the current process and subprocesses. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: March 7, 2020 5 | # URL: https://capturer.readthedocs.io 6 | 7 | """Test suite for the `capturer` package.""" 8 | 9 | # Standard library modules. 10 | import os 11 | import subprocess 12 | import sys 13 | import tempfile 14 | import unittest 15 | 16 | # External dependencies. 17 | from humanfriendly.terminal import clean_terminal_output 18 | from humanfriendly.testing import TestCase, random_string, retry 19 | 20 | # The module we're testing. 21 | from capturer import CaptureOutput, Stream 22 | 23 | 24 | class CapturerTestCase(TestCase): 25 | 26 | """Container for the `capturer` test suite.""" 27 | 28 | def test_carriage_return_interpretation(self): 29 | """Sanity check the results of clean_terminal_output().""" 30 | # Simple output should pass through unharmed. 31 | assert clean_terminal_output('foo') == ['foo'] 32 | # Simple output should pass through unharmed. 33 | assert clean_terminal_output('foo\nbar') == ['foo', 'bar'] 34 | # Carriage returns and preceding substrings should be stripped. 35 | assert clean_terminal_output('foo\rbar\nbaz') == ['bar', 'baz'] 36 | # Trailing empty lines should be stripped. 37 | assert clean_terminal_output('foo\nbar\nbaz\n\n\n') == ['foo', 'bar', 'baz'] 38 | 39 | def test_error_handling(self): 40 | """Test error handling code paths.""" 41 | # Nested CaptureOutput.start_capture() calls should raise an exception. 42 | capturer = CaptureOutput() 43 | capturer.start_capture() 44 | try: 45 | self.assertRaises(TypeError, capturer.start_capture) 46 | finally: 47 | # Make sure not to start swallowing output here ;-). 48 | capturer.finish_capture() 49 | # Nested Stream.redirect() calls should raise an exception. 50 | stream = Stream(sys.stdout.fileno()) 51 | stream.redirect(sys.stderr.fileno()) 52 | self.assertRaises(TypeError, stream.redirect, sys.stderr.fileno()) 53 | 54 | def test_stdout_capture_same_process(self): 55 | """Test standard output capturing from the same process.""" 56 | expected_stdout = random_string() 57 | with CaptureOutput() as capturer: 58 | print(expected_stdout) 59 | assert expected_stdout in capturer.get_lines() 60 | 61 | def test_stderr_capture_same_process(self): 62 | """Test standard error capturing from the same process.""" 63 | expected_stderr = random_string() 64 | with CaptureOutput() as capturer: 65 | sys.stderr.write(expected_stderr + "\n") 66 | assert expected_stderr in capturer.get_lines() 67 | 68 | def test_combined_capture_same_process(self): 69 | """Test combined standard output and error capturing from the same process.""" 70 | expected_stdout = random_string() 71 | expected_stderr = random_string() 72 | with CaptureOutput() as capturer: 73 | sys.stdout.write(expected_stdout + "\n") 74 | sys.stderr.write(expected_stderr + "\n") 75 | assert expected_stdout in capturer.get_lines() 76 | assert expected_stderr in capturer.get_lines() 77 | 78 | def test_stdout_capture_subprocess(self): 79 | """Test standard output capturing from subprocesses.""" 80 | expected_stdout = random_string() 81 | with CaptureOutput() as capturer: 82 | subprocess.call([ 83 | sys.executable, 84 | '-c', 85 | ';'.join([ 86 | 'import sys', 87 | 'sys.stdout.write(%r)' % (expected_stdout + '\n'), 88 | ]), 89 | ]) 90 | assert expected_stdout in capturer.get_lines() 91 | 92 | def test_stderr_capture_subprocess(self): 93 | """Test standard error capturing from subprocesses.""" 94 | expected_stderr = random_string() 95 | with CaptureOutput() as capturer: 96 | subprocess.call([ 97 | sys.executable, 98 | '-c', 99 | ';'.join([ 100 | 'import sys', 101 | 'sys.stderr.write(%r)' % (expected_stderr + '\n'), 102 | ]), 103 | ]) 104 | assert expected_stderr in capturer.get_lines() 105 | 106 | def test_combined_capture_subprocess(self): 107 | """Test combined standard output and error capturing from subprocesses.""" 108 | expected_stdout = random_string() 109 | expected_stderr = random_string() 110 | with CaptureOutput() as capturer: 111 | subprocess.call([ 112 | sys.executable, 113 | '-c', 114 | ';'.join([ 115 | 'import sys', 116 | 'sys.stdout.write(%r)' % (expected_stdout + '\n'), 117 | 'sys.stderr.write(%r)' % (expected_stderr + '\n'), 118 | ]), 119 | ]) 120 | assert expected_stdout in capturer.get_lines() 121 | assert expected_stderr in capturer.get_lines() 122 | 123 | def test_combined_current_and_subprocess(self): 124 | """Test combined standard output and error capturing from the same process and subprocesses.""" 125 | # Some unique strings that are not substrings of each other. 126 | cur_stdout_1 = "Some output from Python's print statement" 127 | cur_stdout_2 = "Output from Python's sys.stdout.write() method" 128 | cur_stdout_3 = "More output from Python's print statement" 129 | cur_stderr = "Output from Python's sys.stderr.write() method" 130 | sub_stderr = "Output from subprocess stderr stream" 131 | sub_stdout = "Output from subprocess stdout stream" 132 | with CaptureOutput() as capturer: 133 | # Emit multiple lines on both streams from current process and subprocess. 134 | print(cur_stdout_1) 135 | sys.stderr.write("%s\n" % cur_stderr) 136 | subprocess.call(["sh", "-c", "echo %s 1>&2" % sub_stderr]) 137 | subprocess.call(["echo", sub_stdout]) 138 | sys.stdout.write("%s\n" % cur_stdout_2) 139 | print(cur_stdout_3) 140 | # Verify that all of the expected lines were captured. 141 | assert all(l in capturer.get_lines() for l in ( 142 | cur_stdout_1, cur_stderr, sub_stderr, 143 | sub_stdout, cur_stdout_2, cur_stdout_3, 144 | )) 145 | 146 | def test_partial_read(self): 147 | """Test that partial reading works as expected.""" 148 | # This test method uses retry logic because `partial=True' makes these 149 | # tests prone to race conditions (this is the whole reason why 150 | # `partial=False' by default :-). 151 | initial_part = random_string() 152 | later_part = random_string() 153 | with CaptureOutput() as capturer: 154 | sys.stderr.write("%s\n" % initial_part) 155 | retry(lambda: initial_part in capturer.get_lines(partial=True)) 156 | sys.stderr.write("%s\n" % later_part) 157 | retry(lambda: later_part in capturer.get_lines(partial=True)) 158 | 159 | def test_non_interpreted_lines_capture(self): 160 | """Test that interpretation of special characters can be disabled.""" 161 | expected_output = random_string() 162 | with CaptureOutput() as capturer: 163 | print(expected_output) 164 | assert expected_output in capturer.get_lines(interpreted=False) 165 | 166 | def test_text_capture(self): 167 | """Test that capturing of all output as a single string is supported.""" 168 | expected_output = random_string() 169 | with CaptureOutput() as capturer: 170 | print(expected_output) 171 | assert expected_output in capturer.get_text() 172 | 173 | def test_save_to_path(self): 174 | """Test that captured output can be stored in a file.""" 175 | expected_output = random_string() 176 | with CaptureOutput() as capturer: 177 | print(expected_output) 178 | fd, temporary_file = tempfile.mkstemp() 179 | try: 180 | capturer.save_to_path(temporary_file) 181 | with open(temporary_file, 'r') as handle: 182 | assert expected_output in handle.read() 183 | finally: 184 | os.unlink(temporary_file) 185 | 186 | def test_unmerged_capture(self): 187 | """Test that standard output and error can be captured separately.""" 188 | expected_stdout = random_string() 189 | expected_stderr = random_string() 190 | with CaptureOutput(merged=False) as capturer: 191 | sys.stdout.write(expected_stdout + "\n") 192 | sys.stderr.write(expected_stderr + "\n") 193 | assert expected_stdout in capturer.stdout.get_lines() 194 | assert expected_stderr in capturer.stderr.get_lines() 195 | 196 | 197 | if __name__ == '__main__': 198 | unittest.main() 199 | -------------------------------------------------------------------------------- /docs/api.rst: -------------------------------------------------------------------------------- 1 | API documentation 2 | ================= 3 | 4 | The following API documentation was automatically generated from the source 5 | code of `capturer` |release|: 6 | 7 | .. contents:: 8 | :local: 9 | 10 | :mod:`capturer` 11 | --------------- 12 | 13 | .. automodule:: capturer 14 | :members: 15 | -------------------------------------------------------------------------------- /docs/changelog.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../CHANGELOG.rst 2 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # Easily capture stdout/stderr of the current process and subprocesses. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: March 7, 2020 5 | # URL: https://capturer.readthedocs.io 6 | 7 | """Sphinx documentation configuration for the `capturer` project.""" 8 | 9 | import os 10 | import sys 11 | 12 | # Add the capturer source distribution's root directory to the module path. 13 | sys.path.insert(0, os.path.abspath('..')) 14 | 15 | # -- General configuration ----------------------------------------------------- 16 | 17 | # Sphinx extension module names. 18 | extensions = [ 19 | 'sphinx.ext.autodoc', 20 | 'sphinx.ext.intersphinx', 21 | 'sphinx.ext.viewcode', 22 | 'humanfriendly.sphinx', 23 | ] 24 | 25 | # Paths that contain templates, relative to this directory. 26 | templates_path = ['templates'] 27 | 28 | # The suffix of source filenames. 29 | source_suffix = '.rst' 30 | 31 | # The master toctree document. 32 | master_doc = 'index' 33 | 34 | # General information about the project. 35 | project = 'capturer' 36 | copyright = '2020, Peter Odding' 37 | 38 | # The version info for the project you're documenting, acts as replacement for 39 | # |version| and |release|, also used in various other places throughout the 40 | # built documents. 41 | 42 | # Find the package version and make it the release. 43 | from capturer import __version__ as capturer_version # NOQA 44 | 45 | # The short X.Y version. 46 | version = '.'.join(capturer_version.split('.')[:2]) 47 | 48 | # The full version, including alpha/beta/rc tags. 49 | release = capturer_version 50 | 51 | # The language for content autogenerated by Sphinx. Refer to documentation 52 | # for a list of supported languages. 53 | language = 'en' 54 | 55 | # List of patterns, relative to source directory, that match files and 56 | # directories to ignore when looking for source files. 57 | exclude_patterns = ['build'] 58 | 59 | # If true, '()' will be appended to :func: etc. cross-reference text. 60 | add_function_parentheses = True 61 | 62 | # http://sphinx-doc.org/ext/autodoc.html#confval-autodoc_member_order 63 | autodoc_member_order = 'bysource' 64 | 65 | # The name of the Pygments (syntax highlighting) style to use. 66 | pygments_style = 'sphinx' 67 | 68 | # Refer to the Python standard library. 69 | # From: http://twistedmatrix.com/trac/ticket/4582. 70 | intersphinx_mapping = dict( 71 | humanfriendly=('https://humanfriendly.readthedocs.io/en/latest/', None), 72 | python=('https://docs.python.org/2/', None), 73 | ) 74 | 75 | # -- Options for HTML output --------------------------------------------------- 76 | 77 | # The theme to use for HTML and HTML Help pages. See the documentation for 78 | # a list of builtin themes. 79 | html_theme = 'nature' 80 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | capturer: Easily capture stdout/stderr of the current process and subprocesses 2 | ============================================================================== 3 | 4 | Welcome to the documentation of `capturer` version |release|! 5 | The following sections are available: 6 | 7 | .. contents:: 8 | :local: 9 | 10 | User documentation 11 | ------------------ 12 | 13 | The readme is the best place to start reading: 14 | 15 | .. toctree:: 16 | readme.rst 17 | 18 | API documentation 19 | ----------------- 20 | 21 | The following API documentation is automatically generated from the source code: 22 | 23 | .. toctree:: 24 | api.rst 25 | 26 | Change log 27 | ---------- 28 | 29 | The change log lists notable changes to the project: 30 | 31 | .. toctree:: 32 | changelog.rst 33 | -------------------------------------------------------------------------------- /docs/readme.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../README.rst 2 | -------------------------------------------------------------------------------- /requirements-checks.txt: -------------------------------------------------------------------------------- 1 | # Python packages required to run `make check'. 2 | flake8 >= 2.6.0 3 | flake8-docstrings >= 0.2.8 4 | pyflakes >= 1.2.3 5 | -------------------------------------------------------------------------------- /requirements-tests.txt: -------------------------------------------------------------------------------- 1 | coverage >= 4.2 2 | pytest >= 3.0.4 3 | pytest-cov >= 2.4.0 4 | -------------------------------------------------------------------------------- /requirements-travis.txt: -------------------------------------------------------------------------------- 1 | --requirement=requirements-checks.txt 2 | --requirement=requirements-tests.txt 3 | --requirement=requirements.txt 4 | coveralls 5 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | humanfriendly >= 8.0 2 | -------------------------------------------------------------------------------- /scripts/travis.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -e 2 | 3 | # Even though Travis CI supports Mac OS X [1] and several Python interpreters 4 | # are installed out of the box, the Python environment cannot be configured in 5 | # the Travis CI build configuration [2]. 6 | # 7 | # As a workaround the build configuration file specifies a single Mac OS X job 8 | # with `language: generic' that runs this script to create and activate a 9 | # Python virtual environment. 10 | # 11 | # Recently the `virtualenv' command seems to no longer come pre-installed on 12 | # the MacOS workers of Travis CI [3] so when this situation is detected we 13 | # install it ourselves. 14 | # 15 | # [1] https://github.com/travis-ci/travis-ci/issues/216 16 | # [2] https://github.com/travis-ci/travis-ci/issues/2312 17 | # [3] https://travis-ci.org/xolox/python-humanfriendly/jobs/411396506 18 | 19 | main () { 20 | if [ "$TRAVIS_OS_NAME" = osx ]; then 21 | local environment="$HOME/virtualenv/python2.7" 22 | if [ -x "$environment/bin/python" ]; then 23 | msg "Activating virtual environment ($environment) .." 24 | source "$environment/bin/activate" 25 | else 26 | if ! which virtualenv &>/dev/null; then 27 | msg "Installing 'virtualenv' in per-user site-packages .." 28 | pip install --user virtualenv 29 | msg "Figuring out 'bin' directory of per-user site-packages .." 30 | LOCAL_BINARIES=$(python -c 'import os, site; print(os.path.join(site.USER_BASE, "bin"))') 31 | msg "Prefixing '$LOCAL_BINARIES' to PATH .." 32 | export PATH="$LOCAL_BINARIES:$PATH" 33 | fi 34 | msg "Creating virtual environment ($environment) .." 35 | virtualenv "$environment" 36 | msg "Activating virtual environment ($environment) .." 37 | source "$environment/bin/activate" 38 | msg "Checking if 'pip' executable works .." 39 | if ! pip --version; then 40 | msg "Bootstrapping working 'pip' installation using get-pip.py .." 41 | curl -s https://bootstrap.pypa.io/get-pip.py | python - 42 | fi 43 | fi 44 | fi 45 | msg "Running command: $*" 46 | eval "$@" 47 | } 48 | 49 | msg () { 50 | echo "[travis.sh] $*" >&2 51 | } 52 | 53 | main "$@" 54 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | # Enable building of universal wheels so we can publish wheel 2 | # distribution archives to PyPI (the Python package index) 3 | # that are compatible with Python 2 as well as Python 3. 4 | 5 | [wheel] 6 | universal=1 7 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Setup script for the 'capturer' package. 4 | # 5 | # Author: Peter Odding 6 | # Last Change: March 7, 2020 7 | # URL: https://capturer.readthedocs.io 8 | 9 | """ 10 | Setup script for the `capturer` package. 11 | 12 | **python setup.py install** 13 | Install from the working directory into the current Python environment. 14 | 15 | **python setup.py sdist** 16 | Build a source distribution archive. 17 | 18 | **python setup.py bdist_wheel** 19 | Build a wheel distribution archive. 20 | """ 21 | 22 | # Standard library modules. 23 | import codecs 24 | import os 25 | import re 26 | 27 | # De-facto standard solution for Python packaging. 28 | from setuptools import find_packages, setup 29 | 30 | 31 | def get_contents(*args): 32 | """Get the contents of a file relative to the source distribution directory.""" 33 | with codecs.open(get_absolute_path(*args), "r", "UTF-8") as handle: 34 | return handle.read() 35 | 36 | 37 | def get_version(*args): 38 | """Extract the version number from a Python module.""" 39 | contents = get_contents(*args) 40 | metadata = dict(re.findall("__([a-z]+)__ = ['\"]([^'\"]+)", contents)) 41 | return metadata["version"] 42 | 43 | 44 | def get_requirements(*args): 45 | """Get requirements from pip requirement files.""" 46 | requirements = set() 47 | with open(get_absolute_path(*args)) as handle: 48 | for line in handle: 49 | # Strip comments. 50 | line = re.sub(r"^#.*|\s#.*", "", line) 51 | # Ignore empty lines 52 | if line and not line.isspace(): 53 | requirements.add(re.sub(r"\s+", "", line)) 54 | return sorted(requirements) 55 | 56 | 57 | def get_absolute_path(*args): 58 | """Transform relative pathnames into absolute pathnames.""" 59 | return os.path.join(os.path.dirname(os.path.abspath(__file__)), *args) 60 | 61 | 62 | setup( 63 | name="capturer", 64 | version=get_version("capturer", "__init__.py"), 65 | description="Easily capture stdout/stderr of the current process and subprocesses", 66 | long_description=get_contents("README.rst"), 67 | url="https://capturer.readthedocs.io", 68 | author="Peter Odding", 69 | author_email="peter@peterodding.com", 70 | license="MIT", 71 | packages=find_packages(), 72 | test_suite="capturer.tests", 73 | install_requires=get_requirements("requirements.txt"), 74 | python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*", 75 | classifiers=[ 76 | "Development Status :: 4 - Beta", 77 | "Environment :: Console", 78 | "Intended Audience :: Developers", 79 | "Intended Audience :: Information Technology", 80 | "Intended Audience :: System Administrators", 81 | "License :: OSI Approved :: MIT License", 82 | "Operating System :: POSIX", 83 | "Operating System :: Unix", 84 | "Natural Language :: English", 85 | "Programming Language :: Python", 86 | "Programming Language :: Python :: 2", 87 | "Programming Language :: Python :: 2.7", 88 | "Programming Language :: Python :: 3", 89 | "Programming Language :: Python :: 3.5", 90 | "Programming Language :: Python :: 3.6", 91 | "Programming Language :: Python :: 3.7", 92 | "Programming Language :: Python :: 3.8", 93 | "Programming Language :: Python :: Implementation :: CPython", 94 | "Programming Language :: Python :: Implementation :: PyPy", 95 | "Topic :: Communications", 96 | "Topic :: Scientific/Engineering :: Human Machine Interfaces", 97 | "Topic :: Software Development", 98 | "Topic :: Software Development :: Libraries", 99 | "Topic :: Software Development :: Libraries :: Python Modules", 100 | "Topic :: Software Development :: User Interfaces", 101 | "Topic :: System :: Shells", 102 | "Topic :: System :: System Shells", 103 | "Topic :: System :: Systems Administration", 104 | "Topic :: Terminals", 105 | "Topic :: Text Processing :: General", 106 | ], 107 | ) 108 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | # Tox (http://tox.testrun.org/) is a tool for running tests in multiple 2 | # virtualenvs. This configuration file will run the test suite on all supported 3 | # python versions. To use it, "pip install tox" and then run "tox" from this 4 | # directory. 5 | 6 | [tox] 7 | envlist = py27, py35, py36, py37, py38, pypy 8 | 9 | [testenv] 10 | deps = -rrequirements-tests.txt 11 | commands = py.test {posargs} 12 | 13 | [pytest] 14 | addopts = --verbose 15 | python_files = capturer/tests.py 16 | 17 | [flake8] 18 | exclude = .tox 19 | ignore = D205,D211,D400,D402 20 | max-line-length = 120 21 | --------------------------------------------------------------------------------