├── .appveyor.yml ├── .travis.yml ├── ChangeLog ├── Dockerfile ├── LICENSE ├── MANIFEST.in ├── README.md ├── _posixsubprocess.c ├── _posixsubprocess_config.h.in ├── _posixsubprocess_helpers.c ├── build-manylinux2014-wheel.sh ├── configure ├── configure.ac ├── python3_redirect └── __init__.py ├── setup.cfg ├── setup.py ├── subprocess32.py ├── test ├── test_subprocess32.py └── testdata ├── fd_status.py ├── input_reader.py ├── qcat.py ├── qgrep.py └── sigchild_ignore.py /.appveyor.yml: -------------------------------------------------------------------------------- 1 | environment: 2 | matrix: 3 | - PYTHON: "C:\\Python27-x64" 4 | PYTHON_VERSION: "2.7.x" 5 | PYTHON_ARCH: "64" 6 | 7 | install: 8 | - "python --version" 9 | 10 | build_script: 11 | - "python -m pip install ." 12 | 13 | test_script: 14 | - "python -m unittest discover -v" 15 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | os: linux 3 | # Each attribute in the matrix will override the global attribute. 4 | matrix: 5 | fast_finish: true 6 | # When MacOS gets to be too high latency or flaky, uncomment this. 7 | #allow_failures: 8 | # - os: osx 9 | include: 10 | - python: 2.7 11 | deploy: 12 | provider: pypi 13 | user: "GregoryPSmith" 14 | password: $TEST_PYPI_PASSWORD 15 | server: https://test.pypi.org/legacy/ 16 | on: 17 | tags: true 18 | branch: master 19 | # No wheels for Linux (manylinux1 is too old) 20 | distributions: sdist 21 | - python: 2.6 22 | # Python 2.6 does not have test discovery. 23 | script: python test_subprocess32.py 24 | env: 25 | - LANG=C 26 | - python: 3.5 27 | # Just check that the dummy shim installs and works on 3.x. 28 | script: | 29 | cd / # We cannot be in the directory with subprocess32.py in it. 30 | python -c 'import os, subprocess32, subprocess, sys; print("working directory:", os.getcwd()); print("subprocess32 module is", subprocess32); sys.exit(subprocess32 is not subprocess)' 31 | - os: osx 32 | # Travis does not have a Python runtime for MacOS. 33 | # https://pythonhosted.org/CodeChat/.travis.yml.html 34 | language: generic 35 | before_install: 36 | - pkgutil --pkgs | grep -F "org.python.Python.PythonFramework-2.7" || curl -L -o /tmp/Python.pkg https://www.python.org/ftp/python/2.7.15/python-2.7.15-macosx10.6.pkg && sudo installer -pkg /tmp/Python.pkg -target / 37 | - pkgutil --pkgs | grep -F "org.python.Python.PythonFramework" 38 | - export PATH="/Library/Frameworks/Python.framework/Versions/2.7/bin:${PATH}" 39 | - which pip2 40 | - pip2 install --upgrade pip 41 | - pip2 install --upgrade twine 42 | - which twine 43 | install: 44 | - pip2 -v install . 45 | deploy: 46 | provider: pypi 47 | user: "GregoryPSmith" 48 | password: $TEST_PYPI_PASSWORD 49 | server: https://test.pypi.org/legacy/ 50 | on: 51 | tags: true 52 | branch: master 53 | distributions: bdist_wheel 54 | install: 55 | - pip install . 56 | script: 57 | - which python ; python --version 58 | - python -m unittest discover -v 59 | -------------------------------------------------------------------------------- /ChangeLog: -------------------------------------------------------------------------------- 1 | ----------------- 2 | 2019-05-20 3.5.4 3 | ----------------- 4 | * Promoted 3.5.4rc2 to become 3.5.4. 5 | 6 | ----------------- 7 | 2019-05-13 3.5.4rc2 8 | ----------------- 9 | * GitHub [#57]: TimeoutExpired and CalledProcessError exceptions can now 10 | be unpickled. 11 | 12 | ----------------- 13 | 2019-05-10 3.5.4rc1 14 | ----------------- 15 | * GitHub [#61]: Backport the fixes for https://bugs.python.org/issue10963, 16 | https://bugs.python.org/issue19612, and https://bugs.python.org/issue30418. 17 | When a child process was so short lived it dies or closes its pipes before 18 | you call communicate(). When supplying stdin or reading its output in 19 | this scenario, do not raise an unexpected broken pipe or interrupted 20 | system call exception. 21 | 22 | ----------------- 23 | 2018-10-09 3.5.3 24 | ----------------- 25 | * Disallow ridiculously large numbers (millions) of arguments. [#54] 26 | 27 | ----------------- 28 | 2018-06-07 3.5.2 29 | ----------------- 30 | * Explicitly include in _posixsubprocess_helpers.c; it already 31 | gets configure checked and pulled in via Python's own in many 32 | circumstances but it is better to be explicit. #IWYU 33 | If you were using subprocess32 on a Python interpreter built *without* 34 | the --with-fpectl configure option enabled, restore_signals is now 35 | useful rather than a no-op. I do not know if such builds were common. 36 | * Adds a functional test for restore_signals=True behavior. 37 | 38 | ----------------- 39 | 2018-05-21 3.5.1 40 | ----------------- 41 | * Fix AttributeError: 'module' object has no attribute 'Signals' when 42 | constructing a CalledProcessError exception. [#49] 43 | 44 | ----------------- 45 | 2018-05-13 3.5.0 (3.5.0rc3) 46 | ----------------- 47 | 48 | * Fixed the packaging problem where the stdlib python3_redirect shim is 49 | supposed to be installed on Python 3. 50 | * Renamed _posixsubprocess to _posixsubprocess32 for consistency. 51 | * Unset CLOEXEC on file descriptors given to Popen pass_fds. (GH #4) 52 | * Drop support for Python 2.4 and 2.5. 53 | * Adds a configure script - run by setup.py - to supply the missing feature 54 | #define's for the _posixsubprocess32 extension module for the things that 55 | Python 2's own configure generated pyconfig.h does not already provide. 56 | 57 | ----------------- 58 | 2017-10-18 3.5.0rc1 59 | ----------------- 60 | 61 | * Backport the new subprocess.run() API from Python 3.5. 62 | * Backport subprocess.DEVNULL support from 3.3. 63 | * Allow stderr to be redirected to stdout even when stdout is not redirected. 64 | https://bugs.python.org/issue22274 65 | * Fix subprocess.Popen.wait() when the child process has exited to a 66 | a stopped instead of terminated state (ex: when under ptrace). 67 | https://bugs.python.org/issue29335 68 | * Include the private API needed by the multiprocessing module for people who 69 | want to drop subprocess32 in as a replacement for their standard library 70 | subprocess module. 71 | * Fix a compilation issue regarding O_CLOEXEC not being defined on ancient 72 | Linux distros such as RHEL 5. 73 | 74 | ----------------- 75 | 2015-11-15 3.2.7 76 | ----------------- 77 | 78 | * Issue #6973: When we know a subprocess.Popen process has died, do 79 | not allow the send_signal(), terminate(), or kill() methods to do 80 | anything as they could potentially signal a different process. 81 | * Issue #23564: Fixed a partially broken sanity check in the _posixsubprocess 82 | internals regarding how fds_to_pass were passed to the child. The bug had 83 | no actual impact as subprocess32.py already avoided it. 84 | 85 | ----------------- 86 | 2015-11-14 3.2.7rc2 87 | ----------------- 88 | 89 | * Moved the repository from code.google.com to github. 90 | * Added a _WAIT_TIMEOUT to satisfy the unsupported people entirely on 91 | their own trying to use this on Windows. 92 | * Similarly: Updated setup.py to not build the extension on non-posix. 93 | 94 | ----------------- 95 | 2014-06-01 3.2.7rc1 96 | ----------------- 97 | 98 | * Issue #21618: The subprocess module could fail to close open fds that were 99 | inherited by the calling process and already higher than POSIX resource 100 | limits would otherwise allow. On systems with a functioning /proc/self/fd 101 | or /dev/fd interface the max is now ignored and all fds are closed. 102 | 103 | ----------------- 104 | 2014-04-23 3.2.6 105 | ----------------- 106 | 107 | * Fixes issue #21291: Popen.wait() is now thread safe so that multiple 108 | threads may be calling wait() or poll() on a Popen instance at the same time 109 | without losing the Popen.returncode value. 110 | * Fixes issue #14396: Handle the odd rare case of waitpid returning 0 when not 111 | expected in Popen.wait(). 112 | * Fixes issue #16962: Use getdents64 instead of the obsolete getdents syscall 113 | on Linux. Some architectures do not implement the latter. 114 | 115 | ----------------- 116 | 2013-12-10 3.2.5 117 | ----------------- 118 | 119 | * Fixes issue #15798: subprocess.Popen() no longer fails if file 120 | descriptor 0, 1 or 2 is closed. 121 | * Fixes issue #18763: close_fd file descriptors are now closed after 122 | any preexec_fn call. 123 | 124 | ----------------- 125 | 2013-06-15 3.2.5rc1 126 | ----------------- 127 | 128 | * Fixes issue #16650 - Don't reference ECHILD from outside the local scope. 129 | * Unittests no longer spew any test data for human verification to stdout. 130 | * Remove a bare print to stdout that could have happened if the child process 131 | wrote garbage to its pre-exec error pipe. 132 | * Fixes issue #16327 - the subprocess module no longer leaks file descriptors 133 | used for stdin/stdout/stderr pipes to the child when the fork() fails. It 134 | also no longer potentially double closes these pipe fds. 135 | * Correct the Python version check around use of imp_module to specify 2.6.3 136 | as the minimum version that exists in. Why is anyone using such an old 2.6? 137 | * Fixes Issue #16114: The subprocess module no longer provides a misleading 138 | error message stating that args[0] did not exist when either the cwd or 139 | executable keyword arguments specified a path that did not exist. 140 | * Add more Popen cwd tests. 141 | * Handle errno.ECHILD in poll. 142 | * Don't leak a reference to the gc module on capi use error. 143 | * Check return value to avoid a crash if the capi were misused. 144 | * Check result of PyObject_IsTrue(). 145 | * Adds test_universal_newlines_communicate_input_none. 146 | * Most everything above consists of backports. See the hg logs for their 147 | upstream hg.python.org cpython revision numbers. 148 | 149 | ---------------- 150 | 2012-06-10 3.2.3 151 | ---------------- 152 | 153 | * Fixes the references to the 'surrogateescape' unicode encoding error 154 | handler that does not exist in Python 2.x. 'strict' is used so that 155 | a UnicodeEncodeError exception is raised in these situations. These 156 | MAY occur if your sys.getfilesystemencoding() is not UTF-8 and 157 | attempt to use a non-ascii executable, args or env values. Prior to 158 | this change, those would result in a hard to debug LookupError for 159 | surrogateescape. 160 | * Issue #15000: Support the "unique" x32 architecture in _posixsubprocess.c. 161 | * Fix a compilation problem when O_CLOEXEC is not defined. 162 | 163 | ------------------ 164 | 2012-02-18 3.2.3b1 165 | ------------------ 166 | 167 | This release brings in the last year and a half's worth of bugfixes and 168 | improvements to Python 3.2's subprocess module: 169 | 170 | Off the top of my head, some major bugfix highlights include: 171 | * Timeout support on the APIs. 172 | * close_fds=True is now the default (as it is in 3.2) and performs much faster. 173 | * Fixed EINTR handling. 174 | * Fixed SIGCHLD handling. 175 | * Fixed several race conditions. 176 | * Many more bug fixes too numerous to list. 177 | 178 | You can grep out the full list of improvements related to subprocess in: 179 | http://hg.python.org/cpython/file/9ce5d456138b/Misc/NEWS 180 | 181 | ------------- 182 | 2010-06 3.2.0 183 | ------------- 184 | 185 | This was the first release. Roughly equivalent to Python 3.2.0a1. 186 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM quay.io/pypa/manylinux2014_x86_64 2 | # Python 2.7.5 exists in this image, but not pip. Fix that. 3 | RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py 4 | RUN python2 get-pip.py 5 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | A. HISTORY OF THE SOFTWARE 2 | ========================== 3 | 4 | Python was created in the early 1990s by Guido van Rossum at Stichting 5 | Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands 6 | as a successor of a language called ABC. Guido remains Python's 7 | principal author, although it includes many contributions from others. 8 | 9 | In 1995, Guido continued his work on Python at the Corporation for 10 | National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) 11 | in Reston, Virginia where he released several versions of the 12 | software. 13 | 14 | In May 2000, Guido and the Python core development team moved to 15 | BeOpen.com to form the BeOpen PythonLabs team. In October of the same 16 | year, the PythonLabs team moved to Digital Creations (now Zope 17 | Corporation, see http://www.zope.com). In 2001, the Python Software 18 | Foundation (PSF, see http://www.python.org/psf/) was formed, a 19 | non-profit organization created specifically to own Python-related 20 | Intellectual Property. Zope Corporation is a sponsoring member of 21 | the PSF. 22 | 23 | All Python releases are Open Source (see http://www.opensource.org for 24 | the Open Source Definition). Historically, most, but not all, Python 25 | releases have also been GPL-compatible; the table below summarizes 26 | the various releases. 27 | 28 | Release Derived Year Owner GPL- 29 | from compatible? (1) 30 | 31 | 0.9.0 thru 1.2 1991-1995 CWI yes 32 | 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes 33 | 1.6 1.5.2 2000 CNRI no 34 | 2.0 1.6 2000 BeOpen.com no 35 | 1.6.1 1.6 2001 CNRI yes (2) 36 | 2.1 2.0+1.6.1 2001 PSF no 37 | 2.0.1 2.0+1.6.1 2001 PSF yes 38 | 2.1.1 2.1+2.0.1 2001 PSF yes 39 | 2.2 2.1.1 2001 PSF yes 40 | 2.1.2 2.1.1 2002 PSF yes 41 | 2.1.3 2.1.2 2002 PSF yes 42 | 2.2.1 2.2 2002 PSF yes 43 | 2.2.2 2.2.1 2002 PSF yes 44 | 2.2.3 2.2.2 2003 PSF yes 45 | 2.3 2.2.2 2002-2003 PSF yes 46 | 2.3.1 2.3 2002-2003 PSF yes 47 | 2.3.2 2.3.1 2002-2003 PSF yes 48 | 2.3.3 2.3.2 2002-2003 PSF yes 49 | 2.3.4 2.3.3 2004 PSF yes 50 | 2.3.5 2.3.4 2005 PSF yes 51 | 2.4 2.3 2004 PSF yes 52 | 2.4.1 2.4 2005 PSF yes 53 | 2.4.2 2.4.1 2005 PSF yes 54 | 2.4.3 2.4.2 2006 PSF yes 55 | 2.4.4 2.4.3 2006 PSF yes 56 | 2.5 2.4 2006 PSF yes 57 | 2.5.1 2.5 2007 PSF yes 58 | 2.5.2 2.5.1 2008 PSF yes 59 | 2.5.3 2.5.2 2008 PSF yes 60 | 2.6 2.5 2008 PSF yes 61 | 2.6.1 2.6 2008 PSF yes 62 | 2.6.2 2.6.1 2009 PSF yes 63 | 2.6.3 2.6.2 2009 PSF yes 64 | 2.6.4 2.6.3 2009 PSF yes 65 | 2.6.5 2.6.4 2010 PSF yes 66 | 3.0 2.6 2008 PSF yes 67 | 3.0.1 3.0 2009 PSF yes 68 | 3.1 3.0.1 2009 PSF yes 69 | 3.1.1 3.1 2009 PSF yes 70 | 3.1.2 3.1 2010 PSF yes 71 | 72 | Footnotes: 73 | 74 | (1) GPL-compatible doesn't mean that we're distributing Python under 75 | the GPL. All Python licenses, unlike the GPL, let you distribute 76 | a modified version without making your changes open source. The 77 | GPL-compatible licenses make it possible to combine Python with 78 | other software that is released under the GPL; the others don't. 79 | 80 | (2) According to Richard Stallman, 1.6.1 is not GPL-compatible, 81 | because its license has a choice of law clause. According to 82 | CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1 83 | is "not incompatible" with the GPL. 84 | 85 | Thanks to the many outside volunteers who have worked under Guido's 86 | direction to make these releases possible. 87 | 88 | 89 | B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON 90 | =============================================================== 91 | 92 | PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 93 | -------------------------------------------- 94 | 95 | 1. This LICENSE AGREEMENT is between the Python Software Foundation 96 | ("PSF"), and the Individual or Organization ("Licensee") accessing and 97 | otherwise using this software ("Python") in source or binary form and 98 | its associated documentation. 99 | 100 | 2. Subject to the terms and conditions of this License Agreement, PSF hereby 101 | grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, 102 | analyze, test, perform and/or display publicly, prepare derivative works, 103 | distribute, and otherwise use Python alone or in any derivative version, 104 | provided, however, that PSF's License Agreement and PSF's notice of copyright, 105 | i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 106 | Python Software Foundation; All Rights Reserved" are retained in Python alone or 107 | in any derivative version prepared by Licensee. 108 | 109 | 3. In the event Licensee prepares a derivative work that is based on 110 | or incorporates Python or any part thereof, and wants to make 111 | the derivative work available to others as provided herein, then 112 | Licensee hereby agrees to include in any such work a brief summary of 113 | the changes made to Python. 114 | 115 | 4. PSF is making Python available to Licensee on an "AS IS" 116 | basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR 117 | IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND 118 | DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS 119 | FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT 120 | INFRINGE ANY THIRD PARTY RIGHTS. 121 | 122 | 5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON 123 | FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS 124 | A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, 125 | OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 126 | 127 | 6. This License Agreement will automatically terminate upon a material 128 | breach of its terms and conditions. 129 | 130 | 7. Nothing in this License Agreement shall be deemed to create any 131 | relationship of agency, partnership, or joint venture between PSF and 132 | Licensee. This License Agreement does not grant permission to use PSF 133 | trademarks or trade name in a trademark sense to endorse or promote 134 | products or services of Licensee, or any third party. 135 | 136 | 8. By copying, installing or otherwise using Python, Licensee 137 | agrees to be bound by the terms and conditions of this License 138 | Agreement. 139 | 140 | 141 | BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0 142 | ------------------------------------------- 143 | 144 | BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1 145 | 146 | 1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an 147 | office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the 148 | Individual or Organization ("Licensee") accessing and otherwise using 149 | this software in source or binary form and its associated 150 | documentation ("the Software"). 151 | 152 | 2. Subject to the terms and conditions of this BeOpen Python License 153 | Agreement, BeOpen hereby grants Licensee a non-exclusive, 154 | royalty-free, world-wide license to reproduce, analyze, test, perform 155 | and/or display publicly, prepare derivative works, distribute, and 156 | otherwise use the Software alone or in any derivative version, 157 | provided, however, that the BeOpen Python License is retained in the 158 | Software, alone or in any derivative version prepared by Licensee. 159 | 160 | 3. BeOpen is making the Software available to Licensee on an "AS IS" 161 | basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR 162 | IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND 163 | DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS 164 | FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT 165 | INFRINGE ANY THIRD PARTY RIGHTS. 166 | 167 | 4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE 168 | SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS 169 | AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY 170 | DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 171 | 172 | 5. This License Agreement will automatically terminate upon a material 173 | breach of its terms and conditions. 174 | 175 | 6. This License Agreement shall be governed by and interpreted in all 176 | respects by the law of the State of California, excluding conflict of 177 | law provisions. Nothing in this License Agreement shall be deemed to 178 | create any relationship of agency, partnership, or joint venture 179 | between BeOpen and Licensee. This License Agreement does not grant 180 | permission to use BeOpen trademarks or trade names in a trademark 181 | sense to endorse or promote products or services of Licensee, or any 182 | third party. As an exception, the "BeOpen Python" logos available at 183 | http://www.pythonlabs.com/logos.html may be used according to the 184 | permissions granted on that web page. 185 | 186 | 7. By copying, installing or otherwise using the software, Licensee 187 | agrees to be bound by the terms and conditions of this License 188 | Agreement. 189 | 190 | 191 | CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1 192 | --------------------------------------- 193 | 194 | 1. This LICENSE AGREEMENT is between the Corporation for National 195 | Research Initiatives, having an office at 1895 Preston White Drive, 196 | Reston, VA 20191 ("CNRI"), and the Individual or Organization 197 | ("Licensee") accessing and otherwise using Python 1.6.1 software in 198 | source or binary form and its associated documentation. 199 | 200 | 2. Subject to the terms and conditions of this License Agreement, CNRI 201 | hereby grants Licensee a nonexclusive, royalty-free, world-wide 202 | license to reproduce, analyze, test, perform and/or display publicly, 203 | prepare derivative works, distribute, and otherwise use Python 1.6.1 204 | alone or in any derivative version, provided, however, that CNRI's 205 | License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) 206 | 1995-2001 Corporation for National Research Initiatives; All Rights 207 | Reserved" are retained in Python 1.6.1 alone or in any derivative 208 | version prepared by Licensee. Alternately, in lieu of CNRI's License 209 | Agreement, Licensee may substitute the following text (omitting the 210 | quotes): "Python 1.6.1 is made available subject to the terms and 211 | conditions in CNRI's License Agreement. This Agreement together with 212 | Python 1.6.1 may be located on the Internet using the following 213 | unique, persistent identifier (known as a handle): 1895.22/1013. This 214 | Agreement may also be obtained from a proxy server on the Internet 215 | using the following URL: http://hdl.handle.net/1895.22/1013". 216 | 217 | 3. In the event Licensee prepares a derivative work that is based on 218 | or incorporates Python 1.6.1 or any part thereof, and wants to make 219 | the derivative work available to others as provided herein, then 220 | Licensee hereby agrees to include in any such work a brief summary of 221 | the changes made to Python 1.6.1. 222 | 223 | 4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" 224 | basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR 225 | IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND 226 | DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS 227 | FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT 228 | INFRINGE ANY THIRD PARTY RIGHTS. 229 | 230 | 5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON 231 | 1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS 232 | A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, 233 | OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 234 | 235 | 6. This License Agreement will automatically terminate upon a material 236 | breach of its terms and conditions. 237 | 238 | 7. This License Agreement shall be governed by the federal 239 | intellectual property law of the United States, including without 240 | limitation the federal copyright law, and, to the extent such 241 | U.S. federal law does not apply, by the law of the Commonwealth of 242 | Virginia, excluding Virginia's conflict of law provisions. 243 | Notwithstanding the foregoing, with regard to derivative works based 244 | on Python 1.6.1 that incorporate non-separable material that was 245 | previously distributed under the GNU General Public License (GPL), the 246 | law of the Commonwealth of Virginia shall govern this License 247 | Agreement only as to issues arising under or with respect to 248 | Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this 249 | License Agreement shall be deemed to create any relationship of 250 | agency, partnership, or joint venture between CNRI and Licensee. This 251 | License Agreement does not grant permission to use CNRI trademarks or 252 | trade name in a trademark sense to endorse or promote products or 253 | services of Licensee, or any third party. 254 | 255 | 8. By clicking on the "ACCEPT" button where indicated, or by copying, 256 | installing or otherwise using Python 1.6.1, Licensee agrees to be 257 | bound by the terms and conditions of this License Agreement. 258 | 259 | ACCEPT 260 | 261 | 262 | CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 263 | -------------------------------------------------- 264 | 265 | Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, 266 | The Netherlands. All rights reserved. 267 | 268 | Permission to use, copy, modify, and distribute this software and its 269 | documentation for any purpose and without fee is hereby granted, 270 | provided that the above copyright notice appear in all copies and that 271 | both that copyright notice and this permission notice appear in 272 | supporting documentation, and that the name of Stichting Mathematisch 273 | Centrum or CWI not be used in advertising or publicity pertaining to 274 | distribution of the software without specific, written prior 275 | permission. 276 | 277 | STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO 278 | THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND 279 | FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE 280 | FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 281 | WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 282 | ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT 283 | OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 284 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include *.md 2 | include LICENSE 3 | include setup.py 4 | include setup.cfg 5 | include ChangeLog 6 | include MANIFEST.in 7 | 8 | include configure.ac configure *.h.in 9 | 10 | include *.c *.py 11 | include testdata/* 12 | include python3_redirect/* 13 | 14 | prune build 15 | prune dist 16 | prune config.* 17 | prune *.pyc 18 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | subprocess32 2 | ------------ 3 | [![PyPI version](https://badge.fury.io/py/subprocess32.svg)](https://badge.fury.io/py/subprocess32) 4 | [![POSIX Build Status](https://travis-ci.org/google/python-subprocess32.svg?branch=master)](https://travis-ci.org/google/python-subprocess32) 5 | [![Windows Build Status](https://ci.appveyor.com/api/projects/status/53apbb2jk1uslj0m?svg=true 6 | )](https://ci.appveyor.com/project/gpshead/python-subprocess32) 7 | 8 | This is a backport of the Python 3 subprocess module for use on Python 2. 9 | This code has not been tested on Windows or other non-POSIX platforms. 10 | 11 | subprocess32 includes many important reliability bug fixes relevant on 12 | POSIX platforms. The most important of which is a C extension module 13 | used internally to handle the code path between fork() and exec(). 14 | This module is reliable when an application is using threads. 15 | 16 | Refer to the 17 | [Python 3.5 subprocess documentation](https://docs.python.org/3.5/library/subprocess.html) 18 | for usage information. 19 | 20 | * Timeout support backported from Python 3.3 is included. 21 | * The run() API from Python 3.5 was backported in subprocess32 3.5.0. 22 | * Otherwise features are frozen at the 3.2 level. 23 | 24 | Usage 25 | ----- 26 | 27 | The recommend pattern for cross platform code is to use the following: 28 | 29 | ```python 30 | if os.name == 'posix' and sys.version_info[0] < 3: 31 | import subprocess32 as subprocess 32 | else: 33 | import subprocess 34 | ``` 35 | 36 | Or if you fully control your POSIX Python 2.7 installation, this can serve 37 | as a replacement for its subprocess module. Users will thank you by not 38 | filing concurrency bugs. 39 | 40 | Got Bugs? 41 | --------- 42 | 43 | Try to reproduce them on the latest Python 3.x itself and file bug 44 | reports on [bugs.python.org](https://bugs.python.org/). 45 | Add gregory.p.smith to the Nosy list. 46 | 47 | If you have can prove that the issue is specifically with this backport and not 48 | a problem in Python 3 itself, feel free to use the github issue tracker for 49 | others to collaborate with you on a fix, but do not expect any support. 50 | 51 | Python 2 has reached EOL, as has this project. 52 | 53 | -- Gregory P. Smith _greg@krypto.org_ [Google LLC] 54 | -------------------------------------------------------------------------------- /_posixsubprocess.c: -------------------------------------------------------------------------------- 1 | /* Authors: Gregory P. Smith & Jeffrey Yasskin */ 2 | 3 | /* We use our own small autoconf to fill in for things that were not checked 4 | * for in Python 2's configure and thus pyconfig.h. 5 | * 6 | * This comes before Python.h on purpose. 2.7's Python.h redefines critical 7 | * defines such as _POSIX_C_SOURCE with undesirable old values impacting system 8 | * which header defines are available. 9 | */ 10 | #include "_posixsubprocess_config.h" 11 | #ifdef HAVE_SYS_CDEFS_H 12 | #include 13 | #endif 14 | 15 | #define PY_SSIZE_T_CLEAN 16 | #include "Python.h" 17 | 18 | #include 19 | #include 20 | #ifdef HAVE_SYS_TYPES_H 21 | #include 22 | #endif 23 | #if defined(HAVE_SYS_STAT_H) && defined(__FreeBSD__) 24 | #include 25 | #endif 26 | #ifdef HAVE_SYS_SYSCALL_H 27 | #include 28 | #endif 29 | #ifdef HAVE_DIRENT_H 30 | #include 31 | #endif 32 | 33 | /* TODO: Some platform conditions below could move into configure.ac. */ 34 | 35 | #if defined(__ANDROID__) && !defined(SYS_getdents64) 36 | /* Android doesn't expose syscalls, add the definition manually. */ 37 | # include 38 | # define SYS_getdents64 __NR_getdents64 39 | #endif 40 | 41 | #include "_posixsubprocess_helpers.c" 42 | 43 | #if (PY_VERSION_HEX < 0x02060300) 44 | /* These are not public API fuctions until 2.6.3. */ 45 | static void _PyImport_AcquireLock(void); 46 | static int _PyImport_ReleaseLock(void); 47 | #endif 48 | 49 | #if defined(sun) 50 | /* readdir64 is used to work around Solaris 9 bug 6395699. */ 51 | # define readdir readdir64 52 | # define dirent dirent64 53 | # if !defined(HAVE_DIRFD) 54 | /* Some versions of Solaris lack dirfd(). */ 55 | # define dirfd(dirp) ((dirp)->dd_fd) 56 | # define HAVE_DIRFD 57 | # endif 58 | #endif 59 | 60 | #if defined(__FreeBSD__) || (defined(__APPLE__) && defined(__MACH__)) 61 | # define FD_DIR "/dev/fd" 62 | #else 63 | # define FD_DIR "/proc/self/fd" 64 | #endif 65 | 66 | #define POSIX_CALL(call) if ((call) == -1) goto error 67 | 68 | 69 | /* Given the gc module call gc.enable() and return 0 on success. */ 70 | static int 71 | _enable_gc(PyObject *gc_module) 72 | { 73 | PyObject *result; 74 | result = PyObject_CallMethod(gc_module, "enable", NULL); 75 | if (result == NULL) 76 | return 1; 77 | Py_DECREF(result); 78 | return 0; 79 | } 80 | 81 | 82 | /* Convert ASCII to a positive int, no libc call. no overflow. -1 on error. */ 83 | static int 84 | _pos_int_from_ascii(char *name) 85 | { 86 | int num = 0; 87 | while (*name >= '0' && *name <= '9') { 88 | num = num * 10 + (*name - '0'); 89 | ++name; 90 | } 91 | if (*name) 92 | return -1; /* Non digit found, not a number. */ 93 | return num; 94 | } 95 | 96 | 97 | #if defined(__FreeBSD__) 98 | /* When /dev/fd isn't mounted it is often a static directory populated 99 | * with 0 1 2 or entries for 0 .. 63 on FreeBSD, NetBSD and OpenBSD. 100 | * NetBSD and OpenBSD have a /proc fs available (though not necessarily 101 | * mounted) and do not have fdescfs for /dev/fd. MacOS X has a devfs 102 | * that properly supports /dev/fd. 103 | */ 104 | static int 105 | _is_fdescfs_mounted_on_dev_fd() 106 | { 107 | struct stat dev_stat; 108 | struct stat dev_fd_stat; 109 | if (stat("/dev", &dev_stat) != 0) 110 | return 0; 111 | if (stat(FD_DIR, &dev_fd_stat) != 0) 112 | return 0; 113 | if (dev_stat.st_dev == dev_fd_stat.st_dev) 114 | return 0; /* / == /dev == /dev/fd means it is static. #fail */ 115 | return 1; 116 | } 117 | #endif 118 | 119 | 120 | /* Returns 1 if there is a problem with fd_sequence, 0 otherwise. */ 121 | static int 122 | _sanity_check_python_fd_sequence(PyObject *fd_sequence) 123 | { 124 | Py_ssize_t seq_idx, seq_len = PySequence_Length(fd_sequence); 125 | long prev_fd = -1; 126 | for (seq_idx = 0; seq_idx < seq_len; ++seq_idx) { 127 | PyObject* py_fd = PySequence_Fast_GET_ITEM(fd_sequence, seq_idx); 128 | long iter_fd = PyLong_AsLong(py_fd); 129 | if (iter_fd < 0 || iter_fd <= prev_fd || iter_fd > INT_MAX) { 130 | /* Negative, overflow, not a Long, unsorted, too big for a fd. */ 131 | return 1; 132 | } 133 | prev_fd = iter_fd; 134 | } 135 | return 0; 136 | } 137 | 138 | 139 | /* Is fd found in the sorted Python Sequence? */ 140 | static int 141 | _is_fd_in_sorted_fd_sequence(int fd, PyObject *fd_sequence) 142 | { 143 | /* Binary search. */ 144 | Py_ssize_t search_min = 0; 145 | Py_ssize_t search_max = PySequence_Length(fd_sequence) - 1; 146 | if (search_max < 0) 147 | return 0; 148 | do { 149 | long middle = (search_min + search_max) / 2; 150 | long middle_fd = PyLong_AsLong( 151 | PySequence_Fast_GET_ITEM(fd_sequence, middle)); 152 | if (fd == middle_fd) 153 | return 1; 154 | if (fd > middle_fd) 155 | search_min = middle + 1; 156 | else 157 | search_max = middle - 1; 158 | } while (search_min <= search_max); 159 | return 0; 160 | } 161 | 162 | 163 | /* Get the maximum file descriptor that could be opened by this process. 164 | * This function is async signal safe for use between fork() and exec(). 165 | */ 166 | static long 167 | safe_get_max_fd(void) 168 | { 169 | long local_max_fd; 170 | #if defined(__NetBSD__) 171 | local_max_fd = fcntl(0, F_MAXFD); 172 | if (local_max_fd >= 0) 173 | return local_max_fd; 174 | #endif 175 | #ifdef _SC_OPEN_MAX 176 | local_max_fd = sysconf(_SC_OPEN_MAX); 177 | if (local_max_fd == -1) 178 | #endif 179 | local_max_fd = 256; /* Matches legacy Lib/subprocess.py behavior. */ 180 | return local_max_fd; 181 | } 182 | 183 | /* While uncommon in Python 2 applications, this makes sure the 184 | * close on exec flag is unset on the subprocess32.Popen pass_fds. 185 | * https://github.com/google/python-subprocess32/issues/4. 186 | */ 187 | static void 188 | _unset_cloexec_on_fds(PyObject *py_fds_to_keep, int errpipe_write) 189 | { 190 | #ifdef FD_CLOEXEC 191 | Py_ssize_t num_fds_to_keep = PySequence_Length(py_fds_to_keep); 192 | Py_ssize_t keep_seq_idx; 193 | /* As py_fds_to_keep is sorted we can loop through the list closing 194 | * fds inbetween any in the keep list falling within our range. */ 195 | for (keep_seq_idx = 0; keep_seq_idx < num_fds_to_keep; ++keep_seq_idx) { 196 | PyObject* py_keep_fd = PySequence_Fast_GET_ITEM(py_fds_to_keep, 197 | keep_seq_idx); 198 | // We just keep going on errors below, there is nothing we can 199 | // usefully do to report them. This is best effort. 200 | long fd = PyLong_AsLong(py_keep_fd); 201 | if (fd < 0) continue; 202 | if (fd == errpipe_write) continue; // This one keeps its CLOEXEC. 203 | // We could use ioctl FIONCLEX, but that is a more modern API 204 | // not available everywhere and we are a single threaded child. 205 | int old_flags = fcntl(fd, F_GETFD); 206 | if (old_flags != -1) { 207 | fcntl(fd, F_SETFD, old_flags & ~FD_CLOEXEC); 208 | } 209 | } 210 | #endif 211 | } 212 | 213 | /* Close all file descriptors in the range from start_fd and higher 214 | * except for those in py_fds_to_keep. If the range defined by 215 | * [start_fd, safe_get_max_fd()) is large this will take a long 216 | * time as it calls close() on EVERY possible fd. 217 | * 218 | * It isn't possible to know for sure what the max fd to go up to 219 | * is for processes with the capability of raising their maximum. 220 | */ 221 | static void 222 | _close_fds_by_brute_force(long start_fd, PyObject *py_fds_to_keep) 223 | { 224 | long end_fd = safe_get_max_fd(); 225 | Py_ssize_t num_fds_to_keep = PySequence_Length(py_fds_to_keep); 226 | Py_ssize_t keep_seq_idx; 227 | int fd_num; 228 | /* As py_fds_to_keep is sorted we can loop through the list closing 229 | * fds inbetween any in the keep list falling within our range. */ 230 | for (keep_seq_idx = 0; keep_seq_idx < num_fds_to_keep; ++keep_seq_idx) { 231 | PyObject* py_keep_fd = PySequence_Fast_GET_ITEM(py_fds_to_keep, 232 | keep_seq_idx); 233 | int keep_fd = PyLong_AsLong(py_keep_fd); 234 | if (keep_fd < start_fd) 235 | continue; 236 | for (fd_num = start_fd; fd_num < keep_fd; ++fd_num) { 237 | while (close(fd_num) < 0 && errno == EINTR); 238 | } 239 | start_fd = keep_fd + 1; 240 | } 241 | if (start_fd <= end_fd) { 242 | for (fd_num = start_fd; fd_num < end_fd; ++fd_num) { 243 | while (close(fd_num) < 0 && errno == EINTR); 244 | } 245 | } 246 | } 247 | 248 | 249 | #if defined(__linux__) && defined(HAVE_SYS_SYSCALL_H) 250 | /* It doesn't matter if d_name has room for NAME_MAX chars; we're using this 251 | * only to read a directory of short file descriptor number names. The kernel 252 | * will return an error if we didn't give it enough space. Highly Unlikely. 253 | * This structure is very old and stable: It will not change unless the kernel 254 | * chooses to break compatibility with all existing binaries. Highly Unlikely. 255 | */ 256 | struct linux_dirent64 { 257 | unsigned long long d_ino; 258 | long long d_off; 259 | unsigned short d_reclen; /* Length of this linux_dirent */ 260 | unsigned char d_type; 261 | char d_name[256]; /* Filename (null-terminated) */ 262 | }; 263 | 264 | /* Close all open file descriptors in the range from start_fd and higher 265 | * Do not close any in the sorted py_fds_to_keep list. 266 | * 267 | * This version is async signal safe as it does not make any unsafe C library 268 | * calls, malloc calls or handle any locks. It is _unfortunate_ to be forced 269 | * to resort to making a kernel system call directly but this is the ONLY api 270 | * available that does no harm. opendir/readdir/closedir perform memory 271 | * allocation and locking so while they usually work they are not guaranteed 272 | * to (especially if you have replaced your malloc implementation). A version 273 | * of this function that uses those can be found in the _maybe_unsafe variant. 274 | * 275 | * This is Linux specific because that is all I am ready to test it on. It 276 | * should be easy to add OS specific dirent or dirent64 structures and modify 277 | * it with some cpp #define magic to work on other OSes as well if you want. 278 | */ 279 | static void 280 | _close_open_fds_safe(int start_fd, PyObject* py_fds_to_keep) 281 | { 282 | int fd_dir_fd; 283 | #ifdef O_CLOEXEC 284 | fd_dir_fd = open(FD_DIR, O_RDONLY | O_CLOEXEC, 0); 285 | #else 286 | fd_dir_fd = open(FD_DIR, O_RDONLY, 0); 287 | #ifdef FD_CLOEXEC 288 | { 289 | int old = fcntl(fd_dir_fd, F_GETFD); 290 | if (old != -1) 291 | fcntl(fd_dir_fd, F_SETFD, old | FD_CLOEXEC); 292 | } 293 | #endif 294 | #endif 295 | if (fd_dir_fd == -1) { 296 | /* No way to get a list of open fds. */ 297 | _close_fds_by_brute_force(start_fd, py_fds_to_keep); 298 | return; 299 | } else { 300 | char buffer[sizeof(struct linux_dirent64)]; 301 | int bytes; 302 | while ((bytes = syscall(SYS_getdents64, fd_dir_fd, 303 | (struct linux_dirent64 *)buffer, 304 | sizeof(buffer))) > 0) { 305 | struct linux_dirent64 *entry; 306 | int offset; 307 | for (offset = 0; offset < bytes; offset += entry->d_reclen) { 308 | int fd; 309 | entry = (struct linux_dirent64 *)(buffer + offset); 310 | if ((fd = _pos_int_from_ascii(entry->d_name)) < 0) 311 | continue; /* Not a number. */ 312 | if (fd != fd_dir_fd && fd >= start_fd && 313 | !_is_fd_in_sorted_fd_sequence(fd, py_fds_to_keep)) { 314 | while (close(fd) < 0 && errno == EINTR); 315 | } 316 | } 317 | } 318 | while (close(fd_dir_fd) < 0 && errno == EINTR); 319 | } 320 | } 321 | 322 | #define _close_open_fds _close_open_fds_safe 323 | 324 | #else /* NOT (defined(__linux__) && defined(HAVE_SYS_SYSCALL_H)) */ 325 | 326 | 327 | /* Close all open file descriptors from start_fd and higher. 328 | * Do not close any in the sorted py_fds_to_keep list. 329 | * 330 | * This function violates the strict use of async signal safe functions. :( 331 | * It calls opendir(), readdir() and closedir(). Of these, the one most 332 | * likely to ever cause a problem is opendir() as it performs an internal 333 | * malloc(). Practically this should not be a problem. The Java VM makes the 334 | * same calls between fork and exec in its own UNIXProcess_md.c implementation. 335 | * 336 | * readdir_r() is not used because it provides no benefit. It is typically 337 | * implemented as readdir() followed by memcpy(). See also: 338 | * http://womble.decadent.org.uk/readdir_r-advisory.html 339 | */ 340 | static void 341 | _close_open_fds_maybe_unsafe(long start_fd, PyObject* py_fds_to_keep) 342 | { 343 | DIR *proc_fd_dir; 344 | #ifndef HAVE_DIRFD 345 | while (_is_fd_in_sorted_fd_sequence(start_fd, py_fds_to_keep)) { 346 | ++start_fd; 347 | } 348 | /* Close our lowest fd before we call opendir so that it is likely to 349 | * reuse that fd otherwise we might close opendir's file descriptor in 350 | * our loop. This trick assumes that fd's are allocated on a lowest 351 | * available basis. */ 352 | while (close(start_fd) < 0 && errno == EINTR); 353 | ++start_fd; 354 | #endif 355 | 356 | #if defined(__FreeBSD__) 357 | if (!_is_fdescfs_mounted_on_dev_fd()) 358 | proc_fd_dir = NULL; 359 | else 360 | #endif 361 | proc_fd_dir = opendir(FD_DIR); 362 | if (!proc_fd_dir) { 363 | /* No way to get a list of open fds. */ 364 | _close_fds_by_brute_force(start_fd, py_fds_to_keep); 365 | } else { 366 | struct dirent *dir_entry; 367 | #ifdef HAVE_DIRFD 368 | int fd_used_by_opendir = dirfd(proc_fd_dir); 369 | #else 370 | int fd_used_by_opendir = start_fd - 1; 371 | #endif 372 | errno = 0; 373 | while ((dir_entry = readdir(proc_fd_dir))) { 374 | int fd; 375 | if ((fd = _pos_int_from_ascii(dir_entry->d_name)) < 0) 376 | continue; /* Not a number. */ 377 | if (fd != fd_used_by_opendir && fd >= start_fd && 378 | !_is_fd_in_sorted_fd_sequence(fd, py_fds_to_keep)) { 379 | while (close(fd) < 0 && errno == EINTR); 380 | } 381 | errno = 0; 382 | } 383 | if (errno) { 384 | /* readdir error, revert behavior. Highly Unlikely. */ 385 | _close_fds_by_brute_force(start_fd, py_fds_to_keep); 386 | } 387 | closedir(proc_fd_dir); 388 | } 389 | } 390 | 391 | #define _close_open_fds _close_open_fds_maybe_unsafe 392 | 393 | #endif /* else NOT (defined(__linux__) && defined(HAVE_SYS_SYSCALL_H)) */ 394 | 395 | 396 | /* 397 | * This function is code executed in the child process immediately after fork 398 | * to set things up and call exec(). 399 | * 400 | * All of the code in this function must only use async-signal-safe functions, 401 | * listed at `man 7 signal` or 402 | * http://www.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_04.html. 403 | * 404 | * This restriction is documented at 405 | * http://www.opengroup.org/onlinepubs/009695399/functions/fork.html. 406 | */ 407 | static void 408 | child_exec(char *const exec_array[], 409 | char *const argv[], 410 | char *const envp[], 411 | const char *cwd, 412 | int p2cread, int p2cwrite, 413 | int c2pread, int c2pwrite, 414 | int errread, int errwrite, 415 | int errpipe_read, int errpipe_write, 416 | int close_fds, int restore_signals, 417 | int call_setsid, 418 | PyObject *py_fds_to_keep, 419 | PyObject *preexec_fn, 420 | PyObject *preexec_fn_args_tuple) 421 | { 422 | int i, saved_errno, unused, reached_preexec = 0; 423 | PyObject *result; 424 | const char* err_msg = ""; 425 | /* Buffer large enough to hold a hex integer. We can't malloc. */ 426 | char hex_errno[sizeof(saved_errno)*2+1]; 427 | 428 | /* Close parent's pipe ends. */ 429 | if (p2cwrite != -1) { 430 | POSIX_CALL(close(p2cwrite)); 431 | } 432 | if (c2pread != -1) { 433 | POSIX_CALL(close(c2pread)); 434 | } 435 | if (errread != -1) { 436 | POSIX_CALL(close(errread)); 437 | } 438 | POSIX_CALL(close(errpipe_read)); 439 | 440 | /* When duping fds, if there arises a situation where one of the fds is 441 | either 0, 1 or 2, it is possible that it is overwritten (#12607). */ 442 | if (c2pwrite == 0) 443 | POSIX_CALL(c2pwrite = dup(c2pwrite)); 444 | if (errwrite == 0 || errwrite == 1) 445 | POSIX_CALL(errwrite = dup(errwrite)); 446 | 447 | /* Dup fds for child. 448 | dup2() removes the CLOEXEC flag but we must do it ourselves if dup2() 449 | would be a no-op (issue #10806). */ 450 | if (p2cread == 0) { 451 | int old = fcntl(p2cread, F_GETFD); 452 | if (old != -1) 453 | fcntl(p2cread, F_SETFD, old & ~FD_CLOEXEC); 454 | } else if (p2cread != -1) { 455 | POSIX_CALL(dup2(p2cread, 0)); /* stdin */ 456 | } 457 | if (c2pwrite == 1) { 458 | int old = fcntl(c2pwrite, F_GETFD); 459 | if (old != -1) 460 | fcntl(c2pwrite, F_SETFD, old & ~FD_CLOEXEC); 461 | } else if (c2pwrite != -1) { 462 | POSIX_CALL(dup2(c2pwrite, 1)); /* stdout */ 463 | } 464 | if (errwrite == 2) { 465 | int old = fcntl(errwrite, F_GETFD); 466 | if (old != -1) 467 | fcntl(errwrite, F_SETFD, old & ~FD_CLOEXEC); 468 | } else if (errwrite != -1) { 469 | POSIX_CALL(dup2(errwrite, 2)); /* stderr */ 470 | } 471 | 472 | /* Close pipe fds. Make sure we don't close the same fd more than */ 473 | /* once, or standard fds. */ 474 | if (p2cread > 2) { 475 | POSIX_CALL(close(p2cread)); 476 | } 477 | if (c2pwrite > 2 && c2pwrite != p2cread) { 478 | POSIX_CALL(close(c2pwrite)); 479 | } 480 | if (errwrite != c2pwrite && errwrite != p2cread && errwrite > 2) { 481 | POSIX_CALL(close(errwrite)); 482 | } 483 | 484 | if (cwd) 485 | POSIX_CALL(chdir(cwd)); 486 | 487 | if (restore_signals) 488 | _Py_RestoreSignals(); 489 | 490 | #ifdef HAVE_SETSID 491 | if (call_setsid) 492 | POSIX_CALL(setsid()); 493 | #endif 494 | 495 | reached_preexec = 1; 496 | if (preexec_fn != Py_None && preexec_fn_args_tuple) { 497 | /* This is where the user has asked us to deadlock their program. */ 498 | result = PyObject_Call(preexec_fn, preexec_fn_args_tuple, NULL); 499 | if (result == NULL) { 500 | /* Stringifying the exception or traceback would involve 501 | * memory allocation and thus potential for deadlock. 502 | * We've already faced potential deadlock by calling back 503 | * into Python in the first place, so it probably doesn't 504 | * matter but we avoid it to minimize the possibility. */ 505 | err_msg = "Exception occurred in preexec_fn."; 506 | errno = 0; /* We don't want to report an OSError. */ 507 | goto error; 508 | } 509 | /* Py_DECREF(result); - We're about to exec so why bother? */ 510 | } 511 | 512 | _unset_cloexec_on_fds(py_fds_to_keep, errpipe_write); 513 | if (close_fds) { 514 | /* TODO HP-UX could use pstat_getproc() if anyone cares about it. */ 515 | _close_open_fds(3, py_fds_to_keep); 516 | } 517 | 518 | /* This loop matches the Lib/os.py _execvpe()'s PATH search when */ 519 | /* given the executable_list generated by Lib/subprocess.py. */ 520 | saved_errno = 0; 521 | for (i = 0; exec_array[i] != NULL; ++i) { 522 | const char *executable = exec_array[i]; 523 | if (envp) { 524 | execve(executable, argv, envp); 525 | } else { 526 | execv(executable, argv); 527 | } 528 | if (errno != ENOENT && errno != ENOTDIR && saved_errno == 0) { 529 | saved_errno = errno; 530 | } 531 | } 532 | /* Report the first exec error, not the last. */ 533 | if (saved_errno) 534 | errno = saved_errno; 535 | 536 | error: 537 | saved_errno = errno; 538 | /* Report the posix error to our parent process. */ 539 | /* We ignore all write() return values as the total size of our writes is 540 | * less than PIPEBUF and we cannot do anything about an error anyways. */ 541 | if (saved_errno) { 542 | char *cur; 543 | unused = write(errpipe_write, "OSError:", 8); 544 | cur = hex_errno + sizeof(hex_errno); 545 | while (saved_errno != 0 && cur > hex_errno) { 546 | *--cur = "0123456789ABCDEF"[saved_errno % 16]; 547 | saved_errno /= 16; 548 | } 549 | unused = write(errpipe_write, cur, hex_errno + sizeof(hex_errno) - cur); 550 | unused = write(errpipe_write, ":", 1); 551 | if (!reached_preexec) { 552 | /* Indicate to the parent that the error happened before exec(). */ 553 | unused = write(errpipe_write, "noexec", 6); 554 | } 555 | /* We can't call strerror(saved_errno). It is not async signal safe. 556 | * The parent process will look the error message up. */ 557 | } else { 558 | unused = write(errpipe_write, "RuntimeError:0:", 15); 559 | unused = write(errpipe_write, err_msg, strlen(err_msg)); 560 | } 561 | if (unused) return; /* silly? yes! avoids gcc compiler warning. */ 562 | } 563 | 564 | 565 | static PyObject * 566 | subprocess_fork_exec(PyObject* self, PyObject *args) 567 | { 568 | PyObject *gc_module = NULL; 569 | PyObject *executable_list, *py_close_fds, *py_fds_to_keep; 570 | PyObject *env_list, *preexec_fn; 571 | PyObject *process_args, *converted_args = NULL, *fast_args = NULL; 572 | PyObject *preexec_fn_args_tuple = NULL; 573 | int p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite; 574 | int errpipe_read, errpipe_write, close_fds, restore_signals; 575 | int call_setsid; 576 | PyObject *cwd_obj, *cwd_obj2; 577 | const char *cwd; 578 | pid_t pid; 579 | int need_to_reenable_gc = 0; 580 | char *const *exec_array, *const *argv = NULL, *const *envp = NULL; 581 | Py_ssize_t arg_num; 582 | 583 | if (!PyArg_ParseTuple( 584 | args, "OOOOOOiiiiiiiiiiO:fork_exec", 585 | &process_args, &executable_list, &py_close_fds, &py_fds_to_keep, 586 | &cwd_obj, &env_list, 587 | &p2cread, &p2cwrite, &c2pread, &c2pwrite, 588 | &errread, &errwrite, &errpipe_read, &errpipe_write, 589 | &restore_signals, &call_setsid, &preexec_fn)) 590 | return NULL; 591 | 592 | close_fds = PyObject_IsTrue(py_close_fds); 593 | if (close_fds < 0) 594 | return NULL; 595 | if (close_fds && errpipe_write < 3) { /* precondition */ 596 | PyErr_SetString(PyExc_ValueError, "errpipe_write must be >= 3"); 597 | return NULL; 598 | } 599 | if (PySequence_Length(py_fds_to_keep) < 0) { 600 | PyErr_SetString(PyExc_ValueError, "cannot get length of fds_to_keep"); 601 | return NULL; 602 | } 603 | if (_sanity_check_python_fd_sequence(py_fds_to_keep)) { 604 | PyErr_SetString(PyExc_ValueError, "bad value(s) in fds_to_keep"); 605 | return NULL; 606 | } 607 | 608 | /* We need to call gc.disable() when we'll be calling preexec_fn */ 609 | if (preexec_fn != Py_None) { 610 | PyObject *result; 611 | gc_module = PyImport_ImportModule("gc"); 612 | if (gc_module == NULL) 613 | return NULL; 614 | result = PyObject_CallMethod(gc_module, "isenabled", NULL); 615 | if (result == NULL) { 616 | Py_DECREF(gc_module); 617 | return NULL; 618 | } 619 | need_to_reenable_gc = PyObject_IsTrue(result); 620 | Py_DECREF(result); 621 | if (need_to_reenable_gc == -1) { 622 | Py_DECREF(gc_module); 623 | return NULL; 624 | } 625 | result = PyObject_CallMethod(gc_module, "disable", NULL); 626 | if (result == NULL) { 627 | Py_DECREF(gc_module); 628 | return NULL; 629 | } 630 | Py_DECREF(result); 631 | } 632 | 633 | exec_array = _PySequence_BytesToCharpArray(executable_list); 634 | if (!exec_array) { 635 | Py_XDECREF(gc_module); 636 | return NULL; 637 | } 638 | 639 | /* Convert args and env into appropriate arguments for exec() */ 640 | /* These conversions are done in the parent process to avoid allocating 641 | or freeing memory in the child process. */ 642 | if (process_args != Py_None) { 643 | Py_ssize_t num_args; 644 | /* Equivalent to: */ 645 | /* tuple(PyUnicode_FSConverter(arg) for arg in process_args) */ 646 | fast_args = PySequence_Fast(process_args, "argv must be a tuple"); 647 | if (fast_args == NULL) 648 | goto cleanup; 649 | num_args = PySequence_Fast_GET_SIZE(fast_args); 650 | converted_args = PyTuple_New(num_args); 651 | if (converted_args == NULL) 652 | goto cleanup; 653 | for (arg_num = 0; arg_num < num_args; ++arg_num) { 654 | PyObject *borrowed_arg, *converted_arg; 655 | borrowed_arg = PySequence_Fast_GET_ITEM(fast_args, arg_num); 656 | if (PyUnicode_FSConverter(borrowed_arg, &converted_arg) == 0) 657 | goto cleanup; 658 | PyTuple_SET_ITEM(converted_args, arg_num, converted_arg); 659 | } 660 | 661 | argv = _PySequence_BytesToCharpArray(converted_args); 662 | Py_CLEAR(converted_args); 663 | Py_CLEAR(fast_args); 664 | if (!argv) 665 | goto cleanup; 666 | } 667 | 668 | if (env_list != Py_None) { 669 | envp = _PySequence_BytesToCharpArray(env_list); 670 | if (!envp) 671 | goto cleanup; 672 | } 673 | 674 | if (preexec_fn != Py_None) { 675 | preexec_fn_args_tuple = PyTuple_New(0); 676 | if (!preexec_fn_args_tuple) 677 | goto cleanup; 678 | _PyImport_AcquireLock(); 679 | } 680 | 681 | if (cwd_obj != Py_None) { 682 | if (PyUnicode_FSConverter(cwd_obj, &cwd_obj2) == 0) 683 | goto cleanup; 684 | cwd = PyString_AsString(cwd_obj2); 685 | } else { 686 | cwd = NULL; 687 | cwd_obj2 = NULL; 688 | } 689 | 690 | pid = fork(); 691 | if (pid == 0) { 692 | /* Child process */ 693 | /* 694 | * Code from here to _exit() must only use async-signal-safe functions, 695 | * listed at `man 7 signal` or 696 | * http://www.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_04.html. 697 | */ 698 | 699 | if (preexec_fn != Py_None) { 700 | /* We'll be calling back into Python later so we need to do this. 701 | * This call may not be async-signal-safe but neither is calling 702 | * back into Python. The user asked us to use hope as a strategy 703 | * to avoid deadlock... */ 704 | PyOS_AfterFork(); 705 | } 706 | 707 | child_exec(exec_array, argv, envp, cwd, 708 | p2cread, p2cwrite, c2pread, c2pwrite, 709 | errread, errwrite, errpipe_read, errpipe_write, 710 | close_fds, restore_signals, call_setsid, 711 | py_fds_to_keep, preexec_fn, preexec_fn_args_tuple); 712 | _exit(255); 713 | return NULL; /* Dead code to avoid a potential compiler warning. */ 714 | } 715 | Py_XDECREF(cwd_obj2); 716 | 717 | if (pid == -1) { 718 | /* Capture the errno exception before errno can be clobbered. */ 719 | PyErr_SetFromErrno(PyExc_OSError); 720 | } 721 | if (preexec_fn != Py_None && 722 | _PyImport_ReleaseLock() < 0 && !PyErr_Occurred()) { 723 | PyErr_SetString(PyExc_RuntimeError, 724 | "not holding the import lock"); 725 | } 726 | 727 | /* Parent process */ 728 | if (envp) 729 | _Py_FreeCharPArray(envp); 730 | if (argv) 731 | _Py_FreeCharPArray(argv); 732 | _Py_FreeCharPArray(exec_array); 733 | 734 | /* Reenable gc in the parent process (or if fork failed). */ 735 | if (need_to_reenable_gc && _enable_gc(gc_module)) { 736 | Py_XDECREF(gc_module); 737 | return NULL; 738 | } 739 | Py_XDECREF(preexec_fn_args_tuple); 740 | Py_XDECREF(gc_module); 741 | 742 | if (pid == -1) 743 | return NULL; /* fork() failed. Exception set earlier. */ 744 | 745 | return PyLong_FromPid(pid); 746 | 747 | cleanup: 748 | if (envp) 749 | _Py_FreeCharPArray(envp); 750 | if (argv) 751 | _Py_FreeCharPArray(argv); 752 | _Py_FreeCharPArray(exec_array); 753 | Py_XDECREF(converted_args); 754 | Py_XDECREF(fast_args); 755 | Py_XDECREF(preexec_fn_args_tuple); 756 | 757 | /* Reenable gc if it was disabled. */ 758 | if (need_to_reenable_gc) 759 | _enable_gc(gc_module); 760 | Py_XDECREF(gc_module); 761 | return NULL; 762 | } 763 | 764 | 765 | PyDoc_STRVAR(subprocess_fork_exec_doc, 766 | "fork_exec(args, executable_list, close_fds, cwd, env,\n\ 767 | p2cread, p2cwrite, c2pread, c2pwrite,\n\ 768 | errread, errwrite, errpipe_read, errpipe_write,\n\ 769 | restore_signals, call_setsid, preexec_fn)\n\ 770 | \n\ 771 | Forks a child process, closes parent file descriptors as appropriate in the\n\ 772 | child and dups the few that are needed before calling exec() in the child\n\ 773 | process.\n\ 774 | \n\ 775 | The preexec_fn, if supplied, will be called immediately before exec.\n\ 776 | WARNING: preexec_fn is NOT SAFE if your application uses threads.\n\ 777 | It may trigger infrequent, difficult to debug deadlocks.\n\ 778 | \n\ 779 | If an error occurs in the child process before the exec, it is\n\ 780 | serialized and written to the errpipe_write fd per subprocess.py.\n\ 781 | \n\ 782 | Returns: the child process's PID.\n\ 783 | \n\ 784 | Raises: Only on an error in the parent process.\n\ 785 | "); 786 | 787 | PyDoc_STRVAR(subprocess_cloexec_pipe_doc, 788 | "cloexec_pipe() -> (read_end, write_end)\n\n\ 789 | Create a pipe whose ends have the cloexec flag set; write_end will be >= 3."); 790 | 791 | static PyObject * 792 | subprocess_cloexec_pipe(PyObject *self, PyObject *noargs) 793 | { 794 | int fds[2]; 795 | int res, saved_errno; 796 | long oldflags; 797 | #if (defined(HAVE_PIPE2) && defined(O_CLOEXEC)) 798 | Py_BEGIN_ALLOW_THREADS 799 | res = pipe2(fds, O_CLOEXEC); 800 | Py_END_ALLOW_THREADS 801 | if (res != 0 && errno == ENOSYS) 802 | { 803 | #endif 804 | /* We hold the GIL which offers some protection from other code calling 805 | * fork() before the CLOEXEC flags have been set but we can't guarantee 806 | * anything without pipe2(). */ 807 | res = pipe(fds); 808 | 809 | if (res == 0) { 810 | oldflags = fcntl(fds[0], F_GETFD, 0); 811 | if (oldflags < 0) res = oldflags; 812 | } 813 | if (res == 0) 814 | res = fcntl(fds[0], F_SETFD, oldflags | FD_CLOEXEC); 815 | 816 | if (res == 0) { 817 | oldflags = fcntl(fds[1], F_GETFD, 0); 818 | if (oldflags < 0) res = oldflags; 819 | } 820 | if (res == 0) 821 | res = fcntl(fds[1], F_SETFD, oldflags | FD_CLOEXEC); 822 | #if (defined(HAVE_PIPE2) && defined(O_CLOEXEC)) 823 | } 824 | #endif 825 | if (res == 0 && fds[1] < 3) { 826 | /* We always want the write end of the pipe to avoid fds 0, 1 and 2 827 | * as our child may claim those for stdio connections. */ 828 | int write_fd = fds[1]; 829 | int fds_to_close[3] = {-1, -1, -1}; 830 | int fds_to_close_idx = 0; 831 | #ifdef F_DUPFD_CLOEXEC 832 | fds_to_close[fds_to_close_idx++] = write_fd; 833 | write_fd = fcntl(write_fd, F_DUPFD_CLOEXEC, 3); 834 | if (write_fd < 0) /* We don't support F_DUPFD_CLOEXEC / other error */ 835 | #endif 836 | { 837 | /* Use dup a few times until we get a desirable fd. */ 838 | for (; fds_to_close_idx < 3; ++fds_to_close_idx) { 839 | fds_to_close[fds_to_close_idx] = write_fd; 840 | write_fd = dup(write_fd); 841 | if (write_fd >= 3) 842 | break; 843 | /* We may dup a few extra times if it returns an error but 844 | * that is okay. Repeat calls should return the same error. */ 845 | } 846 | if (write_fd < 0) res = write_fd; 847 | if (res == 0) { 848 | oldflags = fcntl(write_fd, F_GETFD, 0); 849 | if (oldflags < 0) res = oldflags; 850 | if (res == 0) 851 | res = fcntl(write_fd, F_SETFD, oldflags | FD_CLOEXEC); 852 | } 853 | } 854 | saved_errno = errno; 855 | /* Close fds we tried for the write end that were too low. */ 856 | for (fds_to_close_idx=0; fds_to_close_idx < 3; ++fds_to_close_idx) { 857 | int temp_fd = fds_to_close[fds_to_close_idx]; 858 | while (temp_fd >= 0 && close(temp_fd) < 0 && errno == EINTR); 859 | } 860 | errno = saved_errno; /* report dup or fcntl errors, not close. */ 861 | fds[1] = write_fd; 862 | } /* end if write fd was too small */ 863 | 864 | if (res != 0) 865 | return PyErr_SetFromErrno(PyExc_OSError); 866 | return Py_BuildValue("(ii)", fds[0], fds[1]); 867 | } 868 | 869 | /* module level code ********************************************************/ 870 | 871 | #define MIN_PY_VERSION_WITH_PYIMPORT_ACQUIRELOCK 0x02060300 872 | #if (PY_VERSION_HEX < MIN_PY_VERSION_WITH_PYIMPORT_ACQUIRELOCK) 873 | static PyObject* imp_module; 874 | 875 | static void 876 | _PyImport_AcquireLock(void) 877 | { 878 | PyObject *result; 879 | result = PyObject_CallMethod(imp_module, "acquire_lock", NULL); 880 | if (result == NULL) { 881 | fprintf(stderr, "imp.acquire_lock() failed.\n"); 882 | return; 883 | } 884 | Py_DECREF(result); 885 | } 886 | 887 | static int 888 | _PyImport_ReleaseLock(void) 889 | { 890 | PyObject *result; 891 | result = PyObject_CallMethod(imp_module, "release_lock", NULL); 892 | if (result == NULL) { 893 | fprintf(stderr, "imp.release_lock() failed.\n"); 894 | return -1; 895 | } 896 | Py_DECREF(result); 897 | return 0; 898 | } 899 | #endif /* Python <= 2.5 */ 900 | 901 | 902 | PyDoc_STRVAR(module_doc, 903 | "A POSIX helper for the subprocess module."); 904 | 905 | 906 | static PyMethodDef module_methods[] = { 907 | {"fork_exec", subprocess_fork_exec, METH_VARARGS, subprocess_fork_exec_doc}, 908 | {"cloexec_pipe", subprocess_cloexec_pipe, METH_NOARGS, subprocess_cloexec_pipe_doc}, 909 | {NULL, NULL} /* sentinel */ 910 | }; 911 | 912 | 913 | PyMODINIT_FUNC 914 | init_posixsubprocess32(void) 915 | { 916 | PyObject *m; 917 | 918 | #if (PY_VERSION_HEX < MIN_PY_VERSION_WITH_PYIMPORT_ACQUIRELOCK) 919 | imp_module = PyImport_ImportModule("imp"); 920 | if (imp_module == NULL) 921 | return; 922 | #endif 923 | 924 | m = Py_InitModule3("_posixsubprocess32", module_methods, module_doc); 925 | if (m == NULL) 926 | return; 927 | } 928 | -------------------------------------------------------------------------------- /_posixsubprocess_config.h.in: -------------------------------------------------------------------------------- 1 | /* _posixsubprocess_config.h.in. Generated from configure.ac by autoheader. */ 2 | 3 | /* Define to 1 if you have the header file, and it defines `DIR'. 4 | */ 5 | #undef HAVE_DIRENT_H 6 | 7 | /* Define if you have the 'dirfd' function or macro. */ 8 | #undef HAVE_DIRFD 9 | 10 | /* Define to 1 if you have the header file. */ 11 | #undef HAVE_FCNTL_H 12 | 13 | /* Define to 1 if you have the header file. */ 14 | #undef HAVE_INTTYPES_H 15 | 16 | /* Define to 1 if you have the header file. */ 17 | #undef HAVE_MEMORY_H 18 | 19 | /* Define to 1 if you have the header file, and it defines `DIR'. */ 20 | #undef HAVE_NDIR_H 21 | 22 | /* Define to 1 if you have the `pipe2' function. */ 23 | #undef HAVE_PIPE2 24 | 25 | /* Define to 1 if you have the `setsid' function. */ 26 | #undef HAVE_SETSID 27 | 28 | /* Define to 1 if you have the header file. */ 29 | #undef HAVE_SIGNAL_H 30 | 31 | /* Define to 1 if you have the header file. */ 32 | #undef HAVE_STDINT_H 33 | 34 | /* Define to 1 if you have the header file. */ 35 | #undef HAVE_STDLIB_H 36 | 37 | /* Define to 1 if you have the header file. */ 38 | #undef HAVE_STRINGS_H 39 | 40 | /* Define to 1 if you have the header file. */ 41 | #undef HAVE_STRING_H 42 | 43 | /* Define to 1 if you have the header file. */ 44 | #undef HAVE_SYS_CDEFS_H 45 | 46 | /* Define to 1 if you have the header file, and it defines `DIR'. 47 | */ 48 | #undef HAVE_SYS_DIR_H 49 | 50 | /* Define to 1 if you have the header file, and it defines `DIR'. 51 | */ 52 | #undef HAVE_SYS_NDIR_H 53 | 54 | /* Define to 1 if you have the header file. */ 55 | #undef HAVE_SYS_STAT_H 56 | 57 | /* Define to 1 if you have the header file. */ 58 | #undef HAVE_SYS_SYSCALL_H 59 | 60 | /* Define to 1 if you have the header file. */ 61 | #undef HAVE_SYS_TYPES_H 62 | 63 | /* Define to 1 if you have the header file. */ 64 | #undef HAVE_UNISTD_H 65 | 66 | /* Define to the address where bug reports for this package should be sent. */ 67 | #undef PACKAGE_BUGREPORT 68 | 69 | /* Define to the full name of this package. */ 70 | #undef PACKAGE_NAME 71 | 72 | /* Define to the full name and version of this package. */ 73 | #undef PACKAGE_STRING 74 | 75 | /* Define to the one symbol short name of this package. */ 76 | #undef PACKAGE_TARNAME 77 | 78 | /* Define to the home page for this package. */ 79 | #undef PACKAGE_URL 80 | 81 | /* Define to the version of this package. */ 82 | #undef PACKAGE_VERSION 83 | 84 | /* Define to 1 if you have the ANSI C header files. */ 85 | #undef STDC_HEADERS 86 | 87 | /* Define on OpenBSD to activate all library features */ 88 | #undef _BSD_SOURCE 89 | 90 | /* Define on Irix to enable u_int */ 91 | #undef _BSD_TYPES 92 | 93 | /* Define on Darwin to activate all library features */ 94 | #undef _DARWIN_C_SOURCE 95 | 96 | /* Define on Linux to activate all library features */ 97 | #undef _GNU_SOURCE 98 | 99 | /* Define on NetBSD to activate all library features */ 100 | #undef _NETBSD_SOURCE 101 | 102 | /* Define to activate features from IEEE Stds 1003.1-2008 */ 103 | #undef _POSIX_C_SOURCE 104 | 105 | /* Define to the level of X/Open that your system supports */ 106 | #undef _XOPEN_SOURCE 107 | 108 | /* Define to activate Unix95-and-earlier features */ 109 | #undef _XOPEN_SOURCE_EXTENDED 110 | 111 | /* Define on FreeBSD to activate all library features */ 112 | #undef __BSD_VISIBLE 113 | -------------------------------------------------------------------------------- /_posixsubprocess_helpers.c: -------------------------------------------------------------------------------- 1 | /* Functions and macros from Python 3.2 not found in 2.x. 2 | This file is #included by _posixsubprocess.c and the functions 3 | are declared static to avoid exposing them outside this module. */ 4 | 5 | /* _posixsubprocess_config.h was already included by _posixsubprocess.c 6 | * which is #include'ing us despite the .c name. HAVE_SIGNAL_H comes 7 | * from there. Yes, confusing! */ 8 | #ifdef HAVE_SIGNAL_H 9 | #include 10 | #endif 11 | #include "unicodeobject.h" 12 | 13 | #if (PY_VERSION_HEX < 0x02050000) 14 | #define Py_ssize_t int 15 | #endif 16 | 17 | #define Py_CLEANUP_SUPPORTED 0x20000 18 | 19 | /* Issue #1983: pid_t can be longer than a C long on some systems */ 20 | #if !defined(SIZEOF_PID_T) || SIZEOF_PID_T == SIZEOF_INT 21 | #define PyLong_FromPid PyLong_FromLong 22 | #elif SIZEOF_PID_T == SIZEOF_LONG 23 | #define PyLong_FromPid PyLong_FromLong 24 | #elif defined(SIZEOF_LONG_LONG) && SIZEOF_PID_T == SIZEOF_LONG_LONG 25 | #define PyLong_FromPid PyLong_FromLongLong 26 | #else 27 | #error "sizeof(pid_t) is neither sizeof(int), sizeof(long) or sizeof(long long)" 28 | #endif /* SIZEOF_PID_T */ 29 | 30 | 31 | static PyObject *PyUnicode_EncodeFSDefault(PyObject *unicode) 32 | { 33 | if (Py_FileSystemDefaultEncoding) 34 | return PyUnicode_AsEncodedString(unicode, 35 | Py_FileSystemDefaultEncoding, 36 | "strict"); 37 | else 38 | return PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(unicode), 39 | PyUnicode_GET_SIZE(unicode), 40 | "strict"); 41 | } 42 | 43 | 44 | /* Convert the argument to a bytes object, according to the file 45 | system encoding. The addr param must be a PyObject**. 46 | This is designed to be used with "O&" in PyArg_Parse APIs. */ 47 | 48 | static int 49 | PyUnicode_FSConverter(PyObject* arg, void* addr) 50 | { 51 | PyObject *output = NULL; 52 | Py_ssize_t size; 53 | void *data; 54 | if (arg == NULL) { 55 | Py_DECREF(*(PyObject**)addr); 56 | return 1; 57 | } 58 | if (PyString_Check(arg)) { 59 | output = arg; 60 | Py_INCREF(output); 61 | } 62 | else { 63 | arg = PyUnicode_FromObject(arg); 64 | if (!arg) 65 | return 0; 66 | output = PyUnicode_EncodeFSDefault(arg); 67 | Py_DECREF(arg); 68 | if (!output) 69 | return 0; 70 | if (!PyString_Check(output)) { 71 | Py_DECREF(output); 72 | PyErr_SetString(PyExc_TypeError, "encoder failed to return bytes"); 73 | return 0; 74 | } 75 | } 76 | size = PyString_GET_SIZE(output); 77 | data = PyString_AS_STRING(output); 78 | if (size != strlen(data)) { 79 | PyErr_SetString(PyExc_TypeError, "embedded NUL character"); 80 | Py_DECREF(output); 81 | return 0; 82 | } 83 | *(PyObject**)addr = output; 84 | return Py_CLEANUP_SUPPORTED; 85 | } 86 | 87 | 88 | /* Free's a NULL terminated char** array of C strings. */ 89 | static void 90 | _Py_FreeCharPArray(char *const array[]) 91 | { 92 | Py_ssize_t i; 93 | for (i = 0; array[i] != NULL; ++i) { 94 | free(array[i]); 95 | } 96 | free((void*)array); 97 | } 98 | 99 | 100 | /* 101 | * Flatten a sequence of bytes() objects into a C array of 102 | * NULL terminated string pointers with a NULL char* terminating the array. 103 | * (ie: an argv or env list) 104 | * 105 | * Memory allocated for the returned list is allocated using malloc() and MUST 106 | * be freed by the caller using a free() loop or _Py_FreeCharPArray(). 107 | */ 108 | static char *const * 109 | _PySequence_BytesToCharpArray(PyObject* self) 110 | { 111 | char **array; 112 | Py_ssize_t i, argc; 113 | PyObject *item = NULL; 114 | 115 | argc = PySequence_Size(self); 116 | if (argc == -1) 117 | return NULL; 118 | /* Avoid 32-bit overflows to malloc() from unreasonable values. */ 119 | if (argc > 0x10000000) { 120 | PyErr_NoMemory(); 121 | return NULL; 122 | } 123 | 124 | array = malloc((argc + 1) * sizeof(char *)); 125 | if (array == NULL) { 126 | PyErr_NoMemory(); 127 | return NULL; 128 | } 129 | for (i = 0; i < argc; ++i) { 130 | char *data; 131 | item = PySequence_GetItem(self, i); 132 | data = PyString_AsString(item); 133 | if (data == NULL) { 134 | /* NULL terminate before freeing. */ 135 | array[i] = NULL; 136 | goto fail; 137 | } 138 | array[i] = strdup(data); 139 | if (!array[i]) { 140 | PyErr_NoMemory(); 141 | goto fail; 142 | } 143 | Py_DECREF(item); 144 | } 145 | array[argc] = NULL; 146 | 147 | return array; 148 | 149 | fail: 150 | Py_XDECREF(item); 151 | _Py_FreeCharPArray(array); 152 | return NULL; 153 | } 154 | 155 | 156 | /* Restore signals that the interpreter has called SIG_IGN on to SIG_DFL. 157 | * 158 | * All of the code in this function must only use async-signal-safe functions, 159 | * listed at `man 7 signal` or 160 | * http://www.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_04.html. 161 | */ 162 | static void 163 | _Py_RestoreSignals(void) 164 | { 165 | #ifdef SIGPIPE 166 | PyOS_setsig(SIGPIPE, SIG_DFL); 167 | #endif 168 | #ifdef SIGXFZ 169 | PyOS_setsig(SIGXFZ, SIG_DFL); 170 | #endif 171 | #ifdef SIGXFSZ 172 | PyOS_setsig(SIGXFSZ, SIG_DFL); 173 | #endif 174 | } 175 | -------------------------------------------------------------------------------- /build-manylinux2014-wheel.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | docker pull quay.io/pypa/manylinux2014_x86_64 3 | # This depends on ./Dockerfile 4 | docker build . 5 | # This is the image "name" the above created for me 6 | docker run 89729288db46 pip2 --version 7 | docker run -it -v "$(pwd)":/io 89729288db46 pip2 wheel /io/ -w /io/wheelhouse/ 8 | ls -al wheelhouse/ 9 | docker run -it -v "$(pwd)":/io 89729288db46 auditwheel repair /io/wheelhouse/subprocess32-3.5.4-cp27-cp27mu-linux_x86_64.whl -w /io/wheelhouse 10 | ls -al wheelhouse/ 11 | ../twine_venv/bin/twine check wheelhouse/subprocess32-3.5.4-cp27-cp27mu-manylinux2014_x86_64.whl 12 | ../twine_venv/bin/twine upload wheelhouse/subprocess32-3.5.4-cp27-cp27mu-manylinux2014_x86_64.whl 13 | -------------------------------------------------------------------------------- /configure.ac: -------------------------------------------------------------------------------- 1 | dnl *********************************************** 2 | dnl * Please run autoreconf to test your changes! * 3 | dnl *********************************************** 4 | 5 | AC_PREREQ(2.65) 6 | 7 | AC_INIT(_posixsubprocess32, 3.5, https://github.com/google/python-subprocess32/) 8 | 9 | AC_CONFIG_HEADER(_posixsubprocess_config.h) 10 | 11 | ######## This arcane joy is from 3.5's configure.ac file: ######## 12 | 13 | # The later defininition of _XOPEN_SOURCE disables certain features 14 | # on Linux, so we need _GNU_SOURCE to re-enable them (makedev, tm_zone). 15 | AC_DEFINE(_GNU_SOURCE, 1, [Define on Linux to activate all library features]) 16 | 17 | # The later defininition of _XOPEN_SOURCE and _POSIX_C_SOURCE disables 18 | # certain features on NetBSD, so we need _NETBSD_SOURCE to re-enable 19 | # them. 20 | AC_DEFINE(_NETBSD_SOURCE, 1, [Define on NetBSD to activate all library features]) 21 | 22 | # The later defininition of _XOPEN_SOURCE and _POSIX_C_SOURCE disables 23 | # certain features on FreeBSD, so we need __BSD_VISIBLE to re-enable 24 | # them. 25 | AC_DEFINE(__BSD_VISIBLE, 1, [Define on FreeBSD to activate all library features]) 26 | 27 | # The later defininition of _XOPEN_SOURCE and _POSIX_C_SOURCE disables 28 | # u_int on Irix 5.3. Defining _BSD_TYPES brings it back. 29 | AC_DEFINE(_BSD_TYPES, 1, [Define on Irix to enable u_int]) 30 | 31 | # The later defininition of _XOPEN_SOURCE and _POSIX_C_SOURCE disables 32 | # certain features on Mac OS X, so we need _DARWIN_C_SOURCE to re-enable 33 | # them. 34 | AC_DEFINE(_DARWIN_C_SOURCE, 1, [Define on Darwin to activate all library features]) 35 | 36 | define_xopen_source=yes 37 | 38 | # Some systems cannot stand _XOPEN_SOURCE being defined at all; they 39 | # disable features if it is defined, without any means to access these 40 | # features as extensions. For these systems, we skip the definition of 41 | # _XOPEN_SOURCE. Before adding a system to the list to gain access to 42 | # some feature, make sure there is no alternative way to access this 43 | # feature. Also, when using wildcards, make sure you have verified the 44 | # need for not defining _XOPEN_SOURCE on all systems matching the 45 | # wildcard, and that the wildcard does not include future systems 46 | # (which may remove their limitations). 47 | dnl quadrigraphs "@<:@" and "@:>@" produce "[" and "]" in the output 48 | case $ac_sys_system/$ac_sys_release in 49 | # On OpenBSD, select(2) is not available if _XOPEN_SOURCE is defined, 50 | # even though select is a POSIX function. Reported by J. Ribbens. 51 | # Reconfirmed for OpenBSD 3.3 by Zachary Hamm, for 3.4 by Jason Ish. 52 | # In addition, Stefan Krah confirms that issue #1244610 exists through 53 | # OpenBSD 4.6, but is fixed in 4.7. 54 | OpenBSD/2.* | OpenBSD/3.* | OpenBSD/4.@<:@0123456@:>@) 55 | define_xopen_source=no 56 | # OpenBSD undoes our definition of __BSD_VISIBLE if _XOPEN_SOURCE is 57 | # also defined. This can be overridden by defining _BSD_SOURCE 58 | # As this has a different meaning on Linux, only define it on OpenBSD 59 | AC_DEFINE(_BSD_SOURCE, 1, [Define on OpenBSD to activate all library features]) 60 | ;; 61 | OpenBSD/*) 62 | # OpenBSD undoes our definition of __BSD_VISIBLE if _XOPEN_SOURCE is 63 | # also defined. This can be overridden by defining _BSD_SOURCE 64 | # As this has a different meaning on Linux, only define it on OpenBSD 65 | AC_DEFINE(_BSD_SOURCE, 1, [Define on OpenBSD to activate all library features]) 66 | ;; 67 | # Defining _XOPEN_SOURCE on NetBSD version prior to the introduction of 68 | # _NETBSD_SOURCE disables certain features (eg. setgroups). Reported by 69 | # Marc Recht 70 | NetBSD/1.5 | NetBSD/1.5.* | NetBSD/1.6 | NetBSD/1.6.* | NetBSD/1.6@<:@A-S@:>@) 71 | define_xopen_source=no;; 72 | # From the perspective of Solaris, _XOPEN_SOURCE is not so much a 73 | # request to enable features supported by the standard as a request 74 | # to disable features not supported by the standard. The best way 75 | # for Python to use Solaris is simply to leave _XOPEN_SOURCE out 76 | # entirely and define __EXTENSIONS__ instead. 77 | SunOS/*) 78 | define_xopen_source=no;; 79 | # On UnixWare 7, u_long is never defined with _XOPEN_SOURCE, 80 | # but used in /usr/include/netinet/tcp.h. Reported by Tim Rice. 81 | # Reconfirmed for 7.1.4 by Martin v. Loewis. 82 | OpenUNIX/8.0.0| UnixWare/7.1.@<:@0-4@:>@) 83 | define_xopen_source=no;; 84 | # On OpenServer 5, u_short is never defined with _XOPEN_SOURCE, 85 | # but used in struct sockaddr.sa_family. Reported by Tim Rice. 86 | SCO_SV/3.2) 87 | define_xopen_source=no;; 88 | # On FreeBSD 4, the math functions C89 does not cover are never defined 89 | # with _XOPEN_SOURCE and __BSD_VISIBLE does not re-enable them. 90 | FreeBSD/4.*) 91 | define_xopen_source=no;; 92 | # On MacOS X 10.2, a bug in ncurses.h means that it craps out if 93 | # _XOPEN_EXTENDED_SOURCE is defined. Apparently, this is fixed in 10.3, which 94 | # identifies itself as Darwin/7.* 95 | # On Mac OS X 10.4, defining _POSIX_C_SOURCE or _XOPEN_SOURCE 96 | # disables platform specific features beyond repair. 97 | # On Mac OS X 10.3, defining _POSIX_C_SOURCE or _XOPEN_SOURCE 98 | # has no effect, don't bother defining them 99 | Darwin/@<:@6789@:>@.*) 100 | define_xopen_source=no;; 101 | Darwin/1@<:@0-9@:>@.*) 102 | define_xopen_source=no;; 103 | # On AIX 4 and 5.1, mbstate_t is defined only when _XOPEN_SOURCE == 500 but 104 | # used in wcsnrtombs() and mbsnrtowcs() even if _XOPEN_SOURCE is not defined 105 | # or has another value. By not (re)defining it, the defaults come in place. 106 | AIX/4) 107 | define_xopen_source=no;; 108 | AIX/5) 109 | if test `uname -r` -eq 1; then 110 | define_xopen_source=no 111 | fi 112 | ;; 113 | # On QNX 6.3.2, defining _XOPEN_SOURCE prevents netdb.h from 114 | # defining NI_NUMERICHOST. 115 | QNX/6.3.2) 116 | define_xopen_source=no 117 | ;; 118 | 119 | esac 120 | 121 | if test $define_xopen_source = yes 122 | then 123 | # X/Open 7, incorporating POSIX.1-2008 124 | AC_DEFINE(_XOPEN_SOURCE, 700, 125 | Define to the level of X/Open that your system supports) 126 | 127 | # On Tru64 Unix 4.0F, defining _XOPEN_SOURCE also requires 128 | # definition of _XOPEN_SOURCE_EXTENDED and _POSIX_C_SOURCE, or else 129 | # several APIs are not declared. Since this is also needed in some 130 | # cases for HP-UX, we define it globally. 131 | AC_DEFINE(_XOPEN_SOURCE_EXTENDED, 1, 132 | Define to activate Unix95-and-earlier features) 133 | 134 | AC_DEFINE(_POSIX_C_SOURCE, 200809L, Define to activate features from IEEE Stds 1003.1-2008) 135 | fi 136 | 137 | ######## The stuff we _actually_ want to care about. ######## 138 | 139 | AC_HEADER_STDC 140 | AC_CHECK_HEADERS(unistd.h fcntl.h signal.h \ 141 | sys/cdefs.h sys/types.h sys/stat.h sys/syscall.h) 142 | AC_HEADER_DIRENT 143 | 144 | AC_CHECK_FUNCS(pipe2 setsid) 145 | AC_CHECK_DECL(dirfd, 146 | AC_DEFINE(HAVE_DIRFD, 1, 147 | Define if you have the 'dirfd' function or macro.), , 148 | [#include 149 | #include ]) 150 | 151 | AC_OUTPUT 152 | -------------------------------------------------------------------------------- /python3_redirect/__init__.py: -------------------------------------------------------------------------------- 1 | """When installed as subprocess32, this sets up a redirect to subprocess.""" 2 | 3 | import subprocess 4 | import sys 5 | if sys.version_info[:2] < (3,3): 6 | raise ImportError('Ancient Python 3 versions are not supported.') 7 | # Doing this could crash some older Python interpreters due to the module 8 | # reference going away before the import is complete? 9 | sys.modules['subprocess32'] = subprocess 10 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [sdist] 2 | formats=gztar 3 | 4 | [bdist_rpm] 5 | release = 1 6 | group = Development/Languages/Python 7 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python2 2 | 3 | import os 4 | import stat 5 | import sys 6 | from setuptools import setup, Extension, Command 7 | from setuptools.command.build_ext import build_ext 8 | 9 | 10 | class BuildConfigure(Command): 11 | description = 'Generate _posixsubprocess_config.h via ./configure.' 12 | 13 | def run(self): 14 | config_h = '_posixsubprocess_config.h' 15 | status = 'config.status' 16 | configure = './configure' 17 | # Here we implement time stamp checking logic like make, manually... 18 | if os.path.exists(config_h) and os.path.exists(status): 19 | status_mtime = os.stat(status)[stat.ST_MTIME] 20 | configure_mtime = os.stat(configure)[stat.ST_MTIME] 21 | if configure_mtime < status_mtime: 22 | print(' ' + config_h + ' is already up to date.') 23 | return 24 | configure_command = 'sh ' + configure 25 | if os.system(configure_command): 26 | raise RuntimeError(configure_command + ' failed.') 27 | 28 | # Abstract methods we don't need but are required to define. 29 | def initialize_options(self): pass 30 | def finalize_options(self): pass 31 | 32 | 33 | class BuildExtensionAfterConfigure(build_ext): 34 | """Executes ./configure before the actual extension is built.""" 35 | 36 | # See https://github.com/python/cpython/blob/2.7/Lib/distutils/cmd.py#L328 37 | sub_commands = [('build_configure', None)] + build_ext.sub_commands 38 | 39 | def run(self): 40 | for command in self.get_sub_commands(): 41 | self.run_command(command) 42 | build_ext.run(self) 43 | 44 | 45 | def main(): 46 | ext_modules = [] 47 | py_modules = [] 48 | packages = [] 49 | package_dir = {} 50 | cmdclass = {} 51 | if sys.version_info[0] == 2: # PY2 52 | py_modules.append('subprocess32') 53 | if os.name == 'posix': 54 | ext = Extension('_posixsubprocess32', ['_posixsubprocess.c'], 55 | depends=['_posixsubprocess_helpers.c', 56 | '_posixsubprocess_config.h']) 57 | ext_modules.append(ext) 58 | # Cause ./configure to be run before we build the extension. 59 | cmdclass['build_configure'] = BuildConfigure 60 | cmdclass['build_ext'] = BuildExtensionAfterConfigure 61 | else: # PY3 62 | # Install a redirect that makes subprocess32 == subprocess on import. 63 | packages.append('subprocess32') 64 | package_dir['subprocess32'] = 'python3_redirect' 65 | sys.stderr.write('subprocess32 == subprocess on Python 3.\n') 66 | 67 | setup( 68 | name='subprocess32', 69 | version='3.5.4', 70 | description='A backport of the subprocess module from Python 3 for use on 2.x.', 71 | long_description="""\ 72 | This is a backport of the subprocess standard library module from 73 | Python 3.2 - 3.5 for use on Python 2. 74 | 75 | It includes bugfixes and some new features. On POSIX systems it is 76 | guaranteed to be reliable when used in threaded applications. 77 | It includes timeout support from Python 3.3 and the run() API from 3.5 78 | but otherwise matches 3.2's API. 79 | 80 | It has not been tested by the author on Windows. 81 | 82 | On Python 3, it merely redirects the subprocess32 name to subprocess.""", 83 | license='PSF license', 84 | 85 | maintainer='Gregory P. Smith', 86 | maintainer_email='greg@krypto.org', 87 | url='https://github.com/google/python-subprocess32', 88 | 89 | ext_modules=ext_modules, 90 | py_modules=py_modules, 91 | packages=packages, 92 | package_dir=package_dir, 93 | cmdclass=cmdclass, 94 | 95 | # We don't actually "support" 3.3+, we just allow installation there as 96 | # we install a stub redirecting to the standard library subprocess module 97 | # under the subprocess32 name. 98 | python_requires='>=2.6, !=3.0.*, !=3.1.*, !=3.2.*, <4', 99 | 100 | classifiers=[ 101 | 'Intended Audience :: Developers', 102 | 'Topic :: Software Development :: Libraries', 103 | 'Development Status :: 5 - Production/Stable', 104 | 'License :: OSI Approved :: Python Software Foundation License', 105 | 'Operating System :: MacOS', 106 | 'Operating System :: MacOS :: MacOS X', 107 | 'Operating System :: POSIX', 108 | 'Operating System :: POSIX :: BSD', 109 | 'Operating System :: POSIX :: Linux', 110 | 'Operating System :: POSIX :: SunOS/Solaris', 111 | 'Programming Language :: Python :: 2.6', 112 | 'Programming Language :: Python :: 2.7', 113 | 'Programming Language :: Python :: 2 :: Only', 114 | 'Programming Language :: Python :: Implementation :: CPython', 115 | ], 116 | ) 117 | 118 | 119 | if __name__ == '__main__': 120 | main() 121 | -------------------------------------------------------------------------------- /subprocess32.py: -------------------------------------------------------------------------------- 1 | # subprocess - Subprocesses with accessible I/O streams 2 | # 3 | # For more information about this module, see PEP 324. 4 | # 5 | # Copyright (c) 2003-2005 by Peter Astrand 6 | # 7 | # Licensed to PSF under a Contributor Agreement. 8 | # See http://www.python.org/3.3/license for licensing details. 9 | 10 | r"""Subprocesses with accessible I/O streams 11 | 12 | This module allows you to spawn processes, connect to their 13 | input/output/error pipes, and obtain their return codes. 14 | 15 | For a complete description of this module see the Python documentation. 16 | 17 | Main API 18 | ======== 19 | run(...): Runs a command, waits for it to complete, then returns a 20 | CompletedProcess instance. 21 | Popen(...): A class for flexibly executing a command in a new process 22 | 23 | Constants 24 | --------- 25 | DEVNULL: Special value that indicates that os.devnull should be used 26 | PIPE: Special value that indicates a pipe should be created 27 | STDOUT: Special value that indicates that stderr should go to stdout 28 | 29 | 30 | Older API 31 | ========= 32 | call(...): Runs a command, waits for it to complete, then returns 33 | the return code. 34 | check_call(...): Same as call() but raises CalledProcessError() 35 | if return code is not 0 36 | check_output(...): Same as check_call() but returns the contents of 37 | stdout instead of a return code 38 | """ 39 | 40 | import sys 41 | mswindows = (sys.platform == "win32") 42 | 43 | import os 44 | import errno 45 | import exceptions 46 | import types 47 | import time 48 | import traceback 49 | import gc 50 | import signal 51 | 52 | # Exception classes used by this module. 53 | class SubprocessError(Exception): pass 54 | 55 | 56 | class CalledProcessError(SubprocessError): 57 | """Raised when run() is called with check=True and the process 58 | returns a non-zero exit status. 59 | 60 | Attributes: 61 | cmd, returncode, stdout, stderr, output 62 | """ 63 | def __init__(self, returncode, cmd, output=None, stderr=None): 64 | self.returncode = returncode 65 | self.cmd = cmd 66 | self.output = output 67 | self.stderr = stderr 68 | super(CalledProcessError, self).__init__(returncode, cmd, 69 | output, stderr) 70 | 71 | def __str__(self): 72 | if self.returncode and self.returncode < 0: 73 | return "Command '%s' died with signal %d." % ( 74 | self.cmd, -self.returncode) 75 | else: 76 | return "Command '%s' returned non-zero exit status %d." % ( 77 | self.cmd, self.returncode) 78 | 79 | #@property 80 | def __stdout_getter(self): 81 | """Alias for output attribute, to match stderr""" 82 | return self.output 83 | 84 | #@stdout.setter # Required Python 2.6 85 | def __stdout_setter(self, value): 86 | # There's no obvious reason to set this, but allow it anyway so 87 | # .stdout is a transparent alias for .output 88 | self.output = value 89 | 90 | stdout = property(__stdout_getter, __stdout_setter) # Python 2.4 91 | 92 | 93 | class TimeoutExpired(SubprocessError): 94 | """This exception is raised when the timeout expires while waiting for a 95 | child process. 96 | 97 | Attributes: 98 | cmd, output, stdout, stderr, timeout 99 | """ 100 | def __init__(self, cmd, timeout, output=None, stderr=None): 101 | self.cmd = cmd 102 | self.timeout = timeout 103 | self.output = output 104 | self.stderr = stderr 105 | super(TimeoutExpired, self).__init__(cmd, timeout, output, stderr) 106 | 107 | def __str__(self): 108 | return ("Command '%s' timed out after %s seconds" % 109 | (self.cmd, self.timeout)) 110 | 111 | #@property 112 | def __stdout_getter(self): 113 | return self.output 114 | 115 | #@stdout.setter # Required Python 2.6 116 | def __stdout_setter(self, value): 117 | # There's no obvious reason to set this, but allow it anyway so 118 | # .stdout is a transparent alias for .output 119 | self.output = value 120 | 121 | stdout = property(__stdout_getter, __stdout_setter) # Python 2.4 122 | 123 | 124 | if mswindows: 125 | import threading 126 | import msvcrt 127 | import _subprocess 128 | class STARTUPINFO: 129 | dwFlags = 0 130 | hStdInput = None 131 | hStdOutput = None 132 | hStdError = None 133 | wShowWindow = 0 134 | class pywintypes: 135 | error = IOError 136 | else: 137 | import select 138 | _has_poll = hasattr(select, 'poll') 139 | import fcntl 140 | import pickle 141 | 142 | try: 143 | import _posixsubprocess32 as _posixsubprocess 144 | except ImportError: 145 | _posixsubprocess = None 146 | import warnings 147 | warnings.warn("The _posixsubprocess module is not being used. " 148 | "Child process reliability may suffer if your " 149 | "program uses threads.", RuntimeWarning) 150 | try: 151 | import threading 152 | except ImportError: 153 | import dummy_threading as threading 154 | 155 | # When select or poll has indicated that the file is writable, 156 | # we can write up to _PIPE_BUF bytes without risk of blocking. 157 | # POSIX defines PIPE_BUF as >= 512. 158 | _PIPE_BUF = getattr(select, 'PIPE_BUF', 512) 159 | 160 | _FD_CLOEXEC = getattr(fcntl, 'FD_CLOEXEC', 1) 161 | 162 | def _set_cloexec(fd, cloexec): 163 | old = fcntl.fcntl(fd, fcntl.F_GETFD) 164 | if cloexec: 165 | fcntl.fcntl(fd, fcntl.F_SETFD, old | _FD_CLOEXEC) 166 | else: 167 | fcntl.fcntl(fd, fcntl.F_SETFD, old & ~_FD_CLOEXEC) 168 | 169 | if _posixsubprocess: 170 | _create_pipe = _posixsubprocess.cloexec_pipe 171 | else: 172 | def _create_pipe(): 173 | fds = os.pipe() 174 | _set_cloexec(fds[0], True) 175 | _set_cloexec(fds[1], True) 176 | return fds 177 | 178 | __all__ = ["Popen", "PIPE", "STDOUT", "call", "check_call", 179 | "check_output", "CalledProcessError"] 180 | 181 | if mswindows: 182 | from _subprocess import (CREATE_NEW_CONSOLE, CREATE_NEW_PROCESS_GROUP, 183 | STD_INPUT_HANDLE, STD_OUTPUT_HANDLE, 184 | STD_ERROR_HANDLE, SW_HIDE, 185 | STARTF_USESTDHANDLES, STARTF_USESHOWWINDOW) 186 | # https://msdn.microsoft.com/en-us/library/windows/desktop/ms687032(v=vs.85).aspx 187 | # Note: In Python 3.3 this constant is found in the _winapi module. 188 | _WAIT_TIMEOUT = 0x102 189 | 190 | __all__.extend(["CREATE_NEW_CONSOLE", "CREATE_NEW_PROCESS_GROUP", 191 | "STD_INPUT_HANDLE", "STD_OUTPUT_HANDLE", 192 | "STD_ERROR_HANDLE", "SW_HIDE", 193 | "STARTF_USESTDHANDLES", "STARTF_USESHOWWINDOW"]) 194 | try: 195 | MAXFD = os.sysconf("SC_OPEN_MAX") 196 | except: 197 | MAXFD = 256 198 | 199 | # This lists holds Popen instances for which the underlying process had not 200 | # exited at the time its __del__ method got called: those processes are wait()ed 201 | # for synchronously from _cleanup() when a new Popen object is created, to avoid 202 | # zombie processes. 203 | _active = [] 204 | 205 | def _cleanup(): 206 | for inst in _active[:]: 207 | res = inst._internal_poll(_deadstate=sys.maxint) 208 | if res is not None: 209 | try: 210 | _active.remove(inst) 211 | except ValueError: 212 | # This can happen if two threads create a new Popen instance. 213 | # It's harmless that it was already removed, so ignore. 214 | pass 215 | 216 | PIPE = -1 217 | STDOUT = -2 218 | DEVNULL = -3 219 | 220 | # This function is only used by multiprocessing, it is here so that people 221 | # can drop subprocess32 in as a replacement for the stdlib subprocess module. 222 | 223 | def _args_from_interpreter_flags(): 224 | """Return a list of command-line arguments reproducing the current 225 | settings in sys.flags and sys.warnoptions.""" 226 | flag_opt_map = { 227 | 'debug': 'd', 228 | # 'inspect': 'i', 229 | # 'interactive': 'i', 230 | 'optimize': 'O', 231 | 'dont_write_bytecode': 'B', 232 | 'no_user_site': 's', 233 | 'no_site': 'S', 234 | 'ignore_environment': 'E', 235 | 'verbose': 'v', 236 | 'bytes_warning': 'b', 237 | 'py3k_warning': '3', 238 | } 239 | args = [] 240 | for flag, opt in flag_opt_map.items(): 241 | v = getattr(sys.flags, flag) 242 | if v > 0: 243 | args.append('-' + opt * v) 244 | if getattr(sys.flags, 'hash_randomization') != 0: 245 | args.append('-R') 246 | for opt in sys.warnoptions: 247 | args.append('-W' + opt) 248 | return args 249 | 250 | 251 | def _eintr_retry_call(func, *args): 252 | while True: 253 | try: 254 | return func(*args) 255 | except (OSError, IOError), e: 256 | if e.errno == errno.EINTR: 257 | continue 258 | raise 259 | 260 | 261 | def _get_exec_path(env=None): 262 | """Returns the sequence of directories that will be searched for the 263 | named executable (similar to a shell) when launching a process. 264 | 265 | *env* must be an environment variable dict or None. If *env* is None, 266 | os.environ will be used. 267 | """ 268 | if env is None: 269 | env = os.environ 270 | return env.get('PATH', os.defpath).split(os.pathsep) 271 | 272 | 273 | if hasattr(os, 'get_exec_path'): 274 | _get_exec_path = os.get_exec_path 275 | 276 | 277 | def call(*popenargs, **kwargs): 278 | """Run command with arguments. Wait for command to complete or 279 | timeout, then return the returncode attribute. 280 | 281 | The arguments are the same as for the Popen constructor. Example: 282 | 283 | retcode = call(["ls", "-l"]) 284 | """ 285 | timeout = kwargs.pop('timeout', None) 286 | p = Popen(*popenargs, **kwargs) 287 | try: 288 | return p.wait(timeout=timeout) 289 | except TimeoutExpired: 290 | p.kill() 291 | p.wait() 292 | raise 293 | 294 | 295 | def check_call(*popenargs, **kwargs): 296 | """Run command with arguments. Wait for command to complete. If 297 | the exit code was zero then return, otherwise raise 298 | CalledProcessError. The CalledProcessError object will have the 299 | return code in the returncode attribute. 300 | 301 | The arguments are the same as for the call function. Example: 302 | 303 | check_call(["ls", "-l"]) 304 | """ 305 | retcode = call(*popenargs, **kwargs) 306 | if retcode: 307 | cmd = kwargs.get("args") 308 | if cmd is None: 309 | cmd = popenargs[0] 310 | raise CalledProcessError(retcode, cmd) 311 | return 0 312 | 313 | 314 | def check_output(*popenargs, **kwargs): 315 | r"""Run command with arguments and return its output as a byte string. 316 | 317 | If the exit code was non-zero it raises a CalledProcessError. The 318 | CalledProcessError object will have the return code in the returncode 319 | attribute and output in the output attribute. 320 | 321 | The arguments are the same as for the Popen constructor. Example: 322 | 323 | >>> check_output(["ls", "-l", "/dev/null"]) 324 | 'crw-rw-rw- 1 root root 1, 3 Oct 18 2007 /dev/null\n' 325 | 326 | The stdout argument is not allowed as it is used internally. 327 | To capture standard error in the result, use stderr=STDOUT. 328 | 329 | >>> check_output(["/bin/sh", "-c", 330 | ... "ls -l non_existent_file ; exit 0"], 331 | ... stderr=STDOUT) 332 | 'ls: non_existent_file: No such file or directory\n' 333 | """ 334 | timeout = kwargs.pop('timeout', None) 335 | if 'stdout' in kwargs: 336 | raise ValueError('stdout argument not allowed, it will be overridden.') 337 | process = Popen(stdout=PIPE, *popenargs, **kwargs) 338 | try: 339 | output, unused_err = process.communicate(timeout=timeout) 340 | except TimeoutExpired: 341 | process.kill() 342 | output, unused_err = process.communicate() 343 | raise TimeoutExpired(process.args, timeout, output=output) 344 | retcode = process.poll() 345 | if retcode: 346 | raise CalledProcessError(retcode, process.args, output=output) 347 | return output 348 | 349 | 350 | class CompletedProcess(object): 351 | """A process that has finished running. 352 | This is returned by run(). 353 | Attributes: 354 | args: The list or str args passed to run(). 355 | returncode: The exit code of the process, negative for signals. 356 | stdout: The standard output (None if not captured). 357 | stderr: The standard error (None if not captured). 358 | """ 359 | def __init__(self, args, returncode, stdout=None, stderr=None): 360 | self.args = args 361 | self.returncode = returncode 362 | self.stdout = stdout 363 | self.stderr = stderr 364 | 365 | def __repr__(self): 366 | args = ['args={!r}'.format(self.args), 367 | 'returncode={!r}'.format(self.returncode)] 368 | if self.stdout is not None: 369 | args.append('stdout={!r}'.format(self.stdout)) 370 | if self.stderr is not None: 371 | args.append('stderr={!r}'.format(self.stderr)) 372 | return "{}({})".format(type(self).__name__, ', '.join(args)) 373 | 374 | def check_returncode(self): 375 | """Raise CalledProcessError if the exit code is non-zero.""" 376 | if self.returncode: 377 | raise CalledProcessError(self.returncode, self.args, self.stdout, 378 | self.stderr) 379 | 380 | 381 | def run(*popenargs, **kwargs): 382 | """Run command with arguments and return a CompletedProcess instance. 383 | The returned instance will have attributes args, returncode, stdout and 384 | stderr. By default, stdout and stderr are not captured, and those attributes 385 | will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them. 386 | If check is True and the exit code was non-zero, it raises a 387 | CalledProcessError. The CalledProcessError object will have the return code 388 | in the returncode attribute, and output & stderr attributes if those streams 389 | were captured. 390 | If timeout is given, and the process takes too long, a TimeoutExpired 391 | exception will be raised. 392 | There is an optional argument "input", allowing you to 393 | pass a string to the subprocess's stdin. If you use this argument 394 | you may not also use the Popen constructor's "stdin" argument, as 395 | it will be used internally. 396 | The other arguments are the same as for the Popen constructor. 397 | If universal_newlines=True is passed, the "input" argument must be a 398 | string and stdout/stderr in the returned object will be strings rather than 399 | bytes. 400 | """ 401 | input = kwargs.pop('input', None) 402 | timeout = kwargs.pop('timeout', None) 403 | check = kwargs.pop('check', False) 404 | if input is not None: 405 | if 'stdin' in kwargs: 406 | raise ValueError('stdin and input arguments may not both be used.') 407 | kwargs['stdin'] = PIPE 408 | 409 | process = Popen(*popenargs, **kwargs) 410 | try: 411 | process.__enter__() # No-Op really... illustrate "with in 2.4" 412 | try: 413 | stdout, stderr = process.communicate(input, timeout=timeout) 414 | except TimeoutExpired: 415 | process.kill() 416 | stdout, stderr = process.communicate() 417 | raise TimeoutExpired(process.args, timeout, output=stdout, 418 | stderr=stderr) 419 | except: 420 | process.kill() 421 | process.wait() 422 | raise 423 | retcode = process.poll() 424 | if check and retcode: 425 | raise CalledProcessError(retcode, process.args, 426 | output=stdout, stderr=stderr) 427 | finally: 428 | # None because our context manager __exit__ does not use them. 429 | process.__exit__(None, None, None) 430 | return CompletedProcess(process.args, retcode, stdout, stderr) 431 | 432 | 433 | def list2cmdline(seq): 434 | """ 435 | Translate a sequence of arguments into a command line 436 | string, using the same rules as the MS C runtime: 437 | 438 | 1) Arguments are delimited by white space, which is either a 439 | space or a tab. 440 | 441 | 2) A string surrounded by double quotation marks is 442 | interpreted as a single argument, regardless of white space 443 | contained within. A quoted string can be embedded in an 444 | argument. 445 | 446 | 3) A double quotation mark preceded by a backslash is 447 | interpreted as a literal double quotation mark. 448 | 449 | 4) Backslashes are interpreted literally, unless they 450 | immediately precede a double quotation mark. 451 | 452 | 5) If backslashes immediately precede a double quotation mark, 453 | every pair of backslashes is interpreted as a literal 454 | backslash. If the number of backslashes is odd, the last 455 | backslash escapes the next double quotation mark as 456 | described in rule 3. 457 | """ 458 | 459 | # See 460 | # http://msdn.microsoft.com/en-us/library/17w5ykft.aspx 461 | # or search http://msdn.microsoft.com for 462 | # "Parsing C++ Command-Line Arguments" 463 | result = [] 464 | needquote = False 465 | for arg in seq: 466 | bs_buf = [] 467 | 468 | # Add a space to separate this argument from the others 469 | if result: 470 | result.append(' ') 471 | 472 | needquote = (" " in arg) or ("\t" in arg) or not arg 473 | if needquote: 474 | result.append('"') 475 | 476 | for c in arg: 477 | if c == '\\': 478 | # Don't know if we need to double yet. 479 | bs_buf.append(c) 480 | elif c == '"': 481 | # Double backslashes. 482 | result.append('\\' * len(bs_buf)*2) 483 | bs_buf = [] 484 | result.append('\\"') 485 | else: 486 | # Normal char 487 | if bs_buf: 488 | result.extend(bs_buf) 489 | bs_buf = [] 490 | result.append(c) 491 | 492 | # Add remaining backslashes, if any. 493 | if bs_buf: 494 | result.extend(bs_buf) 495 | 496 | if needquote: 497 | result.extend(bs_buf) 498 | result.append('"') 499 | 500 | return ''.join(result) 501 | 502 | 503 | _PLATFORM_DEFAULT_CLOSE_FDS = object() 504 | 505 | 506 | class Popen(object): 507 | def __init__(self, args, bufsize=0, executable=None, 508 | stdin=None, stdout=None, stderr=None, 509 | preexec_fn=None, close_fds=_PLATFORM_DEFAULT_CLOSE_FDS, 510 | shell=False, cwd=None, env=None, universal_newlines=False, 511 | startupinfo=None, creationflags=0, 512 | restore_signals=True, start_new_session=False, 513 | pass_fds=()): 514 | """Create new Popen instance.""" 515 | _cleanup() 516 | # Held while anything is calling waitpid before returncode has been 517 | # updated to prevent clobbering returncode if wait() or poll() are 518 | # called from multiple threads at once. After acquiring the lock, 519 | # code must re-check self.returncode to see if another thread just 520 | # finished a waitpid() call. 521 | self._waitpid_lock = threading.Lock() 522 | 523 | self._child_created = False 524 | self._input = None 525 | self._communication_started = False 526 | if not isinstance(bufsize, (int, long)): 527 | raise TypeError("bufsize must be an integer") 528 | 529 | if mswindows: 530 | if preexec_fn is not None: 531 | raise ValueError("preexec_fn is not supported on Windows " 532 | "platforms") 533 | any_stdio_set = (stdin is not None or stdout is not None or 534 | stderr is not None) 535 | if close_fds is _PLATFORM_DEFAULT_CLOSE_FDS: 536 | if any_stdio_set: 537 | close_fds = False 538 | else: 539 | close_fds = True 540 | elif close_fds and any_stdio_set: 541 | raise ValueError( 542 | "close_fds is not supported on Windows platforms" 543 | " if you redirect stdin/stdout/stderr") 544 | else: 545 | # POSIX 546 | if close_fds is _PLATFORM_DEFAULT_CLOSE_FDS: 547 | close_fds = True 548 | if pass_fds and not close_fds: 549 | import warnings 550 | warnings.warn("pass_fds overriding close_fds.", RuntimeWarning) 551 | close_fds = True 552 | if startupinfo is not None: 553 | raise ValueError("startupinfo is only supported on Windows " 554 | "platforms") 555 | if creationflags != 0: 556 | raise ValueError("creationflags is only supported on Windows " 557 | "platforms") 558 | 559 | self.args = args 560 | self.stdin = None 561 | self.stdout = None 562 | self.stderr = None 563 | self.pid = None 564 | self.returncode = None 565 | self.universal_newlines = universal_newlines 566 | 567 | # Input and output objects. The general principle is like 568 | # this: 569 | # 570 | # Parent Child 571 | # ------ ----- 572 | # p2cwrite ---stdin---> p2cread 573 | # c2pread <--stdout--- c2pwrite 574 | # errread <--stderr--- errwrite 575 | # 576 | # On POSIX, the child objects are file descriptors. On 577 | # Windows, these are Windows file handles. The parent objects 578 | # are file descriptors on both platforms. The parent objects 579 | # are -1 when not using PIPEs. The child objects are -1 580 | # when not redirecting. 581 | 582 | (p2cread, p2cwrite, 583 | c2pread, c2pwrite, 584 | errread, errwrite) = self._get_handles(stdin, stdout, stderr) 585 | 586 | if mswindows: 587 | if p2cwrite != -1: 588 | p2cwrite = msvcrt.open_osfhandle(p2cwrite.Detach(), 0) 589 | if c2pread != -1: 590 | c2pread = msvcrt.open_osfhandle(c2pread.Detach(), 0) 591 | if errread != -1: 592 | errread = msvcrt.open_osfhandle(errread.Detach(), 0) 593 | 594 | if p2cwrite != -1: 595 | self.stdin = os.fdopen(p2cwrite, 'wb', bufsize) 596 | if c2pread != -1: 597 | if universal_newlines: 598 | self.stdout = os.fdopen(c2pread, 'rU', bufsize) 599 | else: 600 | self.stdout = os.fdopen(c2pread, 'rb', bufsize) 601 | if errread != -1: 602 | if universal_newlines: 603 | self.stderr = os.fdopen(errread, 'rU', bufsize) 604 | else: 605 | self.stderr = os.fdopen(errread, 'rb', bufsize) 606 | 607 | self._closed_child_pipe_fds = False 608 | exception_cleanup_needed = False 609 | try: 610 | try: 611 | self._execute_child(args, executable, preexec_fn, close_fds, 612 | pass_fds, cwd, env, universal_newlines, 613 | startupinfo, creationflags, shell, 614 | p2cread, p2cwrite, 615 | c2pread, c2pwrite, 616 | errread, errwrite, 617 | restore_signals, start_new_session) 618 | except: 619 | # The cleanup is performed within the finally block rather 620 | # than simply within this except block before the raise so 621 | # that any exceptions raised and handled within it do not 622 | # clobber the exception context we want to propagate upwards. 623 | # This is only necessary in Python 2. 624 | exception_cleanup_needed = True 625 | raise 626 | finally: 627 | if exception_cleanup_needed: 628 | for f in filter(None, (self.stdin, self.stdout, self.stderr)): 629 | try: 630 | f.close() 631 | except EnvironmentError: 632 | pass # Ignore EBADF or other errors 633 | 634 | if not self._closed_child_pipe_fds: 635 | to_close = [] 636 | if stdin == PIPE: 637 | to_close.append(p2cread) 638 | if stdout == PIPE: 639 | to_close.append(c2pwrite) 640 | if stderr == PIPE: 641 | to_close.append(errwrite) 642 | if hasattr(self, '_devnull'): 643 | to_close.append(self._devnull) 644 | for fd in to_close: 645 | try: 646 | os.close(fd) 647 | except EnvironmentError: 648 | pass 649 | 650 | def __enter__(self): 651 | return self 652 | 653 | def __exit__(self, type, value, traceback): 654 | if self.stdout: 655 | self.stdout.close() 656 | if self.stderr: 657 | self.stderr.close() 658 | if self.stdin: 659 | self.stdin.close() 660 | # Wait for the process to terminate, to avoid zombies. 661 | self.wait() 662 | 663 | def _translate_newlines(self, data): 664 | data = data.replace("\r\n", "\n") 665 | data = data.replace("\r", "\n") 666 | return data 667 | 668 | 669 | def __del__(self, _maxint=sys.maxint, _active=_active): 670 | # If __init__ hasn't had a chance to execute (e.g. if it 671 | # was passed an undeclared keyword argument), we don't 672 | # have a _child_created attribute at all. 673 | if not getattr(self, '_child_created', False): 674 | # We didn't get to successfully create a child process. 675 | return 676 | # In case the child hasn't been waited on, check if it's done. 677 | self._internal_poll(_deadstate=_maxint) 678 | if self.returncode is None and _active is not None: 679 | # Child is still running, keep us alive until we can wait on it. 680 | _active.append(self) 681 | 682 | 683 | def _get_devnull(self): 684 | if not hasattr(self, '_devnull'): 685 | self._devnull = os.open(os.devnull, os.O_RDWR) 686 | return self._devnull 687 | 688 | def _stdin_write(self, input): 689 | if input: 690 | try: 691 | self.stdin.write(input) 692 | except EnvironmentError as e: 693 | if e.errno == errno.EPIPE: 694 | # communicate() must ignore broken pipe error 695 | pass 696 | elif e.errno == errno.EINVAL : 697 | # bpo-19612, bpo-30418: On Windows, stdin.write() fails 698 | # with EINVAL if the child process exited or if the child 699 | # process is still running but closed the pipe. 700 | pass 701 | else: 702 | raise 703 | 704 | try: 705 | self.stdin.close() 706 | except EnvironmentError as e: 707 | if e.errno in (errno.EPIPE, errno.EINVAL): 708 | pass 709 | else: 710 | raise 711 | 712 | def communicate(self, input=None, timeout=None): 713 | """Interact with process: Send data to stdin. Read data from 714 | stdout and stderr, until end-of-file is reached. Wait for 715 | process to terminate. The optional input argument should be a 716 | string to be sent to the child process, or None, if no data 717 | should be sent to the child. 718 | 719 | communicate() returns a tuple (stdout, stderr).""" 720 | 721 | if self._communication_started and input: 722 | raise ValueError("Cannot send input after starting communication") 723 | 724 | if timeout is not None: 725 | endtime = time.time() + timeout 726 | else: 727 | endtime = None 728 | 729 | # Optimization: If we are not worried about timeouts, we haven't 730 | # started communicating, and we have one or zero pipes, using select() 731 | # or threads is unnecessary. 732 | if (endtime is None and not self._communication_started and 733 | [self.stdin, self.stdout, self.stderr].count(None) >= 2): 734 | stdout = None 735 | stderr = None 736 | if self.stdin: 737 | self._stdin_write(input) 738 | elif self.stdout: 739 | stdout = _eintr_retry_call(self.stdout.read) 740 | self.stdout.close() 741 | elif self.stderr: 742 | stderr = _eintr_retry_call(self.stderr.read) 743 | self.stderr.close() 744 | self.wait() 745 | return (stdout, stderr) 746 | 747 | try: 748 | stdout, stderr = self._communicate(input, endtime, timeout) 749 | finally: 750 | self._communication_started = True 751 | 752 | sts = self.wait(timeout=self._remaining_time(endtime)) 753 | 754 | return (stdout, stderr) 755 | 756 | 757 | def poll(self): 758 | return self._internal_poll() 759 | 760 | 761 | def _remaining_time(self, endtime): 762 | """Convenience for _communicate when computing timeouts.""" 763 | if endtime is None: 764 | return None 765 | else: 766 | return endtime - time.time() 767 | 768 | 769 | def _check_timeout(self, endtime, orig_timeout): 770 | """Convenience for checking if a timeout has expired.""" 771 | if endtime is None: 772 | return 773 | if time.time() > endtime: 774 | raise TimeoutExpired(self.args, orig_timeout) 775 | 776 | 777 | if mswindows: 778 | # 779 | # Windows methods 780 | # 781 | def _get_handles(self, stdin, stdout, stderr): 782 | """Construct and return tuple with IO objects: 783 | p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite 784 | """ 785 | if stdin is None and stdout is None and stderr is None: 786 | return (-1, -1, -1, -1, -1, -1) 787 | 788 | p2cread, p2cwrite = -1, -1 789 | c2pread, c2pwrite = -1, -1 790 | errread, errwrite = -1, -1 791 | 792 | if stdin is None: 793 | p2cread = _subprocess.GetStdHandle(_subprocess.STD_INPUT_HANDLE) 794 | if p2cread is None: 795 | p2cread, _ = _subprocess.CreatePipe(None, 0) 796 | elif stdin == PIPE: 797 | p2cread, p2cwrite = _subprocess.CreatePipe(None, 0) 798 | elif stdin == DEVNULL: 799 | p2cread = msvcrt.get_osfhandle(self._get_devnull()) 800 | elif isinstance(stdin, int): 801 | p2cread = msvcrt.get_osfhandle(stdin) 802 | else: 803 | # Assuming file-like object 804 | p2cread = msvcrt.get_osfhandle(stdin.fileno()) 805 | p2cread = self._make_inheritable(p2cread) 806 | 807 | if stdout is None: 808 | c2pwrite = _subprocess.GetStdHandle(_subprocess.STD_OUTPUT_HANDLE) 809 | if c2pwrite is None: 810 | _, c2pwrite = _subprocess.CreatePipe(None, 0) 811 | elif stdout == PIPE: 812 | c2pread, c2pwrite = _subprocess.CreatePipe(None, 0) 813 | elif stdout == DEVNULL: 814 | c2pwrite = msvcrt.get_osfhandle(self._get_devnull()) 815 | elif isinstance(stdout, int): 816 | c2pwrite = msvcrt.get_osfhandle(stdout) 817 | else: 818 | # Assuming file-like object 819 | c2pwrite = msvcrt.get_osfhandle(stdout.fileno()) 820 | c2pwrite = self._make_inheritable(c2pwrite) 821 | 822 | if stderr is None: 823 | errwrite = _subprocess.GetStdHandle(_subprocess.STD_ERROR_HANDLE) 824 | if errwrite is None: 825 | _, errwrite = _subprocess.CreatePipe(None, 0) 826 | elif stderr == PIPE: 827 | errread, errwrite = _subprocess.CreatePipe(None, 0) 828 | elif stderr == STDOUT: 829 | errwrite = c2pwrite 830 | elif stderr == DEVNULL: 831 | errwrite = msvcrt.get_osfhandle(self._get_devnull()) 832 | elif isinstance(stderr, int): 833 | errwrite = msvcrt.get_osfhandle(stderr) 834 | else: 835 | # Assuming file-like object 836 | errwrite = msvcrt.get_osfhandle(stderr.fileno()) 837 | errwrite = self._make_inheritable(errwrite) 838 | 839 | return (p2cread, p2cwrite, 840 | c2pread, c2pwrite, 841 | errread, errwrite) 842 | 843 | 844 | def _make_inheritable(self, handle): 845 | """Return a duplicate of handle, which is inheritable""" 846 | return _subprocess.DuplicateHandle(_subprocess.GetCurrentProcess(), 847 | handle, _subprocess.GetCurrentProcess(), 0, 1, 848 | _subprocess.DUPLICATE_SAME_ACCESS) 849 | 850 | 851 | def _find_w9xpopen(self): 852 | """Find and return absolut path to w9xpopen.exe""" 853 | w9xpopen = os.path.join( 854 | os.path.dirname(_subprocess.GetModuleFileName(0)), 855 | "w9xpopen.exe") 856 | if not os.path.exists(w9xpopen): 857 | # Eeek - file-not-found - possibly an embedding 858 | # situation - see if we can locate it in sys.exec_prefix 859 | w9xpopen = os.path.join(os.path.dirname(sys.exec_prefix), 860 | "w9xpopen.exe") 861 | if not os.path.exists(w9xpopen): 862 | raise RuntimeError("Cannot locate w9xpopen.exe, which is " 863 | "needed for Popen to work with your " 864 | "shell or platform.") 865 | return w9xpopen 866 | 867 | 868 | def _execute_child(self, args, executable, preexec_fn, close_fds, 869 | pass_fds, cwd, env, universal_newlines, 870 | startupinfo, creationflags, shell, 871 | p2cread, p2cwrite, 872 | c2pread, c2pwrite, 873 | errread, errwrite, 874 | unused_restore_signals, unused_start_new_session): 875 | """Execute program (MS Windows version)""" 876 | 877 | assert not pass_fds, "pass_fds not supported on Windows." 878 | 879 | if not isinstance(args, types.StringTypes): 880 | args = list2cmdline(args) 881 | 882 | # Process startup details 883 | if startupinfo is None: 884 | startupinfo = STARTUPINFO() 885 | if -1 not in (p2cread, c2pwrite, errwrite): 886 | startupinfo.dwFlags |= _subprocess.STARTF_USESTDHANDLES 887 | startupinfo.hStdInput = p2cread 888 | startupinfo.hStdOutput = c2pwrite 889 | startupinfo.hStdError = errwrite 890 | 891 | if shell: 892 | startupinfo.dwFlags |= _subprocess.STARTF_USESHOWWINDOW 893 | startupinfo.wShowWindow = _subprocess.SW_HIDE 894 | comspec = os.environ.get("COMSPEC", "cmd.exe") 895 | args = comspec + " /c " + '"%s"' % args 896 | if (_subprocess.GetVersion() >= 0x80000000L or 897 | os.path.basename(comspec).lower() == "command.com"): 898 | # Win9x, or using command.com on NT. We need to 899 | # use the w9xpopen intermediate program. For more 900 | # information, see KB Q150956 901 | # (http://web.archive.org/web/20011105084002/http://support.microsoft.com/support/kb/articles/Q150/9/56.asp) 902 | w9xpopen = self._find_w9xpopen() 903 | args = '"%s" %s' % (w9xpopen, args) 904 | # Not passing CREATE_NEW_CONSOLE has been known to 905 | # cause random failures on win9x. Specifically a 906 | # dialog: "Your program accessed mem currently in 907 | # use at xxx" and a hopeful warning about the 908 | # stability of your system. Cost is Ctrl+C wont 909 | # kill children. 910 | creationflags |= _subprocess.CREATE_NEW_CONSOLE 911 | 912 | # Start the process 913 | try: 914 | try: 915 | hp, ht, pid, tid = _subprocess.CreateProcess(executable, args, 916 | # no special security 917 | None, None, 918 | int(not close_fds), 919 | creationflags, 920 | env, 921 | cwd, 922 | startupinfo) 923 | except pywintypes.error, e: 924 | # Translate pywintypes.error to WindowsError, which is 925 | # a subclass of OSError. FIXME: We should really 926 | # translate errno using _sys_errlist (or similar), but 927 | # how can this be done from Python? 928 | raise WindowsError(*e.args) 929 | finally: 930 | # Child is launched. Close the parent's copy of those pipe 931 | # handles that only the child should have open. You need 932 | # to make sure that no handles to the write end of the 933 | # output pipe are maintained in this process or else the 934 | # pipe will not close when the child process exits and the 935 | # ReadFile will hang. 936 | if p2cread != -1: 937 | p2cread.Close() 938 | if c2pwrite != -1: 939 | c2pwrite.Close() 940 | if errwrite != -1: 941 | errwrite.Close() 942 | if hasattr(self, '_devnull'): 943 | os.close(self._devnull) 944 | 945 | # Retain the process handle, but close the thread handle 946 | self._child_created = True 947 | self._handle = hp 948 | self.pid = pid 949 | ht.Close() 950 | 951 | def _internal_poll(self, _deadstate=None, 952 | _WaitForSingleObject=_subprocess.WaitForSingleObject, 953 | _WAIT_OBJECT_0=_subprocess.WAIT_OBJECT_0, 954 | _GetExitCodeProcess=_subprocess.GetExitCodeProcess): 955 | """Check if child process has terminated. Returns returncode 956 | attribute. 957 | 958 | This method is called by __del__, so it can only refer to objects 959 | in its local scope. 960 | 961 | """ 962 | if self.returncode is None: 963 | if _WaitForSingleObject(self._handle, 0) == _WAIT_OBJECT_0: 964 | self.returncode = _GetExitCodeProcess(self._handle) 965 | return self.returncode 966 | 967 | 968 | def wait(self, timeout=None, endtime=None): 969 | """Wait for child process to terminate. Returns returncode 970 | attribute.""" 971 | if endtime is not None: 972 | timeout_millis = self._remaining_time(endtime) 973 | if timeout is None: 974 | timeout_millis = _subprocess.INFINITE 975 | else: 976 | timeout_millis = int(timeout * 1000) 977 | if self.returncode is None: 978 | result = _subprocess.WaitForSingleObject(self._handle, 979 | timeout_millis) 980 | if result == _WAIT_TIMEOUT: 981 | raise TimeoutExpired(self.args, timeout) 982 | self.returncode = _subprocess.GetExitCodeProcess(self._handle) 983 | return self.returncode 984 | 985 | 986 | def _readerthread(self, fh, buffer): 987 | buffer.append(fh.read()) 988 | fh.close() 989 | 990 | 991 | def _communicate(self, input, endtime, orig_timeout): 992 | # Start reader threads feeding into a list hanging off of this 993 | # object, unless they've already been started. 994 | if self.stdout and not hasattr(self, "_stdout_buff"): 995 | self._stdout_buff = [] 996 | self.stdout_thread = \ 997 | threading.Thread(target=self._readerthread, 998 | args=(self.stdout, self._stdout_buff)) 999 | self.stdout_thread.daemon = True 1000 | self.stdout_thread.start() 1001 | if self.stderr and not hasattr(self, "_stderr_buff"): 1002 | self._stderr_buff = [] 1003 | self.stderr_thread = \ 1004 | threading.Thread(target=self._readerthread, 1005 | args=(self.stderr, self._stderr_buff)) 1006 | self.stderr_thread.daemon = True 1007 | self.stderr_thread.start() 1008 | 1009 | if self.stdin: 1010 | self._stdin_write(input) 1011 | 1012 | # Wait for the reader threads, or time out. If we time out, the 1013 | # threads remain reading and the fds left open in case the user 1014 | # calls communicate again. 1015 | if self.stdout is not None: 1016 | self.stdout_thread.join(self._remaining_time(endtime)) 1017 | if self.stdout_thread.isAlive(): 1018 | raise TimeoutExpired(self.args, orig_timeout) 1019 | if self.stderr is not None: 1020 | self.stderr_thread.join(self._remaining_time(endtime)) 1021 | if self.stderr_thread.isAlive(): 1022 | raise TimeoutExpired(self.args, orig_timeout) 1023 | 1024 | # Collect the output from and close both pipes, now that we know 1025 | # both have been read successfully. 1026 | stdout = None 1027 | stderr = None 1028 | if self.stdout: 1029 | stdout = self._stdout_buff 1030 | self.stdout.close() 1031 | if self.stderr: 1032 | stderr = self._stderr_buff 1033 | self.stderr.close() 1034 | 1035 | # All data exchanged. Translate lists into strings. 1036 | if stdout is not None: 1037 | stdout = stdout[0] 1038 | if stderr is not None: 1039 | stderr = stderr[0] 1040 | 1041 | # Translate newlines, if requested. We cannot let the file 1042 | # object do the translation: It is based on stdio, which is 1043 | # impossible to combine with select (unless forcing no 1044 | # buffering). 1045 | if self.universal_newlines and hasattr(file, 'newlines'): 1046 | if stdout: 1047 | stdout = self._translate_newlines(stdout) 1048 | if stderr: 1049 | stderr = self._translate_newlines(stderr) 1050 | 1051 | return (stdout, stderr) 1052 | 1053 | def send_signal(self, sig): 1054 | """Send a signal to the process.""" 1055 | # Don't signal a process that we know has already died. 1056 | if self.returncode is not None: 1057 | return 1058 | if sig == signal.SIGTERM: 1059 | self.terminate() 1060 | elif sig == signal.CTRL_C_EVENT: 1061 | os.kill(self.pid, signal.CTRL_C_EVENT) 1062 | elif sig == signal.CTRL_BREAK_EVENT: 1063 | os.kill(self.pid, signal.CTRL_BREAK_EVENT) 1064 | else: 1065 | raise ValueError("Unsupported signal: %s" % sig) 1066 | 1067 | def terminate(self): 1068 | """Terminates the process.""" 1069 | # Don't terminate a process that we know has already died. 1070 | if self.returncode is not None: 1071 | return 1072 | _subprocess.TerminateProcess(self._handle, 1) 1073 | 1074 | kill = terminate 1075 | 1076 | else: 1077 | # 1078 | # POSIX methods 1079 | # 1080 | def _get_handles(self, stdin, stdout, stderr): 1081 | """Construct and return tuple with IO objects: 1082 | p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite 1083 | """ 1084 | p2cread, p2cwrite = -1, -1 1085 | c2pread, c2pwrite = -1, -1 1086 | errread, errwrite = -1, -1 1087 | 1088 | if stdin is None: 1089 | pass 1090 | elif stdin == PIPE: 1091 | p2cread, p2cwrite = _create_pipe() 1092 | elif stdin == DEVNULL: 1093 | p2cread = self._get_devnull() 1094 | elif isinstance(stdin, int): 1095 | p2cread = stdin 1096 | else: 1097 | # Assuming file-like object 1098 | p2cread = stdin.fileno() 1099 | 1100 | if stdout is None: 1101 | pass 1102 | elif stdout == PIPE: 1103 | c2pread, c2pwrite = _create_pipe() 1104 | elif stdout == DEVNULL: 1105 | c2pwrite = self._get_devnull() 1106 | elif isinstance(stdout, int): 1107 | c2pwrite = stdout 1108 | else: 1109 | # Assuming file-like object 1110 | c2pwrite = stdout.fileno() 1111 | 1112 | if stderr is None: 1113 | pass 1114 | elif stderr == PIPE: 1115 | errread, errwrite = _create_pipe() 1116 | elif stderr == STDOUT: 1117 | if c2pwrite != -1: 1118 | errwrite = c2pwrite 1119 | else: # child's stdout is not set, use parent's stdout 1120 | errwrite = sys.__stdout__.fileno() 1121 | elif stderr == DEVNULL: 1122 | errwrite = self._get_devnull() 1123 | elif isinstance(stderr, int): 1124 | errwrite = stderr 1125 | else: 1126 | # Assuming file-like object 1127 | errwrite = stderr.fileno() 1128 | 1129 | return (p2cread, p2cwrite, 1130 | c2pread, c2pwrite, 1131 | errread, errwrite) 1132 | 1133 | 1134 | if hasattr(os, 'closerange'): # Introduced in 2.6 1135 | @staticmethod 1136 | def _closerange(fd_low, fd_high): 1137 | os.closerange(fd_low, fd_high) 1138 | else: 1139 | @staticmethod 1140 | def _closerange(fd_low, fd_high): 1141 | for fd in xrange(fd_low, fd_high): 1142 | while True: 1143 | try: 1144 | os.close(fd) 1145 | except (OSError, IOError), e: 1146 | if e.errno == errno.EINTR: 1147 | continue 1148 | break 1149 | 1150 | 1151 | def _close_fds(self, but): 1152 | self._closerange(3, but) 1153 | self._closerange(but + 1, MAXFD) 1154 | 1155 | 1156 | def _close_all_but_a_sorted_few_fds(self, fds_to_keep): 1157 | # precondition: fds_to_keep must be sorted and unique 1158 | start_fd = 3 1159 | for fd in fds_to_keep: 1160 | if fd >= start_fd: 1161 | self._closerange(start_fd, fd) 1162 | start_fd = fd + 1 1163 | if start_fd <= MAXFD: 1164 | self._closerange(start_fd, MAXFD) 1165 | 1166 | 1167 | def _execute_child(self, args, executable, preexec_fn, close_fds, 1168 | pass_fds, cwd, env, universal_newlines, 1169 | startupinfo, creationflags, shell, 1170 | p2cread, p2cwrite, 1171 | c2pread, c2pwrite, 1172 | errread, errwrite, 1173 | restore_signals, start_new_session): 1174 | """Execute program (POSIX version)""" 1175 | 1176 | if isinstance(args, types.StringTypes): 1177 | args = [args] 1178 | else: 1179 | args = list(args) 1180 | 1181 | if shell: 1182 | args = ["/bin/sh", "-c"] + args 1183 | if executable: 1184 | args[0] = executable 1185 | 1186 | if executable is None: 1187 | executable = args[0] 1188 | orig_executable = executable 1189 | 1190 | # For transferring possible exec failure from child to parent. 1191 | # Data format: "exception name:hex errno:description" 1192 | # Pickle is not used; it is complex and involves memory allocation. 1193 | errpipe_read, errpipe_write = _create_pipe() 1194 | try: 1195 | try: 1196 | 1197 | if _posixsubprocess: 1198 | fs_encoding = sys.getfilesystemencoding() 1199 | def fs_encode(s): 1200 | """Encode s for use in the env, fs or cmdline.""" 1201 | if isinstance(s, str): 1202 | return s 1203 | else: 1204 | return s.encode(fs_encoding, 'strict') 1205 | 1206 | # We must avoid complex work that could involve 1207 | # malloc or free in the child process to avoid 1208 | # potential deadlocks, thus we do all this here. 1209 | # and pass it to fork_exec() 1210 | 1211 | if env is not None: 1212 | env_list = [fs_encode(k) + '=' + fs_encode(v) 1213 | for k, v in env.items()] 1214 | else: 1215 | env_list = None # Use execv instead of execve. 1216 | if os.path.dirname(executable): 1217 | executable_list = (fs_encode(executable),) 1218 | else: 1219 | # This matches the behavior of os._execvpe(). 1220 | path_list = _get_exec_path(env) 1221 | executable_list = (os.path.join(dir, executable) 1222 | for dir in path_list) 1223 | executable_list = tuple(fs_encode(exe) 1224 | for exe in executable_list) 1225 | fds_to_keep = set(pass_fds) 1226 | fds_to_keep.add(errpipe_write) 1227 | self.pid = _posixsubprocess.fork_exec( 1228 | args, executable_list, 1229 | close_fds, sorted(fds_to_keep), cwd, env_list, 1230 | p2cread, p2cwrite, c2pread, c2pwrite, 1231 | errread, errwrite, 1232 | errpipe_read, errpipe_write, 1233 | restore_signals, start_new_session, preexec_fn) 1234 | self._child_created = True 1235 | else: 1236 | # Pure Python implementation: It is not thread safe. 1237 | # This implementation may deadlock in the child if your 1238 | # parent process has any other threads running. 1239 | 1240 | gc_was_enabled = gc.isenabled() 1241 | # Disable gc to avoid bug where gc -> file_dealloc -> 1242 | # write to stderr -> hang. See issue1336 1243 | gc.disable() 1244 | try: 1245 | self.pid = os.fork() 1246 | except: 1247 | if gc_was_enabled: 1248 | gc.enable() 1249 | raise 1250 | self._child_created = True 1251 | if self.pid == 0: 1252 | # Child 1253 | reached_preexec = False 1254 | try: 1255 | # Close parent's pipe ends 1256 | if p2cwrite != -1: 1257 | os.close(p2cwrite) 1258 | if c2pread != -1: 1259 | os.close(c2pread) 1260 | if errread != -1: 1261 | os.close(errread) 1262 | os.close(errpipe_read) 1263 | 1264 | # When duping fds, if there arises a situation 1265 | # where one of the fds is either 0, 1 or 2, it 1266 | # is possible that it is overwritten (#12607). 1267 | if c2pwrite == 0: 1268 | c2pwrite = os.dup(c2pwrite) 1269 | if errwrite == 0 or errwrite == 1: 1270 | errwrite = os.dup(errwrite) 1271 | 1272 | # Dup fds for child 1273 | def _dup2(a, b): 1274 | # dup2() removes the CLOEXEC flag but 1275 | # we must do it ourselves if dup2() 1276 | # would be a no-op (issue #10806). 1277 | if a == b: 1278 | _set_cloexec(a, False) 1279 | elif a != -1: 1280 | os.dup2(a, b) 1281 | _dup2(p2cread, 0) 1282 | _dup2(c2pwrite, 1) 1283 | _dup2(errwrite, 2) 1284 | 1285 | # Close pipe fds. Make sure we don't close the 1286 | # same fd more than once, or standard fds. 1287 | closed = set() 1288 | for fd in [p2cread, c2pwrite, errwrite]: 1289 | if fd > 2 and fd not in closed: 1290 | os.close(fd) 1291 | closed.add(fd) 1292 | 1293 | if cwd is not None: 1294 | os.chdir(cwd) 1295 | 1296 | # This is a copy of Python/pythonrun.c 1297 | # _Py_RestoreSignals(). If that were exposed 1298 | # as a sys._py_restoresignals func it would be 1299 | # better.. but this pure python implementation 1300 | # isn't likely to be used much anymore. 1301 | if restore_signals: 1302 | signals = ('SIGPIPE', 'SIGXFZ', 'SIGXFSZ') 1303 | for sig in signals: 1304 | if hasattr(signal, sig): 1305 | signal.signal(getattr(signal, sig), 1306 | signal.SIG_DFL) 1307 | 1308 | if start_new_session and hasattr(os, 'setsid'): 1309 | os.setsid() 1310 | 1311 | reached_preexec = True 1312 | if preexec_fn: 1313 | preexec_fn() 1314 | 1315 | # Close all other fds, if asked for - after 1316 | # preexec_fn(), which may open FDs. 1317 | if close_fds: 1318 | if pass_fds: 1319 | fds_to_keep = set(pass_fds) 1320 | fds_to_keep.add(errpipe_write) 1321 | self._close_all_but_a_sorted_few_fds( 1322 | sorted(fds_to_keep)) 1323 | else: 1324 | self._close_fds(but=errpipe_write) 1325 | 1326 | if env is None: 1327 | os.execvp(executable, args) 1328 | else: 1329 | os.execvpe(executable, args, env) 1330 | 1331 | except: 1332 | try: 1333 | exc_type, exc_value = sys.exc_info()[:2] 1334 | if isinstance(exc_value, OSError): 1335 | errno_num = exc_value.errno 1336 | else: 1337 | errno_num = 0 1338 | if not reached_preexec: 1339 | exc_value = "noexec" 1340 | message = '%s:%x:%s' % (exc_type.__name__, 1341 | errno_num, exc_value) 1342 | os.write(errpipe_write, message) 1343 | except Exception: 1344 | # We MUST not allow anything odd happening 1345 | # above to prevent us from exiting below. 1346 | pass 1347 | 1348 | # This exitcode won't be reported to applications 1349 | # so it really doesn't matter what we return. 1350 | os._exit(255) 1351 | 1352 | # Parent 1353 | if gc_was_enabled: 1354 | gc.enable() 1355 | finally: 1356 | # be sure the FD is closed no matter what 1357 | os.close(errpipe_write) 1358 | 1359 | # A pair of non -1s means we created both fds and are 1360 | # responsible for closing them. 1361 | # self._devnull is not always defined. 1362 | devnull_fd = getattr(self, '_devnull', None) 1363 | if p2cread != -1 and p2cwrite != -1 and p2cread != devnull_fd: 1364 | os.close(p2cread) 1365 | if c2pwrite != -1 and c2pread != -1 and c2pwrite != devnull_fd: 1366 | os.close(c2pwrite) 1367 | if errwrite != -1 and errread != -1 and errwrite != devnull_fd: 1368 | os.close(errwrite) 1369 | if devnull_fd is not None: 1370 | os.close(devnull_fd) 1371 | # Prevent a double close of these fds from __init__ on error. 1372 | self._closed_child_pipe_fds = True 1373 | 1374 | # Wait for exec to fail or succeed; possibly raising exception 1375 | # exception (limited in size) 1376 | errpipe_data = '' 1377 | while True: 1378 | part = _eintr_retry_call(os.read, errpipe_read, 50000) 1379 | errpipe_data += part 1380 | if not part or len(errpipe_data) > 50000: 1381 | break 1382 | finally: 1383 | # be sure the FD is closed no matter what 1384 | os.close(errpipe_read) 1385 | 1386 | if errpipe_data != "": 1387 | try: 1388 | _eintr_retry_call(os.waitpid, self.pid, 0) 1389 | except OSError, e: 1390 | if e.errno != errno.ECHILD: 1391 | raise 1392 | try: 1393 | exception_name, hex_errno, err_msg = ( 1394 | errpipe_data.split(':', 2)) 1395 | except ValueError: 1396 | exception_name = 'RuntimeError' 1397 | hex_errno = '0' 1398 | err_msg = ('Bad exception data from child: ' + 1399 | repr(errpipe_data)) 1400 | child_exception_type = getattr( 1401 | exceptions, exception_name, RuntimeError) 1402 | if issubclass(child_exception_type, OSError) and hex_errno: 1403 | errno_num = int(hex_errno, 16) 1404 | child_exec_never_called = (err_msg == "noexec") 1405 | if child_exec_never_called: 1406 | err_msg = "" 1407 | if errno_num != 0: 1408 | err_msg = os.strerror(errno_num) 1409 | if errno_num == errno.ENOENT: 1410 | if child_exec_never_called: 1411 | # The error must be from chdir(cwd). 1412 | err_msg += ': ' + repr(cwd) 1413 | else: 1414 | err_msg += ': ' + repr(orig_executable) 1415 | raise child_exception_type(errno_num, err_msg) 1416 | try: 1417 | exception = child_exception_type(err_msg) 1418 | except Exception: 1419 | exception = RuntimeError( 1420 | 'Could not re-raise %r exception from the' 1421 | ' child with error message %r' % 1422 | (child_exception_type, err_msg)) 1423 | raise exception 1424 | 1425 | 1426 | def _handle_exitstatus(self, sts, _WIFSIGNALED=os.WIFSIGNALED, 1427 | _WTERMSIG=os.WTERMSIG, _WIFEXITED=os.WIFEXITED, 1428 | _WEXITSTATUS=os.WEXITSTATUS, _WIFSTOPPED=os.WIFSTOPPED, 1429 | _WSTOPSIG=os.WSTOPSIG): 1430 | """All callers to this function MUST hold self._waitpid_lock.""" 1431 | # This method is called (indirectly) by __del__, so it cannot 1432 | # refer to anything outside of its local scope.""" 1433 | if _WIFSIGNALED(sts): 1434 | self.returncode = -_WTERMSIG(sts) 1435 | elif _WIFEXITED(sts): 1436 | self.returncode = _WEXITSTATUS(sts) 1437 | elif _WIFSTOPPED(sts): 1438 | self.returncode = -_WSTOPSIG(sts) 1439 | else: 1440 | # Should never happen 1441 | raise RuntimeError("Unknown child exit status!") 1442 | 1443 | 1444 | def _internal_poll(self, _deadstate=None, _waitpid=os.waitpid, 1445 | _WNOHANG=os.WNOHANG, _os_error=os.error, _ECHILD=errno.ECHILD): 1446 | """Check if child process has terminated. Returns returncode 1447 | attribute. 1448 | 1449 | This method is called by __del__, so it cannot reference anything 1450 | outside of the local scope (nor can any methods it calls). 1451 | 1452 | """ 1453 | if self.returncode is None: 1454 | if not self._waitpid_lock.acquire(False): 1455 | # Something else is busy calling waitpid. Don't allow two 1456 | # at once. We know nothing yet. 1457 | return None 1458 | try: 1459 | try: 1460 | if self.returncode is not None: 1461 | return self.returncode # Another thread waited. 1462 | pid, sts = _waitpid(self.pid, _WNOHANG) 1463 | if pid == self.pid: 1464 | self._handle_exitstatus(sts) 1465 | except _os_error, e: 1466 | if _deadstate is not None: 1467 | self.returncode = _deadstate 1468 | elif e.errno == _ECHILD: 1469 | # This happens if SIGCLD is set to be ignored or 1470 | # waiting for child processes has otherwise been 1471 | # disabled for our process. This child is dead, we 1472 | # can't get the status. 1473 | # http://bugs.python.org/issue15756 1474 | self.returncode = 0 1475 | finally: 1476 | self._waitpid_lock.release() 1477 | return self.returncode 1478 | 1479 | 1480 | def _try_wait(self, wait_flags): 1481 | """All callers to this function MUST hold self._waitpid_lock.""" 1482 | try: 1483 | (pid, sts) = _eintr_retry_call(os.waitpid, self.pid, wait_flags) 1484 | except OSError, e: 1485 | if e.errno != errno.ECHILD: 1486 | raise 1487 | # This happens if SIGCLD is set to be ignored or waiting 1488 | # for child processes has otherwise been disabled for our 1489 | # process. This child is dead, we can't get the status. 1490 | pid = self.pid 1491 | sts = 0 1492 | return (pid, sts) 1493 | 1494 | 1495 | def wait(self, timeout=None, endtime=None): 1496 | """Wait for child process to terminate. Returns returncode 1497 | attribute.""" 1498 | if self.returncode is not None: 1499 | return self.returncode 1500 | 1501 | # endtime is preferred to timeout. timeout is only used for 1502 | # printing. 1503 | if endtime is not None or timeout is not None: 1504 | if endtime is None: 1505 | endtime = time.time() + timeout 1506 | elif timeout is None: 1507 | timeout = self._remaining_time(endtime) 1508 | 1509 | if endtime is not None: 1510 | # Enter a busy loop if we have a timeout. This busy loop was 1511 | # cribbed from Lib/threading.py in Thread.wait() at r71065. 1512 | delay = 0.0005 # 500 us -> initial delay of 1 ms 1513 | while True: 1514 | if self._waitpid_lock.acquire(False): 1515 | try: 1516 | if self.returncode is not None: 1517 | break # Another thread waited. 1518 | (pid, sts) = self._try_wait(os.WNOHANG) 1519 | assert pid == self.pid or pid == 0 1520 | if pid == self.pid: 1521 | self._handle_exitstatus(sts) 1522 | break 1523 | finally: 1524 | self._waitpid_lock.release() 1525 | remaining = self._remaining_time(endtime) 1526 | if remaining <= 0: 1527 | raise TimeoutExpired(self.args, timeout) 1528 | delay = min(delay * 2, remaining, .05) 1529 | time.sleep(delay) 1530 | else: 1531 | while self.returncode is None: 1532 | self._waitpid_lock.acquire() 1533 | try: 1534 | if self.returncode is not None: 1535 | break # Another thread waited. 1536 | (pid, sts) = self._try_wait(0) 1537 | # Check the pid and loop as waitpid has been known to 1538 | # return 0 even without WNOHANG in odd situations. 1539 | # http://bugs.python.org/issue14396. 1540 | if pid == self.pid: 1541 | self._handle_exitstatus(sts) 1542 | finally: 1543 | self._waitpid_lock.release() 1544 | return self.returncode 1545 | 1546 | 1547 | def _communicate(self, input, endtime, orig_timeout): 1548 | if self.stdin and not self._communication_started: 1549 | # Flush stdio buffer. This might block, if the user has 1550 | # been writing to .stdin in an uncontrolled fashion. 1551 | self.stdin.flush() 1552 | if not input: 1553 | self.stdin.close() 1554 | 1555 | if _has_poll: 1556 | stdout, stderr = self._communicate_with_poll(input, endtime, 1557 | orig_timeout) 1558 | else: 1559 | stdout, stderr = self._communicate_with_select(input, endtime, 1560 | orig_timeout) 1561 | 1562 | self.wait(timeout=self._remaining_time(endtime)) 1563 | 1564 | # All data exchanged. Translate lists into strings. 1565 | if stdout is not None: 1566 | stdout = ''.join(stdout) 1567 | if stderr is not None: 1568 | stderr = ''.join(stderr) 1569 | 1570 | # Translate newlines, if requested. We cannot let the file 1571 | # object do the translation: It is based on stdio, which is 1572 | # impossible to combine with select (unless forcing no 1573 | # buffering). 1574 | if self.universal_newlines and hasattr(file, 'newlines'): 1575 | if stdout: 1576 | stdout = self._translate_newlines(stdout) 1577 | if stderr: 1578 | stderr = self._translate_newlines(stderr) 1579 | 1580 | return (stdout, stderr) 1581 | 1582 | 1583 | def _communicate_with_poll(self, input, endtime, orig_timeout): 1584 | stdout = None # Return 1585 | stderr = None # Return 1586 | 1587 | if not self._communication_started: 1588 | self._fd2file = {} 1589 | 1590 | poller = select.poll() 1591 | def register_and_append(file_obj, eventmask): 1592 | poller.register(file_obj.fileno(), eventmask) 1593 | self._fd2file[file_obj.fileno()] = file_obj 1594 | 1595 | def close_unregister_and_remove(fd): 1596 | poller.unregister(fd) 1597 | self._fd2file[fd].close() 1598 | self._fd2file.pop(fd) 1599 | 1600 | if self.stdin and input: 1601 | register_and_append(self.stdin, select.POLLOUT) 1602 | 1603 | # Only create this mapping if we haven't already. 1604 | if not self._communication_started: 1605 | self._fd2output = {} 1606 | if self.stdout: 1607 | self._fd2output[self.stdout.fileno()] = [] 1608 | if self.stderr: 1609 | self._fd2output[self.stderr.fileno()] = [] 1610 | 1611 | select_POLLIN_POLLPRI = select.POLLIN | select.POLLPRI 1612 | if self.stdout: 1613 | register_and_append(self.stdout, select_POLLIN_POLLPRI) 1614 | stdout = self._fd2output[self.stdout.fileno()] 1615 | if self.stderr: 1616 | register_and_append(self.stderr, select_POLLIN_POLLPRI) 1617 | stderr = self._fd2output[self.stderr.fileno()] 1618 | 1619 | # Save the input here so that if we time out while communicating, 1620 | # we can continue sending input if we retry. 1621 | if self.stdin and self._input is None: 1622 | self._input_offset = 0 1623 | self._input = input 1624 | if self.universal_newlines and isinstance(self._input, unicode): 1625 | self._input = self._input.encode( 1626 | self.stdin.encoding or sys.getdefaultencoding()) 1627 | 1628 | while self._fd2file: 1629 | try: 1630 | ready = poller.poll(self._remaining_time(endtime)) 1631 | except select.error, e: 1632 | if e.args[0] == errno.EINTR: 1633 | continue 1634 | raise 1635 | self._check_timeout(endtime, orig_timeout) 1636 | 1637 | for fd, mode in ready: 1638 | if mode & select.POLLOUT: 1639 | chunk = self._input[self._input_offset : 1640 | self._input_offset + _PIPE_BUF] 1641 | self._input_offset += os.write(fd, chunk) 1642 | if self._input_offset >= len(self._input): 1643 | close_unregister_and_remove(fd) 1644 | elif mode & select_POLLIN_POLLPRI: 1645 | data = os.read(fd, 4096) 1646 | if not data: 1647 | close_unregister_and_remove(fd) 1648 | self._fd2output[fd].append(data) 1649 | else: 1650 | # Ignore hang up or errors. 1651 | close_unregister_and_remove(fd) 1652 | 1653 | return (stdout, stderr) 1654 | 1655 | 1656 | def _communicate_with_select(self, input, endtime, orig_timeout): 1657 | if not self._communication_started: 1658 | self._read_set = [] 1659 | self._write_set = [] 1660 | if self.stdin and input: 1661 | self._write_set.append(self.stdin) 1662 | if self.stdout: 1663 | self._read_set.append(self.stdout) 1664 | if self.stderr: 1665 | self._read_set.append(self.stderr) 1666 | 1667 | if self.stdin and self._input is None: 1668 | self._input_offset = 0 1669 | self._input = input 1670 | if self.universal_newlines and isinstance(self._input, unicode): 1671 | self._input = self._input.encode( 1672 | self.stdin.encoding or sys.getdefaultencoding()) 1673 | 1674 | stdout = None # Return 1675 | stderr = None # Return 1676 | 1677 | if self.stdout: 1678 | if not self._communication_started: 1679 | self._stdout_buff = [] 1680 | stdout = self._stdout_buff 1681 | if self.stderr: 1682 | if not self._communication_started: 1683 | self._stderr_buff = [] 1684 | stderr = self._stderr_buff 1685 | 1686 | while self._read_set or self._write_set: 1687 | try: 1688 | (rlist, wlist, xlist) = \ 1689 | select.select(self._read_set, self._write_set, [], 1690 | self._remaining_time(endtime)) 1691 | except select.error, e: 1692 | if e.args[0] == errno.EINTR: 1693 | continue 1694 | raise 1695 | 1696 | # According to the docs, returning three empty lists indicates 1697 | # that the timeout expired. 1698 | if not (rlist or wlist or xlist): 1699 | raise TimeoutExpired(self.args, orig_timeout) 1700 | # We also check what time it is ourselves for good measure. 1701 | self._check_timeout(endtime, orig_timeout) 1702 | 1703 | if self.stdin in wlist: 1704 | chunk = self._input[self._input_offset : 1705 | self._input_offset + _PIPE_BUF] 1706 | try: 1707 | bytes_written = os.write(self.stdin.fileno(), chunk) 1708 | except EnvironmentError as e: 1709 | if e.errno == errno.EPIPE: 1710 | self._write_set.remove(self.stdin) 1711 | self.stdin.close() 1712 | else: 1713 | raise 1714 | else: 1715 | self._input_offset += bytes_written 1716 | if self._input_offset >= len(self._input): 1717 | self.stdin.close() 1718 | self._write_set.remove(self.stdin) 1719 | 1720 | if self.stdout in rlist: 1721 | data = os.read(self.stdout.fileno(), 1024) 1722 | if data == "": 1723 | self.stdout.close() 1724 | self._read_set.remove(self.stdout) 1725 | stdout.append(data) 1726 | 1727 | if self.stderr in rlist: 1728 | data = os.read(self.stderr.fileno(), 1024) 1729 | if data == "": 1730 | self.stderr.close() 1731 | self._read_set.remove(self.stderr) 1732 | stderr.append(data) 1733 | 1734 | return (stdout, stderr) 1735 | 1736 | 1737 | def send_signal(self, sig): 1738 | """Send a signal to the process 1739 | """ 1740 | # Skip signalling a process that we know has already died. 1741 | if self.returncode is None: 1742 | os.kill(self.pid, sig) 1743 | 1744 | def terminate(self): 1745 | """Terminate the process with SIGTERM 1746 | """ 1747 | self.send_signal(signal.SIGTERM) 1748 | 1749 | def kill(self): 1750 | """Kill the process with SIGKILL 1751 | """ 1752 | self.send_signal(signal.SIGKILL) 1753 | -------------------------------------------------------------------------------- /test: -------------------------------------------------------------------------------- 1 | #!/bin/bash -ex 2 | 3 | # This is for my own convenience, edit it for your own environment. 4 | 5 | PYTHON=python2 6 | "$PYTHON" -V 7 | "$PYTHON" setup.py build 8 | export PYTHONPATH=build/lib.linux-x86_64-2.7 9 | exec "$PYTHON" test_subprocess32.py 10 | 11 | PYTHON=python3 12 | rm -r vvv dist 13 | mkdir vvv 14 | # Test this in a virtualenv installed from the sdist tarball becaue of: 15 | # https://github.com/google/python-subprocess32/issues/44 16 | virtualenv -p "$PYTHON" vvv 17 | PYTHON=vvv/bin/python3 18 | PIP=vvv/bin/pip 19 | "$PYTHON" -V 20 | "$PYTHON" setup.py sdist 21 | "$PIP" install dist/*.tar.gz 22 | # Merely test that it imports. Under Python 3, setup.py should've installed a 23 | # redirection hack making subprocess32 == subprocess. We cd as we must not run 24 | # this from the directory subprocess32.py sources live in. 25 | (cd $(dirname "$PYTHON") && \ 26 | ./python3 -c "import subprocess as s, subprocess32 as s32; assert s is s32, (s, s32)") 27 | 28 | -------------------------------------------------------------------------------- /testdata/fd_status.py: -------------------------------------------------------------------------------- 1 | """When called as a script, print a comma-separated list of the open 2 | file descriptors on stdout. 3 | 4 | Usage: 5 | fd_stats.py: check all file descriptors 6 | fd_status.py fd1 fd2 ...: check only specified file descriptors 7 | """ 8 | 9 | import errno 10 | import os 11 | import stat 12 | import sys 13 | 14 | if __name__ == "__main__": 15 | fds = [] 16 | if len(sys.argv) == 1: 17 | try: 18 | _MAXFD = os.sysconf("SC_OPEN_MAX") 19 | except: 20 | _MAXFD = 256 21 | test_fds = range(0, _MAXFD) 22 | else: 23 | test_fds = map(int, sys.argv[1:]) 24 | for fd in test_fds: 25 | try: 26 | st = os.fstat(fd) 27 | except OSError, e: 28 | if e.errno == errno.EBADF: 29 | continue 30 | raise 31 | # Ignore Solaris door files 32 | if not hasattr(stat, 'S_ISDOOR') or not stat.S_ISDOOR(st.st_mode): 33 | fds.append(fd) 34 | print ','.join(map(str, fds)) 35 | -------------------------------------------------------------------------------- /testdata/input_reader.py: -------------------------------------------------------------------------------- 1 | """When called as a script, consumes the input""" 2 | 3 | import sys 4 | 5 | if __name__ == "__main__": 6 | for line in sys.stdin: 7 | pass 8 | -------------------------------------------------------------------------------- /testdata/qcat.py: -------------------------------------------------------------------------------- 1 | """When ran as a script, simulates cat with no arguments.""" 2 | 3 | import sys 4 | 5 | if __name__ == "__main__": 6 | for line in sys.stdin: 7 | sys.stdout.write(line) 8 | -------------------------------------------------------------------------------- /testdata/qgrep.py: -------------------------------------------------------------------------------- 1 | """When called with a single argument, simulated fgrep with a single 2 | argument and no options.""" 3 | 4 | import sys 5 | 6 | if __name__ == "__main__": 7 | pattern = sys.argv[1] 8 | for line in sys.stdin: 9 | if pattern in line: 10 | sys.stdout.write(line) 11 | -------------------------------------------------------------------------------- /testdata/sigchild_ignore.py: -------------------------------------------------------------------------------- 1 | import signal, subprocess32, sys, time 2 | # On Linux this causes os.waitpid to fail with OSError as the OS has already 3 | # reaped our child process. The wait() passing the OSError on to the caller 4 | # and causing us to exit with an error is what we are testing against. 5 | sig_child = getattr(signal, 'SIGCLD', None) 6 | if sig_child is None: 7 | sig_child = getattr(signal, 'SIGCHLD') 8 | signal.signal(sig_child, signal.SIG_IGN) 9 | subprocess32.Popen([sys.executable, '-c', 'print("albatross")']).wait() 10 | # Also ensure poll() handles an errno.ECHILD appropriately. 11 | p = subprocess32.Popen([sys.executable, '-c', 'print("albatross")']) 12 | num_polls = 0 13 | while p.poll() is None: 14 | # Waiting for the process to finish. 15 | time.sleep(0.01) # Avoid being a CPU busy loop. 16 | num_polls += 1 17 | if num_polls > 3000: 18 | raise RuntimeError('poll should have returned 0 within 30 seconds') 19 | --------------------------------------------------------------------------------