├── CODE_OF_CONDUCT.md ├── LICENSE ├── README.rst └── pipstrap.py /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Community Participation Guidelines 2 | 3 | This repository is governed by Mozilla's code of conduct and etiquette guidelines. 4 | For more details, please read the 5 | [Mozilla Community Participation Guidelines](https://www.mozilla.org/about/governance/policies/participation/). 6 | 7 | ## How to Report 8 | For more information on how to report violations of the Community Participation Guidelines, please read our '[How to Report](https://www.mozilla.org/about/governance/policies/participation/reporting/)' page. 9 | 10 | 16 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2016 Erik Rose 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies 7 | of the Software, and to permit persons to whom the Software is furnished to do 8 | so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in all 11 | copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 19 | SOFTWARE. 20 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | ======== 2 | Pipstrap 3 | ======== 4 | 5 | Pipstrap is a small script that acts as a trust root for installing pip 8 or 6 | greater. 7 | 8 | Embed this in your project, and your VCS checkout is all you have to trust. In 9 | a post-`peep `_ era, this lets you claw 10 | your way to a `hash-checking version of pip 11 | `_, with which you can install the rest of your dependencies safely. All 13 | it assumes is Python 2.6 or better and *some* trustworthy version of pip 14 | already installed. If anything goes wrong, it will exit with a non-zero status 15 | code. 16 | 17 | Invoke it like this:: 18 | 19 | my-virtual-env/bin/python my-project/pipstrap.py 20 | 21 | At that point, pinned, hash-checked versions of pip, setuptools, and wheel will 22 | be installed. (If your existing version of pip was already new enough, pipstrap 23 | will have exited happily without doing anything.) 24 | 25 | .. note:: 26 | 27 | ``pip`` is invoked as a module of the python executable that was used to 28 | run pipstrap, so everything will be installed into that executable's 29 | environment. 30 | 31 | How do I trust this? 32 | ==================== 33 | 34 | An astute question! 35 | 36 | Pipstrap is short; read it. Validate its embedded hashes by downloading from 37 | various locations (to defeat local MITMs), checking the PGP signature on pip, 38 | comparing with version-control checkouts of all three packages, and talking 39 | with friends. Check it into your project, sign your commits, and verify 40 | signatures on deploy. 41 | 42 | Also, if you trust me and my key-management practices, you can check any of the 43 | release tags against my GPG signature. 44 | 45 | Why? 46 | ==== 47 | 48 | * get-pip.py is yuckily large to embed, and it hits the network to fetch the 49 | latest pip, setuptools, and wheel, leaving the question of their genuineness 50 | up to HTTPS. (Its embedded pip is used only to download those new versions.) 51 | * Continuing to embed peep just to bootstrap pip 8 is unwieldy, with an extra 52 | requirements file and a longer-than-necessary script. Plus, now that I've got 53 | hash-checking into pip, I don't want to continue maintaining a script that 54 | breaks whenever pip's private APIs change. 55 | 56 | For how long? 57 | ============= 58 | 59 | When your OS packages pip 8 or you otherwise get a copy of pip 8 you trust onto 60 | your servers, you can dispense with pipstrap. 61 | 62 | What about firewalls? 63 | ===================== 64 | 65 | To use pipstrap from a machine that cannot access https://pypi.python.org/, set 66 | the ``PIP_INDEX_URL`` environment variable to point to a PyPI mirror, either 67 | public or internal. (See `the pip documentation on this option 68 | `_ for 69 | details.) Pipstrap needs a ``packages`` directory which is a copy of the one on 70 | PyPI proper, and it will look for it in 2 places: 71 | 72 | 1. If ``PIP_INDEX_URL`` ends with "/simple" (as it often does in the real world 73 | to satisfy pip), pipstrap will look at ``${PIP_INDEX_URL}/../packages``. In 74 | other words, ``packages`` is expected to sit right next to ``simple``. 75 | 2. Otherwise, it will look at ``${PIP_INDEX_URL}/packages``. 76 | 77 | Trust in the mirror is not necessary, as SHA-256 hash-checking suffices to 78 | prove authenticity. Also, pipstrap doesn't care whether ``PIP_INDEX_URL`` has a 79 | trailing slash. 80 | 81 | Version History 82 | =============== 83 | 84 | 2.0 85 | * Execute pip as a module from the python executable used to call pipstrap, 86 | instead of resolving pip from the ``PATH``. 87 | 88 | 1.5.1 89 | * Revert our 2-phase installation procedure, which was causing setuptools not 90 | to be upgraded on Debian Wheezy. We now install a modern pip and setuptools 91 | all in one pip invocation. (The problem the 2-phase install was meant to 92 | solve has been fixed in recent versions of Ubuntu 16.04, and we don't know 93 | of any other distros having the problem.) 94 | 95 | 1.5 96 | * Update to setuptools 29.0.1, the newest version that doesn't drop support 97 | for any Python versions. This allows use of the ``python_requires`` keyword 98 | arg. 99 | 100 | 1.4 101 | * Add support for PyPI mirrors. 102 | * Do the installation in 2 phases: first pip; then argparse, wheel, and 103 | setuptools. This dodges an obscure bug: 104 | https://github.com/certbot/certbot/issues/4938. (bmw) 105 | 106 | 1.3 107 | * Update pip to 9.0.1 so we can support manylinux1 wheels. 108 | * Restore Python 2.6 compatibility. 109 | 110 | 1.2 111 | * Don't do anything if the pip version is already new enough. 112 | * Disable the pip cache to avoid ownership warnings, which don't apply since 113 | we're passing in files and not using the cache. 114 | * Fix a bytes/string mismatch under Python 3. 115 | 116 | 1.1.1 117 | * Under Python 2.6 don't pass the CalledProcessError exception the ``output`` 118 | kwarg, which hasn't been invented yet. 119 | 120 | 1.1 121 | * Support Python 2.6. I feel so dirty, but Let's Encrypt needs it. 122 | * Update to pip 8.0.3, wheel 0.29, and setuptools 20.2.2. 123 | 124 | 1.0.1 125 | * Make flake8-compliant so you can embed it without having to make exceptions 126 | for it. 127 | 128 | 1.0 129 | * Initial release. Before I signed the release, I verified all 3 embedded 130 | hashes from various network locations and the pip one against dstufft's GPG 131 | signature. 132 | -------------------------------------------------------------------------------- /pipstrap.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """A small script that can act as a trust root for installing pip >=8 3 | 4 | Embed this in your project, and your VCS checkout is all you have to trust. In 5 | a post-peep era, this lets you claw your way to a hash-checking version of pip, 6 | with which you can install the rest of your dependencies safely. All it assumes 7 | is Python 2.6 or better and *some* version of pip already installed. If 8 | anything goes wrong, it will exit with a non-zero status code. 9 | 10 | """ 11 | # This is here so embedded copies are MIT-compliant: 12 | # Copyright (c) 2016 Erik Rose 13 | # 14 | # Permission is hereby granted, free of charge, to any person obtaining a copy 15 | # of this software and associated documentation files (the "Software"), to 16 | # deal in the Software without restriction, including without limitation the 17 | # rights to use, copy, modify, merge, publish, distribute, sublicense, and/or 18 | # sell copies of the Software, and to permit persons to whom the Software is 19 | # furnished to do so, subject to the following conditions: 20 | # 21 | # The above copyright notice and this permission notice shall be included in 22 | # all copies or substantial portions of the Software. 23 | from __future__ import print_function 24 | from distutils.version import StrictVersion 25 | from hashlib import sha256 26 | from os import environ 27 | from os.path import join 28 | from pipes import quote 29 | from shutil import rmtree 30 | try: 31 | from subprocess import check_output 32 | except ImportError: 33 | from subprocess import CalledProcessError, PIPE, Popen 34 | 35 | def check_output(*popenargs, **kwargs): 36 | if 'stdout' in kwargs: 37 | raise ValueError('stdout argument not allowed, it will be ' 38 | 'overridden.') 39 | process = Popen(stdout=PIPE, *popenargs, **kwargs) 40 | output, unused_err = process.communicate() 41 | retcode = process.poll() 42 | if retcode: 43 | cmd = kwargs.get("args") 44 | if cmd is None: 45 | cmd = popenargs[0] 46 | raise CalledProcessError(retcode, cmd) 47 | return output 48 | from sys import exit, version_info, executable 49 | from tempfile import mkdtemp 50 | try: 51 | from urllib2 import build_opener, HTTPHandler, HTTPSHandler 52 | except ImportError: 53 | from urllib.request import build_opener, HTTPHandler, HTTPSHandler 54 | try: 55 | from urlparse import urlparse 56 | except ImportError: 57 | from urllib.parse import urlparse # 3.4 58 | 59 | 60 | __version__ = 2, 0, 0 61 | PIP_VERSION = '9.0.1' 62 | DEFAULT_INDEX_BASE = 'https://pypi.python.org' 63 | 64 | 65 | # wheel has a conditional dependency on argparse: 66 | maybe_argparse = ( 67 | [('18/dd/e617cfc3f6210ae183374cd9f6a26b20514bbb5a792af97949c5aacddf0f/' 68 | 'argparse-1.4.0.tar.gz', 69 | '62b089a55be1d8949cd2bc7e0df0bddb9e028faefc8c32038cc84862aefdd6e4')] 70 | if version_info < (2, 7, 0) else []) 71 | 72 | 73 | PACKAGES = maybe_argparse + [ 74 | # Pip has no dependencies, as it vendors everything: 75 | ('11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/' 76 | 'pip-{0}.tar.gz'.format(PIP_VERSION), 77 | '09f243e1a7b461f654c26a725fa373211bb7ff17a9300058b205c61658ca940d'), 78 | # This version of setuptools has only optional dependencies: 79 | ('59/88/2f3990916931a5de6fa9706d6d75eb32ee8b78627bb2abaab7ed9e6d0622/' 80 | 'setuptools-29.0.1.tar.gz', 81 | 'b539118819a4857378398891fa5366e090690e46b3e41421a1e07d6e9fd8feb0'), 82 | ('c9/1d/bd19e691fd4cfe908c76c429fe6e4436c9e83583c4414b54f6c85471954a/' 83 | 'wheel-0.29.0.tar.gz', 84 | '1ebb8ad7e26b448e9caa4773d2357849bf80ff9e313964bcaf79cbf0201a1648') 85 | ] 86 | 87 | 88 | class HashError(Exception): 89 | def __str__(self): 90 | url, path, actual, expected = self.args 91 | return ('{url} did not match the expected hash {expected}. Instead, ' 92 | 'it was {actual}. The file (left at {path}) may have been ' 93 | 'tampered with.'.format(**locals())) 94 | 95 | 96 | def hashed_download(url, temp, digest): 97 | """Download ``url`` to ``temp``, make sure it has the SHA-256 ``digest``, 98 | and return its path.""" 99 | # Based on pip 1.4.1's URLOpener but with cert verification removed. Python 100 | # >=2.7.9 verifies HTTPS certs itself, and, in any case, the cert 101 | # authenticity has only privacy (not arbitrary code execution) 102 | # implications, since we're checking hashes. 103 | def opener(using_https=True): 104 | opener = build_opener(HTTPSHandler()) 105 | if using_https: 106 | # Strip out HTTPHandler to prevent MITM spoof: 107 | for handler in opener.handlers: 108 | if isinstance(handler, HTTPHandler): 109 | opener.handlers.remove(handler) 110 | return opener 111 | 112 | def read_chunks(response, chunk_size): 113 | while True: 114 | chunk = response.read(chunk_size) 115 | if not chunk: 116 | break 117 | yield chunk 118 | 119 | parsed_url = urlparse(url) 120 | response = opener(using_https=parsed_url.scheme == 'https').open(url) 121 | path = join(temp, parsed_url.path.split('/')[-1]) 122 | actual_hash = sha256() 123 | with open(path, 'wb') as file: 124 | for chunk in read_chunks(response, 4096): 125 | file.write(chunk) 126 | actual_hash.update(chunk) 127 | 128 | actual_digest = actual_hash.hexdigest() 129 | if actual_digest != digest: 130 | raise HashError(url, path, actual_digest, digest) 131 | return path 132 | 133 | 134 | def get_index_base(): 135 | """Return the URL to the dir containing the "packages" folder. 136 | 137 | Try to wring something out of PIP_INDEX_URL, if set. Hack "/simple" off the 138 | end if it's there; that is likely to give us the right dir. 139 | 140 | """ 141 | env_var = environ.get('PIP_INDEX_URL', '').rstrip('/') 142 | if env_var: 143 | SIMPLE = '/simple' 144 | if env_var.endswith(SIMPLE): 145 | return env_var[:-len(SIMPLE)] 146 | else: 147 | return env_var 148 | else: 149 | return DEFAULT_INDEX_BASE 150 | 151 | 152 | def main(): 153 | pip_version = StrictVersion(check_output([executable, '-m', 'pip', '--version']) 154 | .decode('utf-8').split()[1]) 155 | min_pip_version = StrictVersion(PIP_VERSION) 156 | if pip_version >= min_pip_version: 157 | return 0 158 | has_pip_cache = pip_version >= StrictVersion('6.0') 159 | index_base = get_index_base() 160 | temp = mkdtemp(prefix='pipstrap-') 161 | try: 162 | downloads = [hashed_download(index_base + '/packages/' + path, 163 | temp, 164 | digest) 165 | for path, digest in PACKAGES] 166 | check_output('{0} -m pip install --no-index --no-deps -U '.format(quote(executable)) + 167 | # Disable cache since we're not using it and it otherwise 168 | # sometimes throws permission warnings: 169 | ('--no-cache-dir ' if has_pip_cache else '') + 170 | ' '.join(quote(d) for d in downloads), 171 | shell=True) 172 | except HashError as exc: 173 | print(exc) 174 | except Exception: 175 | rmtree(temp) 176 | raise 177 | else: 178 | rmtree(temp) 179 | return 0 180 | return 1 181 | 182 | 183 | if __name__ == '__main__': 184 | exit(main()) 185 | --------------------------------------------------------------------------------